How integration patterns impact your microservices architecture

on microservices, Integration, Distributed-Systems

When the world wide web first emerged, integrating different types of operating systems was a core challenge. Hypertext transfer protocol (HTTP) created communication channels by sharing hypertext, these systems started speaking a common language over an accepted protocol, and the internet as we know today was born.

When creating a microservices architecture, the integration challenges are not very different: Multiple implementation technologies are physically separated by a network and need to communicate with each other. Microservices integration plays a vital role in creating a seamless experience of the system, from the end user’s perspective. Correctly integrated systems also help realize the benefits of distributed systems: They enable scaling at the service level and improve efficiency, and have a potential of reducing the infrastructure costs while serving business needs [1].

On the other hand, incorrectly integrated systems completely undermine the benefits of a microservices architecture: It can result in painful data loss and integrity issues. The problems are usually very hard to track down, and in the meantime, users are impacted adversely.

Seamless integration depends on a number of considerations, which we looked at in our previous post, Principles for Microservices Integration. These serve as a guide to choosing the type of integration that will offer the most autonomy and scale. We’ll cover the various options and their pros and cons below.

Database Integration

In this pattern, two or more services read and write data out of one central data store. All of the services go to this central data store. We can illustrate this using the banking application example from our previous post, which takes login, user profile, transactions, notifications, credit score, and spending reports as separate services defined by the business functionalities.
Integration-patterns-diagram---Db-Integration

One of the significant advantages of database integration is simplicity. The transaction management is more straightforward as compared to other patterns. This is perhaps the most widely-used pattern of integration—but also the most abused.

This pattern couples services together undesirably, making the microservices architecture more difficult to change and expensive to scale. Defining the data ownership and updating the schema can become a messy process—every change requires re-compiling and deploying all of the services. It pushes towards highly orchestrated, big bang style deployments. This type of integration can create significant obstacles in maintaining the autonomy of microservices.

Under the needs of high scale, the only option is to throw more hardware at the database, and even then it becomes difficult to avoid deadlocks in the database and row-level contentions.

Ideally, we don’t recommend this pattern for inter-services communication. It can be used in one of the early phases of a phased microservices rollout. In other words: If you use it, lose it soon.

Synchronous API calls

Integration-patterns-diagram---API
In this integration pattern, the services communicate synchronously through an API. All the access to each other’s data is coordinated through an API in a request-response fashion, and the service waits for data from the API to perform its action. In the example above, if the transaction service needs to read the user profile data, it calls the user profile API and gets what it needs.

This provides a decent abstraction over a direct database call and offers excellent flexibility in terms of technical choices. This provides the benefit of hiding many implementation details: The abstraction gives us the freedom to change technologies without affecting our clients. For example, the user profile service could use JAVA and MySQL, while the transaction service could use SQL server and .NET, and they can still easily speak to each other through the API.

However, this is not much different than the direct database integration pattern. Adding another network hop on top of a database call can also inhibit scale: by increasing the workload, performance decreases—it is significantly at odds with most of distributed systems fallacies [2]. This integration pattern also makes transaction management difficult and inhibits autonomy, as services depend on one another’s uptimes. In microservices, if you have to read data synchronously outside of your system boundary, that is a service-oriented architecture smell [3].

In some cases, this integration pattern is the best or unavoidable. Security tokens are a prime use case for synchronous API integration because those tokens are short-lived and can’t be generated before they are required. Synchronous API calls should be used sparingly, if possible. If used, they should be versioned and should be used with a circuit breaking mechanism such as Polly [4].

ETL (Extract, Transform, and Load)

Integration-patterns-diagram---ETL
ETL entails synchronizing data via background processes on a predefined schedule. This data can be pushed or pulled. Only backend ETL processes need database access. It is asynchronous, meaning services can execute without waiting for a “callback.”

This integration pattern also hides implementation details nicely. It provides reasonable decoupling because the services are not dependent upon one another’s uptimes. Live users don’t get affected by the uptimes or the processing time.

The ETL processes have to change with the source and destination databases. With ETL integrations, data consistency depends on the schedule and duration. Figuring out the change delta could get too complicated. In these situations, the teams fall back on pushing the entire dataset out. That makes processes very long-running, significantly undermining their usefulness.

Reporting services are a natural fit for this type of integration. These processes have their place but usually get very involved with time. They should be used only when the stale data is acceptable in the system.

Messaging

Integration-patterns-diagram---Messaging
In this pattern, services exchange meaningful messages with each other through what are called commands or integration events. They are sent over message transports such as RabbitMQs, MSMQs, or Azure Service Bus. In the example above, the transactions service generates an “account balance changed” event and puts it on a message broker. The rewards service, credit score service, and notification service each subscribe to that event and react to it as necessary. This is a publish-subscribe pattern, and there are plenty of other useful patterns for messaging. Enterprise Integration Patterns is an excellent resource for learning more [5].

If done correctly, messaging provides very good decoupling. It offers complete flexibility in terms of technology choices, as long as they can communicate correctly with the transport. Pushing data to subscribers makes the sending part simpler, and the sender remains completely unaffected by the processing details on the subscriber’s side.

Incorrect service or transaction boundaries can complicate messaging implementations. Also, the loose coupling this pattern provides can come at the cost of consistency. It requires a high level of discipline to implement messaging correctly, and inexperienced teams may struggle with it.

Two categories of messaging

Typical messaging solutions are built on top of the properties of the transport. At a high level, these transports could be divided into two categories: message queueing and message streaming.

The queueing solution involves dealing with live data. Once a message is processed successfully, it is off the queue. As long the processing can keep up, the queues won’t build up and won’t require too much space. However, the message ordering can’t be guaranteed in cases of scaled-out subscribers.

When a new subscriber comes up, we have to involve the source to bring it up to speed. They need specific migration path, which could be challenging depending on the scale and the business domain.

In the streaming solution, the messages are stored on a stream as they come in order. That happens on the message transport itself. The subscriber's position on a stream is maintained on the transport. It can reverse forward on the stream as necessary. This is very advantageous for failure and new subscription scenarios. However, this depends on how long the stream is. It requires a lot more configuration from storage perspectives, necessitating the archiving of streams. Azure Event Hubs and Kafka are some of the examples of this. Some databases such as Cassandra support generating events out of transaction log. Kafka is often looked at as a direct replacement of the ETL jobs [6].

The typical use cases here assume that the system is able to work with somewhat stale data. If that’s not the case, we need to analyze the business domain a bit more and understand why. In our example, when the transaction update happens, the credit score update doesn’t necessarily need to happen in a real time. Messaging fits very well here. We can set the user’s expectation accordingly and allow the services to manage their own load.

Parting thoughts

Every microservices architecture will be different, and there are no perfectly prescribed solutions for integration. We need to keep failure scenarios in mind when we use them—that drives to a combination of these integration patterns. For example, Netflix uses messaging to move the data and they fall back synchronous API if messaging is not available or data is still in transit [7]. In each case, the ideal is achieving the most flexible and scalable microservices architecture—but you have to consider the implementation details and your own capabilities first. The chart below shows how some integration patterns are more desirable from a microservices standpoint, but inherit complexities that your development team must be prepared to deal with:
Integration-patterns-diagram---Integration-Final
The key takeaway here is to have asynchronous patterns when needed for scale and autonomy. To achieve this, you need solid service boundaries and clear data ownership. Otherwise, you end up with complex and unsustainable integration scenarios. Modeling the business processes is important. That will let you know which use cases and processes are inherently asynchronous and suitable for messaging. How do we model the business processes and define the boundaries? We will look into that in the future posts.

References

[1] – Microsoft .NET Microservices Architecture E-book
[2] – Fallacies of Distributed Computing
[3] – What are Microservices?
[4] – The Polly Project
[5] – Enterprise Integration Patterns
[6] – ETL is dead
[7] – Mastering Chaos at Netflix

When the world wide web first emerged, integrating different types of operating systems was a core challenge. Hypertext transfer protocol (HTTP) created communication channels by sharing hypertext, these systems started speaking a common language over an accepted protocol, and the internet as we know today was born. When creating a…

Read More

Principles for microservices integration

on microservices, Integration, Distributed-Systems

Out of the many advantages of microservices, the most significant motivations are scale and autonomy for business units. These go hand in hand. However, we still need to create an integrated experience that makes sense for the end user. It’s important to keep both these aims in mind when developing strategies for the interactions between microservices. Those are the strategies that can make or break your effort.

How we map each microservice determines how autonomous it will be. Microservices modeled by bounded contexts [1] or business capabilities are more natural to the autonomy than the ones based on the technical abilities. Let's consider the example of a banking application. Some bounded contexts a typical it can have are Login and Security, Profile Management, Transaction Services (one service for debit and credit because they are closely tied), Spending Reports, and external services such as Credit Report Check or Rewards Check. These contexts may have many technical implementations that are similar: for example, logging. However, if we create logging as its own service, almost all of the other services are going to be dependent on it. It can become a linchpin: You take that down, and the business stops. Instead, we can push the logging implementation into a library, create services based on the contexts, and make use of the logging library if possible.

Mapping services in vertical business slices with their own databases is only the beginning. We still need to integrate them in a way that creates a cohesive experience and share the data between those services. How do we achieve this while maintaining the autonomy? Before we look into how we could integrate, we must first assess the myriad of interactions between individual services that will influence our integration decisions.

Creating loose coupling and high cohesion

To ensure autonomy and scale, individual services should be highly cohesive (grouping similar functionalities) and loosely coupled [2]. “Coupling” in computer science describes the interdependence between modules [3]. The loosely coupled systems share well-defined data in the form of messages, and that’s all. They don’t worry about states, uptimes, performance levels, or technical implementations.

From our banking example, if credit and debit services are separate, they become very dependent on each other because they tend to affect the same data piece: your account balance. If there’s a discrepancy between balances shown, which one is right? These services must be incredibly consistent, which results in a lot of back and forth network chatter. Instead, we can merge these two functionalities into one cohesive service and avoid the complexity.

Iterating business boundaries

Services that depend too much on other services’ data, implementations, and uptimes, could be a symptom of wrong or outdated business boundaries. Businesses always change, which is why we need to revisit boundary assumptions periodically. This ensures we aren’t creating too fine-grained services, i.e., nanoservices [4]. These nanoservices tend to have fragmented logic and poor performance. They add a lot of maintenance overhead. Horizontal services that are based on technical implementation rather than business boundaries fall into this pit. The division of credit and debit functionalities into their own services fits this, too. There is no need to break the cohesion and introduce a network between them. What’s wrong with the network?

Knowing the network limitations

If there is a chance of something breaking down, we need to have a plan to deal with it as a good engineering practice. Communication over the network is a prime example of it [5]. Services across the business boundaries connect with each other over the network. We need to understand the impact of this because very little can be in the maintenance team’s control outside of the business boundary. Hence, we should keep the network communication as minimum as possible. For example, in the banking application, we need to let the Spending Report service know of a debit transaction. An incorrect implementation would have a call to that service asking if such operation is possible or maybe perform validation on the input parameters followed by the actual reporting of the balance change. These two can be easily merged into one cutting the chatter by half. This idea is based on the Tell, Don’t Ask principle [6]. All of this may seem small, but these little things add up quickly. It is essential to understand the implications of putting APIs on HTTP.

Having a contract-oriented mindset

It’s important to think about the consumer of your API all of the time, no matter which kind of integration we decide to go with. The code written with the service’s consumer in mind has better encapsulation and hides the implementation details very well. Test-driven development can be helpful in this regard. With TDD, we can write on consumer contracts first and then code to satisfies those contracts. PACT[7] can help us share these contracts between services.
It becomes tough to draw boundaries in the code that’s not written this way, for example, CRUD or repository pattern based APIs. They are concerned with the database entities. They span across business functionalities producing tighter coupling. At that point, redesigning them first is a better idea.

Understanding CAP theorem and database technologies

The primary goal of distributed systems is to scale better. In an ideal world, the data shared by loosely coupled services could be replicated without any trouble. That would require optimal consistency, availability, and partition tolerance, which means 1) every reader gets the latest write, 2) every request receives a non-error response, and 3) because the network separates microservices, they must be about to handle an arbitrary number of messages getting dropped. However, this is restricted by the CAP theorem [8], which states that only two of these three conditions can be optimally met in any system.

Because availability and partition tolerance are critical in the distributed world; we must deal with weaker consistency, as shown below:
Integration-patterns-diagram---Consistency

However, consistency itself has many levels. Distributed database technologies such as Azure Cosmos DB supports five of them [9]. Google Cloud Spanner technology, on the other hand, is challenging the CAP theorem by claiming to offer high consistency along with availability and partition tolerance [10]. We need to keep these conditions in mind while deciding on database technologies for our systems.

Understanding transactions and transactional boundaries

Distributed transactions across multiple services are hard to get right because they go through multiple phases before the data is committed [11]. They require orchestration that makes the systems very fragile. All of this hassle to get to a place which can’t scale well and the database choices may differ between services. Now what?

Instead, we can let new database technologies such as Cosmos DB or Cloud Spanner handle the complexity behind the scenes. If that’s not an option, we can support transactional guarantees within the service boundary and generate events with Outbox pattern for everyone else to consume [12]. Using our banking example, when a user changes his or her phone number in the profile, we can commit that info in the User Profile service’s own data store and generate events for the other systems to consume. After the successful consumption of that message, our Notification service can notify the user for account changes, as shown below:

Integration-patterns-diagram---Outbox--1-

Being careful with synchronous (blocking) APIs

We must consider the limitations of our networks before putting services there. Synchronous calls across services usually take place over HTTP, which can become very tricky to manage What happens if the HTTP service goes down? How do we know it’s down? How do we handle the failures? How can we rollback synchronously applied changes? Where does the cache live? How many types of caches will be managed? One per consumer? One per call? All this complexity can result in a complex architecture, with everyone calling each other.

Synchronous services have higher expectations on response times, making them more challenging to scale and maintain. Less is more here. Synchronous API calls usually lead to more orchestrated solutions. Sometimes we need physical obstacles to prevent incorrect usages from creeping into the systems [13]. What’s wrong with the orchestration? What can we do instead?

Considering choreography over orchestration

Any system that requires a lot of central management or any service that plays a role of that kind can become problematic. They become too important to go down. Everything is funneled through them, increasing the coupling in the system. This highly coordinated approach is known as orchestration. In contrast, a choreographed approach lets services decide what to do when an event happens. These services don’t need handholding from a central manager. Back to the banking application example: upon debiting money from your account, the Transactions service can call the Rewards services, which can call the Credit Score service, and end it with a notification. In this case, the Transaction service is sitting in the middle of everything playing a traffic cop. Instead, it can just create an “account balance changed” event and let other services subscribe to those events and finish their operations independently. The latter is a much more decoupled approach—a notification can still be sent even if the credit score service is down.

Integration-patterns-diagram---Choreographed
A choreographed approach can be the difference between partial outage vs. full outage, making the services themselves more robust [14].

Tying it all together

Considering your services structure and the complex web of interactions they inherit is the first step to building a robust microservices architecture. There are no silver bullets in software engineering, but each of these principles is a building block in the construction of a full understanding of the interactions between services. In the next part of this series, we will take a look at the types of integrations we can implement to create cohesion within your systems, and for your end users.

References

[1] More on bounded contexts
[2] Service definition from SOA patterns book
[3] This article does an excellent job explaining different levels of coupling
[4] More on the nanoservices antipattern
[5] Read about the fallacies of network computing
[6] The "Tell, don’t ask principle" explained with C# example
[7] Details on PACT
[8] CAP Theorem explained
[9] Consistency levels supported by Azure Cosmos DB
[10] Google Cloud Spanner
[11] Distributed transactions two-phase commit protocol
[12] Learn about Outbox Pattern
[13] Hear Eric Evans’ argument for physical separation of services in his GOTO Conference talk
[14] The routing slip pattern for messaging can also lead to more choreographed solutions

And to learn more about dealing with the fallacies of network computing in distributed systems, check out this video from Headspring's Chief Architect Jimmy Bogard: Building Distributed Systems.
(This is a cross post from my post on Headspring's blog)

Out of the many advantages of microservices, the most significant motivations are scale and autonomy for business units. These go hand in hand. However, we still need to create an integrated experience that makes sense for the end user. It’s important to keep both these aims in mind when…

Read More

Business case for observability

on microservices, Distributed-Systems, SOA, Monitoring, Observability

In the previous post, we took a look at what observability is and how to build it into your application. Just to recap, observability is developing insights into the system based on external signs. For example, the car dashboard has a service engine light, low-pressure indicators, RPM for the engine, etc. All of this can quickly help us determine if the car is in a condition to drive. We don’t need to actually look at every individual part of the car every single day.

As with anything in our industry, software exists for the business needs. We always need to be cognizant of that. Any initiative that doesn't align with the business needs loses steam quickly and rightly so. After all, we have a limited amount of resources to spend at any given time. The domain driven design movement plays right into it. So, we need to understand why having a more observable system can help us.

Confidence in the system

When we hear people say their system works, what does that mean? Does it mean their smoke tests worked? Does it mean there are zero errors? Can it withstand Chaos Monkey? I highly recommend taking a look at Principles of Chaos Engineering to better understand the Chaos Monkey tool. That deserves a separate discussion in and of itself for some other time.

Sometimes a system may not be throwing any errors but may actually be doing something it is not supposed to. How do we catch that? Very often, this unexpected behavior is not captured in the logs or the monitoring tools. So, going by a no news is good news attitude, the team claims that the system is working. It is similar to walking into a room full of mess with the lights turned off: since I can't see anything, I am conveniently going to assume everything is just great.

Having multiple levels of health checks, connectivity checks, and performance checks along with observing demo data points helps us provide a basis for the "it’s working" claim. It is not a claim anymore, since we have data to back it up. It becomes a fact. This can reduce the number of customer complaints related to code issues and help the business in building confidence in the system.

For example: If I have an e-commerce site along with typical telemetries around errors, availability, and response times, we can create a dummy order and check the data around it periodically to make sure everything is fine. If any part of the order process goes beyond the acceptable thresholds we can raise the appropriate alarms.

This is how we can build confidence in our system with knowledge instead of guesses.

Faster Deployments

Higher confidence in the system paves the way for more advanced deployment models such as canary deployment and blue-green deployment. Both of these deployment models require some level of testing before the new features go completely live. If we can subject new nodes to production loads and observe how the system behaves with the new changes, we reduce the friction between existing codebase and the new changes coming in. All of this means we can deploy new code more reliably and rapidly with minimum to no downtime, thus achieving a true continuous deployment for the system.

Understand changes that affect business KPIs (key performance indicators)

KPIs tell us how the business is doing, as they point to key health issues that need to be addressed. Some examples of KPIs could be the number of active customers, cost per customer, customer attrition, etc. Let’s think of customer attrition from the perspective of a social media site, Twitter. What if it took more than 30-40 seconds to make a tweet live? That would significantly impact how many tweets can be generated, affect Twitter’s popularity, and eventually cause customer attrition. In this case, we can see a relation between latency and customer attrition. Understanding this, we can see why Twitter must have made a move to migrate to Scala and JVM from Rails. This is not a trivial undertaking and the company's existence can depend on it. Could they have done it without gathering performance metrics? Could anyone do this without having a before and after picture?

Observability brings the backend problems to the forefront by making them measurable, which is beneficial because those problems can be absolutely detrimental to the business. On the other hand, fixing those issues proactively can drive the business forward.

Tangible targets

We saw a relation between technical metrics and KPIs. This means if I want to improve my KPIs, I can target my technical metrics because I can tie them to a piece of code. So when I hear the system is slow, I can quickly create an understanding of what that means based on the metrics and start the analysis.

Let’s take a look at an example.If my response time for a request averages 100 milliseconds over a week but jumps to 2-3 seconds after I push a new feature, then I know the system's performance has been affected significantly. It might not be a rollback worthy deployment but is certainly worth taking a look to ensure it doesn’t get any worse. I am likely to know the cause as well based on the timelines and other useful observability metrics and logging. What does that do?

I can now clearly see what I need to achieve. If I can throw hardware at something, I can try that and I have ways to test that. If it requires a code change, I can push those changes through the same rigor. I know that I am not done until I bring the response time down to the acceptable range. Without realistically knowing that range, this would have been hard to achieve.

Faster software reflects positively on sales

With a clear understanding of what fast or slow means for the application, let’s take a look how that can affect the bottom line of the business.

On a social media site, how expensive is the complete shutdown of the site? How many users are lost by the site with just regular sluggish performance? It is very easy to get a bad reputation and the downward spiral begins there. Would you buy claims of performance tuning from a software consulting firm if their own site often suffers from some serious lag, random crashes, etc. I may see an item I am interested in on a vendor site but end up on Amazon anyways because they made it incredibly easy and fast to search items and purchase them. They couldn't have arrived to a premier experience without gathering tons of metrics about usages, the performance of the pages, etc. What would happen if Amazon experiences just 10% slow down in their checkout process? How many carts will get abandoned? All of this has a direct impact on the business and its viability. As software engineers, it is important to understand the impact of these things on the business.

As explained by an Amazon Developer here, Amazon found that revenue increased by 1% for every 100 ms of load time improvement. You can see how Amazon Prime day 2017 fared for them in this [post])(http://blog.catchpoint.com/2017/07/14/performance-report-amazon-prime-day-2017/)

Here’s a result from Pinterest’s frontend performance project in March 2017: 40% drop in perceived wait time, 15% increase in SEO traffic yielded a 15% increase in signups. They go into the details in this Pinterest Engineering blog post

Both of these examples were originally mentioned in the Practical Monitoring book by Mike Julian.

Business priorities determination

It is very common for development teams to be in discussion with product teams about the next set of features. We can always make these conversations more data-driven if we have the usage statistics. In our e-commerce example, if we find that most users are utilizing the search functionality on the site for discovery instead of the navigation system, we can use this insight to make the search faster and more useful. We reduce the priority on everything related to the navigation system.

Justification for refactoring

Everyone cringes when they look at their old code. Engineers want to immediately start refactoring. Business doesn't see any value in it as they think nothing changes from the end users perspective. The maintenance argument works only with folks with some development experience. The friction creeps in. How can we allow time for this activity from the business perspective? We can find a middle ground in the observability metrics. For example, we can prioritize activities that are going to improve performance of the login process over other types of changes.

Generate a complete picture for A/B testing

The observable nature of the system can contribute to A/B test experiments effectively. Technical data metrics can be tied to usage to generate information that can help understand the stress points in the system. Fortunately, there are tons of tools to conduct these experiments. Optimizely is one such tool I have seen being used effectively.

Provide steps towards auto-healing

Observability is knowing the speed of your car by looking at the speedometer and not the spinning wheel. It won’t necessarily fix problems but it will provide good insight into those problems that could prove crucial in resolving. Auto-healing could be hard to generalize because it can change per context, per architecture, per tech stack, etc. To come up with a truly auto-healing system can be a daunting task but the path to that destination goes through observability.

Parting thoughts

We have seen how observability helps you build better software to drive value for users and the business. When we understand a system’s behavior we can operate better. We can deploy faster, build greater confidence in our system, understand KPIs better and drive sales.

In the previous post, we took a look at what observability is and how to build it into your application. Just to recap, observability is developing insights into the system based on external signs. For example, the car dashboard has a service engine light, low-pressure indicators, RPM for the engine,…

Read More

Increasing observability of microservices

on microservices, Distributed-Systems, SOA, Monitoring, Observability

Software engineering is an evidence-based practice. We are always seeking facts. We are logically sequencing them together to generate knowledge out of them. And yet, sometimes we are completely oblivious to the state of our system in production. No amount of testing can guarantee a 100% bug free production software. Things will go wrong. The real test of the system begins when it gets deployed in production. Are we prepared for this?

Typical Enterprise Experience

How often changes are pushed to production without completely collecting data around the issue? We wait to change until someone complains. Then the investigation begins. We look at the logs. We "fix" the problem by making code changes, throwing more hardware at it, etc. and life move on. How reactionary is this!

Then someone says, "We need monitoring!". That's largely because of some random compliance. Some tools are installed, they gather info about CPU and RAM usage. They look at the spikes, they complain, they log a ticket, nothing happens, and everyone moves on.

Then there are alerts. Something goes wrong, a million of them fill up people's inboxes even during a planned outage. They quickly become annoying. Moreover, they are compared to a fire alarm but are anything but. These alerts, unlike a fire alarm, go off during the problem instead of at the first sign of smoke. What’s the point of a loud alarm in the middle of a raging fire? People create rules to ignore these and move on. That whole situation is like this:

alt

Sometimes, they also gather service's uptime using heartbeat reports with some nice charts and put them on TV. Is that enough? Can a service, returning 200 OK with a perfect heartbeat (i.e. it is up), perform its critical actions? I also hear about not making systems any slower than today, however, nobody can tell what slow or fast means?

All of this points toward a lack of value provided by a typical monitoring in the enterprise. These systems were put in place because somebody wanted them, or they came in as part of some compliance. It lacks purpose. So, how do we fix this? How do we change our systems to make them more observable?

What is Observability?

Wikipedia says:

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

Now, if we apply this to some of the scenarios we saw above, we can see how incomplete the external outputs tend to be. They don't scale very well either.

Applying to Microservices

In the distributed systems/Microservices world, we are dealing with eventual consistency, where data may not be in the identical state throughout. We are dealing with services changing constantly at their own pace. Sometimes there could be a situation when certain functionalities are not available. All of this can make triaging problems tricky.

Increasing Observability

Let's see how to achieve this:

1) Come up with service level objectives and indicators: This helps us in putting together clear and practical definition around what a healthy service looks like. Google's Service Reliability Engineering Book has good advice on how to put these numbers together and how to avoid pitfalls. Afterwards, we can gather actual numbers and build intelligence around it.

2) Log aggregation: Most of the software produce a lot of logs. These logs can be aggregated into one place. This one place could be Kibana, Splunk or HoneyComb. If we can make holistic sense of these logs, we should be good.

3) Traceable individual event lifecycles: This can be achieved by generating correlation ids and source of origins on events and they can be logged. We can have messaging headers to pass this information around. NServiceBus supports these types of headers.

4) Business goal tracking: We need to gather user's activity on our application so that we can figure out the importance of features based on its usage. A/B testing can provide great insight into this. Optimizely is one such tool that I have seen used effectively.

5) Host performance tracking: There is value in gathering info about the host running our service. However, to make it effective, we should be able to create enough log information so that any anomalies can be tracked down to a piece of code that caused it.
Otherwise, any issues arising from this, are hard to recreate and resolve.

6) Meaningful notifications: I'd create less but more meaningful notifications. My proposed principle for this would be to generate a notification when something is about to go wrong. This way, the individuals can take necessary actions to avoid further escalations.

7) Custom approach: Sometimes there is no getting away from writing custom few lines of code somewhere to proactively capture data you need. In some of my systems, I had to take this approach to collect what I needed. It worked out better than customers telling us about our problems.

8) Automation: If someone must manually hit a button, that's not good. It is not going to happen. It will be ignored quickly. We need to build a culture of Automation.

9) Autonomy: It can neither bring the system down nor should it go down with the system. In the past, one of our application servers started to slow down because the monitoring tool was consuming too many resources.

Avoid these traps

Cargo cult: Blindly following practices that worked for one successful organization may seem to be working out in the beginning, but that can backfire and introduce unnecessary complications. Next time, when you are using terms such as "Netflix did it that way" or "Amazon does it this way", catch yourself and see if it really applies to you. Some of those ideas are great and noteworthy but they still need to be correctly evaluated for validity in your situation. Every environment is different.

Another form of Cargo Cult is, "We have always done it this way." or "This is our pattern." The fix here is the same as above. We always need to evaluate if past lessons are still applicable or if we can do a better job this time around.

Tool Obsessions: Whenever I had conversations around monitoring, within a few minutes, they turned into a battle of different tools out there. Yes, the capabilities of tools can give us ideas but before spending too much money on these, we need to get some of the basics right and evaluate if those tools can help us or not.

Information I found useful

Sam Newman, in his Principles of Microservices talk briefly touches upon observability being one of the core principles of building microservices. Charity Majors goes into a lot more details in her Monitoring is dead talk. Her book Database reliability engineering also has a couple of good chapters relevant to monitoring. Google's SRE book is another good source as mentioned above. Microsoft's best practices and Practical monitoring book take you deeper into monitoring.

Parting thoughts

With increased observability, when someone says our system is slow, we can add more to it with the information around how slow and the data around the cause. This can be incredibly powerful in fixing problems as they come. It will result in increased resiliency of our system and making a positive impact on the business.

Software engineering is an evidence-based practice. We are always seeking facts. We are logically sequencing them together to generate knowledge out of them. And yet, sometimes we are completely oblivious to the state of our system in production. No amount of testing can guarantee a 100% bug free production software.…

Read More

Message store clean up strategy

on microservices, Distributed-Systems, NServicebus, SOA, Idempotent-Receiver

In the previous post, we saw how to use message store strategy to make a receiver idempotent. As we are using a table in SQL server, it can fill up quickly if you have multiple types of messages getting stored. How do we keep it from filling up?

Instead of relying on any strategy outside our endpoint, I decided to use NServicebus scheduling. I configured it to send a message every 24 hours. This can be configured easily. My send code looks like below:

await endpointInstance.ScheduleEvery(
    timeSpan: TimeSpan.FromHours(24),
    task: context=>
    {
        var message = new CleanupStoreMessage();
        return context.Send(message);
    })
.ConfigureAwait(false);

The CleanupStoreMessageHandler picks it up after. The handler goes back to the configured amount of days and removes them from the store. That code looks like below:

 var messagesToRemove = dbContext.MyMessageStore.Where(
     m =>  m.Timestamp <= DateTime.UtcNow.AddDays(-10));
 
 dbContext.MyMessageStore.RemoveRange(messagesToRemove);
 dbContext.SaveChanges();

The NServicebus scheduling comes with several caveats. A relevant one is below:

Since the Scheduler is non-durable, if the process restarts, all scheduled tasks (that are created during the endpoint's startup) are recreated and given new identifiers.

To get around this, my strategy is to keep the TimeSpan for the ScheduleEvery method small enough so that it gets executed at least once in a day. Since I am going back some days based on the day of the execution, if I receive more than one message, the later messages just wont find any data to cleanup and that's ok for me. My query here goes back 10 days and I am sending the CleanupStoreMessage every 24 hours. If I reduce the TimeSpan to 12, I will get 2 messages per day and the second time, I may not find any data to clean. This way my handler has a greater chance of executing and mitigating the limitation.

Another advantage here is that I am cleaning up my own data without relying on any other clever database archiving strategy or some other process to clean up it. I am less likely to forget about it and it requires less maintenance.

In the previous post, we saw how to use message store strategy to make a receiver idempotent. As we are using a table in SQL server, it can fill up quickly if you have multiple types of messages getting stored. How do we keep it from filling up? Instead of…

Read More