Why should you avoid get calls across microservices?

In the previous posts in this series, we saw what microservices are and how to start the journey towards broken out services from the monolithic application.

In the second post, I talked about not having get calls across microservices. On that, I received some questions. Here's one:

"Nice article on microservices. But i did not get reasoning behind
"You shouldn't make a query/get call to another microservice."

My reply to that is below:

As a service, you should have your own store for the data you need. If you don't, what happens if the service you're using for get calls goes down for an extended period of time? You are blocked. Your function that depends on it fails now. What if you have multiple of these? You just increased your reasons to fail by that factor.

To explain this further, too many blocking get calls may result into a system that can get completely stalled. What if the individual calls that are happening have different SLAs? Do we block the entire request for the slowest call?

One of the projects I worked on in the past had these types of calls spread through out the system. In the worst cases, the http requests took well over a minute to return. Is this acceptable?

To counter this, the team went towards caching the entire relational data stores. Pulling all that information down became very tricky. The data sync was happening once a month. The sync job itself took days to finish, pushing the changes further back. I have seen many project teams going towards this type of solution and eventually running into the scale problem. Then they settle on syncing the data using messaging.

So, what is the trade off? I explained it below:

The data duplication that may come with this is an acceptable trade off for scale. In other words, you are giving up some consistency and moving towards eventual consistency for scale and availability.

One of the teams I worked with did a good job differentiating the types of messages. Some of the changes had to be pushed forward at a higher priority, some required a little more processing, etc. We built different routes for those messages so that we could maintain our SLA for these messages. The data was still cached but we were able to get rid of the sync jobs that were trying to sync all the data at once. Messaging enables different kinds of patterns. I recommend Enterprise Integration Patterns book for those patterns. It goes deep into the different kinds of messaging patterns.

I used this example afterwards:

For a banking app, you can say, there are $100 in your account as of the last sync time instead of $0 because you can't hit that service. People are much more likely to freak out in the later scenario than the one before. This method scales too since we have reduced the consistency level to eventual. It sounds radical but you are trading in strong consistency for scale and user experience. I hope that clears things up for you. I think I should write another post on this :)

I went on further to express the points below:

Another reason is network. You can't take it for granted. In any of these situations, you don't want your customers to get affected by this implementation detail. The way to avoid this is by building your own store beforehand and keep it synced through messaging.

You don't have a distributed system if the system is not resilient enough to consider the network loss. This is something that needs to be considered on day one. Your customers are not going to care that your network failed. They want your app to work.

If your system needs too much of a cross service chatter then it could be a sign of wrong system boundaries. It means that the service is just too fine grained. Your services could be suffering from the nanoservices anti pattern. It is a subtle problem. The SOA patterns book goes into the details of that. Some of the services may require merging.

The Netflix approach

They go the hybrid route. They hit the local cache store first, if it is not available, they fall back on cross boundary calls. Josh Evans explains that very eloquently in this talk.

To summarize, we are giving up some consistency for autonomy. We can solve the consistency problem using data pumps or sync jobs easily but autonomy has to remain as strong as possible. Without autonomy, you won't be able to materialize any benefits of moving towards microservices architecture.

In the previous posts in this series, we saw what microservices are and how to start the journey towards broken out services from the monolithic application. In the second post, I talked about not having get calls across microservices. On that, I received some questions. Here's one: "Nice article on…

Read More

What are microservices?

If you pick up a list of talks for any developer conference, you will find at least one talk related to Microservices. As many, I have been fascinated by them for some time now. The obvious question to me was what are microservices. I started reading up on it. I came across Martin Fowler's post. I listened to industry experts such as Clemens Vasters. In a series of books to read on the topic, I started out with the Building Microservices book. Before we proceed further, it is necessary to understand what a service is.

What's a service?

The SOA patterns book says

Service should provide a distinct business function and it should be a coarse-grained piece of logic. One of the characteristics of the service is autonomy, which means the service should be mainly self-sufficient.

I like this definition. The main point to notice here is autonomy. This means a service can be deployed anytime, multiple times a day without any burden of affecting the consumers. It also tries to keep the failures isolated to itself. Loose coupling between these services is another hallmark of service oriented architecture. So, what are microservices then? What's new about them? Are they just SOA done right?

Microservices definition or properties?

There is no industry-wide, fully agreed, one sentence type of definition for microservices. However, there is a set of properties laid out in Building microservices below:

  • They are autonomous.
  • Their boundaries align with business boundaries.
  • They are small.

Now, how small is small enough? Until it doesn't feel too big? To me, they should be small enough to maintain the cohesion. Business boundaries also give us a good idea on their size. If your service spans across multiple business areas, it is too big. If they are too small and have too many boundaries, then you may end up in the death star architecture like this:

To further the point on autonomy, I should be able to independently scale these services. They should also have their own data sources and don't share a monolithic one. I can then call this data independence. One good point I've heard is

You shouldn't make a query/get call to another microservice.

You can hear Scott Bellware and Scott Hanselman discussing the details here

So, microservices are a combination of these properties. Here are few other great talks that made things very clear to me:

Analogy

I tend to shy away from playing the analogy card too much but if/when I get pushed too hard on one on this topic, I am likely to use this.
That drawing is trying to refer to a gas stove. Let's say we have to make chicken curry, steam some veggies and make Indian roti(flat bread) for dinner. I can cook all of these things together on each burner without really affecting one another. If I mess one up, the other one is not affected. I can take them off the burner at any time. That's autonomy. Each of these gets a steady supply of gas and has some their own ingredients. That's data independence. Those ingredients stay in the pot and don't leave it until I put them on my plate (composite UI). I have my boundaries clearly defined. I can use an extra burner if required for something that I need more of. Now, I am realizing the scalability aspect of the system.

In the real world

Let's apply this to my favorite tennis equipment site.
All the red rectangle areas could be services of their own with their own databases. They can be scaled differently with different deployment policies.

One of the highly scaled backend services apps I worked on had 12 different NServicebus endpoints. They were scaled differently at least from the instances perspective. They were deployed separately. They were sort of fault tolerant. They processed tens of millions of records every day.
A couple of components in the chain of endpoints were doing direct calls to SOAP services with monolithic databases. The stored procedures used were in the zone of 5k-7k lines. No one knew how to optimize them or what rules were in them. The components took longer to process than everything else. If they failed the couldn't process their backlog at all. Was this a microservices system? Not really because they were sharing a monolithic database through RPC call. Breaking that up was not a trivial task. Moreover, the endpoints did not represent business boundaries very well.

Avoid the tech evangelism trap

A good chunk of folks in our industry is trying to promote something. More often than not, it is their product FOO and it is better than everyone else's. It is the exact thing you are going to need for the foreseeable future. How many times do we hear this type of stuff? The container products are pushed very hard these days in the Microservices space. The evangelists/promoters will tell you what's right about those but don't go into space of where it doesn't apply. They tell you enough to push you off the cliff but leaving the flying part to you. Very often people just get hung up on these things. When you ask them which problem these are actually solving, you get crickets back. I am not saying all of these are bad ideas but you don't have to have one to have microservices. For the exhaustive list of what does or what doesn't make a microservice, I'd refer to Jimmy Bogard's post

Where do I begin?

It is usually easier to break up a monolith into microservices than starting out in a green field project. So, how do you divide these monoliths? There is no one silver bullet type of answer. You have to take it on a case by case basis. The good place to start is monitoring your monolith and develop a heat map. Kelsey Hightower says "If you tell me that the app is slow, you got to be able to tell me why." on this episode of hanselminutes. They touch on a number of topics in this area along with horizontal scalability. This is where containers come into the picture.

Another place is to dig deeper into the business domain and start developing some consensus around boundaries there. I’ve seen implementations go completely awry if this was missed.

Parting words

I hope this clarified some of the clouds around the definition of microservices. They are not a solution that you can slap on every problem out there. I wouldn't break up my blog into microservices architecture. We will cover more topics in the area later.

If you pick up a list of talks for any developer conference, you will find at least one talk related to Microservices. As many, I have been fascinated by them for some time now. The obvious question to me was what are microservices. I started reading up on it. I…

Read More