Yogiraj Aradhye - Page 1

Why should you avoid get calls across microservices?

September 18, 2017 on microservices, Distributed-Systems, SOA

In the previous posts in this series, we saw what microservices are and how to start the journey towards broken out services from the monolithic application.

In the second post, I talked about not having get calls across microservices. On that, I received some questions. Here's one:

"Nice article on microservices. But i did not get reasoning behind
"You shouldn't make a query/get call to another microservice."

My reply to that is below:

As a service, you should have your own store for the data you need. If you don't, what happens if the service you're using for get calls goes down for an extended period of time? You are blocked. Your function that depends on it fails now. What if you have multiple of these? You just increased your reasons to fail by that factor.

To explain this further, too many blocking get calls may result into a system that can get completely stalled. What if the individual calls that are happening have different SLAs? Do we block the entire request for the slowest call?

One of the projects I worked on in the past had these types of calls spread through out the system. In the worst cases, the http requests took well over a minute to return. Is this acceptable?

To counter this, the team went towards caching the entire relational data stores. Pulling all that information down became very tricky. The data sync was happening once a month. The sync job itself took days to finish, pushing the changes further back. I have seen many project teams going towards this type of solution and eventually running into the scale problem. Then they settle on syncing the data using messaging.

So, what is the trade off? I explained it below:

The data duplication that may come with this is an acceptable trade off for scale. In other words, you are giving up some consistency and moving towards eventual consistency for scale and availability.

One of the teams I worked with did a good job differentiating the types of messages. Some of the changes had to be pushed forward at a higher priority, some required a little more processing, etc. We built different routes for those messages so that we could maintain our SLA for these messages. The data was still cached but we were able to get rid of the sync jobs that were trying to sync all the data at once. Messaging enables different kinds of patterns. I recommend Enterprise Integration Patterns book for those patterns. It goes deep into the different kinds of messaging patterns.

I used this example afterwards:

For a banking app, you can say, there are $100 in your account as of the last sync time instead of $0 because you can't hit that service. People are much more likely to freak out in the later scenario than the one before. This method scales too since we have reduced the consistency level to eventual. It sounds radical but you are trading in strong consistency for scale and user experience. I hope that clears things up for you. I think I should write another post on this :)

I went on further to express the points below:

Another reason is network. You can't take it for granted. In any of these situations, you don't want your customers to get affected by this implementation detail. The way to avoid this is by building your own store beforehand and keep it synced through messaging.

You don't have a distributed system if the system is not resilient enough to consider the network loss. This is something that needs to be considered on day one. Your customers are not going to care that your network failed. They want your app to work.

If your system needs too much of a cross service chatter then it could be a sign of wrong system boundaries. It means that the service is just too fine grained. Your services could be suffering from the nanoservices anti pattern. It is a subtle problem. The SOA patterns book goes into the details of that. Some of the services may require merging.

The Netflix approach

They go the hybrid route. They hit the local cache store first, if it is not available, they fall back on cross boundary calls. Josh Evans explains that very eloquently in this talk.

To summarize, we are giving up some consistency for autonomy. We can solve the consistency problem using data pumps or sync jobs easily but autonomy has to remain as strong as possible. Without autonomy, you won't be able to materialize any benefits of moving towards microservices architecture.

In the previous posts in this series, we saw what microservices are and how to start the journey towards broken out services from the monolithic application. In the second post, I talked about not having get calls across microservices. On that, I received some questions. Here's one: "Nice article on…

Sending JSON message natively to RabbitMQ for NServiceBus subscribers

September 01, 2017 on Distributed-Systems, NServicebus, RabbitMQ, JSON

I was working on putting together some sample code to demonstrate what data synchronization services may look like. I will write a post about the details and purposes of this in a later post. The TLDR; version for that is, I have a trigger which gets fired on inserting a new row in a table, it inserts a row in the NServiceBus queue in the same database. The handler then picks up that message and puts in the RabbitMQ directly. For further illustration, please see the diagram below:

I followed NServiceBus documentation for sending the message natively to the RabbitMQ. In their example they used XML. In my case, I was working with JSON. After pushing the message on to the queue, I ran into a problem on the subscriber side. Here's the error:

 2017-08-29 15:45:48.269 ERROR NServiceBus.RecoverabilityExecutor Moving message 'd38b59e7-6680-416e-a679-d9a016972240' to the error queue 'error' because processing failed due to an exception:
NServiceBus.MessageDeserializationException: An error occurred while attempting to extract logical messages from transport message d38b59e7-6680-416e-a679-d9a016972240 ---> System.Exception: Could not find metadata for 'Newtonsoft.Json.Linq.JObject'.
Ensure the following:
1. 'Newtonsoft.Json.Linq.JObject' is included in initial scanning. 
2. 'Newtonsoft.Json.Linq.JObject' implements either 'IMessage', 'IEvent' or 'ICommand' or alternatively, if you don't want to implement an interface, you can use 'Unobtrusive Mode'.
 at NServiceBus.Unicast.Messages.MessageMetadataRegistry.GetMessageMetadata(Type messageType)
at NServiceBus.Pipeline.LogicalMessageFactory.Create(Type messageType, Object message)
at NServiceBus.DeserializeLogicalMessagesConnector.Extract(IncomingMessage physicalMessage)
at NServiceBus.DeserializeLogicalMessagesConnector.ExtractWithExceptionHandling(IncomingMessage message)
--- End of inner exception stack trace ---
at NServiceBus.DeserializeLogicalMessagesConnector.ExtractWithExceptionHandling(IncomingMessage message)
at NServiceBus.DeserializeLogicalMessagesConnector.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

This was happening even before my breakpoint in the message handler was hit. I had Newtonsoft.Json referenced correctly. It was getting scanned as well. So, what was happening?

After feverishly scratching my head(and losing a little more hair) and some StackOverflow searches, I started to look at my message on the queue. It looked like below:

{
   "FirstName":"Jane",
   "LastName":"Doe",
    "Id":"44ECBBFD-85B6-47C8-9E08-3DCF2633862C"
}

It looks like a perfectly valid JSON. When NServiceBus picked it up off the queue, it left it as a JObject. Why? Because NServiceBus couldn't figure out the type. The type information was not part of the message. This type property has to be on the top. So, the JSON should look like below:

{
   "$type":"Messages.NewUser"
   "FirstName":"Jane",
   "LastName":"Doe",
    "Id":"44ECBBFD-85B6-47C8-9E08-3DCF2633862C"
}

There are multiple options such as using JsonConverter, adding the type property in the message, etc. I went to a simpler route as below:

 var jObjectFromMessage = JObject.FromObject(message);
 var typeName = typeof(NewUser).FullName;
 jObjectFromMessage.AddFirst(new JProperty("$type", typeName));
 var serializedMessage = jObjectFromMessage.ToString();

And that worked! The complete handler code looks like below:

public async Task Handle(NewUser message, IMessageHandlerContext context)
{
     var factory = new ConnectionFactory
     {
          HostName = "localhost",
          UserName = "guest",
          Password = "guest"
     };

     await Task.Run(() =>
     {
         using (var connection = factory.CreateConnection())
         {
             using (var channel = connection.CreateModel())
             {
                  var properties = channel.CreateBasicProperties();
                  properties.MessageId = Guid.NewGuid().ToString();

                  var jObjectFromMessage = JObject.FromObject(message);
                  var typeName = typeof(NewUser).FullName;
                  jObjectFromMessage.AddFirst(new JProperty("$type", typeName));
                  var serializedMessage = jObjectFromMessage.ToString();                        
                  var messageBytes = Encoding.UTF8.GetBytes(serializedMessage);

                  channel.QueueDeclare("SyncUsers.RabbitMqEndpoint", true, autoDelete: false, exclusive: false);
                  channel.BasicPublish(string.Empty, "SyncUsers.RabbitMqEndpoint", false, properties,  messageBytes);

                }
            }
     });
}

That's it. I hope this helps if you are trying to send JSON messages directly to RabbitMQ for NServiceBus subscribers.

UPDATE 09/04/2017

Simon Cropp on Twitter sent me a note about enclosed message types serialization headers. I tried them out and that worked too. I removed jObjectFromMessage.AddFirst(new JProperty("$type", typeName)); and added NServiceBus.EnclosedMessageTypes header to my message like below:

   var typeName = typeof(NewUser).FullName;
   var properties = channel.CreateBasicProperties();
   properties.MessageId = Guid.NewGuid().ToString();
   properties.Headers =  new Dictionary<string, object> {{"NServiceBus.EnclosedMessageTypes", typeName}};

This could be useful if you don't have control over the serialization.

Tackling a monolith - where to begin?

August 01, 2017 on microservices, Distributed-Systems

In the previous post, we saw what microservices are. We also looked at how they differ from the big monolithic apps.

In the startup/earlier days, you did a file => new project in Visual Studio or your favorite IDE, designed some perfectly normalized SQL database, brought in an ORM on top of it for reasons. It worked great for a while. You brought in teams of people to do heads down development. The app has grown in size along with the user base and the org. With that, you are running into problems that you didn't anticipate before. The application is feeling slower, the complaints of timeout are coming in and bug reports are increasing. The backlog of new features is starting to look bigger. The tech debt is out of control. Engineering teams are frustrated at best because they know there are problems but their hands are tied. They have pushed some of the processing to the scheduled jobs but now those jobs are running into problems. The app is feeling like a giant monolith. The business functions are vertical in nature but the supporting IT teams and solutions are cross-cutting. This has increased coupling and complexity of the system. Even smaller changes take huge app deployment lasting hours which scares everyone. The delivery to production takes longer than ever, adding to the frustration. The current infrastructure has been pushed beyond its limits. The question of scaling further is very real. Breaking this app makes sense based on all the material out there but where to begin? How do we create microservices out of this while delivering features that keep the business viable? Tackling up the monolith seems like a necessary step but how do we justify the costs for breaking it up? How do we convince the business that this is not just another refactoring effort from the engineering team? Very real questions! This is a very common scenario these days. Where do we begin when El Capitan of a monolith is staring you right in the face?
I have been there. Now, let's take a look at some of the ways.

Take clues from refactoring practices

Over a period of time, all codebases gain some tech debts.There are times when you deliberately punt on problems. The technology changes, the nature of problems is never static and yesterday's solution becomes obsolete. One approach I've taken is to refactor code just a little bit at a time, just around the area that your new feature code is going to touch. Of course you'd need to co-ordinate this with your pull request reviewers so that they don't get surprised by more code changes than necessary. This makes your code a little better today than yesterday. Since it is incremental and scoped to the small area, this can become the team's habit in a short amount of time. As opposed to this, I was never able to sell big bang refactoring to the business or project management. Coincidently, I found this supporting post from Steve Smith essentially stating the same thing. So, when it comes to tackling a monolith, it is essential to take an incremental approach instead of a big bang one.

Business Alignment

One other step that can be done in parallel is to really understand the business domain from the perspectives of bounded contexts. We can start isolating the business processes in a way that they don’t leave their boundaries. This is not a trivial task to undertake. Often times, it involves a lot of different people from the org and knowledge of the existing processes. If the app is constructed in vertical slices instead of layers as shown by Jimmy Bogard in this talk, we can consider that our app is ahead of the game.

One reason we want to do this is to understand the misalignment that has happened throughout the organization in terms of teams and functions by horizontal layering. The cross-cutting services for everything may seem appealing at first because of the code reuse, upfront time savings and the costs but it insidiously increases the tight coupling. That happens because the reason for change is not considered. Over a period of time, the gap increase by a natural progression of business processes and the horizontal structure of supporting teams and applications. This leads to too much of cross communication between services. If they are disjointed services, that could end up making the system incredibly chatty.

Another big purpose behind this is to get the organizational buy-in into this type of project. It is important for business to understand why vertical slicing is necessary, how much does it cost currently to deliver a piece of code, how this can directly improve the delivery and the possibilities it opens up from the end user’s perspective. It can also show them places to improve efficiencies by cutting down redundancies.

After identifying these areas, we want to establish the communication patterns between different pieces and start establishing service layer agreements for those. For example, when a user buys an item on an ecommerce site, do we show them a spinner until it gets shipped or is it ok to send them an email saying we will send you updates as the order goes through the different steps of processing.

Choosing the area

It is very important to have monitoring in place for a variety of reasons. In this case, we can generate a heat map to identify which parts of the app are heavily used over the others. This can help us setting some weights on these areas. Highly used areas and the slowest performing could be a good starting place.

One other thing to consider is how many new features are coming down the pipe and if the system is no longer in active development. If there is too much happening at once, changing architecture may delay the delivery. The real answer here is, it depends on the type of changes and how much time we can buy from the business. If the system is no longer under active development, it could be a candidate for retirement or get isolated enough for near future retirement. This is not a good candidate.

Remote procedure calls

This is another area that comes in the way of scaling the apps. You don't want to create an accidental DDOS attack on someone's service because you spawn hundreds of your container instances and you call them directly in your process. Also, the network shouldn't be taken for granted. This is another good area to start analysis right away. Eliminating the read RPC calls may not be a trivial task but identifying this is an important step. Our app is not going to scale very well if we are hardcoding the calls to other services. Service discovery with Consul can be helpful. We will see the details on Consul in the later posts.

Shared relational databases

One of the problematic and yet very common forms of integration patterns is the shared relational database. It creates coupling that becomes very hard to remove. Once we start introducing foreign keys into it, it becomes even harder to scale. Is it fair to say the degree of normalization is inversely proportional to the ability to scale the database? Even if some magical indexing strategy makes the calls very responsive, its maintenance is cumbersome at best. I'd recommend this talk by Pat Helland. He has also written some papers in this area. Knowing this, we can start identifying some of the most heavily used tables, their relationships and how applications are using them. This sort of analysis can give teams ideas about reducing the dependencies on these tables and setting them up for the complete break up in the future.

Containers

Before jumping too much into the container world, you will have to increase the organizational awareness and education around them. This is especially true in the Windows environments since most of the containers come from the Linux world. How do we leverage these? One place to try them out is in your build and test systems as mentioned in this .Net rocks episode.The risk is lower. The faster build and test pipeline can help in delivering the software quicker.

Parting words

All of these can be done, in parallel, with one goal in mind of identifying areas that could come in the way of breaking up a monolith. After identification, the app areas can slowly start making changes for some immediate cheap wins. I hope this helps.

In the previous post, we saw what microservices are. We also looked at how they differ from the big monolithic apps. In the startup/earlier days, you did a file => new project in Visual Studio or your favorite IDE, designed some perfectly normalized SQL database, brought in an ORM on…

What are microservices?

July 23, 2017 on microservices, SOA, Distributed-Systems

If you pick up a list of talks for any developer conference, you will find at least one talk related to Microservices. As many, I have been fascinated by them for some time now. The obvious question to me was what are microservices. I started reading up on it. I came across Martin Fowler's post. I listened to industry experts such as Clemens Vasters. In a series of books to read on the topic, I started out with the Building Microservices book. Before we proceed further, it is necessary to understand what a service is.

What's a service?

The SOA patterns book says

Service should provide a distinct business function and it should be a coarse-grained piece of logic. One of the characteristics of the service is autonomy, which means the service should be mainly self-sufficient.

I like this definition. The main point to notice here is autonomy. This means a service can be deployed anytime, multiple times a day without any burden of affecting the consumers. It also tries to keep the failures isolated to itself. Loose coupling between these services is another hallmark of service oriented architecture. So, what are microservices then? What's new about them? Are they just SOA done right?

Microservices definition or properties?

There is no industry-wide, fully agreed, one sentence type of definition for microservices. However, there is a set of properties laid out in Building microservices below:

They are autonomous.
Their boundaries align with business boundaries.
They are small.

Now, how small is small enough? Until it doesn't feel too big? To me, they should be small enough to maintain the cohesion. Business boundaries also give us a good idea on their size. If your service spans across multiple business areas, it is too big. If they are too small and have too many boundaries, then you may end up in the death star architecture like this:

To further the point on autonomy, I should be able to independently scale these services. They should also have their own data sources and don't share a monolithic one. I can then call this data independence. One good point I've heard is

You shouldn't make a query/get call to another microservice.

You can hear Scott Bellware and Scott Hanselman discussing the details here

So, microservices are a combination of these properties. Here are few other great talks that made things very clear to me:

Analogy

I tend to shy away from playing the analogy card too much but if/when I get pushed too hard on one on this topic, I am likely to use this.
That drawing is trying to refer to a gas stove. Let's say we have to make chicken curry, steam some veggies and make Indian roti(flat bread) for dinner. I can cook all of these things together on each burner without really affecting one another. If I mess one up, the other one is not affected. I can take them off the burner at any time. That's autonomy. Each of these gets a steady supply of gas and has some their own ingredients. That's data independence. Those ingredients stay in the pot and don't leave it until I put them on my plate (composite UI). I have my boundaries clearly defined. I can use an extra burner if required for something that I need more of. Now, I am realizing the scalability aspect of the system.

In the real world

Let's apply this to my favorite tennis equipment site.
All the red rectangle areas could be services of their own with their own databases. They can be scaled differently with different deployment policies.

One of the highly scaled backend services apps I worked on had 12 different NServicebus endpoints. They were scaled differently at least from the instances perspective. They were deployed separately. They were sort of fault tolerant. They processed tens of millions of records every day.
A couple of components in the chain of endpoints were doing direct calls to SOAP services with monolithic databases. The stored procedures used were in the zone of 5k-7k lines. No one knew how to optimize them or what rules were in them. The components took longer to process than everything else. If they failed the couldn't process their backlog at all. Was this a microservices system? Not really because they were sharing a monolithic database through RPC call. Breaking that up was not a trivial task. Moreover, the endpoints did not represent business boundaries very well.

Avoid the tech evangelism trap

A good chunk of folks in our industry is trying to promote something. More often than not, it is their product FOO and it is better than everyone else's. It is the exact thing you are going to need for the foreseeable future. How many times do we hear this type of stuff? The container products are pushed very hard these days in the Microservices space. The evangelists/promoters will tell you what's right about those but don't go into space of where it doesn't apply. They tell you enough to push you off the cliff but leaving the flying part to you. Very often people just get hung up on these things. When you ask them which problem these are actually solving, you get crickets back. I am not saying all of these are bad ideas but you don't have to have one to have microservices. For the exhaustive list of what does or what doesn't make a microservice, I'd refer to Jimmy Bogard's post

Where do I begin?

It is usually easier to break up a monolith into microservices than starting out in a green field project. So, how do you divide these monoliths? There is no one silver bullet type of answer. You have to take it on a case by case basis. The good place to start is monitoring your monolith and develop a heat map. Kelsey Hightower says "If you tell me that the app is slow, you got to be able to tell me why." on this episode of hanselminutes. They touch on a number of topics in this area along with horizontal scalability. This is where containers come into the picture.

Another place is to dig deeper into the business domain and start developing some consensus around boundaries there. I’ve seen implementations go completely awry if this was missed.

Parting words

I hope this clarified some of the clouds around the definition of microservices. They are not a solution that you can slap on every problem out there. I wouldn't break up my blog into microservices architecture. We will cover more topics in the area later.

Continuous integration For a Xamarin App with Jenkins and deployments with Hockey App

June 27, 2016 on Mobile App Development, Continuous Integration, HockeyApp, Jenkins

In the previous post, we saw why Jenkins makes sense for Xamarin build, at least for now. If you have previously worked with continuous integration systems such as Team City, the steps in Jenkins will look familiar to you. The steps follow a logical flow. So, even if you're brand new to this, the learning curve is not that steep.

Jenkins supports a Master/Slave setup. If there is a master running on Windows, perhaps a Mac slave may do the trick. For simplicity, we are going to look at the master running on a Mac.

Jenkins provides a download for Mac and Windows at Jenkins .io. However, I am not going to recommend that. The installer creates a new shared user Jenkins on Mac. This complicates things if Jenkins needs to access anything from the Keychain of the logged in user(likely to be a service account or your own). So, if you are working with TFS Git or any Git repo over HTTPS, you are probably using OSXKeychain Helper or Microsoft's Credential Manager. Jenkins user simply can't access those easily. Creating another user just for Jenkins was not an option for me. We will now see how to get around this issue.

We are going to need Homebrew installed on the Mac. It is a package/application manager for Mac, Mac's Chocolatey from Windows world.
After installation, simply run the commands below:

brew Install Jenkins
brew services start Jenkins

This way Jenkins gets installed for the currently logged in user and the credentials/keychain access problem does not come up.

In order to make Jenkins site available from other machines, follow instructions below:

Open homebrew.mxcl.jenkins.plist located in/usr/local/Cellar/Jenkins/2.7 (your installed version could be different)
Change the -httpListenAddress=127.0.0.1 to 0.0.0.0
Fix the url in Jenkins Configuration

Restart Jenkins using commands below

brew services stop jenkins
brew servics start jenkins

And that's it, you should be able to hit the site installed on the port 8080 if your firewall allows traffic on that port.

Afterwards, I recommend following these very detailed instructions by Jeffry van de Vuurst except for the build script. My modified version of the build script is available on my Github repo.I have updated it to work with the new folder structure Xamarin uses to put the .IPA files in and added a command to restore packages from sources other than the nuget.org feed.

Xamarin also provides documentation on setting up Jenkins. They recommend downloading the Jenkins app which uses the installer instead of Homebrew, so you may run into issues related to credentials.

There are two ways of triggering the builds Polling and Adding Git hooks. This post goes into the details of both. For simplicity, I use polling with Cron schedule of * * * * *.

Hockey App provides a way of pushing the builds on to the devices. It is fairly cheap and simple to setup. You can create multiple teams and push different builds on to their devices. In Jenkins world, "There is a plugin for that!" is very true. They of course have a plugin for the Hockey App. The setup is very straight forward and the details on setting it up can be found in another post by Jeffry.

At this point, the setup is pretty much all set. It works just fine if you have the provisioning profile downloaded on the build machines using Xcode. While setting all that up in the Apple Developer Portal, it is very important to remember to create the certificate on the build machine otherwise the build fails. It can't create an .IPA file because of the private key issues.

Whenever I ran into issues, I always went to the workspace folder of Jenkins, opened the solution in Xamarin Studio and built it. That revealed a lot of details on the problems that lead to a resolution.

Continuous Integration For a Xamarin app, a Microsoft update

June 22, 2016 on Mobile App Development, Continuous Integration, TFS

Recently, Microsoft ended up buying Xamarin. It was very much anticipated, it always felt more of a "when" than "if" and finally it happened. Microsoft made the Xamarin suites (most of it) available with the MSDN licenses. It removed a barrier for a lot of companies and independent developers. Now, if you're a .NET developer and want to build a cross platform application, you may not have to look beyond Xamarin.

In this series of mobile app development, we are going to first tackle the continuous integration.

In today's Enterprise world, the use of TFS is prevalent. It usually tends be an on-prem one. Thankfully, TFS allows us to create Git repositories. For now, it only supports HTTP. SSH is coming to TFS with Update 3.The timeline for features is here. I have also seen the on-prem TFS version is slightly behind the cloud one. The features mentioned here are not available in an on-prem version. There is some starter documentation by Xamarin but it may not be enough to see you though turbulence.

For the Xamarin apps, the Mac TFS build agent has gone through a lot of churn. It used to be VSO Agent.It is being deprecated in favor of the VSTS agent.The new agent didn't support on-prem TFS until version 2.101.0. I learned it the hard way. It is in a preview state as of June 20th, 2016. It lacks community support. If you hit a road block, you may have to dig into code yourself (yay, OSS!) or open up a ticket on GitHub and wait for the team's reply.

Based on the feature gaps between visual studio online and TFS on-prem, lack of community support, confusing MSDN articles and issues with the preview version of Mac Build Agent, I'd recommend not to use TFS for the continuous integration purposes, at least for now. They are moving in the right direction but not fully there yet.

Instead, the better option is Jenkins. It is free. It has a great community behind it. It has rich plugin ecosystem. It has a nice upgrade and downgrade functionality. If for some reason, upgrading a plugin or Jenkins itself is not working the way the previously installed version did, you can simply go back to that version without too much of a hassle. In other words, it just works!

The next post in the series is available at http://aradhye.com/continuous-integration-for-xamarin-app-with-jenkins-and-deployments-with-hockey-app/