Pushing it to the Edge

Many times when I go into a new customer I am going into a place that has attempted to build their own ESB or their own SOA infrastructure. So why are we being brought in? It’s simple. They’ve failed. What’s not so simple are the reasons they have failed.

To begin to understand the problem you first have to understand the people that are usually tasked with building such systems. They are usually very smart and are often very accomplished programmers. Many times they have extremely strong architecture skills. In some cases these folks have had extensive experience with BizTalk or WebSphere or other integration products. So what’s caused people like this to fail?

There are two paths taken but both lead to the same destination and both are traveled because of the same motivations.

The first path, let’s call it the Super Product path is related to the “Golden Hammer” and involves purchasing a platform and making sure all the money spent on that platform is recouped by taking advantage of all its “Enterprise” features and increasing “Governance” by making sure everything flows through the platform. Schemas are built and worshipped and declared sacrosanct. The POC goes well and then everything starts to unravel…

The second path which we’ll call “The Framework” is very similar to the first though its practitioners would often be loathe to admit it and refuse to acknowledge the similarity. They may even practice Agile and have loads of unit tests and mocks and Scrums and Sprints and Fists of Five and other Agile goodness they will say makes them cognizant of and ready to tackle change… This path starts either with a very serious OO design up front or an Open Source product base and then becomes “The Framework”. The creator(s) of “The Framework” presents their masterpiece and makes sure to provide many API’s and configuration options to inject “The Framework” into everything that’s done thereby ensuring everything will benefit from the genius of “The Framework”. The POC goes well and then everything starts to unravel…

So, from whence cometh the unraveling? The Domains. What do I mean by the Domains? Domains are the things that matter in the business and without them there will be no business. They are almost always but not always directly related to the systems and people that are tasked with activities that are directly related to what the business is trying to accomplish. Domains are driven by specialists. Sure there may be pointy haired mangers mucking about but at some level there are specialists and people who know their jobs and tools much better than the IT department ever will. Otherwise, it’s unlikely the company they are all a part of will remain.

Domains appear to always be far from the integration technologist. They appear to always be on the edge. They may be inhabited by fellow IT personnel who’ve been “matrixed” to a department. But they may as well be extraterrestrials because they only know their little world that helps their department accomplish things.

Given that definition of Domains, why did I say almost always instead of always? Well, the one glaring exception is if technology is the company’s business and what kind of technologist would I be if I didn’t let that kind of pedantry interfere with a narrative.

First though to the business…Businesses change. They change all of the time. Obvious! You say. This is why they built the SOA or ESB to begin with using the Super Product or The Framework. It was to create a highly decoupled architecture that allowed for change!

If it’s so obvious then why do both paths lead to the exact opposite? Why do both paths inevitably create a highly coupled architecture where the coupling of systems to systems has been replaced with a more complex coupling to the Super Product or The Framework? Why do I see the same thing all of the time even though I constantly meet smart people at our customer sites?

If I tried to enumerate all of the nuances this blog would become a novel so I’ll attempt to boil it down a little. There are 3 primary reasons this happens

• Struggling against the Domains
• OO and Contract worship applied to integration
• A centralized mindset

One of the most common utterances I hear is “The business is always changing their minds”. Or another way of putting it “I don’t know what they do over there but it’s always changing”…The “over there” part of that statement is the beginning of enlightenment. Much of what people tasked with bringing things together deal with seems constantly “over there”. The integration technologist valiantly struggles to create the proverbial eye while wind howls and cows fly on the edges all around them. They begin to see all of those edges as unruly and the problem that upsets the calm they are trying to bring through their magic schemas and holy contracts and genius frameworks that all shall obey.

But, that unruly edge is where things that really matter happen. The fact is for most businesses the people doing those jobs on the edge and all of the bits flying about are the only reason the technologists are employed. The employees in those Domains are not only more knowledgeable about why they do what they do than their management or the technologists tasked with bringing them together but, they’re also far more subject to the whims of business reality. This means the last thing they want to hear is that you need to make them wait 6 months for a system change or an API change because what they’ve just decided to do cascades through your N-Tiers because they have changed a contract they absolutely double dog promised they wouldn’t in a meeting you had 4 months ago on a Friday afternoon.

For many reading this what I described above makes sense and the first thought may be “Fine. How do we solve it?” We’ll get there I promise. But first we need to go over the next two points and the second one is a doozy and may cause you to flee or for the more aggressive among you submit me to the GOF or WS* fanatics for trial and burning.

Let’s ease into it by agreeing to some basic first principles. Integration problems are generally distributed problems. This means bringing systems together will involve some sort of transport be it network or file system. That medium will have its own metadata and protocol rules and those rules have no knowledge of Domains, Contracts or Hierarchies. A great deal of the time integration problems involve systems over which the technologists charged with tying them together will have no control. Once again those systems will have no knowledge of class design, may not even be written in languages that care about classes and yet still provide critical business value.

If we agree on those first principles then why is the first thing many technologists embark on when charged with leading their companies integration efforts is to create some elaborately, strongly typed, mega design that works perfectly as long nothing changes anything or does anything outside of the extension points they've anticipated…

It’s just bytes flying about people! Many of the places and paths those bytes are going to flow through and end up at don’t give a diseased rat’s behind about your mid tier’s rules. If you’re lucky some of those bytes easily will massage into the form of a SOAP packet. But even there it’s a freaking SOAP packet!!! I see hours and hours spent all of the time insisting on manipulating those blobs of bloated text with strongly typed mapping tools that explode and have to be redeployed if an element is changed or creating an elaborate serialization framework that “saves” developers from having to type icky and impossible things like //*[local-name()=’Foo’ and namespace-uri()=’http://tempuri.org/’] or ToString().ToLower().Contains(“thingy I’m looking for in middle of all that xml goop”).

Trust me there is more than enough work just figuring out the wiring and communications and getting the bytes and bits and blobs and gobs into rudimentarily digestible shapes for each domain without complicating it with a bunch of stuff that assumes loads of good behavior. Neuron and other things like it are called service buses for a reason! Think about buses. Anyone can get on those things. And if for some reason they change a published route the bus is not supposed to spontaneously explode. Actually it’s worse in software to insist on becoming overly intimate with your passengers because a service bus is more like a busplanetraincartractor but calling it that would mess up the acronym consultants.

Ironically, specialized contracts and hierarchies and rules and all of that OO goodness can absolutely provide coherence and benefits to a Domain. So if you are finding all of this talk of unruly bits that could care less about your carefully crafted schemas and hierarchies’ distasteful then head to the edge! Sure you’ll be a flying cow flung about by the insane whims of management and market forces but you’ll be in a world much more conducive to your mindset. It is unlikely the accounting department you’re matrixed to will insist on you writing them a Silverlight driven virtual art gallery or that the dental software company you’re working at will announce they’ve become a furniture consignment provider.

For those of us in the integration world though we can make no such claims. We get asked things like I’d like the CRM system to be aware of who is using the Silverlight art gallery so we can kick off a campaign for our gallery and by the way our parent company is embarking on a radical horizontal strategy and have purchased the world’s leading dental provider and furniture consignment companies and would like your software to help us roll all of that up to one mega GL and make sure the information is put into our data warehouse.

Which sort of leads me to the final mistake of a centralized mindset and while this one is not as egregious as the first two it is nonetheless problematic. Let’s say someone gets past the first two issues. They almost always trip up on this one. That is they see the physical architecture as one thing that is managed completely centrally and controlled by their team. In other words they see a server and things flow through those servers. So what’s the issue? Many times the Domain doesn’t need or want to be held up by having to deploy or develop with a centralized server in mind but they still want the power of all the tools available to the server. Some people hear this and go “Huh?”.

But it’s really not that crazy and it’s not so hard to see what’s wrong with the mindset once we use the example of WCF itself. Would people want to use WCF as much if it only worked in IIS? You can run a service host in your WinForm or in a Windows Service without everything having to flow through one WCF uber server. Would you want to use WCF if it forced you to do so?

So why, would you ever want to create your SOA or ESB in such a way that things only worked if they are hosted in a certain process space? Why should everything share the same config or runtime environment? Wouldn’t it be more advantageous to allow things to flow naturally in such a way that edge specific processing could occur in ways specific to the Domain(s) along that edge?

Can You Smell What the Integration Nerd is Cooking?

The answer to all of these issues becomes more and more apparent. Push it to the Edge! Or more prescriptively an ESB or SOA or Cloud or whatever you’re calling your bits that help glue things together should perform the minimum amount of processing necessary to successfully deliver the payload to the correct Domain in a form the Domain can digest successfully.

Crazy talk coming from a guy who works for an integration product? Not really. I want people to be successful with Neuron. While it’s harder to make the mistakes I’ve described with Neuron it’s exceedingly possible. Neuron provides tons of tools that allow you to massage, manipulate, reshape, reroute and otherwise munge and inspect the bits flowing through it as much as you want. It also provides a ton of help transitioning from protocol to protocol, publishing and subscribing etc. But, if you insist on filling your Neuron server with 10,000 helpers that turn SOAP into objects or you insist on all validation occurring at the mid-tier or you insist on not using the API because of some WS* bigotry then you will likely head down the wrong path.

Or… in words sure to offend the environmentally conscious everywhere… If the guys who processed the trees right after they were chopped down decided everything would end up to be toothpicks or cabinets then we’d have no baseball which would mess up mom and apple pie and therefore America which would kill worldwide democracy and lead to Soylent Green.

So it’s simple.

Ok maybe it’s not. But using Neuron correctly can help make it simpler.