October 31, 2005

Let confusion reign

The Enterprise Service Bus (ESB) debacle is a prevailing sign of the integration industry's utter disorganization and confusion. Customers & vendors do not seem to know or agree upon what they wants in the integration space -- only that it involves some magical mixture of reliable messaging middleware, business process orchestration, and XML-aware routing and data transformation. So, no one really can agree on what an ESB is, other than it's some sort of bundle of features that might be implemented by one or more products and tied together in an "architecture" (whatever that is).

Despite this frustration, I tend to think it might be a good thing (in the long run). Indecision and acrimony is usually is an indication that something is important. How many things that are important in life are nearly impossible to define in an agreed upon manner? What I would like to address today are the arguments against the ESB, and also the caveats to consider when adopting one. Buzzword bingo follows; please try not to cry (much). Also, I come from a biased background (BEA consulting), but that doesn't mean what I'm saying has anything to do with BEA's agenda, it's just my interpretation of the market.

The ESB opponents seem to have three arguments, not always held simultaneously:

a. ESB is not a product, it's a pattern (aka. I can do that stuff today with [insert favorite tool here] )
b. ESB is proprietary, web standards only should be used (aka. the "fabric" approach).
c. ESB is unnecessary, as is all of SOAP and WS-*, we all should be using REST-style XML+HTTP+SSL.

IN short, my answers are:
a. FUD.
b. Standards are absolutely necessary but can sometimes be overrated, or solidified too quickly, before the industry knows what it's doing.
c. B.S.

Argument (a) is a game of FUD to me: vendors and interest groups are trying to protect their turf. For example, Microsoft claims they have everything an ESB has with Biztalk, which is true, but it's disingenuous. Respected developers are falling in love with Biztalk 2004, but its not like this is particularly new -- BEA has had everything Biztalk has (and thus an ESB has) with WebLogic Integration (WLI) 8.1, since mid-2003. Yet BEA doesn't claim WLI is an ESB (though you could build an ESB with it). BEA claims the AquaLogic Service Bus is an ESB. It supports stateless multi-transport / multi-format stateless transformation & routing in an appliance-like manner -- no custom code other than XPath, XQuery and a graphical pipeline language. IBM recently announced 2 ESB's -- one based on WebSphere Application Server (for standards-based interop) and one based on MQ (for proprietary + standards interop). Then there are the plethora of smaller ESB's from Cape Clear, Sonic, Polarlake, IONA, Fiorano, etc. They're all ESB-like and yet none of them fit the broadest definiton. Some are more optimized to certain use cases than others. Some ESB vendors still require code for the "last mile", others need it for cases such as long running processes. And one has to wonder whether BPEL support is required or not to be an ESB, even though it's not even standardized yet!

So the story isn't over yet as to what an ESB should / should not be. The takeaway is that many ESB products make life simpler than using traditional brokers, but no ESB products (yet) cover all popular integration use cases. YMMV.

Argument (b) has some merit to it, which is why I will address it at length.

Many ESB's push their proprietary messaging heritage and have features that virtually ensure lock-in. Cape Clear distinguishes between "service-centric vs. message-centric" ESBs, and have a paper on the subject. It is marketing focused (being a thinly-veiled attack on Sonic), and it pushes the purist "fabric" approach to ESB a bit too much, but there are good points. My view on this debate is to look at it in terms of feature lock-in.

There are two classes of features: fundamental and instrumental. Fundamental features are "required" features to do anything useful with a product. Instrumental features are "tooling", and not required for the core operation of the product.

Fundamental features tend to be the core policies or abstractions exposed by a product. In an ESB, the fundamental features are routing, transformation, security, auditing, SLA enforcement, and management. These features must have, at the very least, a clearly-delineated mode that is on track with a standards-based "least common denominator" approach, or else you will be locked-in to vendor's infrastructure.

For example, Sonic ESB likes to push "distributed SOA" or "itinerary-based routing" as a core feature of their ESB, one that no other vendor can touch. Itinerary-based routing is the idea that a message is like an "agent", it has a series of endpoints in its header and the ESB infrastructure will read this itinerary as it routes the message through the network. This approach to routing has some intuitive appeal, and was also pushed, briefly, by Microsoft with their WS-Routing specification.

However, all of this is irrelevant now -- the standards process has jettisoned WS-Routing in favour of WS-Addressing. Itineraries are inherently insecure because every ESB intermediary must modify the message header as it is going through its itinerary. WS-Addressing adopts the "next hop" approach to routing, the same one that the TCP/IP adopted: the IP header is never modified, and routing decisions are made by intermediaries (such as a Cisco switch or router). Microsoft published a paper in mid-2004 explaining how to handle web services routing with this model.

The moral of this story is that routing is a fundamental feature, and "next hop" routing is the only standard way to approach it in a transport-neutral fashion. Arguments that claim other ESBs or BPM engines like BizTalk are "hub and spoke", and not "truly distributed" are disingenuous. The entire internet is based on "next hop" routing with registry-based lookup (DNS). The web services world will likely follow the same approach when ESB's start integrating with UDDI registries.

Sonic does make one good argument, though flawed, about the benefits of their approach. Itinerary-based routing facilitates a global process view instead of a splintered process view. That is, you can "orchestrate the orchestrations" across multiple intermediaries. The flaw in this is that the argument really isn't about routing, it's about global interaction management. All of these modern ESB/BPM hybrid engines, whether BizTalk, WLI, or BPEL-based are about "orchestration". They all require a central conductor to manage the process state. BPEL is just-another-way to implement an orchestration -- something you could also do with BizTalk's XLANG or WLI's JPD. But one doesn't need the drawbacks of itinerary-based routing to get a global view, one just needs a a contractual set of interactions -- something also known as choreography. Perhaps WS-CDL (Choreography Description Language) will eventually catch on to fill this void. Perhaps a future BPEL extension will -- I've noticed that IBM has released a WS-BPEL 2.0 sub-process extension just a few weeks ago. Until the industry figures out how it wants to handle choreography, which will probably require a number of years, itinerary-based routing is 100% proprietary, will only work on a single vendor's ESB (though you might be able to write a lot of custom code to bridge the gap) and will likely never be standardized. Use it if it makes sense to you, but understand the risks.

Instrumental features, as I mentioned, tend to be more pluggable -- they're tooling, they're "nice to have", but they're not absolutely necessary for the product to operate. That is, assuming one can separate the policy from the underlying implementation, supporting proprietary protocols, data formats or transports can effectively (but not completely) become instrumental. The key is to ensure that there is a very clear demarcation between what is core to the ESB and what is not, and any dependency on instrumental features has a clear abstraction. This is arguably where Java EE has always shined -- in creating a market for pluggable device drivers, whether database (JDBC), messaging (JMS), or general connectivity (JCA), allowing you to choose whatever core programmatic model you'd like for your application.

So, in an ESB built on Java EE, reliability and security could be made pluggable between SSL and WS-Security, or JMS or WS-ReliableMessaging -- assuming your ESB vendor chooses to do so in its core framework. One can use JMS in the short term, and move to a WS-ReliableMessaging endpoint once it is more widely adopted. An ESB should allow this without requiring any code changes. Routing can be generalized to rely on any metadata in any one of transport-specific headers (HTTP, JMS, etc.), SOAP standard headers, or on the content of the message itself. ESB intermediary should be able to expose both a REST-based XML+HTTPS endpoint and a SOAP-based WS-Security endpoint to the same service -- so long as the underlying service has some known way of handling security, and the ESB knows how to translate between approaches. Last-mile connectivity to legacy enterprise systems or packaged applications can be made effectively (but again, not completely) instrumental by using API-level standards such as the Java Connector Architecture (JCA), again assuming there is a general way of mapping non-XML data into XML data and vice-versa.

One key fundamental feature in the ESB market that is lacking good standards support is in the manageability and SLA enforcement space. The standards here (such as WSDM) are rather poorly adopted and lack a lot of what is needed. It will likely be years before standards evolve here, so every ESB vendor will have the opportunity to at least "try" to provide some level of interoperability with sub-optimal standards like JMX or SNMP.

A trend you'll notice here is that API-level standards are very flexible but potentially have a lot of labour associated with them. This to me is contrary whole point of an ESB -- to reduce the amount of labour required to integrate applications! If your ESB requires you to write a lot of custom code, it's not doing its job as well as it should.

Another other trend is that the ESB vendor has to expose its own set of abstractions to implement transformation, routing, and management. Consistent with my earlier point, these abstractions should NOT have their primary exposure through an API, they should be exposed themselves as standard services or through some form of management interface. But, having said that, there will probably be some level of lock-in on how well the ESB vendor manages the distinction between the fundamental features of management, transformation, routing, and endpoint bindings.

There have been early attempts at standardizing this, with mixed results. There's the BPEL 1.1 draft and Apache WSIF, both of which are useful, have some teething problems, and probably will never be adopted by a standards body in their current form. OASIS is working on WS-BPEL 2.0 which has some very significant changes over 1.1. And I believe Java Business Integration (JBI) is hoped to be a generalized alternative to WSIF. Nevertheless, in theory, you can port a BPEL 1.1 + WSIF process between Oracle and IBM's BPEL engines, though I'd be curious about how well that would work in practice. But both WSIF and JBI assume your ESB is implemented on Java EE! There will be no standard way to port a BPEL 1.1 or WS-BPEL 2.0 process that uses WSIF or JBI onto BizTalk, for example.

The moral of the standards story: this stuff is too new to expect a truly portable ESB execution language. WS-BPEL 2.0 will be close, but in practise it probably will only be portable among Java EE based containers. That might be OK -- SQL is a standard that isn't the same everywhere and certainly isn't portable, but it has been a success in terms of adoption. But WS-BPEL is arguably not appropriate for stateless ESB's like AquaLogic. Should an ESB vendor adopt BPEL for all message exchange patterns, or should it have seperate products that optimize stateful vs. stateless processing? We have a cart-before-the-horse standard, yet again! Hence proprietary extensions will abound.

My final point on argument (b) is that the "fabric" approach seems to only be pushed by small vendors that have nothing to lose, but also don't have a long track record. The WS-* standards aren't completely ready yet, so there needs to be the ability for an enterprise to choose de facto and/or proprietary standards that are suitable for them in the short run. This of course is only appropriate for intra-enterprise services, or tighly-coupled cross-enterprise integration -- which is why SaaS proponents often discount this usage of ESBs!

One claim is that only "pure" WS-ReliableMessaging implementations should be adopted. Bigger vendors are basing their WS-RM implementations on their older MOM technology, such as MQ Series or JMS, and this is somehow a bad thing. I don't understand this line of reasoning at all. The infrastructure underneath a high speed reliable messaging protocol is both sophisticated and requires a lot of investment to develop. Older MOM's are proven. Why throw them out? The lesson of recent years is that Interoperability is achieved at the protocol layer, not the API. Who really cares if my underlying MOM has a JMS binding? The point is that they must eventually expose WS-ReliableMessaging over TCP, UDP, or HTTP to be interoperable. Having said that, WS-ReliableMessaging alone has had 3 major revisions since 2003, and all the various ESB/Fabric players support it at varying levels. Until the big vendors such as IBM, Microsoft, BEA, and Oracle have their WS-RM implementations shipping, this standard is too new to be your "sole" approach to reliable messaging.

Cape Clear actually turns API-based pluggability into a feature: it is pluggable with any JMS-based middleware engine. For some, this might be compelling, especially if their MOM vendor has uses proprietary approaches to fundamental features. It has the downside of (potentially) being less performing than a fully-integrated stack. For example, I'd actually be interested to see how Cape Clear performs on WebLogic Server 9 vs. the AquaLogic Service Bus 2.x. It would indicate how much of the performance increase ALSB is showing is due to WLS9 vs. path-length and memory allocation improvements over WLI 8.1's dynamic transformation and routing.

Argument (c) to me is irrelevant if an ESB supports REST-style XML+HTTP+SSL. I believe this is the case with some vendors (though not all), so I that's all I will say there.

Posted by stu at 11:14 AM

The Web 2.0 programmer hype

I've noticed a trend lately as part of the Web 2.0 hype. Programmers are latching on to this movement and trying to project it into their world, suggesting that the "programmer experience" should also change drastically. I thought Web 2.0 was supposedly more about user experience and collaborative agility than the substance behind how you build the stuff, but hey, people want their shot at glory I guess.

So there have been a lot of wild claims floating around lately: from claims that AJAX will kill application servers, all infrastructure will be open source infrastructure, all infrastructure will be outsourced, Ruby on Rails will kill J2EE, SOAP v. REST, RDBMS' are unnecessary to manage meaning, ESB's are bad, Portals are passe', and SOA is DOA. I can't possibly tackle these all, though I will tackle the first few. And I will post a separate entry on ESBs, a topic close to my current work.

Bottom line, these are generally ridiculous claims, on many levels -- unless (of course) you're pushing an agenda. AJAX only kills application servers if you believe that Software as a Service (SaaS) will destroy enterprise IT as we know it. This seems to be the message that both Phil Wainewright and Nick Carr like to push. Similarly, programmers sick of their existing languages and environments are also searching for new ways to do Enterprise IT, and seem to feed this line of reasoning -- they believe that SaaS will allow them to join a startup and use "language du jour" such as Lisp or Ruby on Rails to teach the Java vendors & Microsoft a lesson! This view seems to have started from Paul Graham in his various essays.

From a Service Consumer's perspective, the argument makes a fair amount of sense. From a provider's perspective, or even an "ecosystem" perspective, things start to get murky. The two-second version, from a service provider's perspective, follows -- enterprise IT still needs server side code , but a world of SaaS changes everything to a simple distinction between 'service consumers' and 'service providers', where the consumers (apparently) only need the thinest layer of HTML and JavaScript to tie together their applications. And even if you do need server side code, all you need is a bit of Lisp, PHP, or Ruby. No application server. No .NET runtime. No transaction processing or reliable messaging. Maybe an RDBMS (if you insist). Obviously I'm exaggerating the extent of this strawman somewhat, but from a high enough vantage point, this really seems to be what the pundits are arguing for.

I enjoy punditry, within reason, because it makes us think. Pundits have the odd good point, but this latest wave of SaaS and Web 2.0 stuff seem to be generating a lot more strawmen than usual. There are counter arguments that are being ignored because they don't fit into the broader agenda.

"All infrastructure will be open source infrastructure" seems to be a variant of "service contracts make implementations irrelevant", and is pushed as a way of destroying the last bit of enterprise IT that SaaS doesn't kill -- software infrastructure companies like IBM, BEA, or Oracle. But it makes no sense considering how much custom development is going on behind the scenes to make these Web 2.0 companies scale and perform. Strawmen arguments like to point to Web 2.0 companies who rely mostly on open source, but this seems like bullshit to me. Last I checked, Amazon.com used Oracle, eBay used WebSphere, and Google uses Java (in spots) -- and all rely on tonnes of custom (closed!) code. And let's not forget that Microsoft is making a huge push into this world with .NET 2.0 and Indigo. Perhaps the "long tail" of Web 2.0 services will only need the small bits of infrastructure, and only the big boys will need the "beefy" application servers. It might mean you'll need to do a rewrite if you grow beyond your expectations, it might not. The upstart SaaS infrastructure players will find some successes, certainly. But let's not get carried away. "Commercial quality OSS" is still the exception, not the rule.

Now, a service consumer certainly doesn't care if you use J2EE or PHP or .NET or Ruby under the covers, so long as you meet your service contract and/or SLA. But to claim that one cannot compete based on differentiated service features is quite disingenuous. Unless the world adopts vertical-industry vocabularies on mass, service contracts will vary , with many different kinds of features and many ways to differentiate among providers. The underlying infrastructure will have an impact on the ability to deliver these features in an agile, scalable, and performing manner. Thus there will continue to be a highly competitive market in infrastructure, probably with a mix of open source and proprietary technologies.

Ruby on Rails killing J2EE seems enthrone the "web site backed by a database" as the ultimate use case. And it was - in 2002. Enterprise IT has always been about eliminating silos and creating shared frameworks and environments. The RoR camp wants a beach head, but this will create yet another set of silos! Frankly I think IT shops are getting sick of this. Yes, the productivity is useful, but the lack of interoperability with a broader aggregation / syndication strategy is going to hold adoption. Architectural governance is getting better, and I think it will be hard to push RoR unless it fits into a broader interoperability (WSRP) and manageability framework (perhaps JRuby?)

Lots of people enjoy saying that complex technologies like SOAP and WSDL will never catch on (all evidence I've seen says they have, at least inside companies), I can point to a number of counter-arguments. "Complex" is in the eye of the beholder. Is UNIX/Linux complex? To some people yes, to others no. In the server world it is popular, but to the desktop world it is pretty complex. Does this mean it's doomed?

How about Microsoft COM / ActiveX? Tonnes of software was built on this layer. Let's just enumerate the surface of complexity here: automation vs. early binding, IDL vs. type libraries, custom marshalling, apartment threading, etc. Sure, it didn't take over the world, but it certainly took over the Microsoft world (which isn't small)!

A more universal example might be SQL and the RDBMS. Despite all the trends and hype in the Java community about ORMs and people hiding their SQL, the majority of database-using applications in the Java, PHP, .NET, Perl, Python, or even good old COBOL or C++ are still programmed with heavy reliance on SQL and stored procedures. Yet lots of people still aren't able to think in terms of sets, and DBA's are still paid a lot of money to make this stuff perform because programmers can't be bothered to understand the database. This situation isn't changing any time soon.

SOAP and WS-* are not that complex by any historical measure of comparable specifications, in my view. They're composable -- you use what you need, and don't use what you don't need. WSDL is a bit of a disappointment but is being fixed. XML Schema is probably the biggest disappointment in terms of unnecessary complexity, but it looks like we're stuck with it due to the lack of interest in alternatives like Relax NG.

I see an increase in complexity all right -- not in the individual technologies, but in the splintering of the marketplace and communities. No one wants to sit down and learn what exists, everyone wants to re-invent the world in their image, their one shot at greater glory. Not that I blame them! I'm just skeptical that the collaborative spirit of Web 2.0 will bring the talent to bear to meet what the industry needs.

Posted by stu at 10:14 AM