On Layers

Something I saw somewhere about cutting layers out from a software stack reminded me of something else that has been niggling me for a while. Most likely what I saw was someone talking about the Vishal Sikka keynote at SAP's TechEd conference, talking about SAP's HANA in-memory application server removing layers from the software stack.

SAP has been doing this layer-collapsing move a fair amount lately and I think this is attributable to their innovation push over the last few years. I think that's all good, though we'll see where it brings us. More on that later. But what niggles me is this idea that removing layers is always a good thing. Them there layers are actually doing something you know!

So what are layers? On the server side of a client-server (two layers right there) architecture we might be talking about database servers, application servers, and front-end rendering servers. In application the application space maybe we talk about model, view, controller architectures. Every system of any complexity has some sort of layering concept.

Why do these layers exist? What are they doing? Primarily lowering costs and allowing people to get their work done. Layers server as an abstraction, as a way to get work done within a layer without having to worry too much about what is going on outside of the layer. They are a protective bubble that allows me to do things like write this post or tweet without experiencing (increasingly) existential angst about how the bits are stored.

Yes, layers look really complex when viewed from a systems perspective, but that is usually because they are codifying an incredibly complex system into a form that we can actually comprehend. Layers force structure onto a system in order to make spaces where complexity is hidden and people can live and work.

Which brings us to the point that it's not really about what is in the layers so much as it is about the transitions between the layers - those shields that create the spaces where actual work and life gets done. Transitions between layers are interfaces of some sort. They are negotiated ways for layers to talk to each other. In programs, these are called APIs. In real life, this is called bureacracy.

APIs and bureacracy share a few things in common:

  • They have lots of rules about who you have to talk to and how you have to talk to them
  • Everyone thinks they are terribly designed
  • They're usually pretty slow compared to going "straight to the source"
  • If they didn't exist and everyone went "straight to the source" then there would be total chaos

In short, everyone hates them but nothing would get done without them.

Sometimes, layers outlive their usefulness and those who hate them overwhelm the inherent inertia of mass acceptance. When this happens in technology, it's a "technical revolution". When it happens to companies, it's a "reorganization". And when it happens to governments, it's a real "revolution.

Here's the thing: Revolutions are pretty nasty affairs.

In the political sphere, lots of people die. In the industrial sphere, people lose their jobs. And in technology people experience angst about bits.

Seriously, there have been a ton of questions about how HANA persists data. People are literally experiencing angst over how their bits are stored.

So, I know that tearing apart layers is sometimes necessary in order to get past some institutionalized road-blocks that have outlived their usefulness. Just remember that those layers you just ripped through actually were doing something useful.

Downloads for Developers

The news here isn’t that the 'new king-makers', as Savio put it, look a lot like the old kingmakers: developers. The news is that management may finally be realizing it.

- Stephen O'Grady, Redmonk

Developers, developers, [...] developers

- Steve Balmer

Most software platform companies at least partially get it these days - developers drive adoption and quality among technical groups. The software that has quality developers on its side will look better to both business and technical interest groups than the same software that is dragged down by developer indifference or animosity.

These points can be debated to a certain extent.

There are plenty of non-technical power-centers that drive adoption of software platforms in the enterprise. Preferred-vendor arrangements are common and historically were often negotiated at the CIO or higher level, with little developer involvement. Further, application vendors attempt (often with good success) to sell into business units rather than IT groups.

But in both cases developers still drive quality and adoption. Business units that buy applications directly often find themselves in need of connections to other systems or extensions to the application. This means developers are involved, either via an IT group, or as outside consultants.

Meanwhile, preferred-vendor agreements are constantly undermined and even when they are successful they may very well promote homogeneity and management ease at the expense of long-term quality in people and software. In order to make good software development and vendor management decisions, one must be well aware of the world beyond a single vendor bubble.

In order to bring developers on board with a vendor's offerings, to increase general awareness, and to drive sales, vendors need to get their software into the hands of developers. In the case of open source vendors, this is mostly an issue of getting the word out, as the software itself is only a download away. But in the case of more traditional enterprise vendors this can be a complicated proposition. Most enterprise vendors now provide downloads of some version of most of their platform software. Some vendors provide downloads of much of their application software as well.

Some example download sites:

These downloads are usually made under a fairly restrictive license and are usually not available for all parts of the application or platform software. Because of vendors' business model it is somewhat costly to provide these downloads because they appear in a format that is not the standard distribution format for the vendor's software. There are also legal costs associated with writing and maintaining the developer licenses that are applied to these downloads.

I believe that vendors have a tendency to see these sorts of downloads as an overhead cost. They are not. They are a key step in driving both sales and developer adoption, which are closely linked.

Here's how:

  • Ability to prototype before purchasing is a key part of the software selection process for responsible companies.
  • Today's developers will guide tomorrow's purchasing decisions.
  • A healthy developer ecosystem is necessary condition for a strong third-party application ecosystem.
  • A skilled, and preferably large, pool of developers is necessary for good project success rates.

In his article "The CIO is the last to know", Billy Marshall talks about the CIO of a financial services company who is surprised to find that his operations people are running Red Hat Linux. This CIO was handed a decision via bottom-up fiat. It is a story that is played out again and again in the enterprise space.

The point is not that CIOs aren't doing their jobs. It's that the decision is inevitably influenced from a different level: the level of those actually carrying out development and operations. Maybe these people aren't actually making the purchasing decisions, but they talk to the people who are. And if someone makes a purchasing decision that development and operations disagree with or are unable to execute, that person is going to hear it. And they'll probably feel it when their group's productivity falls off a cliff.

CIOs either are or should be listening to their developers' opinions. It would be wise for enterprise vendors to divert some sales attention into making sure that those developers have good opinions of their software.

The first step is getting that software into developers' hands quickly and with a minimum of developer effort.

SAP's HANA and "the Overall Confusion"

I threw together a very long response to a very long question on the SCN forums, regarding SAP's HANA application and its impact on business intelligence and datawarehousing activities. The original thread is here and I'm sure it will continue to grow. But since my response was pretty thorough and contains a ton of relevant links, I thought I would reformat it and post it here as well. In order to get a good overview of the HANA situation, I strongly recommend that anyone interested check out the following blogs and articles by several people, myself included:

Some of these blogs are using out of date terminology, which is hard to avoid since SAP seems to change its product names every 6 months. But hopefully if you read them they will give you some insight into the overall situation unfolding around HANA. With regards to DW/BI and HANA, these blogs address many of those issues as well. Now, to try answering the questions:

1. Does SAP HANA replace BI?

It's worth noting that HANA is actually a bundle of a few technologies on a specific hardware platform. It includes ETL (Sybase Replication Server and BusinessObject Data Services), Database and database-level modeling tools (ICE, or whatever it's called today), and reporting interfaces (SQL, MDX, and possibly bundled BusinessObjects BI reporting tools). So, in the sense that your question is "does anything change as far as needing to do ETL, modeling, and reporting work to develop BI solutions?", then the answer is no. If you are asking about SAP's overall strategy regarding BW, then this is open to change and I think the blogs above will give you some answers. The short answer is that I see SAP supporting both the scenario of using BW as a DW toolkit (running on top of BWA or HANA) as well as the scenario of using loosely coupled tools (HANA alone, or the database of your choice with BusinessObjects tools) for the foreseeable future. At least I hope this is the case, as I think it would be a mistake to do otherwise.

2. Will SAP continue 5-10 years down the road to support "Traditional BI"?

I hope so. If you read my last blog listed above you will see that HANA actually solves none of the traditional BI problems, and addresses only a few of them. So we still need "traditional" (read "good old hard work") approaches to address these problems.

3. What does this mean for our RDBMS, meaning Oracle?

Very interesting question. For a long time, SAP has supported competitive products to Oracle offerings. In my view, this was to give SAP and its customers options other than the major database vendors, and to give itself an out in the event that contract negotiations with a major vendor went south. So in a sense, HANA can be seen as maintaining this alternative offering. Of course, SAP says HANA is more than that, and I think they are right. Analytic DBMSes have been relatively slow catching on and as SAP's business slants more and more towards BI, the fact is that the continued use of traditional RDBMSes in BI and DW contexts has done a lot of damage by making it difficult to achieve good performance. It's a lot easier to sell fast reports than slow reports :-) So that is another driver. Personally, I don't agree with SAP's rhetoric about HANA being revolutionary or changing the industry. The technologies and approaches used in the ICE are not new, as far as I have seen. As far as changing the industry from a performance or TCO perspective, I'm reserving judgement on that until SAP releases some repeatable benchmarks against competing products. I doubt that HANA will significantly outperform competitive columnar in-memory databases like Exasol and ParAccel. If you are Oracle, you have a rejuvenated, and perhaps slightly more frightening competitor. I don't think anyone really thought that MaxDB was a danger to Oracle, but HANA holds more potential as a competitor to Exadata. Licensing discussions could get interesting.

4. Is HANA going to be adopted and implemented more quickly on the ECC side than BI side first?

Everything I have seen has indicated that SAP will be driving adoption in BI/Analytic scenarios first and then in the ECC/Business Suite scenario once everyone is satisfied with the stability of the solution. Keep in mind, the first version of HANA is still in ramp-up. SAP is usually very conservative in certifying databases to run Business Suite applications.

What is a commodity?

Lately I've seen several discussions of commoditization in the enterprise software space. Or perhaps "commoditisation", depending on which side of the Great English-language Divide on which you happened to spend your school years. Specifically the claim has been made that applications (ERP, BI, analytics) or skills (programming, project management, for example) are becoming commoditized.

I'm not going to link to any of these claims because my claim is that the word is being subject to rampant misuse and I'd rather not call out anyone specific. The misuse is widespread and I don't think it would be fair to name individuals.

This misuse is a shame. There is a lot of really useful economic and social theory around commoditization. Incorrectly labeling a market trend as "commoditization" creates the incorrect impression that these bodies of theory are applicable, and this can result in incorrect analysis.

See, for example, the insightful discussion of commoditization as a competitive strategy in this blog about Facebook's Open Compute Project by Marco Arment. When we use the term incorrectly, we poison the well from which this sort of analysis is drawn.

So, what is commoditization? Wikipedia, as usual, has a fairly good definition. To break it down, a commodity is:

  1. Undifferentiated - a commodity from supplier X is basically the same as the same product from supplier Y
  2. Fungible - an instance of a commodity can easily be switched for another instance of the same commodity without significant impact on the user of the commodity - in other words, a commodity has low switching costs

When a product is a commodity it will usually have a price that is determined by a market of exchange. Markets are not always efficient, and very few products are truly commodities, but there are some products that come fairly close. Wikipedia gives several examples, but here are a couple for your consideration:

  • Salt - All salt is basically the same, and switching from one brand of salt to another has no noticeable impact on the user (not withstanding "premium" salts, for example sea-salt).
  • Unskilled labor - The initial pay of grocery store shelf-stockers, for example, is determined primarily by market forces or minimum wage as the labor pool is considered fairly undifferentiated and the cost of hiring a different employee is fairly low in some labor markets.

So what about enterprise software? I can't think of anything that's a commodity in enterprise software. Maybe servers, as the advent of virtualization and cloud computing begins to lower switching costs (improving fungibility).

What is not a commodity in enterprise software? Lots of stuff. Here are some examples:

  • ERP and BI software - Different offerings are still quite differentiated, and switching costs are astronomical because of the amount of customization required of all solutions. Additionally, cloud vendors are now creating data lock-in scenarios that can make it very difficult to migrate old data to a new solution.
  • Switching from "build" to "buy" does not commoditize a market - An IT department switching from a "build" to a "buy" approach, or a vendor pushing solutions that require less customization, does not result in commoditization of a market. This is because different solutions are still differentiated based on features, performance, or ease-of-use, and because switching costs remain high. Switching costs should be lower when going from a custom solution to an "off-the-self" solution than the other way around, perhaps making the product more commodity-like, but this is a stretch. Implementation costs are still going to be quite high.
  • Developers and consultants - There is lots of suspect research talking about how the best developers are some multiplier (usually 6-20X) more productive than the average developer. In fact, it's probably worse because this research tends to focus on long-term employment. Because of the time taken for on-boarding (switching costs) and the increased administration costs that come with a larger team, hiring a middling developer or consultant for your project can often make the project progress even more slowly than hiring no one at all!
  • Tool-kits - Development tool-kits have a real impact on the performance of custom development. The choice to use language A versus language B for a given development project is not academic. The differentiation equation depends on your existing skill-set as well as features of the tool-kit and switching costs are high due to the need for retraining and reorientation.

Commodity theory does not apply to any of these areas. The goods are not fungible and the products are differentiated.

Any vendor who says otherwise is probably peddling a subpar product (or labor). Any IT department that believes this is probably making some bad purchasing decisions. And every purchasing department likely talks to their suppliers about how this is a commodity market ... because they are trying to negotiate a better price.