Stephen Few recently published a blog that takes Forrester to task over the value of its publications on the topic of visual business intelligence, and by extension on all topics that Forrester weighs in on. Meanwhile, over on ZDNet, we've got Ed Bott asking why the IT industry continues to listen to Gartner.

I'm not particularly comfortable with the personal tone that Stephen Few takes in some of his diatribes, but both of these posts are fundamentally correct that the big IT analyst firms are strikingly bad at predicting the future and at evaluating the current state of the market. I'll venture that most people think these are exactly the two skills that analyst firms are selling in their reports, and so it is rather surprising that analyst firms like Forrester and Gartner are no better (and often worse) than the rest of us in these areas.

Of course, this isn't actually what these firms are doing with these reports. As far as I can tell, there are three main customers of large IT analyst firm reports:

Large enterprises (i.e. enterprise software customers)
Consulting firms
Vendors

Notably missing from the list are experts in the fields that these firms are covering. Yes, some of these experts work at consulting companies, vendors, and customers, but they make up a surprisingly small percentage of the employees of these companies. I don't think these are the people buying the reports.

My take: Software customers buy these reports to justify buying decisions. This is the 21st century version of "no one gets fired for buying IBM", but now it reads "no one gets fired for buying a leader on the Gartner Magic Quadrant". Consulting firms buy these reports so that they appear knowledgeable and know what products they can recommend to their clients who are reading the same reports. Vendors buy distribution rights to the reports that list them as industry leaders so they can use the reports to reassure their nervous customers during the extensive sales process.

The emperor has no clothes and everyone already knows it, including the emperor. It's just convenient for everyone to keep their mouths shut about the topic. Stephen Few and Ed Bott are hardly the naive child in this scenario. They have not suddenly realized that the emperor has no clothes, but pointing out this fact every once in a while is a good way to advertise that they are selling something different.

Meanwhile, IT analyst firms churn out reports that are designed to be incredibly conservative, informed by research and sales processes that inherently support the status quo. Of course, they pretend to be cutting edge with an eye on disruptive innovation. They've got to, because the idea of disruptive innovation has become the status quo. But in reality it is almost impossible for one of these reports to recognize and recommend a disruptive technology because the very methodology of the reporting process precludes the analysis of truly disruptive vendors.

It's a sad situation, especially as I know some really bright minds work for these firms. These are people who do understand the industry and do have a feeling for disruptions and innovations. I'm quite glad that Gartner and Forrester now have many of their analysts blogging on these topics, because this is a platform where these people can actually give us a more honest idea of what's going on. But it is telling that when independent analysts join these firms, their blogging tends to drop off as more of their time is plowed into the production of reports and client inquiries.

As it is, I now primarily follow independent analysts (like Curt Monash or Horace Dediu) and commentators for informative reports on the BI, mobile, and data spaces. And I end up doing a lot of my own research. It seems to me that it's unfortunately difficult to pay for good information these days, so you're often better off not paying.