We see this a lot in the data analysis space, but it's worth remembering that it really matters what you measure.

Rarely have I seen such a clear example of that as the claim today from the app developer Snappli that the percentage of people using Apple Maps on iOS6 has dropped from 35% to 4%. The Guardian nails the story, noting that the claim may well be without merit.

What happened?

Well, it turns out that Snappli was measuring not the percentage of people using the maps application each day, but the percentage of people using maps data every day. In iOS5, these percentages were effectively the same because the Maps application almost always had to download new map tiles every time it was opened.

So Snappli happily thought they had a good proxy for maps usage and over time they may have forgotten that their measurement was a proxy. They may have begun to assume that they were measuring actual maps usage. This happens all the time and it's really easy to do. But it's pretty embarrassing when something changes, causing our assumption to become invalid, and we don't realize it. That's when we start making false claims and looking pretty foolish.

That is what happened to Snappli. Because (surprise!) the Maps app in iOS6 doesn't download data very often. I just now threw my phone into airplane mode and zoomed into the last three cities I'd looked at over the last week and a half: Madison, WI; Minneapolis, MN; and Chicago, IL. In each of these cities I could see streets down to the lowest level of detail, including shops and points of interest. No data use.

So it seems that what Snappli scored as a strike against the new Maps app should actually be counted as a point in it's favor. And this whole saga can serve as a good reminder to us in the data business that we need to keep the assumptions behind our metrics and measurements in plain view as much as possible.

P.S. I realize there are some major problems with the data behind the maps app on iOS6. However, I also think it's important to focus on actual problems and not made up problems.