Organizing and synthesizing the world's energy data

Energy is a sector potentially well-suited to be mined with big data—the accumulation, integration, synthesis and interpretation of enormous amounts of data from disparate sources—Stuebi writes. 

One of the most hyped trends in high-tech is “big data”:  the accumulation, integration, synthesis and interpretation of enormous amounts of data from disparate sources.

Big data is being touted not only as a driver of increased efficiencies for companies, but also increased revenues:  as customers indicate or reveal their preferences through their behavior and choices, companies can then develop new products or services to offer in a very targeted and efficient manner to those most likely to purchase. 

With technology enabling faster mass-computation at lower cost, and a growing set of data available from all sorts of places and easily collectable from a thoroughly-connected world, big data could be a boon — especially for those industries for which cost reduction and efficiency opportunities are limited, and for companies desperately searching for new revenue streams.

Energy is a sector potentially well-suited to be mined with big data.  The industry is largely quantitative already, with scads of measured (and if not measured now, measurable) parameters across vast geographies with innumerable supply sources, processing facilities, transportation nodes, and consumption points.  And, for the most part, energy companies are largely stuck in a mode of selling commodity products, and need new twists and differentiators for which a premium can be charged.

Proactively, I went searching the Internet for the best insight, wisdom and perspective on the incursion of big data into energy.  Frankly, I didn’t find terribly much — which tells me that the space at the intersection of big data and energy is ripe for innovation.

One of the better pieces I found was this posting from early 2012 by Katie Fehrenbacher, “10 Ways Big Data is Remaking Energy”, which identifies the 10 types of data that will increasingly be mined in the energy sector:

1. Weather
2. Cell phone usage
3. Thermostats
4. Hadoop
5. Solar/wind sources
6. Electric cars
7. Power lines
8. Real estate
9. Variable energy prices
10. Behavioral analytics

Although this is a good list of the innovative places from which data will be gathered, it still leaves open for entrepreneurs to identify specific customer needs to be met and value propositions that can be developed from big data.

It seems that a big challenge will be in making data from disparate sources mesh accurately.  As this article from Intelligent Utility indicates, integrating “unstructured data” — especially from hand-written and other manually filled out forms — will be particularly difficult.

Big data in the energy sector will likely be a large opportunity for the major players in the IT sector — IBM, Intel, SAP, Oracle, Google, Microsoft, and so on.  My advice to them is that they will need to be patient:  the big players in big energy move glacially, and big data will take a long time to penetrate in a major way.  But, the opportunity is vast, so it’s probably worth the effort and the waiting…and waiting…and waiting.