Big Data – Let’s Put It Into Perspective


When reading about Big Data it often strikes me how many different industries are championing the buzzword, whether that be companies trying to sell analytic services to said industries or big players trumpeting how they are utilising Big Data to better serve their customers. This is despite the fact that at this point in time the potential impact of Big Data is revolutionary in some circles while in others the tangible benefits are more limited. For example, outside the Oil & Gas industry you might be hard pressed to believe that Big Data is the brainchild of the social media marketing sector. Every day we see new reports tagged with Big Data highlighting the latest trends across the social media platforms. Sure, it sounds impressive when we learn that so many millions of photos are being uploaded to Instagram by XYZ demographic around a certain time, but what tangible, revenue generating good can we immediately produce from this?

Yes, social media produces petabytes of data, and it IS Big Data, but we need to put things into perspective when reporting on the phenomena. Big Data provides some great insights for marketers, but they often struggle to convey the impact of their digital efforts in dollar terms, whereas in the Oil & Gas industry, where companies live and die by the dollar, the impact of Big Data is immediate and clear. According to GE, utilising Big Data to reduce capital expenditures by as little as 1% would save over $90 Billion of accumulated costs over a 15 year period. That is, Big Data has a clear, tangible impact on the goods produced by the Oil & Gas industry; the impact of which is far reaching, as oil and gas currently powers most of the world.

Despite the significant difference in tangible benefits available today, the amount of Big Data generated by the Oil & Gas industry is actually miniscule compared to what social media produces. On one hand this is the consequence of design, as social media by nature is completely virtual and connected, while Oil & Gas operations are essentially physical and only driven by insights from data and other areas. This paradigm will soon shift, though, as the cost of measuring Oil & Gas operations, and thus producing Big Data, is going down fast. According to Goldman Sachs, the price of the average connected sensor for measurement will have dropped from $1.30 in 2004 to nearly $0.50 in 2016. Thus the drop in price enables each and every new operational endpoint to have more sensors, creating exponentially more data for Oil & Gas operators to exploit to drive better decisions.

I would go as far as arguing that Big Data produced by Oil & Gas is far more manageable in the field, too. For example, in Oil & Gas we know that the reservoir data has a low correlation with data from the refinery processing plant. This means we don’t need to combine them in the same bucket; looking for “emerging relationships” (from data mining and predictive analytics). Therefore, even with the sheer amount of data collected, it can be decomposed into more manageable sub-sets that we can quickly utilise to make decisions. Social media delivers a lot of big data surrounding demographics and behavioural patterns; information which can’t be utilised as quickly as adjusting a drilling operation in the field, for example.

Despite the overarching positives surrounding Big Data in Oil & Gas, I am a little concerned about the popular mindset that we should be rushing to acquire more and more data. Though, this is coming more from technology providers than from the data analytics folks. More data is great, but only if we can analyse and use it to make real operational decisions. Data that is collected and not utilised is simply adding to the noise, and operators shouldn’t be paying to make their own analytics more difficult, costly or time consuming.

In summary, I hope I have demonstrated that there is big data, and then there is BIG DATA. We need to be aware of this and make the distinction in the overall discussion, as right now each industry’s utilisation of Big Data differs too much to be represented or explained by the single buzzword.

Long live Big Data in Oil & Gas!

– Carlos & Michael

Leave a Reply