Big Data Crystal Ball – Looking Forward

A guest post by Clif Triplett, ex-CIO, Baker-Hughes and member of Flutura advisory board.

“Big Data”; another big idea like “Cloud”; is it something that we should be embracing right now or be lost forever? Is this another vague term that can be applied to countless projects? Everyone is doing it our vendors will tell us. You may have the question, “Am I actually running a “Big Data” project or do I just have to deal with a lot of data?” What is “Big”?

As I look forward, I think Big Data will begin to take different distinct forms as this other concept of “Cloud” has matured over recent years. Business cases will become more obvious and solutions better defined. The enablers to address “Big Data” opportunities will also become more obvious as we see the dynamics of tomorrow dramatically evolve compared to just a few years ago.

The most influential contributors emerging on the horizon that will change the game are the following:

  • Explosion of sensors and digital inputs

  • Very large memory machines (Terabytes of memory and more)

  • Easy access to thousands of processors from Cloud providers like Azure and Amazon

  • HPC environments and Cloud operating systems that can organize computing across many independent machines

  • The continued explosion of readily available data on the internet (e.g. weather, stock market, government records, news, social media)

  • Web crawlers that can locate and index information on the web Predictive Analytic engines

  • Open Source software (e.g. Predictive Analytic engines, search engines, cloud operating systems, and I anticipate the emergence of applications like those that might be able to judge and predict sentiment about topics)

  • Proven use cases by industry

  • Commercial-Off-The-Shelf (CoTS) package solutions connecting multiple data sources and answering specific business challenges e.g. sales forecasts, predictive maintenance, process control, inventory management.

This convergence of technology and information now make it possible to define sub-classes of Big Data projects, such as Demand Forecasting, Failure or Defect Management, Market Sentiment, Situational Alternative Simulation, and many others I am sure.

Big Data product segments will also be clearer and strong leaders will emerge. One of the product segments where we will see specialization will be hardware platforms. We will see specialized machines such as HANA platforms and Solid State storage optimized for very large data sets. We will also see Big Data Cloud platforms leveraging the momentum of two major technology trends; Cloud and Big Data. There will be many analytic platforms; both open source and proprietary. We will see boutique solutions to specific problems like product launch predictive sales analytics and advanced maintenance management programs.

The other big player to emerge will be taxonomy platforms that assist in connecting disparate data sources and applying context to support integration. Today most major applications provide APIs to their applications. I foresee them in the future also providing taxonomy context to allow more dynamic integration of systems.

So, what are some of the issues that this future of Big Data will bring upon us in the near term? Organizational change management will be top of the list. This emerging trend surrounding Big Data will cause many new roles, new skill demands, new systems, and new approaches to business. In short; change.

First of all, we will face the issue of where is our Master Data and who is the owner? This is a problem that has been with us for some time, but now it will emerge as a very strategic decision and not just an IT annoyance. Second will be the issue of data quality and how do we clean up all this conflicting or missing data we have uncovered. Our list will continue to grow and here are a few more to consider:

  • Development of Business Cases to clean up data

  • Engaging the business in data governance

  • New Technologies for solving this new class of Big Data business solutions

  • New skills required and a shortage of top talent

  • Technology and Business roadmaps – what does it look like?

  • Many new vendors emerging and the old favorites trying new things they have never done before – how do we know who knows what they are doing?

  • Project estimating and run away projects

We have seen many of these challenges before and many of us will see them again. It will be exciting times and we better start preparing now. The planets are aligning and Big Data is arriving. It still may be a bit still too far away for some to see it clearly, but keep your eyes open for it will be upon you soon.



Vivamus suscipit tortor eget felis porttitor volutpat. Praesent sapien massa, egestas non nisi.