Over the past couple of months we have met with a number of oil and gas executives to demonstrate our Oil and Gas Solution built on Data-Tactics’ Big Data Engine (BDE). During these conversations it has become obvious that the very mention of “Big Data” produces an involuntary physiological response among business leaders – eye rolls and yawns. It appears that big data has reached the Gartner “trough of disillusionment”. These executives have heard from a bewildering array of software, hardware and services vendors trying to sell them a big data solution. Essentially, every salesperson has presented: I have your solution, now tell me about your problem. (Of course this is not a new sales strategy in the world of corporate IT.) There is an inherent assumption that every business needs big data. However, the use cases touted for the retail world just don’t apply to upstream oil and gas. These business leaders are desperate to get past the technical jargon and talk about real solutions to real business problems.
Upstream oil and gas organizations are sophisticated, complex organizations with a rich and diverse IT landscape. They already have search tools. They already have master datamanagement, data warehouses and BI tools. They already have visualization tools. They have many, many systems that generate and manage operational and geospatial data. They don’t need another shiny new tool. Just what exactly can big data do for them that their current solutions can’t?
This is not an easy question to answer. To answer this question, you have to know a lot about the current set of tools and their limitations. You have to understand the possibilities that exist within the next generation of tools. You have to know how the old and the new can work together. And you have to understand how their business operates and where the challenges or opportunities exist. We at Sullexis have a deep background in BI, MDM and data warehousing. We understand the current IT landscapes and the business challenges our clients in oil and gas face. We are able to facilitate these discussions to define the use cases for big data solutions.
So the yawns stop. And the hard discussions begin. Our clients raise the challenges they have either faced or read about: a big data solution requires a complex implementation with expensive resources. The challenges of implementation surrounding the big data technologies include:
- the challenge of internal funding and sponsorship for a platform that could serve many groups within the organization and could offer benefits that may not yet be quantified,
- the technical difficulty of standing up the big data stack,
- the cost and scarcity of experienced resources (both technical and data scientists),
- the support challenges,
- the lack of domain knowledge,
- and the vast and confusing eco-system of big data products and services.
Our partnership with Data-Tactics and the use of their BDE platform is demonstrating that many of these challenges can be overcome very quickly and cost effectively. BDE is the product of many years of partnership between Data-Tactics and various intelligence and military branches of the U.S. government. Data-Tactics has wrapped the open source Hadoop framework, along with many essential open-source tools, to provide an enterprise grade, production-resilient, multi-tenant platform that offers accelerators and tools to support high-volume data ingestion, data indexing, management, search and discovery. The platform includes an out-of-the-box user interface to discover and visualize data. The platform is extensible to support visualizations and analytics using D3, R or other Hadoop compatible tools. With this platform, it is possible to start loading data on day 1 and have end users begin exploring the data on day 2. This model lends itself to an agile and iterative approach to involve end users in the visualization and use case development process.
By leveraging the BDE cloud environment, we have been able to build relatively low cost and quick proof of concept applications to help clients in proving out the benefits of their use cases and recruiting internal support and sponsorship to move the big data platform forward. During the proof of concept phase, we have been able to provide previously unseen insights to our client’s data within a matter of days.
In one recent example, we completed a hugely successful proof-of-concept (POC) application that was able to ‘slide’ the BDE in alongside a number of existing systems (SQL Server, Exchange, SharePoint, File Server, structured database systems, etc.). We were able to ingest some data and index in place other data from these systems and provide search capabilities and advanced analytics for the end users. In this data fusion approach, we loaded >600 MBs of both structured (databases), semi-structured (text files, email) and unstructured (PDF, MS Word, images) data from more than 20 data sources in less then 6 days. By the end of day 21, answers were being provided to questions that had been first asked 18 months ago but traditional BI approaches were unable to answer.
By eliminating the need for up-front data modeling and embracing diverse datasets being generated in most enterprises (e.g. sensor and operations data, email, SharePoint data, web/reference data), this platform opens the door to new kinds of analysis that can connect the dots across all of this data. This capability has caught our clients’ attention and the initial yawns have been stifled. Most of our conversations are now about developing POCs to illustrate the performance, lower cost of ownership, scale, and the speed of deployment to apply this new technology to an existing, hard to solve problem. Bringing together disparate data platforms that were previously silo-ed and providing the ability to report, explore or run analytics on the full data set is indeed a solution for many problems.