The Challenge
Sullexis helped a premier midstream company implement a real-time, data streaming platform and analytics platform.
- Our client’s existing real-time data platform had evolved over many years, incorporating assets from acquisitions and point-to-point integrations set up to meet specific needs.
- The client’s 17 distinct midstream assets and associated SCADA systems are streaming data from over 2 million tags and generating over 3TB of data per year.
- Demands by both internal and external customers were increasing, database instances growing and database licensing costs exploding.
- This increased demand was straining system scalability and the ability of IT to support the increasing requests.
- Developing a purpose-built, real-time, streaming data platform was seen as a way to manage the increase in demand and growing data volumes while keeping IT support costs constant and potentially drive lower
The Solution
- We designed and implemented a state-of-the-art, real-time, streaming data platform built on Confluent’s Event Streaming Platform (Kafka), MongoDB, InfluxDB, with data visualization provided by PowerBI and Grafana, and reporting via Microsoft SSRS.
- The solution was implemented for 2 of the assets and a roadmap developed for implementation across the other 15 assets.
- The real-time data acquisition system was re-engineered to stream JSON data to Kafka topics, with the original data being persisted in MongoDB. Multi data consumers (sinks) were created to meet specific business needs, data aggregation for business reporting (MongoBD, PowerBI and SSRS), data analytics (MongoDB and PowerBI), real-time data visualization (InfluxDB and Grafana).
The Result
- The solution is currently receiving data from 2 assets and 130,000 tags, while the roadmap for implementation across the other 15 assets is executed.
- In addition, new data streams for specific business needs are being provisioned within a matter of days, enabling IT to meet the growing demands of end-users. The solution has laid the foundation for advanced analytics and machine learning, opening-up new opportunities for our client to increase uptime and utilization of its assets.