A large energy company contacted Kforce Consulting Solutions for help in designing and building data streams to monitor and administer natural gas pipelines. The client wanted to improve their ability to predict demand, direct their supply and foresee outages and other problems.
Before creating the proper AI solution, they first needed to establish best practices in data quality and governance.
• Establish data quality, architecture and governance to build effective machine learning data dreams
• Use machine learning models to gain insight into sensor data
• Leverage data to generate reliable predictive analytics
• Provide the client with confidence that the reports and KPIs they see are verifiable and reliable
Our data and analytics practice partnered with the client to develop a data and AI architecture that provided the necessary foundation for effective AI and machine learning elements.
• Data ingestion
• Feature engineering
• Automated model training/storage
• Inference pipelines for predictive analytics
The initial plan was for the Kforce Consulting Solutions practice leader to work closely with the Kforce machine learning engineering team on feature engineering and model architecture. However, through data and process exploration, the practice lead discovered numerous data quality and governance problems around data ingestion into the client data stores, as well as numerical and statistical problems around the machine learning models themselves.
As a result, the practice lead spent much of his time instituting data quality and governance practices with client ETL engineers and consulting with client data scientists to improve the statistical methods in the machine learning processes. The machine learning engineering team conducted much of their work independently, while full stack developers embedded in client teams and worked to iteratively improve dashboards and analytic displays based on stakeholder feedback.
• Implementing anomaly detection for sensor data to prevent data artifacts from impacting prediction accuracy
• Instituting data normalization and other data quality protocols for the AI/ML data pipeline
• Creating source and transformation documentation standards for the data pipeline
• Automating and standardizing all data governance and quality processes for data related to predictive models
• Creating and documenting data formatting and ingestion protocols for the predictive models
Ultimately, this AI solution operated on batched data from more than thirty sensors. The predictive model enabled client stakeholders to see usage, supply predictions and assess outage risks in near real-time.
• Real-time statistical analysis and anomaly detection for data quality assurance
• Python modules in Sagemaker (XG Boost and Gradient Boosted Trees) to predict demand and station pressures in distribution
• Real-time data ingestion (Kafka/SparkFlow) from sensor data combined into 5-minute batch intervals for dashboarding/reporting
• Continuous model evaluation, retraining using AWS technologies
Machine learning platform : AWS Sagemaker
Data platform : SQL server, SSAS, AWS Redshift
Machine learning models: Gradient Boosted Trees, XG Boost
As you move from exploration to impact, we’ll help you understand how AI can provide value to your organization. We’ll work with you to create scalable solutions that drive efficiency, growth and measurable ROI.
