Skip to Main Content

The HOW Channel: Business answers you need in the time you have.Stream Now

Choose a Regional Site
Insight

Chevron’s Supply Chain Cloud Transformation

Paul Brenzel
Paul Brenzel
Karen Kirwan
Karen Kirwan

Chevron Corporation’s (CVX), global refining system manufactures fuels and other products sold by the company’s marketing, lubricants, and supply and trading organizations under the Chevron®, Texaco® and Caltex® brands. These refining systems are capable of processing heavy crude oils and producing a variety of high-value products, including transportation fuels, chemicals, and lubricants. This value chain supports 2.4 million barrels/day worth of annual products sold by seven refineries across a network of 8,000 retail stations.

The Commodity Supply Chain Management (CSCM) Data and Insights team is responsible for managing more than 200 data pipelines to ingest data from various internal and external sources and standardize it in the Supply Chain Data Foundation (SCDF). SCDF is central to driving data integration and related initiatives across the many functions responsible for managing the flow of crude oil and refined products, including:

  • replenishment planning and scheduling
  • inventory management
  • price and demand forecasting
  • contract planning
  • product quality and blending
  • margin analysis
  • common master data
The immediate priority of the SCDF project was to replace the existing legacy on-premise data platform with an equivalent data platform deployed in Microsoft Azure®.

The Chevron team envisioned that the migration to Azure could go beyond the Company’s existing data capabilities to enable aspirational and advanced capabilities, such as

  • improved operational efficiency
  • improved, agile business decision making
  • end-to-end profitability

The Publicis Sapient Product & Data Engineering team partnered with Chevron CSCM to build a new Platform as a Service (PaaS) solution in Azure to deliver on this vision.

The Why: The Benefits of Data in the Cloud

Migrating the data foundation to Azure has substantially:

  • Minimized support and disruption costs with a streamlined, fit-for-purpose architecture
  • Improved capability to enhance and scale the data platform much more rapidly than the legacy on-premise system
  • Enabled future advanced capabilities in Azure including artificial intelligence (AI), streaming/device (IoT) data, and management of unstructured data (PDFs, emails, legal)
  • Improved ability to develop, test, and deploy changes quickly via modern DevOps practices

It gave more than 400 manufacturing and value chain optimization users access to:

  • All supply chain data integrated into a central, Chevron-wide instance of a data lake to enable cross-department collaboration
  • Self-service BI to allow citizen users seamless data exploration and analysis, improving business insight generation
  • Supply Chain apps consume data more efficiently from the platform leading to fewer disruptions and an enhanced user experience

Costs

Traditional on-premise systems incur significant legacy costs due to:

1. Licensing
2. Infrastructure
3. Support

The new PaaS SCDF platform has enabled significant reduction of these legacy costs. The only costs incurred in the cloud are related to resource consumption. Some cloud services have licensing costs, but those services can be rapidly turned on or off as needed enabling low-stakes, fit-gap analysis as new requirements are detected.

The commercial costs and barriers of identifying, assessing, and integrating new technologies are eliminated with the availability of the wide-ranging Azure cloud services.

Data collaboration

Before the project, Chevron used an on-premise data warehouse solution. Providing direct user access was cumbersome due to network connectivity restrictions and client software setup issues.

The project team deployed Synapse, Azure’s data warehousing and analytics solution, to securely share datasets, provision access to those with data needs, and enable the connection of datasets otherwise impossible.

Improved performance

Queries generally complete 45 percent faster on Synapse compared with the previous on-premise solution with queries frequently timing out or failing. For fixed resource on-premise systems, too many simultaneous or resource intensive queries could cause system-wide performance issues. Dynamic cloud resource allocation allows minimization of disruption costs by scaling the platform to the resource level needs at the moment.

Reduced operational overhead

The previous solution required a full-time (Database Administrator) DBA team on staff to oversee 24x7 operational support for database issues, including:

  • disruptive queries
  • backup issues
  • space/resource allocation
  • user permission issues
  • reduced  vendor support & data center support

With Synapse, these support activities are covered by Microsoft cloud support, with a smaller local support team focusing on the functional nature of the platform.

Agile delivery

Agile work processes are enabled by removing infrastructure/administrative dependencies for simple tasks, such as taking database back-ups for testing purposes. Improved self-sufficiency of developers enables rapid development and deployment cycles, minimizing development cost and time.

Platform enhancements

Enhancements to data management services such as Azure Data Factory and Synapse are continually deployed by Microsoft and don’t require large-scale planning efforts or costly outages associated with comparable upgrades to on-premise systems.

The How: Platform Technology

It gave more than 400 manufacturing and value chain optimization users access to:

Integration

More than 200 data integration jobs were converted from the previous solution to Azure Data Factory (ADF). The sequencing of the delivery of these had to be carefully managed due to the internal dependencies of the data platform.
However, to enable delivery and a consistent pattern, multiple frameworks were developed early in the project to facilitate:

  • Massive parallel processing using Synapse dedicated pools
  • Parameterized pipelines to enable re-use for common data patterns
  • Change data capture framework ensuring management of data change
  • High performance data warehouse load using polybase connector
  • Python unit case (Pytest) for scalable, automated test cases saving effort and improving testing quality

Performance

While the requirements of the timeliness of data depend on the part of the supply chain being supported, critical 24/7 operations like refinery, blending, and scheduling have more strict data needs. In addition, proper design and management of resources in the cloud is crucial to ensure excess Azure costs are not incurred. Major components include:

  • Databricks pool cluster is provided to critical business functions on-demand to eliminate typical cloud spin-up time
  • Databricks multithreading and Spark integration to handle massive tables efficiently
  • Polybase load to data warehouse (Synapse) to optimize writing to the Synapse data warehouse
  • Table distribution and results set caching to optimize large read operations on the data warehouse
  • Workload management to classify the importance and prioritization of certain jobs

Analytics and consumption

In addition to storing the data, the data furthermore needed to be transformed and served to the business function needing the data. Streamlined access and proper preparation of the data are crucial to keep the data consumers from experiencing disruption as the system migrates from on-premise to the cloud. Major components include:

  • Automated DB database refreshes enabled by DevOps to reset the data to a neutral state to enable agile unit testing and structured QA script execution
  • Data quality engine enabled by rules defined in tables, executed in ADF, and reported in Power BI
  • Azure Web APIs developed to enable consumers to consume the data efficiently and on their own terms

The goal

In less than one year, the Publicis Sapient delivery and Chevron CSCM Platform teams were tasked to migrate the following components to Azure:

  • 200 data integration pipelines to Azure Data Factory/Databricks/ LogicApps
  • 400 tables modelled and migrated to Azure Synapse
  • 450 stored procedures and queries to Synapse
  • Conversion of these statements to end-state architecture was facilitated by Microsoft Tradurre, a Microsoft product not yet publically available.
  • A data quality engine with 400 rules to Synapse/ADF
  • 20 sets of reports migrated to Synapse integrated Power BI Reports

With traditional project management techniques, the SCDF project would have taken years.

However, the Publicis Sapient team, partnering with the larger Chevron platform team leveraging Microsoft workflow and integration tools like Azure DevOps (ADO), combined the following practices to enable a successful project completion, all in less than one year.

Scaling Up Agile

Given the size and scope of the project, the coordination required between the Publicis Sapient delivery team and the larger Chevron CSCM platform team was a significant endeavor. Azure DevOps was used to facilitate both internal iteration planning, in addition to managing cross-team dependencies.

More than 20 supply chain application teams were needed to test, plan, and migrate their integrations to the new platform simultaneous with the decommissioning of the legacy on-premise platform. The delivery team embraced Chevron’s newly deployed Scaled Agile process within the CSCM platform in ADO to plan out quarterly PI increments, map dependencies, and monitor execution via ADO dashboards and progress reports to ensure delivery adhered to the schedule.

DevOps

Agile Program Management and modern DevOps practices go hand-in-hand; one is essential to enable the other and neither are successful independently. While many believe DevOps is a technology or a skillset, the Publicis Sapient delivery team embraced DevOps as a mindset to enable the rapid development, integration, testing, and deployment cycle necessary to meet tight timelines. Without this DevOps mindset, enabled by ADO, Azure Repos, and Azure Automated Deployment Pipelines, the delivery of the platform in less than a year would have been unrealistic.

Project Outcome

The new platform was deployed in production using Microsoft Project to coordinate the cutover,” a carefully sequenced set of steps to ensure that the project is deployed simultaneously with all data publishers and consumers reconnecting to the new platform.

Remarkably, the team was able to deliver the platform on time at high quality, despite a surge in the Delta variant of COVID-19 during 2021 which, at times, reduced total team capacity by 30 percent as team members took off for illness and to care for their families .

Paul Brenzel
Paul Brenzel
Business Consulting Manager
Karen Kirwan
Karen Kirwan
Executive Client Partner, Energy & Commodities Practice