Program to Manage, Monitor & Govern the Snowflake Service

Snowflake governance is a core platform project managed by a dedicated small team that focuses on overseeing and optimizing the usage of Snowflake within the organization. It encompasses monitoring, access control, cost management, and security to ensure the platform’s efficient and secure operation while aligning with the firm’s data management and compliance goals.

Background

The client, a leading international investment management firm, aims to enhance their reference data management procedures and is interested in deploying a cloud-based data warehouse. This solution will empower the firm to perform robust analytics on extensive historical data, ultimately enabling more informed business decision-making.

Imperative for Change

The governance framework is put in place to support critical aspects of Snowflake’s usage within the organization, including:

• Monitoring: The platform team is responsible for continuously monitoring Snowflake’s performance, health, and security. They track system usage, query performance, and resource utilization to promptly identify and address any issues
• Access Control: Snowflake governance involves defining and enforcing access policies and permissions to ensure that only authorized personnel can access and manipulate the data stored in Snowflake
• Cost Management: Controlling costs associated with Snowflake usage is crucial, especially in a cloud environment. The platform team sets up cost monitoring and optimization strategies to track expenses and recommend cost-efficient resource allocation
• Compliance and Security: Snowflake governance includes compliance with regulatory standards and security best practices. The platform team implements security measures, encryption, and auditing to safeguard data and ensure the organization complies with relevant regulations

The Transformative Solution

The solution involved the following components:

• Role-based Access Control (RBAC)
• Automation tools/utilities help establish a robust governance framework to manage and monitor Snowflake
• Lambda-based monitoring of Snowflake accounts to detect and notify of policy or security violations
• DBT models for data transformation and retaining data from Snowflake share
• Resource Monitors: Resource monitoring to track credit usage and receive alerts for potential credit consumption issues

The Impact

• We have a small 3-member team from PS helping clients to build and develop the tools around monitoring and managing the Snowflake platform
• PS partnered with the client to build automation tools/utilities to help establish a robust governance framework to manage and monitor Snowflake
• PS helped the client create a governance dashboard in Tableau that provides real- time insights into Snowflake usage, cost, and security compliance
• As the Snowflake usage within the firm increases, PS is helping other projects to onboard onto the platform and creating Roles, Databases, and Schema with appropriate privileges for different user groups

Program to Build a Market Reference Data Platform

Snowflake governance is a core platform project managed by a dedicated small team that focuses on overseeing and optimizing the usage of Snowflake within the organization. It encompasses monitoring, access control, cost management, and security to ensure the platform’s efficient and secure operation while aligning with the firm’s data management and compliance goals.

Background

The client is a prominent global investment management firm seeking to optimize their reference data management processes. Confronted with issues surrounding data quality, restricted data coverage, intricate data consumption approaches, and the need for thorough auditing, the company visioned to overhaul its reference data platform. The objective was to enhance overall efficiency and facilitate more informed decision-making.

Imperative for Change

The project aims to easily enable data acquisition with a broad range of external vendor datasets to support firms’ market reference data needs, thereby ensuring steadfast adherence to vendor data consistency and quality principles. The primary focal points of the project encompassed the following central objectives:

• Enhanced understanding of Data Lineage and Utilization
• Remove vendor data processing from Oracle and use Snowflake to store the vendor data
• Improve vendor data quality validations and transparency back to the vendor file to ensure vendor data across the firm has been validated before use
• Expose all vendor data in query accessible source for all consumers across the firm

The Transformative Solution

Build a modern reference data platform using Snowflake’s cloud-based data warehousing solution combined with AWS services.

• Snowflake Data Warehouse
• AWS Services Integration
• Snowpipe for Real-time Ingestion
• Data Transformation and Quality Checks
• Data Access and Consumption

The Impact

• We have a 30-member team from Engineering, Experience design, QA research, and product working across five tracks to build the next-generation reference data platform
• PS partnered with the customer to build an in-house, highly configurable vendor data onboarding application tool, which replaced their legacy third-party tools, saving on licensing fees and faster onboarding of any new vendor data
• PS partnered with the customer to build a metadata-based data processing application that empowers analysis to write business logic to quickly process and transform vendor data without any developer involvement
• PS partnered with the customer to create a new security setup service, significantly sped up the process of setting up a new instrument within the company. This service is now heavily leveraged by the entire firm
• PS Quality Engineering team collaborated with the customer to automate testing and release processes. This collaboration led to a noteworthy enhancement in their ability to quickly introduce new features to the production

Program to Build a Next Gen Advisor Platform

Detailed technical analysis and technical design to develop the advisor platform and big data engine powering the data needs

Background

Client is building a next-gen advisor platform solution that needs a solid pillar of data and abilities to migrate multiple data sources including hundreds of legacy OLTP systems to cloud computing systems. Advisor should be able to view and recommend insights to their clients based on latest asset trends and reports that are built out of the data platform.

Imperative for Change

• Detailed data and technical requirements needed to be driven out for the Extract, Transform, Load (ETL) design part of cloning the 40 applications running in the retained Bank
• Requirements gathering, functional analysis, technical architecture and detailed data design / engineering of source data, data extraction, transformation and manual data processes
• Provide valuable input for the courses of actions that the bank needs to take in order to succeed in the transformation programme including data governance
• Baseline Data Architecture, and Baseline Application Architecture to inform decisions and subsequent work for cutover and migration activities
• Provide functional and technical support during transformed target values, system testing, system integration, proving and user acceptance testing, data cuts and sign off

The Transformative Solution

• During the onboarding phase, Publicis Sapient proactively performed on evaluation of two major Big Data platforms that the client was reviewing and made recommendations that were greatly received
• Publicis Sapient’s team performed multiple POCs over the technology evaluation period and consistently enabled the client in making informed and right decisions
• In the current development phase, the team is helping build capabilities across the depths of the data using technologies like Snowflake, DBT, Airflow and Power BI to name a few
• Publicis Sapient is building solutions involving Java, Python and testing using automation frameworks

The Impact

• We currently have 24 team members from experience, engineering, research and product working across 6 different tracks of work on the next generation advisor platform
• Publicis Sapient made quick turn arounds and enabled multiple Power BI reports which were previously not possible with legacy systems resulting in client confidence and trust
• Even as the company is expanding its in-house development team, we continue to focus on designing their strategically important next generation data platform and contributing to their application development initiatives