Data Product Architecture

Data Architecture Layers

In a progressive data architecture approach, raw data is transformed into refined and well-structured information through a series of layers. These layers are commonly referred to as bronze, silver, and gold.

Bronze Layer

The bronze layer, also known as the landing stage, is the initial phase of the data architecture. During this stage, raw data is ingested from various sources and stored in an immutable (read-only) manner. The data retains its original structure from the source systems, allowing for historical reloads, debugging, and testing. Historical data in the bronze layer is based on delivery timelines, enabling tracking of changes over time.

Silver Layer

Moving up the data architecture hierarchy, the silver layer represents the stage where data undergoes cleansing and filtering processes. This layer aims to refine the data by removing inconsistencies, errors, and irrelevant information. The data in the silver layer is historized using Slowly Changing Dimension Type 2 (SCD2) techniques, enabling the tracking of historical changes. Additionally, the data is modeled according to subject areas to facilitate better organization and analysis. The silver layer is primarily utilized by operational analytical teams for their specific needs.

Gold Layer

At the top of the data architecture hierarchy, the gold layer signifies the highest level of governance and refinement. The highly-governed data in this layer adheres to quality standards and regulatory requirements. It is well-structured and extensively documented to provide a clear understanding of its content and usage. Historical data in the gold layer is historized based on specific use cases, allowing for a comprehensive view of its evolution. Furthermore, complex business rules are applied to enhance the value and relevance of the data. The gold layer serves various use cases including being consumer-ready for serving data to end-users and being utilized for reporting and analysis purposes.

Each layer plays a vital role in ensuring the reliability, accuracy, and usability of data products within the data architecture framework. With the progressive approach of bronze, silver, and gold layers, raw data is transformed into refined and well-structured information that meets quality standards and is valuable for business needs.

Further reading

Decentralized data management

Dimensional modeling

Relational modeling

Data vault modeling