Data Platform Design
Why Data Platforms Exist
Every company that takes data seriously goes through the same arc. First, one data engineer writes a few cron jobs that copy production tables into a warehouse. Then analysts ask for more sources, so the data engineer writes more jobs. Then the ML team needs features, product needs event analytics, finance needs reconciled revenue, and suddenly there are three hundred pipelines, two warehouses, a lake nobody trusts, and a central data team with a six-week ticket backlog.
The bottleneck is not compute or storage. It is people. A central team that builds every pipeline by hand cannot scale linearly with the number of questions the business asks. A data platform exists to break that coupling: the platform team builds the paved road — ingestion, storage, compute, governance, and serving as self-service primitives — and the teams that actually understand the data use those primitives to build their own pipelines and data products.
flowchart LR
subgraph Before["Central Team Model"]
direction TB
R1[Marketing request] --> Q[Ticket queue]
R2[Finance request] --> Q
R3[ML request] --> Q
R4[Product request] --> Q
Q --> DT[Central data team<br/>builds every pipeline]
DT --> WH1[(Warehouse)]
end
subgraph After["Platform Model"]
direction TB
PT[Platform team<br/>builds primitives] --> PR[Ingestion / Storage /<br/>Compute / Catalog / Serving]
PR --> D1[Marketing builds<br/>own pipelines]
PR --> D2[Finance builds<br/>own pipelines]
PR --> D3[ML builds<br/>own pipelines]
D1 --> LH[(Shared lakehouse)]
D2 --> LH
D3 --> LH
end
style Before fill:#fee2e2,stroke:#ef4444
style After fill:#dcfce7,stroke:#22c55e
A data platform is to data what an internal developer platform is to services: a product whose customers are other engineers and analysts, whose job is to make the right thing the easy thing.
What a Platform Is Not
| Misconception | Reality |
|---|---|
| "A platform is the tools we bought" | Tools are components. A platform is the opinionated integration of those components plus defaults, templates, and guardrails. |
| "A platform means no central team" | Self-service shifts what the central team builds, not whether it exists. Platform teams are usually 15-25% of total data headcount. |
| "Data mesh is the platform" | Data mesh is an organizational model. It requires a strong self-service platform to work at all. |
| "We need a platform on day one" | With fewer than ~5 data producers and ~50 pipelines, a well-run central team is cheaper and faster. |
Platform Architecture
A modern data platform decomposes into five horizontal layers plus two vertical concerns that cut across all of them.
graph TB
subgraph Sources["Sources"]
OLTP[(Operational DBs)]
EV[Event streams]
SaaS[SaaS APIs]
Files[Files / partner drops]
end
subgraph Ingestion["1. Ingestion"]
CDC[CDC<br/>Debezium]
ELT[Managed ELT<br/>Fivetran / Airbyte]
STR[Streaming ingest<br/>Kafka Connect]
end
subgraph Storage["2. Storage"]
OBJ[Object storage<br/>S3 / GCS / ADLS]
TF[Open table format<br/>Iceberg / Delta]
end
subgraph Compute["3. Compute & Transformation"]
SPK[Batch<br/>Spark / dbt]
FLK[Streaming<br/>Flink]
ORC[Orchestration<br/>Airflow / Dagster]
end
subgraph Serving["4. Serving"]
SQL[Interactive SQL<br/>Trino / Snowflake]
OLAP[Real-time OLAP<br/>ClickHouse / Pinot]
FS[Feature store]
RETL[Reverse ETL / APIs]
end
subgraph Consumption["5. Consumption"]
BI[BI dashboards]
ML[ML training]
APP[Product features]
end
subgraph Cross["Cross-cutting"]
CAT[Catalog, lineage, governance]
OBS[Observability, quality, cost]
end
Sources --> Ingestion --> Storage --> Compute --> Serving --> Consumption
Compute --> Storage
CAT -.-> Ingestion
CAT -.-> Storage
CAT -.-> Serving
OBS -.-> Ingestion
OBS -.-> Compute
OBS -.-> Serving
Layer 1: Ingestion
Ingestion is where most platforms lose the self-service battle, because every source looks different. The goal is to reduce "add a new source" from a project to a config change.
| Source type | Pattern | Typical tooling | Self-service unit |
|---|---|---|---|
| Operational Postgres/MySQL | Log-based CDC | Debezium, DMS, Datastream | "Register this table for CDC" |
| SaaS (Salesforce, Stripe, HubSpot) | Managed connectors | Fivetran, Airbyte | "Enable connector X with credentials Y" |
| Product events | SDK → event bus | Kafka, Kinesis, Segment, Snowplow | "Publish an event matching schema Z" |
| Partner files | Landing bucket + sensor | S3 events, Airflow sensors | "Drop files in this prefix" |
| Internal APIs | Scheduled extraction | Custom operators, dlt | "Write a small extractor using the SDK" |
The platform's job is to own the contract at the edge: every ingested dataset lands with a schema, an owner, a freshness expectation, and a classification. See CDC Patterns for log-based capture and Incremental Loads for pull-based extraction.
Ingestion Without Ownership Becomes a Swamp
If ingestion is self-service but ownership is optional, you will ingest 2,000 tables in the first year and nobody will know which 200 matter. Make owner and classification mandatory fields that block ingestion when missing.
Layer 2: Storage
The storage layer has converged: cheap object storage plus an open table format. This is the lakehouse pattern, and its main platform benefit is engine independence — Spark, Trino, Flink, DuckDB, and the warehouse can all read the same tables, so you can change compute vendors without migrating data.
Platform-level decisions for storage:
- One table format, not three. Pick Iceberg or Delta and standardize. Supporting multiple formats multiplies catalog, compaction, and access-control work. See Open Table Formats.
- Layered zones. Raw (immutable, as ingested), cleaned (typed, deduplicated), and curated (business-modeled) — the medallion architecture. Each zone has different retention, access, and quality rules.
- Namespacing by domain.
finance.curated.revenue_daily, notanalytics.revenue_daily_v2_final. The namespace encodes ownership. - Table maintenance as a platform service. Compaction, snapshot expiry, and orphan-file cleanup should run automatically for every table. Domain teams should never write compaction jobs.
Layer 3: Compute and Transformation
Compute is where domain teams spend most of their time, so this layer gets the most opinionated golden paths:
| Workload | Golden path | Why |
|---|---|---|
| SQL transformations | dbt on the warehouse or Trino | Analysts already know SQL; dbt gives tests, docs, lineage for free |
| Heavy batch / Python | Spark on Kubernetes or managed (EMR, Dataproc, Databricks) | Scale-out for joins over billions of rows |
| Streaming | Flink or Spark Structured Streaming | Event-time semantics, exactly-once sinks |
| Orchestration | One scheduler (Airflow or Dagster) with a shared deployment | A single place to see every dependency |
The critical design choice is how domain code gets deployed. The lowest-friction pattern is a monorepo or template repo per domain with CI that builds, tests, and registers DAGs automatically. See Pipeline Orchestration and Stream Processing.
Layer 4: Serving
Different consumers need different access paths, and forcing everything through one engine is a common failure:
| Consumer | Latency need | Serving path |
|---|---|---|
| Analysts, BI | Seconds | Warehouse or Trino over curated tables |
| Customer-facing dashboards | Sub-second at high concurrency | Real-time OLAP — see Real-Time Analytics |
| ML training | Minutes, high throughput | Direct Parquet/Iceberg reads |
| ML inference | Milliseconds | Online feature store (Redis, DynamoDB) |
| Operational tools (CRM, support) | Minutes | Reverse ETL |
| Other services | Milliseconds | Data APIs backed by a cache or OLAP store |
Cross-Cutting: Observability, Quality, Cost
Observability for data has three dimensions that service monitoring does not:
- Freshness — when did this table last receive data, versus its promised SLA?
- Volume and distribution — did today's row count drop 60%? Did a column's null rate jump?
- Lineage-aware impact — if this source is late, which dashboards and models are now wrong?
The platform should compute these for every table automatically. See Data Quality Checks, Data Lineage, and Pipeline Monitoring.
Self-Service Tooling
"Self-service" is meaningless unless you can say what a domain team does without filing a ticket. A useful test: list the ten most common requests your central team receives and measure how many can be completed by the requester alone in under an hour.
The Self-Service Surface
flowchart TD
DEV[Domain engineer / analyst] --> SPEC[Write data product spec<br/>YAML in domain repo]
SPEC --> PR[Pull request]
PR --> CI{Platform CI}
CI -->|schema lint| C1[Contract valid?]
CI -->|policy check| C2[Owner, classification,<br/>retention present?]
CI -->|dry run| C3[SQL compiles,<br/>tests pass?]
C1 & C2 & C3 -->|all pass| MERGE[Merge]
CI -->|any fail| FB[Actionable feedback<br/>in PR]
FB --> SPEC
MERGE --> PROV[Provisioner]
PROV --> NS[Namespace + tables]
PROV --> IAM[Access grants]
PROV --> DAG[Scheduled pipeline]
PROV --> CATR[Catalog entry + lineage]
PROV --> MON[Freshness + quality monitors]
The key idea: a single declarative spec drives everything. The domain team describes what they want; the platform derives namespaces, permissions, schedules, monitors, and catalog entries from it.
The Data Product Spec
# domains/payments/products/settled_transactions.yaml
apiVersion: dataplatform/v1
kind: DataProduct
metadata:
name: settled_transactions
domain: payments
owner: team-payments-data
oncall: payments-data-oncall
description: >
One row per settled card transaction, deduplicated and converted
to USD at the settlement-date FX rate.
spec:
classification: confidential # public | internal | confidential | restricted
pii_columns: [cardholder_email] # masked for roles without pii_reader
retention_days: 2555 # 7 years, finance requirement
inputs:
- source: cdc.payments_db.transactions
- source: finance.curated.fx_rates_daily
transform:
engine: dbt
model: models/settled_transactions.sql
schedule: "15 * * * *"
output:
table: payments.curated.settled_transactions
format: iceberg
partition_by: [days(settled_at)]
schema:
- { name: transaction_id, type: string, nullable: false, unique: true }
- { name: merchant_id, type: string, nullable: false }
- { name: amount_usd, type: "decimal(18,2)", nullable: false }
- { name: settled_at, type: timestamp, nullable: false }
- { name: cardholder_email, type: string, nullable: true }
sla:
freshness_minutes: 90
completeness_pct: 99.9
consumers:
- finance.revenue_reporting
- risk.fraud_features
This is a data contract plus deployment manifest in one file. The contract part (schema, SLA) is what consumers rely on; the deployment part (schedule, partitioning) is what the platform executes.
The Provisioner
The provisioner turns specs into infrastructure. It must be idempotent — running it twice with the same spec produces no changes — because it runs on every merge.
# platform/provisioner/apply.py
from __future__ import annotations
import logging
from dataclasses import dataclass
import yaml
from platform.clients import catalog, iam, orchestrator, monitors, warehouse
log = logging.getLogger(__name__)
CLASSIFICATION_READERS = {
"public": ["role:all_employees"],
"internal": ["role:all_employees"],
"confidential": ["role:{domain}_readers", "role:data_platform_admin"],
"restricted": ["role:{domain}_owners"],
}
@dataclass(frozen=True)
class Spec:
name: str
domain: str
owner: str
classification: str
pii_columns: list[str]
output_table: str
schema: list[dict]
schedule: str
freshness_minutes: int
inputs: list[str]
@classmethod
def load(cls, path: str) -> "Spec":
with open(path) as f:
raw = yaml.safe_load(f)
md, sp = raw["metadata"], raw["spec"]
if sp["classification"] not in CLASSIFICATION_READERS:
raise ValueError(f"{path}: unknown classification {sp['classification']!r}")
if not md.get("owner"):
raise ValueError(f"{path}: metadata.owner is required")
return cls(
name=md["name"],
domain=md["domain"],
owner=md["owner"],
classification=sp["classification"],
pii_columns=sp.get("pii_columns", []),
output_table=sp["output"]["table"],
schema=sp["schema"],
schedule=sp["transform"]["schedule"],
freshness_minutes=sp["sla"]["freshness_minutes"],
inputs=[i["source"] for i in sp["inputs"]],
)
def apply(spec: Spec, dry_run: bool = False) -> list[str]:
"""Reconcile platform state with the spec. Returns a list of planned/applied changes."""
changes: list[str] = []
def step(desc: str, fn, *args, **kwargs):
changes.append(desc)
if not dry_run:
fn(*args, **kwargs)
# 1. Table: create or evolve (additive changes only; breaking changes fail CI earlier)
current = warehouse.describe_table(spec.output_table)
if current is None:
step(f"create table {spec.output_table}", warehouse.create_table,
spec.output_table, spec.schema)
else:
for col in spec.schema:
if col["name"] not in current.columns:
step(f"add column {col['name']}", warehouse.add_column,
spec.output_table, col)
# 2. Access: grants derived from classification, PII masking policy
readers = [r.format(domain=spec.domain) for r in CLASSIFICATION_READERS[spec.classification]]
for role in readers:
if not iam.has_grant(role, spec.output_table, "SELECT"):
step(f"grant SELECT to {role}", iam.grant, role, spec.output_table, "SELECT")
for col in spec.pii_columns:
step(f"mask {col} unless role:pii_reader", iam.apply_mask,
spec.output_table, col, unmask_role="role:pii_reader")
# 3. Catalog + lineage
step("register catalog entry", catalog.upsert, spec.output_table,
owner=spec.owner, domain=spec.domain, classification=spec.classification)
for upstream in spec.inputs:
step(f"lineage {upstream} -> {spec.output_table}",
catalog.add_lineage, upstream, spec.output_table)
# 4. Pipeline + monitors
step(f"schedule {spec.name} @ {spec.schedule}", orchestrator.upsert_job,
spec.name, spec.domain, spec.schedule)
step(f"freshness monitor {spec.freshness_minutes}m", monitors.upsert_freshness,
spec.output_table, spec.freshness_minutes, notify=spec.owner)
log.info("%s: %d changes (%s)", spec.name, len(changes), "dry-run" if dry_run else "applied")
return changes
Make the Dry Run the PR Comment
Run apply(spec, dry_run=True) in CI and post the change list as a PR comment. Domain engineers see exactly what the platform will do — "grant SELECT to role:payments_readers, add column refund_reason" — before merging. This one feature does more for trust in the platform than any amount of documentation.
Guardrails in CI
Self-service without guardrails is just distributed chaos. The CI stage enforces platform policy before anything reaches production:
# platform/ci/check_contract.py
"""Fail the PR if a data product change breaks downstream consumers."""
import sys
from platform.provisioner.apply import Spec
from platform.clients import warehouse
BREAKING = "breaking"
def diff_schema(old: list[dict], new: list[dict]) -> list[tuple[str, str]]:
old_by_name = {c["name"]: c for c in old}
new_by_name = {c["name"]: c for c in new}
findings = []
for name, col in old_by_name.items():
if name not in new_by_name:
findings.append((BREAKING, f"column {name} removed"))
continue
if new_by_name[name]["type"] != col["type"]:
findings.append((BREAKING, f"column {name} type {col['type']} -> {new_by_name[name]['type']}"))
if col.get("nullable") is False and new_by_name[name].get("nullable", True):
findings.append((BREAKING, f"column {name} became nullable"))
for name, col in new_by_name.items():
if name not in old_by_name and col.get("nullable") is False:
findings.append((BREAKING, f"new column {name} is NOT NULL (existing rows invalid)"))
return findings
def main(path: str) -> int:
spec = Spec.load(path)
current = warehouse.describe_table(spec.output_table)
if current is None:
return 0 # new product, nothing to break
findings = diff_schema(current.schema, spec.schema)
breaking = [msg for kind, msg in findings if kind == BREAKING]
if breaking:
print(f"Breaking changes to {spec.output_table}:")
for msg in breaking:
print(f" - {msg}")
print("Publish a new version (e.g. settled_transactions_v2) and deprecate the old one.")
return 1
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1]))
The rule it encodes is the core of schema evolution: additive changes are free, breaking changes require a new version and a deprecation window.
Measuring Self-Service
| Metric | Definition | Healthy target |
|---|---|---|
| Time to first data product | New team onboarding → first product in production | < 1 week |
| Ticket ratio | Platform tickets ÷ data products shipped | Falling every quarter |
| Golden path adoption | % of pipelines using platform templates | > 80% |
| Unowned tables | Tables with no owner or no reads in 90 days | < 5% |
| Freshness SLA attainment | % of product-hours within SLA | > 99% |
| Platform NPS | Quarterly survey of domain engineers | Positive and rising |
Data Catalogs and Governance
A catalog answers three questions: what data exists, who owns it, and can I trust it? Governance answers a fourth: am I allowed to use it, and for what?
Catalog Capabilities
| Capability | What it gives you | Tools |
|---|---|---|
| Technical metadata | Schemas, partitions, file stats | Iceberg REST catalog, Unity Catalog, Glue, Polaris |
| Business metadata | Descriptions, glossary terms, owners | DataHub, OpenMetadata, Atlan, Collibra |
| Lineage | Column-level upstream/downstream graph | OpenLineage, dbt artifacts, DataHub |
| Usage | Who queries what, how often | Query log ingestion |
| Quality status | Last test results, freshness | Integration with dbt tests, Soda, Great Expectations |
Two catalogs are normal: a technical catalog that engines talk to (the Iceberg/Unity catalog that resolves table names to files) and a discovery catalog that humans browse. Keep them in sync automatically — the provisioner above writes to both — rather than asking people to document things by hand.
Manual Documentation Rots
Catalogs that depend on people writing descriptions after the fact reach ~20% coverage and stop. Catalogs that pull descriptions from the data product spec (which CI requires) reach near-100% because documentation is a precondition for shipping, not an afterthought.
Governance Model
flowchart LR
subgraph Policy["Policy (central)"]
CL[Classification levels]
RET[Retention rules]
PII[PII handling rules]
REG[Regulatory mapping<br/>GDPR / SOC 2 / PCI]
end
subgraph Enforcement["Enforcement (platform)"]
TAG[Column tags]
ABAC[Attribute-based access]
MASK[Dynamic masking]
AUD[Audit log]
DEL[Deletion workflows]
end
subgraph Decisions["Decisions (domain)"]
CLS[Classify own columns]
APR[Approve access requests]
PURP[Declare purpose of use]
end
Policy --> Enforcement
Decisions --> Enforcement
Enforcement --> AUD
The pattern that scales is federated computational governance: policies are defined centrally, enforced automatically by the platform, and decisions about specific data are made by the owning domain.
- Attribute-based access control (ABAC) beats role-per-table. Tag columns with
pii=emailand write one policy: "maskpii=*unless the role haspii_reader". Ten thousand tables, one policy. - Access requests route to owners, not to the platform team. The platform provides the workflow; the payments team approves access to payments data.
- Right-to-be-forgotten is a platform workflow. Domain teams declare which column identifies a data subject; the platform runs deletes across every table carrying that tag. See GDPR Engineering.
- Every read is auditable. Query logs land in the lake and feed both the usage catalog and audit logging.
Attribute-Based Masking Example
-- Snowflake-style masking policy applied by tag, not by table
CREATE OR REPLACE MASKING POLICY governance.mask_pii_string
AS (val STRING) RETURNS STRING ->
CASE
WHEN IS_ROLE_IN_SESSION('PII_READER') THEN val
WHEN val IS NULL THEN NULL
ELSE SHA2(val, 256) -- stable hash keeps joins working
END;
CREATE TAG IF NOT EXISTS governance.pii ALLOWED_VALUES 'email', 'phone', 'name', 'address';
ALTER TAG governance.pii SET MASKING POLICY governance.mask_pii_string;
-- The provisioner only needs to tag columns; the policy follows the tag.
ALTER TABLE payments.curated.settled_transactions
MODIFY COLUMN cardholder_email SET TAG governance.pii = 'email';
Hashing rather than nulling keeps the column joinable for analysts who need distinct counts but not raw values.
Team and Ownership Models
The architecture is the easy part. The hard part is deciding who owns what. There are four common models, and most organizations move through them in order.
flowchart LR
A["Centralized<br/>One data team<br/>owns everything"] --> B["Hub and spoke<br/>Central platform +<br/>embedded analysts"]
B --> C["Platform + domains<br/>Domains own pipelines,<br/>platform owns primitives"]
C --> D["Data mesh<br/>Domains own data products<br/>end to end"]
style A fill:#fee2e2,stroke:#ef4444
style B fill:#fef3c7,stroke:#f59e0b
style C fill:#dbeafe,stroke:#3b82f6
style D fill:#dcfce7,stroke:#22c55e
| Model | Who builds pipelines | Who owns quality | Works well when | Breaks when |
|---|---|---|---|---|
| Centralized | Central data team | Central data team | < 50 pipelines, few sources, one product | Backlog grows faster than headcount |
| Hub and spoke | Central team, analysts embedded in domains | Shared, often unclear | Domains need analysis but not engineering | Embedded analysts become pipeline engineers without support |
| Platform + domains | Domain engineers using platform templates | Domain team | Domains have engineering capacity | Platform is too immature to self-serve |
| Data mesh | Domain teams, end to end | Domain team, measured as product | Many domains, strong engineering culture, mature platform | Domains lack capacity or incentive to own data |
Data Mesh: What It Actually Requires
Data mesh rests on four principles: domain ownership, data as a product, a self-serve platform, and federated computational governance. Teams that adopt it often underestimate the third and fourth.
What data mesh buys you:
- Ownership sits with the people who understand the data's semantics.
- The central bottleneck disappears — each domain ships at its own pace.
- Quality improves when producers feel the pain of bad data directly.
What data mesh costs you:
- Duplicated skills. Every domain needs at least one person who understands data modeling, incremental loads, and backfills. That is 10 domains × 1-2 engineers you may not have.
- Inconsistent modeling. Without strong conventions,
customer_idmeans four different things in four domains. - Cross-domain queries get harder. Joining payments, marketing, and support data requires shared keys and conformed dimensions that nobody owns by default.
- A larger platform investment, not a smaller one. Self-service must be genuinely good, because domains will not tolerate a platform that requires tickets.
War Story (illustrative)
A company with eight product domains declared a data mesh after a reorg. Each domain was told it now owned its data products. Within six months, three domains had built excellent products; three had copied raw tables into their namespace with _product suffixes and no SLA; two had built nothing because their engineers were fully allocated to features.
The fix was not to abandon the mesh but to admit the platform was not ready. They kept domain ownership for the three mature domains, moved the other five back to a platform-plus-domains model where a central team built pipelines to the domain's spec, and set a bar — templated ingestion, automated monitors, contract CI — before any domain could take full ownership. Two years later, seven of eight domains owned their products end to end.
Platform Team Responsibilities
Whatever model you choose, define the boundary explicitly:
| Platform team owns | Domain team owns |
|---|---|
| Ingestion framework and connectors | Choosing which sources to ingest |
| Storage, table format, maintenance jobs | Table design and partitioning within guidelines |
| Orchestrator deployment and upgrades | DAG logic and schedules |
| Catalog, lineage, access-control enforcement | Descriptions, classification, access approvals |
| Freshness/quality monitoring framework | Thresholds and responding to their own alerts |
| Cost visibility and chargeback | Staying within budget |
| Golden-path templates and docs | Using them, or justifying exceptions |
The single most important line in that table is the last row of the "monitoring" pair: domains respond to their own data alerts. If the platform team is paged when finance's revenue table is late, you have a centralized model with extra steps.
Build vs Buy
Every layer has managed options, open-source options, and the option of writing it yourself. The decision is less about features and more about where you want your engineers' time to go.
| Layer | Buy (managed) | Run (open source) | Build | Default recommendation |
|---|---|---|---|---|
| SaaS ingestion | Fivetran, Stitch | Airbyte, dlt | Custom extractors | Buy for long-tail SaaS; run Airbyte if volume makes per-row pricing painful |
| CDC | DMS, Datastream, Confluent | Debezium on Kafka Connect | — | Run Debezium if you already run Kafka; otherwise managed |
| Storage + table format | Databricks, Snowflake (Iceberg tables) | Iceberg / Delta on S3 | — | Open table format either way; managed if team < 10 |
| Warehouse / SQL | Snowflake, BigQuery, Redshift | Trino, StarRocks | — | Buy until compute bill exceeds ~2 engineers' salary, then evaluate |
| Orchestration | MWAA, Astronomer, Dagster Cloud | Airflow, Dagster | — | Managed: scheduler upgrades are pure toil |
| Catalog / discovery | Atlan, Collibra, Unity Catalog | DataHub, OpenMetadata | — | Open source if you will integrate deeply; managed for governance-heavy orgs |
| Data quality | Monte Carlo, Bigeye | dbt tests, Soda Core, Great Expectations | Thin wrapper | Open source plus a thin platform wrapper |
| Provisioning / specs | — | — | Build | Build — this is your differentiation |
The pattern: buy commodity infrastructure, build the integration layer. Nobody else can write your data product spec, your provisioner, or your guardrails, because they encode your organization's policies. Everything below that layer is undifferentiated heavy lifting.
Total Cost of Ownership
A useful rule of thumb when comparing a managed service to self-hosting:
Self-hosting Airflow for 500 DAGs typically consumes 0.5-1 FTE in upgrades, scaling, and incident response. At a loaded cost of $200K, that is $100-200K/year before infrastructure — often more than a managed offering. The calculation flips at very large scale, or when compliance requires running inside your own network.
The Vendor Lock-in Question
Lock-in is real but often misjudged. Being locked into a warehouse's compute is cheap to escape if your data lives in Iceberg on your own bucket. Being locked into a vendor's proprietary storage format means a migration project measured in quarters. Prioritize open storage; be pragmatic about compute.
Cost Visibility
Self-service compute without cost attribution leads to surprise bills. Tag every query and job with domain and product, then publish a per-domain cost table:
-- Daily warehouse spend per domain and data product
-- (query tags set by the orchestrator: domain=payments;product=settled_transactions)
SELECT
DATE_TRUNC('day', start_time) AS day,
SPLIT_PART(SPLIT_PART(query_tag, 'domain=', 2), ';', 1) AS domain,
SPLIT_PART(SPLIT_PART(query_tag, 'product=', 2), ';', 1) AS product,
ROUND(SUM(credits_used_cloud_services + credits_attributed_compute) * 3.0, 2)
AS est_cost_usd
FROM platform.usage.query_history
WHERE start_time >= DATEADD('day', -30, CURRENT_DATE)
GROUP BY 1, 2, 3
ORDER BY est_cost_usd DESC;
Showback (publishing the numbers) changes behavior by itself; chargeback (billing domains) is usually only needed once spend is significant. See Cost Allocation.
Maturity Stages
The data engineering maturity model describes pipeline capabilities. Platform maturity is about how much of that capability is self-service.
| Stage | Name | What exists | Who builds pipelines | Signal you are ready for the next stage |
|---|---|---|---|---|
| 0 | Scripts | Cron jobs, one warehouse, no catalog | One or two data engineers | More than ~3 teams asking for data |
| 1 | Managed stack | Managed ELT, warehouse, dbt, one orchestrator | Central team | Backlog > 4 weeks; same patterns rebuilt repeatedly |
| 2 | Templated | Pipeline templates, shared CI, automated freshness checks | Central team plus power users in domains | Domains want to ship without waiting |
| 3 | Self-service | Data product specs, provisioner, catalog from specs, ABAC | Domain teams on golden paths | Cross-domain consistency and cost become the main problems |
| 4 | Federated | Data mesh with computational governance, chargeback, product SLAs | Domain teams end to end | — |
flowchart LR
S0[Stage 0<br/>Scripts] --> S1[Stage 1<br/>Managed stack]
S1 --> S2[Stage 2<br/>Templated]
S2 --> S3[Stage 3<br/>Self-service]
S3 --> S4[Stage 4<br/>Federated]
S0 -.->|invest in| I0[Pick managed ELT<br/>+ warehouse + dbt]
S1 -.->|invest in| I1[Templates, CI,<br/>freshness monitors]
S2 -.->|invest in| I2[Specs, provisioner,<br/>catalog automation]
S3 -.->|invest in| I3[Governance policies,<br/>cost attribution]
Skip Stages at Your Peril
The most common failure is jumping from Stage 1 to Stage 4 because a conference talk made data mesh sound inevitable. Each stage builds the foundation the next one assumes. Domain ownership without templates (Stage 2) produces ten different ways to do incremental loads; specs without automated catalogs (Stage 3) produce documentation nobody trusts.
Reference Architecture: Mid-Size Company
A concrete example for a company with ~300 engineers, ~40 data practitioners, and 8 product domains at Stage 3:
graph TB
subgraph Ingest["Ingestion"]
DBZ[Debezium → Kafka<br/>12 Postgres DBs]
FIV[Fivetran<br/>35 SaaS sources]
SDK[Event SDK → Kafka<br/>~400 event types]
end
subgraph Lake["Storage: Iceberg on S3"]
RAW[raw.*<br/>immutable, 90d hot]
CLN[domain.cleaned.*]
CUR[domain.curated.*<br/>data products]
end
subgraph Proc["Compute"]
FLINK[Flink<br/>Kafka → raw]
DBT[dbt on Trino + Snowflake<br/>~1,800 models]
AF[Managed Airflow<br/>one DAG per data product]
end
subgraph Serve["Serving"]
SNOW[Snowflake<br/>BI and ad hoc]
CH[ClickHouse<br/>customer dashboards]
REDIS[Redis feature store]
HT[Reverse ETL → CRM]
end
subgraph Gov["Platform services"]
POL[Polaris / Iceberg REST catalog]
DH[DataHub<br/>discovery + lineage]
PROV[Provisioner + contract CI]
MON[Freshness, volume,<br/>cost dashboards]
end
DBZ --> FLINK
SDK --> FLINK
FLINK --> RAW
FIV --> RAW
RAW --> DBT --> CLN --> DBT
DBT --> CUR
AF -.->|schedules| DBT
CUR --> SNOW
CUR --> CH
CUR --> REDIS
CUR --> HT
PROV -.-> POL
PROV -.-> DH
PROV -.-> AF
MON -.-> CUR
| Metric | Value |
|---|---|
| Platform team size | 7 engineers (ingestion 2, compute/orchestration 2, governance/catalog 2, DX 1) |
| Domain data engineers | ~20 across 8 domains |
| Analysts | ~13, writing dbt directly |
| Data products with specs | ~240 |
| Time to onboard a new CDC table | ~15 minutes (spec PR + review) |
| Freshness SLA attainment | 99.4% of product-hours |
Anti-Patterns
| Anti-pattern | Symptom | Fix |
|---|---|---|
| Platform as ticket queue | "Self-service" portal that creates a Jira ticket | Automate the fulfilment, not the request form |
| Golden cage | Golden path is mandatory and cannot handle 20% of real use cases | Allow documented escape hatches; fold common exceptions back into the path |
| Tool zoo | Three orchestrators, two table formats, four BI tools | Standardize per layer; deprecate with dates |
| Ownership theater | Every table has an owner field set to a departed engineer | Owners are teams, not people; validate against the org directory in CI |
| Platform without product management | Platform builds what is interesting, not what is needed | Treat domains as customers: roadmap, surveys, adoption metrics |
| Governance as gatekeeping | Every access request needs a committee | Encode policy in ABAC; route remaining approvals to owners |
| Mesh without a platform | Each domain builds its own stack | Platform maturity Stage 3 before domain ownership |
Decision Framework
flowchart TD
Q1{How many teams<br/>produce data?} -->|1-3| A1[Centralized team<br/>+ managed stack]
Q1 -->|4-10| Q2{Do domains have<br/>engineering capacity?}
Q1 -->|10+| Q3{Is self-service<br/>platform at Stage 3?}
Q2 -->|No| A2[Hub and spoke<br/>+ templates]
Q2 -->|Yes| A3[Platform + domains]
Q3 -->|No| A4[Platform + domains,<br/>invest in platform first]
Q3 -->|Yes| A5[Data mesh with<br/>federated governance]
Key Takeaways
- A data platform exists to remove the central team as the bottleneck — measure success by how many requests domains complete without a ticket.
- Architect in layers (ingestion, storage, compute, serving) with catalog, governance, observability, and cost as cross-cutting concerns.
- Standardize on open storage (one table format) so compute can change without data migration.
- Drive everything from a declarative data product spec: tables, grants, schedules, monitors, and catalog entries are all derived from one file.
- Governance scales through policy-as-code and attribute-based access, with decisions delegated to owning domains.
- Data mesh is an organizational model that demands a mature self-service platform; adopt it after Stage 3, not instead of it.
- Buy commodity infrastructure, build the integration layer that encodes your organization's policies.
Related Pages
- Data Lakehouse — the storage architecture most platforms standardize on
- Medallion Architecture — raw, cleaned, and curated zones
- Query Engines — choosing compute over open tables
- Pipeline Patterns — the patterns domain teams build on the platform
- Data Lineage — impact analysis across domains
- Data Contracts — producer-consumer agreements in depth
- ETL vs ELT — where transformation happens
- Platform Engineering Maturity — the service-platform equivalent of this page
- Developer Experience — measuring and improving platform usability