On this page

Data Platform Design

Why Data Platforms Exist

Every company that takes data seriously goes through the same arc. First, one data engineer writes a few cron jobs that copy production tables into a warehouse. Then analysts ask for more sources, so the data engineer writes more jobs. Then the ML team needs features, product needs event analytics, finance needs reconciled revenue, and suddenly there are three hundred pipelines, two warehouses, a lake nobody trusts, and a central data team with a six-week ticket backlog.

The bottleneck is not compute or storage. It is people. A central team that builds every pipeline by hand cannot scale linearly with the number of questions the business asks. A data platform exists to break that coupling: the platform team builds the paved road — ingestion, storage, compute, governance, and serving as self-service primitives — and the teams that actually understand the data use those primitives to build their own pipelines and data products.

flowchart LR
    subgraph Before["Central Team Model"]
        direction TB
        R1[Marketing request] --> Q[Ticket queue]
        R2[Finance request] --> Q
        R3[ML request] --> Q
        R4[Product request] --> Q
        Q --> DT[Central data team<br/>builds every pipeline]
        DT --> WH1[(Warehouse)]
    end

    subgraph After["Platform Model"]
        direction TB
        PT[Platform team<br/>builds primitives] --> PR[Ingestion / Storage /<br/>Compute / Catalog / Serving]
        PR --> D1[Marketing builds<br/>own pipelines]
        PR --> D2[Finance builds<br/>own pipelines]
        PR --> D3[ML builds<br/>own pipelines]
        D1 --> LH[(Shared lakehouse)]
        D2 --> LH
        D3 --> LH
    end

    style Before fill:#fee2e2,stroke:#ef4444
    style After fill:#dcfce7,stroke:#22c55e

A data platform is to data what an internal developer platform is to services: a product whose customers are other engineers and analysts, whose job is to make the right thing the easy thing.

What a Platform Is Not

MisconceptionReality
"A platform is the tools we bought"Tools are components. A platform is the opinionated integration of those components plus defaults, templates, and guardrails.
"A platform means no central team"Self-service shifts what the central team builds, not whether it exists. Platform teams are usually 15-25% of total data headcount.
"Data mesh is the platform"Data mesh is an organizational model. It requires a strong self-service platform to work at all.
"We need a platform on day one"With fewer than ~5 data producers and ~50 pipelines, a well-run central team is cheaper and faster.

Platform Architecture

A modern data platform decomposes into five horizontal layers plus two vertical concerns that cut across all of them.

graph TB
    subgraph Sources["Sources"]
        OLTP[(Operational DBs)]
        EV[Event streams]
        SaaS[SaaS APIs]
        Files[Files / partner drops]
    end

    subgraph Ingestion["1. Ingestion"]
        CDC[CDC<br/>Debezium]
        ELT[Managed ELT<br/>Fivetran / Airbyte]
        STR[Streaming ingest<br/>Kafka Connect]
    end

    subgraph Storage["2. Storage"]
        OBJ[Object storage<br/>S3 / GCS / ADLS]
        TF[Open table format<br/>Iceberg / Delta]
    end

    subgraph Compute["3. Compute & Transformation"]
        SPK[Batch<br/>Spark / dbt]
        FLK[Streaming<br/>Flink]
        ORC[Orchestration<br/>Airflow / Dagster]
    end

    subgraph Serving["4. Serving"]
        SQL[Interactive SQL<br/>Trino / Snowflake]
        OLAP[Real-time OLAP<br/>ClickHouse / Pinot]
        FS[Feature store]
        RETL[Reverse ETL / APIs]
    end

    subgraph Consumption["5. Consumption"]
        BI[BI dashboards]
        ML[ML training]
        APP[Product features]
    end

    subgraph Cross["Cross-cutting"]
        CAT[Catalog, lineage, governance]
        OBS[Observability, quality, cost]
    end

    Sources --> Ingestion --> Storage --> Compute --> Serving --> Consumption
    Compute --> Storage
    CAT -.-> Ingestion
    CAT -.-> Storage
    CAT -.-> Serving
    OBS -.-> Ingestion
    OBS -.-> Compute
    OBS -.-> Serving

Layer 1: Ingestion

Ingestion is where most platforms lose the self-service battle, because every source looks different. The goal is to reduce "add a new source" from a project to a config change.

Source typePatternTypical toolingSelf-service unit
Operational Postgres/MySQLLog-based CDCDebezium, DMS, Datastream"Register this table for CDC"
SaaS (Salesforce, Stripe, HubSpot)Managed connectorsFivetran, Airbyte"Enable connector X with credentials Y"
Product eventsSDK → event busKafka, Kinesis, Segment, Snowplow"Publish an event matching schema Z"
Partner filesLanding bucket + sensorS3 events, Airflow sensors"Drop files in this prefix"
Internal APIsScheduled extractionCustom operators, dlt"Write a small extractor using the SDK"

The platform's job is to own the contract at the edge: every ingested dataset lands with a schema, an owner, a freshness expectation, and a classification. See CDC Patterns for log-based capture and Incremental Loads for pull-based extraction.

Ingestion Without Ownership Becomes a Swamp

If ingestion is self-service but ownership is optional, you will ingest 2,000 tables in the first year and nobody will know which 200 matter. Make owner and classification mandatory fields that block ingestion when missing.

Layer 2: Storage

The storage layer has converged: cheap object storage plus an open table format. This is the lakehouse pattern, and its main platform benefit is engine independence — Spark, Trino, Flink, DuckDB, and the warehouse can all read the same tables, so you can change compute vendors without migrating data.

Platform-level decisions for storage:

  • One table format, not three. Pick Iceberg or Delta and standardize. Supporting multiple formats multiplies catalog, compaction, and access-control work. See Open Table Formats.
  • Layered zones. Raw (immutable, as ingested), cleaned (typed, deduplicated), and curated (business-modeled) — the medallion architecture. Each zone has different retention, access, and quality rules.
  • Namespacing by domain. finance.curated.revenue_daily, not analytics.revenue_daily_v2_final. The namespace encodes ownership.
  • Table maintenance as a platform service. Compaction, snapshot expiry, and orphan-file cleanup should run automatically for every table. Domain teams should never write compaction jobs.

Layer 3: Compute and Transformation

Compute is where domain teams spend most of their time, so this layer gets the most opinionated golden paths:

WorkloadGolden pathWhy
SQL transformationsdbt on the warehouse or TrinoAnalysts already know SQL; dbt gives tests, docs, lineage for free
Heavy batch / PythonSpark on Kubernetes or managed (EMR, Dataproc, Databricks)Scale-out for joins over billions of rows
StreamingFlink or Spark Structured StreamingEvent-time semantics, exactly-once sinks
OrchestrationOne scheduler (Airflow or Dagster) with a shared deploymentA single place to see every dependency

The critical design choice is how domain code gets deployed. The lowest-friction pattern is a monorepo or template repo per domain with CI that builds, tests, and registers DAGs automatically. See Pipeline Orchestration and Stream Processing.

Layer 4: Serving

Different consumers need different access paths, and forcing everything through one engine is a common failure:

ConsumerLatency needServing path
Analysts, BISecondsWarehouse or Trino over curated tables
Customer-facing dashboardsSub-second at high concurrencyReal-time OLAP — see Real-Time Analytics
ML trainingMinutes, high throughputDirect Parquet/Iceberg reads
ML inferenceMillisecondsOnline feature store (Redis, DynamoDB)
Operational tools (CRM, support)MinutesReverse ETL
Other servicesMillisecondsData APIs backed by a cache or OLAP store

Cross-Cutting: Observability, Quality, Cost

Observability for data has three dimensions that service monitoring does not:

  1. Freshness — when did this table last receive data, versus its promised SLA?
  2. Volume and distribution — did today's row count drop 60%? Did a column's null rate jump?
  3. Lineage-aware impact — if this source is late, which dashboards and models are now wrong?

The platform should compute these for every table automatically. See Data Quality Checks, Data Lineage, and Pipeline Monitoring.

Self-Service Tooling

"Self-service" is meaningless unless you can say what a domain team does without filing a ticket. A useful test: list the ten most common requests your central team receives and measure how many can be completed by the requester alone in under an hour.

The Self-Service Surface

flowchart TD
    DEV[Domain engineer / analyst] --> SPEC[Write data product spec<br/>YAML in domain repo]
    SPEC --> PR[Pull request]
    PR --> CI{Platform CI}
    CI -->|schema lint| C1[Contract valid?]
    CI -->|policy check| C2[Owner, classification,<br/>retention present?]
    CI -->|dry run| C3[SQL compiles,<br/>tests pass?]
    C1 & C2 & C3 -->|all pass| MERGE[Merge]
    CI -->|any fail| FB[Actionable feedback<br/>in PR]
    FB --> SPEC
    MERGE --> PROV[Provisioner]
    PROV --> NS[Namespace + tables]
    PROV --> IAM[Access grants]
    PROV --> DAG[Scheduled pipeline]
    PROV --> CATR[Catalog entry + lineage]
    PROV --> MON[Freshness + quality monitors]

The key idea: a single declarative spec drives everything. The domain team describes what they want; the platform derives namespaces, permissions, schedules, monitors, and catalog entries from it.

The Data Product Spec

# domains/payments/products/settled_transactions.yaml
apiVersion: dataplatform/v1
kind: DataProduct
metadata:
  name: settled_transactions
  domain: payments
  owner: team-payments-data
  oncall: payments-data-oncall
  description: >
    One row per settled card transaction, deduplicated and converted
    to USD at the settlement-date FX rate.
spec:
  classification: confidential        # public | internal | confidential | restricted
  pii_columns: [cardholder_email]     # masked for roles without pii_reader
  retention_days: 2555                # 7 years, finance requirement
  inputs:
    - source: cdc.payments_db.transactions
    - source: finance.curated.fx_rates_daily
  transform:
    engine: dbt
    model: models/settled_transactions.sql
    schedule: "15 * * * *"
  output:
    table: payments.curated.settled_transactions
    format: iceberg
    partition_by: [days(settled_at)]
  schema:
    - { name: transaction_id, type: string, nullable: false, unique: true }
    - { name: merchant_id,    type: string, nullable: false }
    - { name: amount_usd,     type: "decimal(18,2)", nullable: false }
    - { name: settled_at,     type: timestamp, nullable: false }
    - { name: cardholder_email, type: string, nullable: true }
  sla:
    freshness_minutes: 90
    completeness_pct: 99.9
  consumers:
    - finance.revenue_reporting
    - risk.fraud_features

This is a data contract plus deployment manifest in one file. The contract part (schema, SLA) is what consumers rely on; the deployment part (schedule, partitioning) is what the platform executes.

The Provisioner

The provisioner turns specs into infrastructure. It must be idempotent — running it twice with the same spec produces no changes — because it runs on every merge.

# platform/provisioner/apply.py
from __future__ import annotations

import logging
from dataclasses import dataclass

import yaml

from platform.clients import catalog, iam, orchestrator, monitors, warehouse

log = logging.getLogger(__name__)

CLASSIFICATION_READERS = {
    "public": ["role:all_employees"],
    "internal": ["role:all_employees"],
    "confidential": ["role:{domain}_readers", "role:data_platform_admin"],
    "restricted": ["role:{domain}_owners"],
}


@dataclass(frozen=True)
class Spec:
    name: str
    domain: str
    owner: str
    classification: str
    pii_columns: list[str]
    output_table: str
    schema: list[dict]
    schedule: str
    freshness_minutes: int
    inputs: list[str]

    @classmethod
    def load(cls, path: str) -> "Spec":
        with open(path) as f:
            raw = yaml.safe_load(f)
        md, sp = raw["metadata"], raw["spec"]
        if sp["classification"] not in CLASSIFICATION_READERS:
            raise ValueError(f"{path}: unknown classification {sp['classification']!r}")
        if not md.get("owner"):
            raise ValueError(f"{path}: metadata.owner is required")
        return cls(
            name=md["name"],
            domain=md["domain"],
            owner=md["owner"],
            classification=sp["classification"],
            pii_columns=sp.get("pii_columns", []),
            output_table=sp["output"]["table"],
            schema=sp["schema"],
            schedule=sp["transform"]["schedule"],
            freshness_minutes=sp["sla"]["freshness_minutes"],
            inputs=[i["source"] for i in sp["inputs"]],
        )


def apply(spec: Spec, dry_run: bool = False) -> list[str]:
    """Reconcile platform state with the spec. Returns a list of planned/applied changes."""
    changes: list[str] = []

    def step(desc: str, fn, *args, **kwargs):
        changes.append(desc)
        if not dry_run:
            fn(*args, **kwargs)

    # 1. Table: create or evolve (additive changes only; breaking changes fail CI earlier)
    current = warehouse.describe_table(spec.output_table)
    if current is None:
        step(f"create table {spec.output_table}", warehouse.create_table,
             spec.output_table, spec.schema)
    else:
        for col in spec.schema:
            if col["name"] not in current.columns:
                step(f"add column {col['name']}", warehouse.add_column,
                     spec.output_table, col)

    # 2. Access: grants derived from classification, PII masking policy
    readers = [r.format(domain=spec.domain) for r in CLASSIFICATION_READERS[spec.classification]]
    for role in readers:
        if not iam.has_grant(role, spec.output_table, "SELECT"):
            step(f"grant SELECT to {role}", iam.grant, role, spec.output_table, "SELECT")
    for col in spec.pii_columns:
        step(f"mask {col} unless role:pii_reader", iam.apply_mask,
             spec.output_table, col, unmask_role="role:pii_reader")

    # 3. Catalog + lineage
    step("register catalog entry", catalog.upsert, spec.output_table,
         owner=spec.owner, domain=spec.domain, classification=spec.classification)
    for upstream in spec.inputs:
        step(f"lineage {upstream} -> {spec.output_table}",
             catalog.add_lineage, upstream, spec.output_table)

    # 4. Pipeline + monitors
    step(f"schedule {spec.name} @ {spec.schedule}", orchestrator.upsert_job,
         spec.name, spec.domain, spec.schedule)
    step(f"freshness monitor {spec.freshness_minutes}m", monitors.upsert_freshness,
         spec.output_table, spec.freshness_minutes, notify=spec.owner)

    log.info("%s: %d changes (%s)", spec.name, len(changes), "dry-run" if dry_run else "applied")
    return changes

Make the Dry Run the PR Comment

Run apply(spec, dry_run=True) in CI and post the change list as a PR comment. Domain engineers see exactly what the platform will do — "grant SELECT to role:payments_readers, add column refund_reason" — before merging. This one feature does more for trust in the platform than any amount of documentation.

Guardrails in CI

Self-service without guardrails is just distributed chaos. The CI stage enforces platform policy before anything reaches production:

# platform/ci/check_contract.py
"""Fail the PR if a data product change breaks downstream consumers."""
import sys

from platform.provisioner.apply import Spec
from platform.clients import warehouse

BREAKING = "breaking"


def diff_schema(old: list[dict], new: list[dict]) -> list[tuple[str, str]]:
    old_by_name = {c["name"]: c for c in old}
    new_by_name = {c["name"]: c for c in new}
    findings = []
    for name, col in old_by_name.items():
        if name not in new_by_name:
            findings.append((BREAKING, f"column {name} removed"))
            continue
        if new_by_name[name]["type"] != col["type"]:
            findings.append((BREAKING, f"column {name} type {col['type']} -> {new_by_name[name]['type']}"))
        if col.get("nullable") is False and new_by_name[name].get("nullable", True):
            findings.append((BREAKING, f"column {name} became nullable"))
    for name, col in new_by_name.items():
        if name not in old_by_name and col.get("nullable") is False:
            findings.append((BREAKING, f"new column {name} is NOT NULL (existing rows invalid)"))
    return findings


def main(path: str) -> int:
    spec = Spec.load(path)
    current = warehouse.describe_table(spec.output_table)
    if current is None:
        return 0  # new product, nothing to break
    findings = diff_schema(current.schema, spec.schema)
    breaking = [msg for kind, msg in findings if kind == BREAKING]
    if breaking:
        print(f"Breaking changes to {spec.output_table}:")
        for msg in breaking:
            print(f"  - {msg}")
        print("Publish a new version (e.g. settled_transactions_v2) and deprecate the old one.")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

The rule it encodes is the core of schema evolution: additive changes are free, breaking changes require a new version and a deprecation window.

Measuring Self-Service

MetricDefinitionHealthy target
Time to first data productNew team onboarding → first product in production< 1 week
Ticket ratioPlatform tickets ÷ data products shippedFalling every quarter
Golden path adoption% of pipelines using platform templates> 80%
Unowned tablesTables with no owner or no reads in 90 days< 5%
Freshness SLA attainment% of product-hours within SLA> 99%
Platform NPSQuarterly survey of domain engineersPositive and rising

Data Catalogs and Governance

A catalog answers three questions: what data exists, who owns it, and can I trust it? Governance answers a fourth: am I allowed to use it, and for what?

Catalog Capabilities

CapabilityWhat it gives youTools
Technical metadataSchemas, partitions, file statsIceberg REST catalog, Unity Catalog, Glue, Polaris
Business metadataDescriptions, glossary terms, ownersDataHub, OpenMetadata, Atlan, Collibra
LineageColumn-level upstream/downstream graphOpenLineage, dbt artifacts, DataHub
UsageWho queries what, how oftenQuery log ingestion
Quality statusLast test results, freshnessIntegration with dbt tests, Soda, Great Expectations

Two catalogs are normal: a technical catalog that engines talk to (the Iceberg/Unity catalog that resolves table names to files) and a discovery catalog that humans browse. Keep them in sync automatically — the provisioner above writes to both — rather than asking people to document things by hand.

Manual Documentation Rots

Catalogs that depend on people writing descriptions after the fact reach ~20% coverage and stop. Catalogs that pull descriptions from the data product spec (which CI requires) reach near-100% because documentation is a precondition for shipping, not an afterthought.

Governance Model

flowchart LR
    subgraph Policy["Policy (central)"]
        CL[Classification levels]
        RET[Retention rules]
        PII[PII handling rules]
        REG[Regulatory mapping<br/>GDPR / SOC 2 / PCI]
    end

    subgraph Enforcement["Enforcement (platform)"]
        TAG[Column tags]
        ABAC[Attribute-based access]
        MASK[Dynamic masking]
        AUD[Audit log]
        DEL[Deletion workflows]
    end

    subgraph Decisions["Decisions (domain)"]
        CLS[Classify own columns]
        APR[Approve access requests]
        PURP[Declare purpose of use]
    end

    Policy --> Enforcement
    Decisions --> Enforcement
    Enforcement --> AUD

The pattern that scales is federated computational governance: policies are defined centrally, enforced automatically by the platform, and decisions about specific data are made by the owning domain.

  • Attribute-based access control (ABAC) beats role-per-table. Tag columns with pii=email and write one policy: "mask pii=* unless the role has pii_reader". Ten thousand tables, one policy.
  • Access requests route to owners, not to the platform team. The platform provides the workflow; the payments team approves access to payments data.
  • Right-to-be-forgotten is a platform workflow. Domain teams declare which column identifies a data subject; the platform runs deletes across every table carrying that tag. See GDPR Engineering.
  • Every read is auditable. Query logs land in the lake and feed both the usage catalog and audit logging.

Attribute-Based Masking Example

-- Snowflake-style masking policy applied by tag, not by table
CREATE OR REPLACE MASKING POLICY governance.mask_pii_string
  AS (val STRING) RETURNS STRING ->
  CASE
    WHEN IS_ROLE_IN_SESSION('PII_READER') THEN val
    WHEN val IS NULL THEN NULL
    ELSE SHA2(val, 256)      -- stable hash keeps joins working
  END;

CREATE TAG IF NOT EXISTS governance.pii ALLOWED_VALUES 'email', 'phone', 'name', 'address';

ALTER TAG governance.pii SET MASKING POLICY governance.mask_pii_string;

-- The provisioner only needs to tag columns; the policy follows the tag.
ALTER TABLE payments.curated.settled_transactions
  MODIFY COLUMN cardholder_email SET TAG governance.pii = 'email';

Hashing rather than nulling keeps the column joinable for analysts who need distinct counts but not raw values.

Team and Ownership Models

The architecture is the easy part. The hard part is deciding who owns what. There are four common models, and most organizations move through them in order.

flowchart LR
    A["Centralized<br/>One data team<br/>owns everything"] --> B["Hub and spoke<br/>Central platform +<br/>embedded analysts"]
    B --> C["Platform + domains<br/>Domains own pipelines,<br/>platform owns primitives"]
    C --> D["Data mesh<br/>Domains own data products<br/>end to end"]

    style A fill:#fee2e2,stroke:#ef4444
    style B fill:#fef3c7,stroke:#f59e0b
    style C fill:#dbeafe,stroke:#3b82f6
    style D fill:#dcfce7,stroke:#22c55e
ModelWho builds pipelinesWho owns qualityWorks well whenBreaks when
CentralizedCentral data teamCentral data team< 50 pipelines, few sources, one productBacklog grows faster than headcount
Hub and spokeCentral team, analysts embedded in domainsShared, often unclearDomains need analysis but not engineeringEmbedded analysts become pipeline engineers without support
Platform + domainsDomain engineers using platform templatesDomain teamDomains have engineering capacityPlatform is too immature to self-serve
Data meshDomain teams, end to endDomain team, measured as productMany domains, strong engineering culture, mature platformDomains lack capacity or incentive to own data

Data Mesh: What It Actually Requires

Data mesh rests on four principles: domain ownership, data as a product, a self-serve platform, and federated computational governance. Teams that adopt it often underestimate the third and fourth.

What data mesh buys you:

  • Ownership sits with the people who understand the data's semantics.
  • The central bottleneck disappears — each domain ships at its own pace.
  • Quality improves when producers feel the pain of bad data directly.

What data mesh costs you:

  • Duplicated skills. Every domain needs at least one person who understands data modeling, incremental loads, and backfills. That is 10 domains × 1-2 engineers you may not have.
  • Inconsistent modeling. Without strong conventions, customer_id means four different things in four domains.
  • Cross-domain queries get harder. Joining payments, marketing, and support data requires shared keys and conformed dimensions that nobody owns by default.
  • A larger platform investment, not a smaller one. Self-service must be genuinely good, because domains will not tolerate a platform that requires tickets.

War Story (illustrative)

A company with eight product domains declared a data mesh after a reorg. Each domain was told it now owned its data products. Within six months, three domains had built excellent products; three had copied raw tables into their namespace with _product suffixes and no SLA; two had built nothing because their engineers were fully allocated to features.

The fix was not to abandon the mesh but to admit the platform was not ready. They kept domain ownership for the three mature domains, moved the other five back to a platform-plus-domains model where a central team built pipelines to the domain's spec, and set a bar — templated ingestion, automated monitors, contract CI — before any domain could take full ownership. Two years later, seven of eight domains owned their products end to end.

Platform Team Responsibilities

Whatever model you choose, define the boundary explicitly:

Platform team ownsDomain team owns
Ingestion framework and connectorsChoosing which sources to ingest
Storage, table format, maintenance jobsTable design and partitioning within guidelines
Orchestrator deployment and upgradesDAG logic and schedules
Catalog, lineage, access-control enforcementDescriptions, classification, access approvals
Freshness/quality monitoring frameworkThresholds and responding to their own alerts
Cost visibility and chargebackStaying within budget
Golden-path templates and docsUsing them, or justifying exceptions

The single most important line in that table is the last row of the "monitoring" pair: domains respond to their own data alerts. If the platform team is paged when finance's revenue table is late, you have a centralized model with extra steps.

Build vs Buy

Every layer has managed options, open-source options, and the option of writing it yourself. The decision is less about features and more about where you want your engineers' time to go.

LayerBuy (managed)Run (open source)BuildDefault recommendation
SaaS ingestionFivetran, StitchAirbyte, dltCustom extractorsBuy for long-tail SaaS; run Airbyte if volume makes per-row pricing painful
CDCDMS, Datastream, ConfluentDebezium on Kafka Connect—Run Debezium if you already run Kafka; otherwise managed
Storage + table formatDatabricks, Snowflake (Iceberg tables)Iceberg / Delta on S3—Open table format either way; managed if team < 10
Warehouse / SQLSnowflake, BigQuery, RedshiftTrino, StarRocks—Buy until compute bill exceeds ~2 engineers' salary, then evaluate
OrchestrationMWAA, Astronomer, Dagster CloudAirflow, Dagster—Managed: scheduler upgrades are pure toil
Catalog / discoveryAtlan, Collibra, Unity CatalogDataHub, OpenMetadata—Open source if you will integrate deeply; managed for governance-heavy orgs
Data qualityMonte Carlo, Bigeyedbt tests, Soda Core, Great ExpectationsThin wrapperOpen source plus a thin platform wrapper
Provisioning / specs——BuildBuild — this is your differentiation

The pattern: buy commodity infrastructure, build the integration layer. Nobody else can write your data product spec, your provisioner, or your guardrails, because they encode your organization's policies. Everything below that layer is undifferentiated heavy lifting.

Total Cost of Ownership

A useful rule of thumb when comparing a managed service to self-hosting:

Self-hosting Airflow for 500 DAGs typically consumes 0.5-1 FTE in upgrades, scaling, and incident response. At a loaded cost of $200K, that is $100-200K/year before infrastructure — often more than a managed offering. The calculation flips at very large scale, or when compliance requires running inside your own network.

The Vendor Lock-in Question

Lock-in is real but often misjudged. Being locked into a warehouse's compute is cheap to escape if your data lives in Iceberg on your own bucket. Being locked into a vendor's proprietary storage format means a migration project measured in quarters. Prioritize open storage; be pragmatic about compute.

Cost Visibility

Self-service compute without cost attribution leads to surprise bills. Tag every query and job with domain and product, then publish a per-domain cost table:

-- Daily warehouse spend per domain and data product
-- (query tags set by the orchestrator: domain=payments;product=settled_transactions)
SELECT
    DATE_TRUNC('day', start_time)                          AS day,
    SPLIT_PART(SPLIT_PART(query_tag, 'domain=', 2), ';', 1)  AS domain,
    SPLIT_PART(SPLIT_PART(query_tag, 'product=', 2), ';', 1) AS product,
    ROUND(SUM(credits_used_cloud_services + credits_attributed_compute) * 3.0, 2)
                                                           AS est_cost_usd
FROM platform.usage.query_history
WHERE start_time >= DATEADD('day', -30, CURRENT_DATE)
GROUP BY 1, 2, 3
ORDER BY est_cost_usd DESC;

Showback (publishing the numbers) changes behavior by itself; chargeback (billing domains) is usually only needed once spend is significant. See Cost Allocation.

Maturity Stages

The data engineering maturity model describes pipeline capabilities. Platform maturity is about how much of that capability is self-service.

StageNameWhat existsWho builds pipelinesSignal you are ready for the next stage
0ScriptsCron jobs, one warehouse, no catalogOne or two data engineersMore than ~3 teams asking for data
1Managed stackManaged ELT, warehouse, dbt, one orchestratorCentral teamBacklog > 4 weeks; same patterns rebuilt repeatedly
2TemplatedPipeline templates, shared CI, automated freshness checksCentral team plus power users in domainsDomains want to ship without waiting
3Self-serviceData product specs, provisioner, catalog from specs, ABACDomain teams on golden pathsCross-domain consistency and cost become the main problems
4FederatedData mesh with computational governance, chargeback, product SLAsDomain teams end to end—
flowchart LR
    S0[Stage 0<br/>Scripts] --> S1[Stage 1<br/>Managed stack]
    S1 --> S2[Stage 2<br/>Templated]
    S2 --> S3[Stage 3<br/>Self-service]
    S3 --> S4[Stage 4<br/>Federated]

    S0 -.->|invest in| I0[Pick managed ELT<br/>+ warehouse + dbt]
    S1 -.->|invest in| I1[Templates, CI,<br/>freshness monitors]
    S2 -.->|invest in| I2[Specs, provisioner,<br/>catalog automation]
    S3 -.->|invest in| I3[Governance policies,<br/>cost attribution]

Skip Stages at Your Peril

The most common failure is jumping from Stage 1 to Stage 4 because a conference talk made data mesh sound inevitable. Each stage builds the foundation the next one assumes. Domain ownership without templates (Stage 2) produces ten different ways to do incremental loads; specs without automated catalogs (Stage 3) produce documentation nobody trusts.

Reference Architecture: Mid-Size Company

A concrete example for a company with ~300 engineers, ~40 data practitioners, and 8 product domains at Stage 3:

graph TB
    subgraph Ingest["Ingestion"]
        DBZ[Debezium → Kafka<br/>12 Postgres DBs]
        FIV[Fivetran<br/>35 SaaS sources]
        SDK[Event SDK → Kafka<br/>~400 event types]
    end

    subgraph Lake["Storage: Iceberg on S3"]
        RAW[raw.*<br/>immutable, 90d hot]
        CLN[domain.cleaned.*]
        CUR[domain.curated.*<br/>data products]
    end

    subgraph Proc["Compute"]
        FLINK[Flink<br/>Kafka → raw]
        DBT[dbt on Trino + Snowflake<br/>~1,800 models]
        AF[Managed Airflow<br/>one DAG per data product]
    end

    subgraph Serve["Serving"]
        SNOW[Snowflake<br/>BI and ad hoc]
        CH[ClickHouse<br/>customer dashboards]
        REDIS[Redis feature store]
        HT[Reverse ETL → CRM]
    end

    subgraph Gov["Platform services"]
        POL[Polaris / Iceberg REST catalog]
        DH[DataHub<br/>discovery + lineage]
        PROV[Provisioner + contract CI]
        MON[Freshness, volume,<br/>cost dashboards]
    end

    DBZ --> FLINK
    SDK --> FLINK
    FLINK --> RAW
    FIV --> RAW
    RAW --> DBT --> CLN --> DBT
    DBT --> CUR
    AF -.->|schedules| DBT
    CUR --> SNOW
    CUR --> CH
    CUR --> REDIS
    CUR --> HT
    PROV -.-> POL
    PROV -.-> DH
    PROV -.-> AF
    MON -.-> CUR
MetricValue
Platform team size7 engineers (ingestion 2, compute/orchestration 2, governance/catalog 2, DX 1)
Domain data engineers~20 across 8 domains
Analysts~13, writing dbt directly
Data products with specs~240
Time to onboard a new CDC table~15 minutes (spec PR + review)
Freshness SLA attainment99.4% of product-hours

Anti-Patterns

Anti-patternSymptomFix
Platform as ticket queue"Self-service" portal that creates a Jira ticketAutomate the fulfilment, not the request form
Golden cageGolden path is mandatory and cannot handle 20% of real use casesAllow documented escape hatches; fold common exceptions back into the path
Tool zooThree orchestrators, two table formats, four BI toolsStandardize per layer; deprecate with dates
Ownership theaterEvery table has an owner field set to a departed engineerOwners are teams, not people; validate against the org directory in CI
Platform without product managementPlatform builds what is interesting, not what is neededTreat domains as customers: roadmap, surveys, adoption metrics
Governance as gatekeepingEvery access request needs a committeeEncode policy in ABAC; route remaining approvals to owners
Mesh without a platformEach domain builds its own stackPlatform maturity Stage 3 before domain ownership

Decision Framework

flowchart TD
    Q1{How many teams<br/>produce data?} -->|1-3| A1[Centralized team<br/>+ managed stack]
    Q1 -->|4-10| Q2{Do domains have<br/>engineering capacity?}
    Q1 -->|10+| Q3{Is self-service<br/>platform at Stage 3?}
    Q2 -->|No| A2[Hub and spoke<br/>+ templates]
    Q2 -->|Yes| A3[Platform + domains]
    Q3 -->|No| A4[Platform + domains,<br/>invest in platform first]
    Q3 -->|Yes| A5[Data mesh with<br/>federated governance]

Key Takeaways

  • A data platform exists to remove the central team as the bottleneck — measure success by how many requests domains complete without a ticket.
  • Architect in layers (ingestion, storage, compute, serving) with catalog, governance, observability, and cost as cross-cutting concerns.
  • Standardize on open storage (one table format) so compute can change without data migration.
  • Drive everything from a declarative data product spec: tables, grants, schedules, monitors, and catalog entries are all derived from one file.
  • Governance scales through policy-as-code and attribute-based access, with decisions delegated to owning domains.
  • Data mesh is an organizational model that demands a mature self-service platform; adopt it after Stage 3, not instead of it.
  • Buy commodity infrastructure, build the integration layer that encodes your organization's policies.