Data pipelines for data-heavy B2B teams
High-cardinality ingestion, schema evolution, and multi-tenant query layers for products where the pipeline is the product.
Domain context
When the data pipeline is the product, the engineering bar is different. A consumer analytics product can absorb a few minutes of late data and a few percent of dropped events. A customer-facing analytics product can't, because the customer is staring at the chart waiting for it to update. Cardinality is the killer. The pipeline that worked at 50 customers and 200 distinct event types breaks at 500 customers and 5,000 event types, and the breakage is rarely loud. It shows up as slow queries, cost overruns on the warehouse, and a dashboard that loads in eight seconds instead of one. We design data pipelines for data products assuming cardinality will grow by an order of magnitude, schemas will evolve weekly, and the customer's tolerance for "the data is still loading" is roughly zero.
Why this combination
Generalist data engineering ships pipelines that work at the scale you have today. Data-product pipelines need to ship for the scale you'll have in eighteen months, with the schema-evolution discipline to get there without a rewrite. The patterns that matter aren't what a generic data team reaches for first: columnar storage tuned to your access patterns, a query layer with per-customer result caching that respects permissions, change-data-capture from customer-managed sources, materialised views with incremental refresh, schema registries with backward-compatibility checks wired into CI. We've shipped them before, and we bring the opinions. Iceberg over plain Parquet for evolving schemas. ClickHouse or DuckDB for the hot query path when the warehouse is too slow. OpenLineage for the multi-source joins your support team will eventually ask about.
“We rebuilt a customer-facing analytics pipeline that was timing out at 14% of dashboard loads. The new stack holds p99 query latency under 800ms across the full customer base, with 4x lower warehouse spend per query.”
Frequently asked
- Kafka or a managed equivalent for ingestion, dbt for transformation, a columnar warehouse (Snowflake, BigQuery, or Databricks) for the cold tier, and a fast query engine (ClickHouse, DuckDB, or Pinot) for the hot tier customers actually hit. Iceberg or Delta as the lakehouse format if you have multiple compute engines reading the same data. The principle: separate the path the customer queries from the path the analyst queries, because their access patterns are different.
- Schema-on-write with a registry that flags backward-incompatible changes in CI before they hit production. Tiered partitioning by customer and time, so per-customer queries don't scan the global dataset. A property-bag column for high-cardinality custom dimensions that doesn't blow up the column count. We've shipped systems with millions of distinct event-type-and-property combinations. The discipline is in the schema, not the storage engine.
- Schemas are versioned and migrations are explicit. A column rename is a deprecation followed by a removal, never a silent change, with a window where both names are queryable so customer dashboards don't break overnight. The schema registry is the source of truth and is wired into both the pipeline and the query API, so a removed column produces a clear error instead of a silently-empty result.
- Checks run inline in the pipeline, not as a nightly batch. Volume anomalies (a sudden drop or spike in event rate per customer), schema anomalies (a previously-required field arriving null), and freshness anomalies (data older than the customer-facing SLA) all alert on the channel the on-call engineer monitors. The customer-facing UI surfaces freshness explicitly, so a delayed pipeline is visible to the customer rather than hidden.
- Strongly in favour. Open formats let you swap compute engines without rewriting data, which is a non-trivial benefit when the right query engine for your hot tier is different from the right one for your warehouse. Iceberg in particular handles schema evolution and time-travel queries cleanly, which matters for a customer-facing product where the auditability of historical numbers is part of the contract. We default to Iceberg unless there's a specific reason not to.