The most dangerous data pipeline problems are not the ones that throw an error. They are the ones that complete successfully and return a wrong answer. A join that silently drops 3% of rows. A schedule that stopped refreshing two weeks ago. A transform that handles nulls differently depending on the source system. These failures are hard to find because nothing breaks visibly.
Why silent failures are more common than loud ones ¶
Most data pipelines are built with error handling for the obvious cases: a source system is down, a file is missing, a schema change breaks a query. The silent failures happen in the gaps: a left join that should be an inner join, a date filter that excludes records from the current day, a deduplication step that uses the wrong key. These pass all the checks and produce a number that is close enough to right that nobody questions it for months.
Start with row counts at every stage ¶
The simplest diagnostic is also the most effective: count the rows at every stage of the pipeline and compare them. If 10,000 records enter a transform and 9,700 come out, find the 300. They are either legitimately excluded, in which case that exclusion should be documented, or they are being dropped by a logic error. Most pipelines we review have at least one stage where the row count has never been checked.
Check the schedules, not just the logic ¶
A pipeline that runs on a schedule can fail silently if the schedule stops. The job completes, the table exists, the dashboard loads. But the data is from two weeks ago. We have seen this cause a full quarter of reporting errors at a healthcare administration client. The fix was a simple freshness check on the final table, alerting if the most recent record is more than 24 hours old. It took an afternoon to build.
Document every join and its intended cardinality ¶
Every join in a pipeline should have a documented expectation: one-to-one, one-to-many, or many-to-many. If the actual cardinality differs from the expectation, that is a finding. Many pipelines have joins that were written as one-to-one but are actually one-to-many in production, which means every downstream aggregate is inflated. Writing down the expected cardinality forces the question to be asked.
Build a pipeline map before you start fixing anything ¶
Before changing anything, draw the full pipeline: every source, every transform, every destination, every schedule. Most organisations do not have this document. Building it takes time but it makes every subsequent decision faster and safer. It also makes it possible to assess the blast radius of a change before making it.
Silent pipeline failures are fixable. The first step is accepting that they probably exist. Most pipelines we review have at least three. Finding them is methodical work, not heroics.