AVTECH AI
← INSIGHTS / DATA
DATA · 6 min read

Why nobody trusts the warehouse

Three departments, three revenue numbers, and a monthly close that begins with a week of reconciliation. A new platform will not fix any of it.

Ask three departments in a large organization for last quarter's revenue and you will often get three numbers. Everybody knows this happens. Most organizations have stopped finding it remarkable, which is itself the problem.

The instinctive response is to buy something. A new warehouse, a new lakehouse, a new BI tool, a migration program with a name. Occasionally this helps, mostly by accident, because the migration forces someone to finally write down what the numbers mean. Usually it does not, because the problem was never where the data was stored.

The meeting where it surfaces

It rarely announces itself as a data problem. It shows up as a meeting.

An executive asks a simple question. Two teams produce two answers. The next forty minutes are spent not deciding anything, but establishing whose number is right — and often ending without resolution, with an action item for someone to reconcile offline.

The cost is not the discrepancy itself. It is that the organization gradually stops using data to decide and starts using it to argue. People learn which number supports their position and bring that one. Analysis becomes advocacy. And the genuinely useful signal in the data goes unread, because nobody trusts any of it enough to act on something surprising.

It is almost never the tooling

Modern data platforms are extremely capable. In the engagements we see, the platform is rarely the binding constraint. The recurring failure modes are these.

There is no agreed definition of the metric — more on that below. Ingestion silently drops or duplicates records, and nobody notices because nothing checks. Transformation logic has accumulated over years across a hundred jobs that no single person can trace end to end. The semantic layer exists in four different BI tools, implemented four different ways by four different teams. And no individual is accountable for any given number being right.

None of these are fixed by a migration. All of them travel with you to the new platform, usually arriving slightly worse for the journey.

Definitions are a governance problem in technical costume

Revenue sounds unambiguous until you actually ask. Gross or net of returns? Recognized when invoiced or when shipped? Does it include intercompany? Which FX rate, as of which date? What about the subsidiary acquired last year that is still on its own chart of accounts? Are cancellations backed out of the original period or the current one?

Every one of those is a legitimate business decision. None of them is a technical decision. Where organizations go wrong is allowing them to be answered implicitly — by whichever engineer wrote each pipeline, at the moment they wrote it, based on whatever they understood at the time.

The fix is unglamorous. For each reported metric: one written definition, one named owner with authority to change it, and one implementation that every downstream consumer reads from. Not one per department. One.

Writing the definition is not the hard part. The hard part is getting finance, operations, and sales into a room to agree on it, because they have each been quietly using a different one and at least two of them will have to change a number they have been reporting. This is why the work stalls in organizations that treat it as an engineering task and assign it to an engineering team. It is a governance task with an engineering implementation.

Test the data, not just the pipeline

Most teams test their code. Considerably fewer test their data. A pipeline can run green every night for a month and produce nonsense the entire time, because the job succeeded — it just processed a source that started sending nulls in a column three weeks ago.

The tests worth having are not exotic. Row counts against expected ranges. Referential integrity across joins. Distribution checks that catch a field quietly changing shape. Freshness thresholds. Uniqueness on anything claiming to be a key. And reconciliation totals against the system of record, run automatically rather than by a person in week one of the close.

The point of all of it is that a bad upstream change fails loudly and immediately, rather than quietly for six weeks until it surfaces in a board pack. Loud failure at 3am is inconvenient. Quiet failure is how an organization loses confidence in its own reporting.

How trust actually gets rebuilt

Trust is not restored by announcing a new platform. It is restored one number at a time, and it is slower than anyone would like.

Pick the two or three reports that actually matter — the ones in the board pack, the ones operations runs on. Agree the definitions with the people who have been arguing about them, and write those definitions down somewhere findable. Rebuild those specific pipelines with tests and traceable lineage. Then run the new numbers in parallel with the old ones until they reconcile, and publish the reconciliation, including the cases where the old number was wrong.

Then decommission the old report. That last step is the one organizations skip, because it is politically awkward and nobody wants to take away a spreadsheet somebody depends on. Skipping it is precisely why so many enterprises now have four ways to produce the same number and no way to say which is correct.

Do this for three metrics and something shifts. People start asking for the governed number by name. That is the point at which the platform investment starts returning, and not one day before.

MORE INSIGHTS

Working on something like this?

We design, build, and run the systems enterprises depend on. Tell us what you are trying to deliver.

Talk to us →