RAG infrastructure across 12,000 SharePoint sources

§ 01

The setting: many teams, one platform

Inside a global energy company, departments had been building their own RAG pipelines — each with its own ingestion scripts, its own index, its own definition of "working". The company was consolidating them into a single shared platform, with an MVP connected to roughly 12,000 SharePoint sources.

At that scale, the interesting engineering problem is no longer the retriever. Retrieval over a curated demo corpus is a solved problem. The problem is everything underneath it:

• thousands of sources owned by people who don't know a RAG platform depends on them;
• documents that are edited, moved, renamed, and deleted daily;
• content in every format SharePoint allows — text, tables, scans, embedded media;
• access rules that must survive the trip from source to retrieved context.

My ownership on the platform was exactly this layer: pipeline health metrics — ingestion, freshness, context delivery — the evaluation framework, and agentic retrieval tooling.

§ 02

Why enterprise RAG fails quietly

A RAG platform almost never fails loudly. The API stays up, the answers keep coming, the demo still works.

It fails quietly: the sync drifts, an index goes stale, and the system starts confidently answering from last month's documents.

The user can't see this. The answer looks the same whether the underlying document was ingested an hour ago or has silently dropped out of the pipeline. By the time someone notices a wrong answer and traces it back, the platform has been degrading for weeks — and trust, once lost this way, is very hard to rebuild.

That's why the central design position of my work there was:

Freshness, ingestion health, and context delivery are product guarantees — not infrastructure trivia. If you can't measure them, you don't have a platform; you have a demo with a large index.

§ 03

Pipeline health as a first-class metric

I built the health-monitoring pipeline for the platform MVP — tracking every stage a document passes through on its way to becoming retrievable context:

• Ingestion — is every connected source actually being read? Which sources are erroring, and for how long?
• Indexing — did ingested content make it into the index, in a form retrieval can use?
• Data freshness — how far behind the source of truth is each part of the index right now?
• Context delivery — when a question is asked, does the retrieved context actually contain the material the answer should be grounded in?

Each of these is a number, per source group, over time — not a feeling. That turns "the RAG seems off lately" into a dashboard question with an owner, and it turns incidents into detections instead of user complaints.

§ 04

Evaluation and regression testing

The second layer was the evaluation framework for the unified platform — the part that answers "did this change make retrieval better or worse?" with measurements instead of opinions. It scored the pipeline on three levels:

• Retrieval quality — does the right material come back for a query?
• Answer accuracy — is the generated answer actually supported by the retrieved context?
• End-to-end task success — does the user get what they came for?

On top of that sat regression test suites on fixed benchmarks. Any change to the pipeline — a new chunking strategy, a parser update, an embedding swap — ran against the same fixed sets before release. A change that improved one department's corpus at the cost of another's was caught in CI, not in production.

The platform ran on the client's Azure estate — Azure ML Studio, Azure OpenAI, Azure AI Search — with Ragas as part of the evaluation stack.

§ 05

What the monitoring caught

The monitoring earned its keep quickly. Among the failure modes it surfaced:

• SharePoint sync drift — sources that had silently stopped syncing, so the platform kept answering from an aging snapshot;
• stale indexes — content ingested but never re-indexed after upstream changes;
• multimodal ingestion failures — documents whose text made it in while their tables, scans, or embedded content quietly didn't.

Every one of these is invisible at the answer level until a user happens to hit exactly the affected material — and every one of them was found by the pipeline's own metrics instead. Fixing them strengthened data freshness and context completeness for every team on the platform at once.

§ 06

The main takeaway

An enterprise RAG platform is data infrastructure, not a vector database with a prompt on top.

What made this one viable as a shared platform wasn't a smarter retriever — it was the layer that made quality observable: health metrics for ingestion, freshness, and context delivery; an evaluation framework that scored retrieval, answers, and task success; and regression suites that gated every pipeline change.

If I had to compress the whole engagement into one rule, it's this: in enterprise RAG, the model is the smallest risk. The data path is the product.