Retrieval-augmented generation — pointing a language model at your own documents so it answers from them rather than from memory — is the most useful pattern in business AI right now. It is also the most over-applied.
The technique is straightforward: find the relevant passages, hand them to the model, and require the answer to cite them. The judgement is in deciding whether your problem is actually shaped like that.
The problems it suits
RAG earns its keep where the same knowledge is looked up repeatedly by different people, the source material is written down, and being slightly wrong is recoverable.
- Internal support and onboarding. Policies, runbooks, product specifications and past tickets. Staff currently interrupt a colleague; now they ask and get a cited answer.
- Sales and bid support. Answering the same product, security and compliance questions from previous responses, with the source visible so it can be checked.
- Field and operational reference. Technical manuals and procedures that are long, structured and rarely read end to end.
The common thread: the answer exists somewhere in your documents, and the expensive part is finding it.
The problems it does not suit
- Anything requiring a precise, auditable calculation. Pricing, tax, payroll, dosing. Use the system of record and deterministic logic. A model can explain the result; it should not produce it.
- Questions about live transactional data. "What is this customer's balance?" is a database query. Wrapping it in a language model adds latency and a new failure mode.
- Undocumented knowledge. If the answer lives only in a senior colleague's head, retrieval has nothing to retrieve. The project you actually need is a documentation project.
- Decisions with legal or safety consequences. Use it to draft and to locate sources, with a named human accountable for the outcome.
What determines whether it works
Retrieval quality, almost always. When these assistants disappoint, the model is rarely the problem — it answered faithfully from the wrong passages.
- Source hygiene. Superseded documents in the index produce confidently outdated answers. Decide what is authoritative before you index anything.
- Sensible chunking. Split on document structure — headings, clauses, sections — rather than a fixed character count that cuts sentences in half.
- Metadata. Document type, date, owner and status let you filter to current material and explain provenance.
- Permissions at query time. The assistant must apply the same access rules as the underlying systems, not a cached copy of them. This is where these projects most often go wrong.
Measure it, or you are guessing
Before launch, write thirty to fifty real questions with known good answers, drawn from the queries people actually ask. Score every change against that set. Track:
- Did it retrieve the right source?
- Is the answer supported by that source?
- Did it correctly refuse when the answer is not in the corpus?
That last one matters most. An assistant that says "I cannot find this — here is who to ask" is far more valuable than one that always produces something.
A realistic path
Start with one domain, one team, and a fortnight of use. Log every question and every thumbs-down, and read them. The failure log tells you whether the fix is better retrieval, better documents, or a different tool entirely — and sometimes it tells you the problem was a search box all along.
Costs are ongoing: embedding and re-indexing as documents change, model inference per query, and someone's time to review the failure log. Budget for the operating cost, not just the build.
The question to answer first
How many hours per week does your organisation currently spend finding information it already owns? If you cannot estimate it, that is the work to do before any AI project starts.
If you would like help working out whether an assistant is worth building — including hearing that it is not — book a 30-minute call and bring an example of the questions you are trying to answer.