What 'RAG' actually means
Retrieval-augmented generation: before the model answers, it searches your own documents for the relevant passages and answers from those, instead of answering from what it memorized during training. That's the difference between a general chatbot and an assistant that actually knows your business — and it's why a good one can show you exactly which document a claim came from.
The failure mode it exists to prevent
A model with no grounding will answer confidently whether or not it actually knows — a hallucination reads identically to a correct answer until someone checks. Retrieval doesn't eliminate that risk, but it gives you something to check against: a citation, a source passage, a 'this isn't in the documents I have' instead of a guess dressed up as one.
When it's worth building
The clearest case is a body of documents that's technically searchable and practically unsearchable: policies, precedents, contracts, past projects, scattered across folders that only the people who filed them can navigate. If the same question gets asked repeatedly and the answer lives in a document somewhere, that's a strong candidate.
The weak case is the opposite: a handful of documents that rarely change. A FAQ page or a good folder structure solves that more cheaply than a retrieval pipeline does.
What to ask before you build one
Who can see what — does the assistant respect the access boundaries that already exist, or does it flatten everyone to one level? If more than one client or matter is involved, does retrieval prove it can't cross between them, or does it just filter results after the fact? What happens when the answer isn't in the documents — does it say so, or does it guess? These questions matter more than which model is behind it.
Related