A founder shows you the ticket: the internal assistant told a customer the wrong return window. Support caught it. The customer didn't. Somebody in the room says the words that cost the most money in AI projects right now — "we need to fine-tune it on our data."
Nine times out of ten, that's the wrong repair. The model didn't lack a personality. It lacked the current policy document. That's a retrieval problem, and fine-tuning is a strange, expensive, slow-moving way to fix it.
The three options in the title are not three grades of the same thing — cheap, medium, expensive. They fix three different failure modes. Pick by symptom, not by budget.
Diagnose the failure before you pick the tool
Use a hiring test. Imagine a competent new hire on day one, and ask which intervention would have prevented the error:
- They'd get it right if you handed them the document. → Retrieval problem. Build RAG or connect a search tool.
- They'd get it right if you handed them the style guide, the rules, and five worked examples. → Instruction problem. Fix the prompt and the context.
- They'd only get it right after 500 reps, and you need those 500 reps to run fast and cheap forever. → Now you have a fine-tuning candidate.
Most production failures land in the first two buckets. That's not a shortcut, it's a diagnosis. And it's the one step teams skip when they jump straight to training runs.
What RAG actually is — and what it doesn't fix
Retrieval-Augmented Generation was named in a 2020 NeurIPS paper by Lewis et al., and the core idea has barely changed: retrieve relevant passages at query time, put them in front of the model, generate the answer from them. The knowledge lives outside the weights. Update the document, and the next answer changes. No retraining.
That's the property operators should care about. Your pricing changes on Tuesday. Your SLA changes when legal says so. A retrieval system tracks that; a set of frozen weights does not.
What RAG doesn't do is make the failure mode disappear. It relocates it. Now you can fail at chunking, at embedding, at ranking, at consolidating conflicting passages. Researchers cataloguing production systems in "Seven Failure Points When Engineering a Retrieval Augmented Generation System" (CAIN 2024) found exactly this: the interesting breakages sit in the pipeline, not the model. Missing content, the right passage ranked below your cutoff, the right passage retrieved but not used.
Two practical consequences that founders discover late:
- Chunking destroys tables and contracts. A fee schedule split across two chunks retrieves as two half-answers. Structured documents need structure-aware ingestion, not a blind 500-token split. Anthropic's contextual retrieval write-up exists because bare chunks lose the context that made them meaningful.
- Your index inherits your permission problem. The moment you embed the HR folder and the board deck into one vector store, retrieval becomes an access-control surface. Permissions have to be enforced at query time, per user — not assumed from the source system. NIST's Generative AI Profile (AI 600-1) lists data privacy and confabulation among the risks that generative systems create or amplify, and both show up here at once.
"Just use a bigger context window" is the tempting escape hatch. It helps, but it isn't free. Anthropic's engineering team describes long-context behavior as a performance gradient rather than a cliff — models stay capable, but precision on retrieval and long-range reasoning can degrade as you stuff more in. Dumping 200 pages into every request also multiplies your token bill on every single call.
Fine-tuning: what it buys, and the risk nobody prices in
Fine-tuning is real and sometimes correct. Per OpenAI's own model optimization guide, it lets you supply more examples than fit in a context window, run shorter prompts (lower token cost and latency at scale), train on proprietary data without repeating it in every request, and push a task down to a smaller, cheaper model.
Read that list again. Every item is about form, consistency, and unit economics. Not one is about knowing today's price list. Fine-tuning teaches a model how to behave, not what is currently true.
Then there's the risk that rarely makes it into the pitch deck. That same OpenAI page currently states the company is winding down its fine-tuning platform, that it is no longer accessible to new users, and that fine-tuned models remain available for inference only until their base models are deprecated. Verify the current status and timelines on the deprecations page before you plan around it — vendor roadmaps move.
The lesson generalizes past any one provider. A fine-tune is an asset with someone else's expiration date on it. Your training data is portable. The trained artifact usually isn't. If your differentiation lives in weights hosted by a vendor, your moat depends on that vendor's product decisions. If it lives in a clean, versioned dataset plus a retrieval layer you control, you can move.
Fine-tuning also has a hidden recurring cost: every base model upgrade means re-running the job, re-evaluating, and re-deploying. Prompts and retrieval indexes usually survive a model swap with a round of testing. Fine-tunes need a rebuild.
Start with evals, or you're guessing
Here's the unglamorous part that decides the project. OpenAI's guide suggests writing evals before prompts, in the spirit of behavior-driven development, then looping: measure, prompt, retrieve, and only fine-tune if the numbers say so.
In operator terms, build a small graded test set first. Fifty to two hundred real questions from real tickets, with the answer you'd accept from a good employee. Run every change against it. Without that, "the new prompt feels better" is the entire quality process, and you'll ship a regression the week your model version updates.
An eval set is also the cheapest instrument for the build-versus-buy call. It tells you whether the gap is 3 points or 30 — and a 3-point gap is almost never worth a training pipeline. If you're standing this up inside a product rather than a chat window, treat evals as part of the software you ship, not a notebook someone runs by hand.
The order of operations
- Write the eval set. Real inputs, accepted outputs, a scoring rule.
- Fix the prompt and the context. Clear instructions, explicit output format, a few strong examples, and the relevant source text supplied at request time. Cheapest change, fastest to reverse, often enough.
- Add retrieval when the failures are knowledge failures. Wrong facts, stale facts, "I don't have access to that." Require citations back to the source document so a human can audit an answer in seconds.
- Consider fine-tuning only for stubborn form and cost problems. Rigid output schemas, a house voice you cannot prompt into existence, or a high-volume task you want running on a smaller model. Confirm the gap on your evals first.
- Instrument production. Log queries, retrieved chunks, and outputs. Most of month three's problems are index freshness and permission drift, and you can only see them if you logged them.
This sequencing is also a scoping tool. Steps one through three are the bulk of what makes an internal assistant trustworthy, and they're mostly plumbing — ingestion, sync, access rules, monitoring — which is closer to business automation work than to machine learning research. If you want an outside read on which step your failure actually sits in, that's the conversation to have in AI consulting, before anyone provisions a training job.
One more constraint: what you say about it publicly
Whatever architecture you pick, your marketing claims about it have to be substantiated. In September 2024 the FTC announced Operation AI Comply, five enforcement actions against companies that used AI hype or sold AI tools deployed deceptively. The agency's framing was blunt: there is no AI exemption from existing law.
Translated for a founder: "our AI is trained on your data" is a claim. "Accurate answers, always" is a claim. If your system is a retrieval layer over a general model, describe it that way. Precision in your architecture and precision in your copy are the same discipline.
The short version
Prompting changes behavior. Retrieval changes knowledge. Fine-tuning changes form and cost. Diagnose which one broke, prove it with an eval set, and spend in that order. Teams that skip the diagnosis pay for the most expensive option first and still ship the wrong return window.
If you're working through this decision now, the reasoning is worth pressure-testing against operators who've already shipped one — that's the kind of thing worth bringing to the community or to an upcoming event. And if your team is still calibrating the vocabulary, start in learn before you start in the console.
Frequently Asked Questions
1. Can RAG and fine-tuning be used together?
Yes, and in mature systems they usually are, because they solve different problems. A common pattern: retrieval supplies the current facts, and a fine-tuned model enforces a rigid output format or a specific voice at high volume. Build the retrieval layer first — if it closes the gap, you've saved yourself a training pipeline and its maintenance.
2. What kinds of projects benefit most from RAG?
Anything where the correct answer changes without warning, and where an answer needs to be traceable to a source. Policy and documentation assistants, support deflection over a live knowledge base, contract and proposal lookup, internal search across systems that don't talk to each other. If a human would answer the question by opening a document, that's a retrieval use case.
3. When is fine-tuning genuinely the right call?
When the failure is consistency or unit economics, not knowledge, and you can prove it on an eval set. Typical cases: a strict output schema the model keeps drifting from, a domain voice that resists prompting, or a high-volume classification task you want to run on a smaller, cheaper model. You also need a curated dataset and the willingness to rebuild it when the base model changes.
4. Isn't a huge context window simpler than building retrieval?
Sometimes — for a small, stable corpus it can be the right answer, and it's far less to maintain. The trade-offs are cost and precision: you pay for those tokens on every request, and precision on retrieval and long-range reasoning can degrade as the context grows. Long context and retrieval also aren't mutually exclusive; retrieval decides what deserves the space.
5. What's the most commonly skipped step?
Evaluation. Teams debate architecture for weeks without a single graded test case, then can't tell whether a change helped. A modest eval set built from real tickets settles most architecture arguments faster than the arguments do.
Visuals
