RAG vs Fine-Tuning: How to Actually Decide (Now That OpenAI Is Shutting Down Its Fine-Tuning API)

On May 7, 2026, OpenAI told developers it is winding down the self-serve fine-tuning API, according to OpenAI's own deprecations documentation. Organizations that had never run a fine-tuning job before that date can no longer start one. By July 2, 2026, that restriction extends to any organization that has gone 60 days without using a fine-tuned model. By January 6, 2027, nobody can create a new fine-tuning job on OpenAI's platform at all, full stop. Inference on models you already fine-tuned keeps working until the underlying base model itself is retired, but the door on creating new ones is closing on a fixed schedule.

That's not a footnote. For years, "should I use RAG or should I fine-tune" was framed as a coin flip between two roughly equal options. OpenAI just removed one of the easiest on-ramps to one side of that coin. If you're building an AI product in 2026, it's worth actually understanding what each approach does mechanically, because the decision matters more now, not less.

What Retrieval Actually Does Under the Hood

Retrieval-Augmented Generation doesn't teach a model anything. It looks things up and hands the model an open book right before it answers. The mechanism has three moving parts: your documents get split into chunks, each chunk gets converted into a vector (a list of numbers) by an embedding model, and those vectors get stored in a vector database. When a user asks a question, the question itself gets embedded into the same vector space, and the database returns the chunks whose vectors sit closest to the question's vector, measured by cosine similarity. Those chunks get stuffed into the prompt, and the model answers using only what's in front of it.

This is why RAG can cite its sources and why swapping in new documents doesn't require touching the model at all: you're changing the library, not the reader. It's also why RAG is the right first move for the "what does this model know about our specific data" problem, current pricing, internal policies, a customer's order history, because none of that needs to survive inside the model's weights.

What Fine-Tuning Actually Does (and Doesn't Do)

Fine-tuning is a different kind of change. Instead of handing the model a document at query time, you show it thousands of example input-output pairs and adjust its internal weights so it produces outputs shaped like those examples going forward. Nearly every practical fine-tuning job in 2026 uses LoRA or QLoRA, which freeze the original model weights and train a small set of additional low-rank matrices alongside them, cutting the GPU memory needed by roughly an order of magnitude compared to updating every weight in the model.

Here's the part that trips people up: fine-tuning is good at teaching a model *how* to respond, tone, output format, a consistent JSON schema, a specific classification behavior, but it's unreliable at teaching a model new *facts*. A model fine-tuned on your product documentation doesn't reliably "know" that documentation the way a retrieval system does; it picks up the style and patterns without dependably recalling specific details on demand. If your problem is "the model doesn't know our return policy," fine-tuning is the wrong tool even before you consider OpenAI's wind-down. If your problem is "the model won't stop writing in the wrong tone" or "I need it to output this exact structure every time," fine-tuning is often the only thing that actually works.

Why Naive RAG Pipelines Fail in Production

RAG's mechanism is simple to describe and easy to get wrong in practice, and almost all of the failure shows up at one step: chunking. Chunk size is the lever most teams never touch after copying a default from a tutorial, and Redis's engineering team has documented why that's a mistake. Chunks that are too large mix multiple ideas into one vector, so the embedding becomes an average that doesn't match anything precisely. Chunks that are too small get embedded cleanly but arrive stripped of the surrounding context the model needs to actually answer with them. One controlled study Redis cites found that shrinking chunk size from 1,024 tokens down to 64 tokens improved fact-based recall by 10 to 15 percentage points on entity-heavy datasets, while the same change would hurt a narrative question that needs a wider passage to make sense.

The fix isn't a universal chunk size, it's tracking retrieval quality directly. A useful diagnostic is hit rate at k: the fraction of test queries where the correct chunk shows up in your top-k results. Above roughly 0.8 you have a workable baseline; below 0.6, the problem is almost certainly your chunking or embedding model, not your prompt. Two other production mistakes compound the same failure: stripping section headings and document titles from chunks before embedding them (so the model loses the "where did this come from" context), and relying on vector similarity alone when a hybrid approach that also scores exact keyword matches (BM25) catches product names and domain terms that embeddings alone tend to miss.

The Free, Open-Source Path for Both Sides

Neither retrieval nor fine-tuning requires a paid API, and OpenAI stepping back from self-serve fine-tuning is arguably pushing more teams toward the open-weight path anyway. For embeddings, BGE-M3 from BAAI is fully open source, supports over 100 languages and inputs up to 8,192 tokens, and scores competitively with OpenAI's paid models on the MTEB benchmark, at the cost of needing your own GPU to run inference instead of paying per token. OpenAI's own pricing page lists text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13 per million tokens, cheap at low volume but a real, recurring line item once you're embedding millions of documents and re-embedding them every time you change your chunking strategy.

For fine-tuning itself, now that OpenAI's self-serve door is closing, Unsloth and Axolotl are the two open-source frameworks doing the most real work here. Both support LoRA and QLoRA fine-tuning of open-weight models like Llama, Mistral, and Qwen, either on your own GPU or a rented one, and Unsloth specifically claims meaningful speed and memory improvements over a vanilla Hugging Face training loop. Neither requires OpenAI's platform at all, which means the wind-down timeline doesn't affect you if you fine-tune an open-weight model directly. We cover the full self-hosted stack, from picking a base model to running inference at reasonable cost, in self-hosting a free ChatGPT alternative with Ollama and Open WebUI.

A Decision Framework You Can Actually Use

Ask these in order, and stop as soon as one gives you a clear answer:

• Does the correct answer depend on information that changes weekly, or that a user brings with them (their own documents, their account data)? Use RAG. Fine-tuning can't keep up with data that changes faster than your retraining cadence, and it can't personalize per user without one fine-tune per user, which nobody does.
• Do you need the model to cite where an answer came from, for compliance, trust, or debugging? Use RAG. A fine-tuned model has no mechanism to point at a source; the "knowledge" is smeared across its weights with no attribution.
• Is the problem entirely about tone, format, or a repeated structural behavior, not facts? Fine-tuning is the right tool, and it's often the only one that reliably fixes it, since prompting alone tends to drift back to default behavior under load.
• Are you trying to cut inference cost and latency by distilling a frontier model's behavior on a narrow task into a smaller, cheaper model? That's fine-tuning's other legitimate use case, and it's unrelated to teaching facts at all.

Most production systems that hold up under real traffic use both: RAG for what the model needs to know that changes, fine-tuning for how it should behave that shouldn't. Treating them as competitors is where the "should I use RAG or fine-tuning" framing usually goes wrong in the first place. We walk through building this stack end to end, including the retrieval, chunking, and evaluation decisions above, in AI Product Development Bootcamp.