Prompting, RAG or fine-tuning? A 2026 decision guide

The first architectural argument on most AI projects is some version of do we prompt this, add retrieval, or fine-tune? In 2026 the balance has tipped further toward prompting than most teams expect. The clearest signal: in May 2026 OpenAI closed its self-serve fine-tuning platform to new organisations, with existing customers able to start new jobs only until 6 January 2027.¹² We read that as a sign of where the frontier is: current base models follow instructions and output formats well enough that many jobs we would once have fine-tuned for now need only a careful prompt.

Fine-tuning has not stopped being useful. Microsoft, AWS and Google still offer it, open-weight tuning has never been easier, and there are jobs where a tuned small model beats any prompt. But the bar has risen. This is the decision framework we use with clients, with the economics worked through.

The short version

• Start with context engineering on a current model: clear instructions, the right examples, structured output, and an evaluation set. Most projects stop here.
• If the model lacks knowledge, add retrieval, not training. Fine-tuning is a poor way to teach facts.
• Try a smaller current model with the same prompt before tuning anything. Base model prices have fallen faster than fine-tuning prices.
• Fine-tune for behaviour — format, style, narrow classification, tool-call accuracy — when evals show prompting has plateaued and the volume justifies the upkeep.
• Budget for re-tuning. A tuned model is tied to a base model that will be retired.

The decision ladder

We walk every project up the same five rungs, and only climb when an evaluation shows the current rung has stopped improving.

Context engineering is the new prompt engineering

Prompt engineering used to mean wordsmithing a system prompt. Anthropic’s framing of context engineering is more useful: curating everything that ends up in the model’s context — instructions, tools, retrieved data and message history — and aiming for the smallest set of high-signal tokens that gets the result.³ Long contexts are not free even when they fit; recall degrades as context grows, so more is not automatically better.³

In practice, a disciplined prompt has the same parts every time: a role and goal, explicit rules, a handful of examples drawn from real data (including the awkward cases), an output schema the downstream code validates, and an escape hatch for inputs the model should refuse or flag.

<role>
You classify inbound support emails for Acme Billing.
</role>

<rules>
- Choose exactly one category from the schema.
- If the email mentions a chargeback or legal action, set escalate=true.
- If you are not confident, use category "other" and explain in reason.
</rules>

<examples>
  <example>
    <email>I was charged twice for March...</email>
    <output>{"category":"billing_error","escalate":false,"reason":"duplicate charge"}</output>
  </example>
  <!-- 4-8 more, chosen from real traffic, including edge cases -->
</examples>

<output_schema>
{"category": "billing_error|refund|account_access|cancellation|other",
 "escalate": boolean, "reason": string}
</output_schema>

Pair that with schema-constrained structured output where your provider supports it, and grade every change against a held-out set rather than a few hand-picked examples. Static instructions and examples go first, variable input last — which is also what makes the next section work.

Prompt caching changed the economics

A classic argument for fine-tuning was that a long prompt full of instructions and examples is expensive to send on every request. Prompt caching has largely removed that argument. Providers charge much less for a repeated prompt prefix:

• Anthropic: cache writes cost 1.25× the base input price for a 5-minute cache (2× for 1 hour); cache reads cost 0.1×, and less on some newer models.⁴
• OpenAI: caching is automatic on supported models, with no write premium; the minimum cacheable prompt is 1,024 tokens on GPT-5.6 and later.⁵
• Google: implicit caching is on by default for Gemini 2.5 and newer, with minimums of 2,048 to 4,096 tokens depending on the model.⁶

The break-even arithmetic with Anthropic’s 5-minute multipliers: a prefix reused N times within the cache window costs 1.25 + 0.1 × (N − 1) units instead of N. Two uses cost 1.35 instead of 2; ten uses cost 2.15 instead of 10. With the 1-hour cache (write 2×), you need three uses to come out ahead. For any endpoint with steady traffic, a 5,000-token prefix of instructions and examples now costs a fraction of its list price.

When fine-tuning still wins

With caching and better base models, the cases for fine-tuning are narrower but real:

• Format and style consistency at volume. House style, rigid templates, domain shorthand — behaviour that is easier to demonstrate with hundreds of examples than to describe.
• Cost and latency through a smaller model. Teaching a small model one job a large model does well, so you can serve it cheaply at high volume.
• Narrow classification and extraction. Stable label sets with plenty of labelled history are where tuned small models are most reliable.
• Tool use and retrieved-context handling. Microsoft lists better tool selection and argument accuracy, and better use of retrieved context, among the goals fine-tuning can serve.⁷

What fine-tuning is not good at: teaching facts the model can look up. You get a model that states the wrong fact more confidently, and you can’t update it without retraining. Microsoft’s own documentation is explicit that fine-tuning doesn’t replace retrieval for current information.⁷

The methods: SFT, DPO, RFT, LoRA and distillation

LoRA and QLoRA are how most open-weight tuning is done. LoRA freezes the base weights and trains small low-rank matrices; on GPT-3 175B its authors reported 10,000× fewer trainable parameters and a third of the GPU memory of full fine-tuning.⁸ QLoRA adds a 4-bit quantized base model, enough to fine-tune a 65B model on a single 48 GB GPU.⁹ The practical upside is operational: adapters are small, so you can keep one base model and swap adapters per task.

DPO replaced the reward-model-plus-RL pipeline of classic RLHF with a simple loss on preference pairs, which is why preference tuning is now a standard managed option.¹⁰ RFT goes the other way: instead of examples, you supply a grader. Bedrock’s implementation, for instance, scores multiple sampled responses per prompt with a Lambda function or a model-as-judge and trains with GRPO.¹¹ Microsoft’s documentation warns that a rising reward doesn’t mean better task performance if the model is exploiting the grader — test graders against deliberately wrong answers first.⁷

Distillation is often the most valuable form of fine-tuning, because the training data is cheap: prompts from your own traffic and answers from a model that already does the job. Bedrock automates this end to end, including generating teacher responses from your prompts or invocation logs.¹²

# SFT example (chat format, one JSON object per line)
{"messages": [
  {"role": "system", "content": "Classify the support email. Reply with JSON."},
  {"role": "user", "content": "I was charged twice for March..."},
  {"role": "assistant", "content": "{\"category\":\"billing_error\",\"escalate\":false}"}
]}

# Preference example (TRL's prompt / chosen / rejected format)
{"prompt": "Summarise this incident for the customer: ...",
 "chosen": "We found the cause and fixed it on 3 Oct. No data was affected...",
 "rejected": "Root cause: race in the ledger worker (see JIRA-4411)..."}

For open-weight models, a LoRA run with TRL is a few lines:¹³

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})

trainer = SFTTrainer(
    model="Qwen/Qwen3-8B",   # any open-weight base you are licensed to tune
    train_dataset=data["train"],
    eval_dataset=data["eval"],
    args=SFTConfig(output_dir="out", num_train_epochs=3, learning_rate=2e-4),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                           task_type="CAUSAL_LM"),
)
trainer.train()

Who offers what (October 2026)

Managed fine-tuning menus change often and differ by model and region. This is the state we verified this month; check the provider’s supported-models table before planning around it.

Data requirements, honestly

Nobody can tell you the right number of examples without seeing the task, and anyone who quotes one confidently is guessing. What we can say:

• Quality beats quantity. The LIMA paper tuned a 65B model on just 1,000 carefully curated examples and got strong results, arguing most capability comes from pretraining.¹⁶ A few hundred consistent examples beat thousands of inconsistent ones.
• Start small and measure. AWS suggests beginning RFT with 100–200 examples, checking baseline rewards first: near 0% means you need SFT first, above 95% means RFT may be unnecessary.¹¹
• Labelling is the real cost. If the examples don’t already exist in your logs or systems, producing and reviewing them dwarfs the training bill.
• Keep a held-out test set that never touches training or prompt examples. Without it you can’t tell improvement from memorisation.

Let evals decide

Every rung on the ladder is an experiment, and the evaluation set is the instrument. Build it before the first prompt iteration: a few hundred real cases, sliced by the categories that matter, with a grading method you trust — exact match where possible, a rubric or an LLM judge you have checked against human labels where not.

def evaluate(model_fn, cases):
    rows = []
    for case in cases:                      # held-out: never in prompts or training
        out = model_fn(case["input"])
        rows.append({
            "slice": case["slice"],
            "schema_ok": validate(out.text, SCHEMA),
            "correct": grade(out.text, case["expected"]),
            "latency_ms": out.latency_ms,
            "cost_usd": out.cost_usd,
        })
    return summarise(rows)   # accuracy per slice, p95 latency, cost per 1k requests

Run the same harness on the prompted baseline, a smaller model with the same prompt, and each fine-tuned checkpoint. Compare checkpoints on held-out tasks rather than training loss — Microsoft’s guidance says the same, and warns against assuming the latest checkpoint is best.⁷

Total cost of ownership

The training run is usually the smallest line item. The full list:

The last row is the one teams forget. OpenAI keeps tuned models serving only until their base model is deprecated, and Microsoft notes retraining can become necessary as a base model approaches retirement.¹⁷ Meanwhile the prompted system gets the new base model’s improvements for the price of an eval run. Budget at least one full re-tune per year.

A worked cost comparison

A support-email classifier, 2 million requests a month. The prices are OpenAI’s published per-million-token rates; everything else is an assumption, labelled.¹⁷

Assumptions: the prompted versions send 1,200 static tokens (instructions and examples), 300 variable tokens and receive 150 output tokens; 90% of static-prefix tokens hit the cache. Output counts include any reasoning tokens — measure this on your own traffic. The fine-tuned versions need only a short instruction: 400 input tokens, no caching. Training: 2,000 examples × 600 tokens × 3 epochs = 3.6M training tokens. Data: 2,000 reviewed examples at an assumed $3 each.

How one row is calculated, for gpt-5.4-mini: static prefix 1,200 × (0.9 × $0.075 + 0.1 × $0.75) / 1M = $0.000171; variable 300 × $0.75 / 1M = $0.000225; output 150 × $4.50 / 1M = $0.000675; total $0.001071 × 2M = $2,142. Training gpt-4.1-mini on 3.6M tokens at $5/1M costs $18; the assumed data costs $6,000.

What this shows:

• The fine-tuned gpt-4.1-mini saves $542 a month against the prompted gpt-5.4-mini, so the assumed data cost alone takes about 11 months to pay back — before evaluation work, iteration or a re-tune.
• The cheapest option is a current small model with a good prompt: prompted gpt-5-nano costs less than half of the fine-tuned nano. If it passes your evals, it wins outright. The newest small models widen the gap: gpt-6-luna lists at $0.10 / $0.50 per million tokens as of October 2026.¹⁷
• Fine-tuned models carry a per-token premium over their own base model (gpt-4.1-mini is $0.40 / $1.60 untuned) and are pinned to an older generation.¹⁷
• These OpenAI fine-tuning prices are a reference point only; new organisations can no longer use that platform, and other providers price training, hosting and inference differently.¹

Fine-tuning wins this comparison only if the small prompted models fail the evals and a tuned small model passes — which does happen, on genuinely specialised tasks. That is exactly the case to prove with data before committing.

Failure modes we see repeatedly

• Fine-tuning to fix a knowledge gap that retrieval would have solved.
• No baseline. The tuned model is compared against a lazy prompt rather than a properly engineered one.
• Not trying a smaller current model before tuning an older one.
• Evaluation leakage: test cases that also appear in training data or prompt examples.
• Rewarding the grader, not the task in RFT.
• No owner for the tuned model when its base model is scheduled for retirement.
• Uncached long prompts, with the dynamic content placed before the static instructions.

Checklist

1. Write the eval set first: real cases, held out, sliced, with a trusted grader.
2. Engineer the context: role, rules, real examples, output schema, escape hatch.
3. Put static content first and confirm cache hits in usage data.
4. If answers need private or changing knowledge, add retrieval.
5. Run the same prompt on a smaller current model and compare.
6. Only then consider fine-tuning — and pick the method that matches your signal (examples, preferences or a grader).
7. Check which provider and region supports that method for that model today.
8. Price the whole lifecycle: data, training, serving, re-tuning.
9. Compare checkpoints on held-out data, not training loss.
10. Name an owner and a re-tune plan before the model goes live.

Most teams that follow this order ship something useful in weeks and keep a system they can evolve as base models improve. When fine-tuning genuinely is the answer, ForthClover’s fine-tuning team can help you prove it with evals before you commit.

Sources

1. OpenAI API documentation, Deprecations — fine-tuning platform (notice of 7 May 2026; stages to 6 Jan 2027) — developers.openai.com/api/docs/deprecations
2. OpenAI API documentation, Model optimization (wind-down notice; SFT, vision, DPO and RFT supported models) — developers.openai.com/api/docs/guides/model-optimization
3. Anthropic Engineering, “Effective context engineering for AI agents,” 29 Sep 2025 — www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
4. Claude API documentation, Prompt caching (write/read multipliers, TTLs, minimum lengths) — platform.claude.com/docs/en/build-with-claude/prompt-caching
5. OpenAI API documentation, Prompt caching — developers.openai.com/api/docs/guides/prompt-caching
6. Google AI for Developers, Gemini API context caching (implicit and explicit) — ai.google.dev/gemini-api/docs/caching
7. Microsoft Learn, Fine-tuning in Microsoft Foundry (methods, training types, supported models, limitations) — learn.microsoft.com/en-us/azure/foundry/fine-tuning/overview
8. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” 2021 — arxiv.org/abs/2106.09685
9. Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs,” 2023 — arxiv.org/abs/2305.14314
10. Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” 2023 — arxiv.org/abs/2305.18290
11. AWS, Customize a model with reinforcement fine-tuning in Amazon Bedrock — docs.aws.amazon.com/bedrock/latest/userguide/reinforcement-fine-tuning.html
12. AWS, Customize a model with distillation in Amazon Bedrock — docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html
13. Hugging Face, TRL documentation (SFT, DPO, GRPO, distillation trainers; PEFT integration) — huggingface.co/docs/trl/index
14. AWS, Customize your model in Amazon Bedrock (methods and training charges) — docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html
15. Google Cloud, Tune Gemini models (Gemini Enterprise Agent Platform, formerly Vertex AI) — docs.cloud.google.com/vertex-ai/generative-ai/docs/models/tune-models
16. Zhou et al., “LIMA: Less Is More for Alignment,” 2023 — arxiv.org/abs/2305.11206
17. OpenAI API pricing (standard models and fine-tuning tables) — developers.openai.com/api/docs/pricing

Provider capabilities and prices were checked against each primary source in October 2026. Fine-tuning availability changes by model and region; confirm before planning.