Practical guide · Model migration
How to switch LLM models without breaking your product
A new model can improve one benchmark and still make your product slower, costlier, or less reliable. Compare the complete workflow on real tasks, shadow production traffic, canary a small share, and keep the old route ready until the new one proves itself.
Published October 21, 2026 · Reviewed October 21, 2026 · 11 minute read
The short answer
Build a fixed set of representative tasks and run the current and candidate models through the same prompts, tools, retrieval, limits, and graders. Score accepted results, failure slices, p95 end-to-end latency, cost per accepted result, and reviewer work. Inspect raw failures, shadow live requests without affecting users, then canary the candidate behind an instant rollback. Keep the old model available until production evidence meets a written release gate.
Freeze the comparison before you run it
A model bake-off becomes hard to interpret when the prompt, tools, retrieval index, timeout, and grader change together. Write the decision first: which route will handle which task, and what evidence would justify changing it? OpenAI's evaluation guidance recommends task-specific evals that include typical, edge, and adversarial cases. Use production examples and keep an untouched holdout rather than tuning on every case you score.
| Part | Keep fixed | Measure |
|---|---|---|
| Tasks | Inputs, references, risk tags, and holdout split | Pass rate and failure rate by task |
| Application | Prompt, tools, retrieval, history, and parser | Schema, tool, citation, and safety failures |
| Budget | Timeout, retries, reasoning level, and output cap | Completed work and exhausted runs |
| Quality gate | Deterministic checks, rubric, and reviewer guidance | Accepted results and grader disagreement |
| Operations | Region, service tier, concurrency, and cache state | p50 and p95 end-to-end latency, errors, and rate limits |
| Economics | Pricing date and included infrastructure | Tokens, retries, review time, and cost per accepted result |
A seven-step model migration workflow
- Name the reason to switch. Record the capability, price, retirement date, latency target, compliance need, or reliability problem. A newer version number is not a release criterion.
- Capture the current baseline. Pin the current model ID and settings. Save accepted-task rate, failure slices, latency distribution, token use, retry rate, and reviewer effort before testing the candidate.
- Run a matched offline bake-off. Use the same inputs and application code. Repeat variable tasks, randomize blinded human comparisons, and report distributions rather than one lucky run.
- Inspect changed failures. Read outputs and traces where one model passes and the other fails. Check tool arguments, refusals, citations, structured output, instruction following, and stopping behavior.
- Test the integration contract. Verify model IDs, request parameters, context and output limits, tool-call formats, stop reasons, streaming events, token accounting, rate limits, region support, and fallback behavior.
- Shadow live traffic. Send a safe copy of sampled production requests to the candidate, discard side effects, protect personal data, and compare outcomes without changing the user response.
- Canary and decide. Route a small bounded share to the candidate, monitor the written gates, expand in stages, and roll back automatically when quality or operational thresholds fail.
Anthropic's model selection guide tells teams to test actual prompts and data across accuracy, response quality, edge cases, speed, and cost. Google documents a similar sequence in its Gemini migration guide: prepare evaluations, run offline checks, tune, load test, optionally evaluate online, and then deploy.
Worked example: a support answer assistant
Suppose an assistant drafts answers from product documentation. The current model costs $0.08 per case and 92 of 100 sampled answers pass review. A candidate costs $0.05 per attempt and returns text sooner. Those numbers do not settle the migration because failed attempts need retries or human repair.
Create 120 cases from real traffic: 60 routine questions, 20 ambiguous questions, 15 missing-evidence cases that should abstain, 15 tool or retrieval failures, and 10 policy-sensitive cases. Keep 20 untouched. Require correct policy, claim-level source support, valid links, approved tone, no restricted data, and an explicit abstention when the documents do not answer.
Run each model three times on the first 100 cases. If the candidate passes 94 cases but adds unsupported claims to three high-risk answers, the overall average hides the release blocker. Repair the prompt only on the development split, rerun both routes, and use the untouched 20 once. Then shadow current traffic and canary 5% if the candidate meets every critical slice.
When the faster model loses
Time to first token matters for interactive chat, while a research agent may spend most of its time searching, calling tools, retrying, and waiting for review. Measure the interval from accepted request to accepted result. Include queue time, cache misses, tool calls, retries, fallback calls, parsing, and reviewer corrections.
Use cost per accepted result for the same reason. A cheaper attempt loses when it needs enough retries, escalations, or manual repair. A more capable model can also lose when it overthinks routine cases and consumes a larger reasoning budget. Test effort settings and routing before replacing every request with one model.
- Choose one model when one route meets every critical slice with acceptable cost and latency.
- Route by task when a cheaper model handles routine work and a stronger model earns its cost on difficult cases.
- Keep a fallback for rate limits and outages, but test it with the same prompt and response contract.
- Delay the switch when a candidate wins an average but fails a safety, compliance, or high-value slice.
Rollout and rollback
Put model selection behind configuration rather than scattering IDs through application code. Record the route, model version, settings, prompt version, tool version, and retrieval version on each trace. A rollback should restore the complete known-good route, including parameters and parsers, because changing only the model string may leave an incompatible request contract in place.
Shadowing must not repeat side effects. Stub email, purchases, database writes, and external actions, or replay recorded tool responses. During the canary, define stop thresholds before traffic starts. Examples include any critical policy failure, a two-point accepted-task drop, p95 latency above the user budget, or fallback errors above a fixed rate.
Limits and common mistakes
- A small eval set can miss rare failures. Add production failures after release and preserve a holdout for later decisions.
- Model judges can favor style or a related model family. Validate them against blinded expert decisions and retain deterministic checks.
- Provider previews, aliases, and routing layers can change beneath one name. Pin versions where possible and record the served version.
- Caches make warm tests look faster than first requests. Report cold and warm paths separately.
- Shadow traffic can duplicate sensitive data. Apply the same retention, region, access, and deletion rules as the live route.
- A fallback can violate the new contract or policy. Test it as a production route rather than assuming any answer is better than an error.
Migration checklist
- □ The current and candidate model IDs and settings are pinned.
- □ Representative tasks, costly failures, and an untouched holdout are present.
- □ Prompts, tools, retrieval, limits, and graders stay matched.
- □ Critical slices pass, not only the overall average.
- □ p50 and p95 end-to-end latency, retries, reviewer time, and cost per accepted result are recorded.
- □ Request, streaming, tool, refusal, token, and fallback contracts are tested.
- □ Shadow runs cannot perform real side effects.
- □ Canary stages, stop thresholds, owner, and rollback route are tested.