What this video shows
Suo uses an analyst task about circuit-board suppliers to separate model speed from useful completion time. The talk follows source collection, financial calculations, reasoning, and review rather than treating tokens per second as the final outcome. It also shows that sparse attention, cache precision, speculative decoding, server load, and reasoning budgets can each change a different part of the wait.
The practical test is a matched run on your own work. Keep the prompt, tools, data, timeout, and acceptance rubric fixed. Measure whether the report supports its calculations, then record time to first useful output, full completion time, retries, review time, and cost per accepted result. A lower token count can still lose when reviewers repair omitted evidence.
Use OpenAI's evaluation guidance to build task-specific cases before comparing routes. Google's Gemini migration guide also requires offline evaluation and load testing before production replacement.
What you will learn
- Measure a complete accepted task, not one flattering speed metric.
- Hold the application around the model constant before attributing a difference to the model.
- Record quality, latency, retries, tokens, cost, and human review together.
- Treat vendor benchmark numbers as hypotheses for your workload.
How to apply this safely
- Choose 30 representative tasks, including long inputs, tool use, and costly failures.
- Run both models several times with the same prompt, tools, data, budgets, and acceptance checks.
- Capture p50 and p95 end-to-end latency, cost per accepted result, unsupported claims, and reviewer minutes.
- Shadow the candidate on live traffic, then canary a small share with an immediate rollback route.
Important limitations
- MiniMax employs the speaker, and the talk introduces an unreleased MiniMax model, so performance claims are vendor evidence.
- The example is one research workflow and cannot establish results for coding, chat, multimodal, or agent workloads.
- A model released after the talk may change before general availability.
- Server load and provider routing can change latency independently of model quality.
Sources to check
- Evaluation best practices OpenAI's guidance on task-specific eval datasets and continuous evaluation.
- Migrate to the latest Gemini models Google's workflow for offline evals, load tests, online checks, and rollout.
- Is a Faster Model Actually Faster? AI Engineer video published October 21, 2026.
Continue learning on Learnetto
Switch LLM models safely
Run a matched bake-off, shadow test, canary, and rollback.
AI evals guide
Build representative cases and release gates.
Backtest AI agents
Prevent future evidence from entering historical tests.