What this video shows
Katz describes an agent harness as the software around a model that supplies tools, context, execution, and controls. He argues that some coding-specific prompts and edit machinery lose value as models learn those behaviors during post-training. He cites recent harness comparisons, then draws a boundary around the parts a model cannot provide by itself: a durable runtime, permissions, secret handling, context delivery, and control over token and cache costs.
The practical lesson is to test each layer rather than remove it because a stronger model arrived. One Apple and EPFL study found that a minimal coding-agent session matched elaborate systems on the ML-engineering tasks it tested, but the paper also limits that result to its models, benchmarks, and time budgets. Learnetto recommends an ablation: remove one layer, rerun the same cases, and retain the simpler system only when task success, cost, latency, and safety checks still pass.
Read the Apple and EPFL harness study for the tested models, benchmark scope, and authors' claims. Inspect the Grok Build repository for a current example of tools, execution, and long-running task support in a coding-agent runtime.
What you will learn
- Separate prompts that compensate for model weaknesses from runtime duties that must exist outside the model.
- A benchmark result applies to its tasks, models, budgets, and grader, so it cannot prove that every harness layer is useless.
- A smaller harness can match task quality while costing more or exposing more risk, which means success rate alone is incomplete.
- Remove one component at a time and preserve the traces needed to explain any change in quality, cost, latency, or unsafe behavior.
How to apply this safely
- List every harness component and name the property it provides, such as context selection, retries, permissions, verification, or caching.
- Choose representative fixed tasks with deterministic checks and record a baseline for success, tokens, elapsed time, interventions, and denied actions.
- Remove or simplify one component, keep the model and task budget fixed, and rerun each case several times because agent outputs vary.
- Keep the change only when the evidence meets the acceptance thresholds and the remaining system still protects credentials, data, and release boundaries.
Important limitations
- Katz worked on Cursor's harness and now works at SpaceXAI, so the product examples and predictions reflect a builder's viewpoint rather than an independent market study.
- The Apple and EPFL paper studies autonomous ML engineering with specific open-weight models. It does not establish the same result for every coding task, model, or production environment.
- The prediction that most agent work will move off laptops within six months is a forecast, not measured evidence.
- Removing prompts or checks can improve speed while weakening security or review. Measure those properties separately from task completion.
Sources to check
- How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering? The Apple and EPFL study cited in the talk, published in October 2026.
- Grok Build SpaceXAI's public terminal coding-agent runtime and tool interface.
- Smarter Models Killed the Harness AI Engineer video published October 19, 2026.
Continue learning on Learnetto
Backtest AI agents
Replay time-ordered cases without letting future evidence enter the run.
AI evals guide
Build fixed cases, graders, traces, and regression gates.
Sandbox AI coding agents
Keep filesystem, network, credential, and approval boundaries explicit.