YC Paper Club argues the harness matters more than the model

Logged at IST: 2026-09-08 19:04 IST

What it is: A Y Combinator Paper Club video titled “Why The Harness Matters More Than The Model.”

Gist: The video’s core thesis is that for agents, the model is increasingly only one component. The surrounding harness often determines whether the system is mediocre or genuinely useful. The useful mental model is:

Agent = model + context + tools + execution loop + memory + environment + evaluation + adaptation

YC’s striking example is ARC-AGI: the same underlying model weights can perform dramatically differently depending on the harness. Rohan’s pasted summary notes cited jumps around 30% to 95%, and another system reaching 100%, largely through changes outside the model itself. The point is not that models do not matter; it is that raw model capability can be badly underused without the right system around it.

Harness evolution: The talk describes a progression from a basic prompt → LLM → answer loop, to context injection, to tool-calling loops, and then to memory, skills, code execution, sub-agents, recursion, and sandboxes. The loop, rather than the single inference call, becomes the product: assemble context, select tools and skills, let the model decide and execute, observe results, update state, and repeat.

Self-improving harnesses: The most interesting idea is moving from static harnesses to harnesses that themselves learn. Instead of a human permanently deciding the system prompt, memory strategy, tool set, topology, and retry policy, an outer meta-harness can generate variants, evaluate them, discard worse versions, and keep better ones. That shifts some learning from training-time model weights into runtime software.

Prime Agent and long-running work: Prime Agent frames agents less as fixed applications and more as systems that adapt their own execution strategy. This matters most for coding, research, and long multi-step tasks, where a strong model can still get lost after 100 steps if context management, state, recovery, and evaluation mechanisms are weak.

OpenJarvis and local personal AI: OpenJarvis asks what happens if a personal AI runs largely on personal devices. The broader idea is a distributed intelligence stack: local models and memory on phones, laptops, or workstations; a harness that routes work; and frontier cloud models used only when required. The talk claims large cost and latency improvements, including an example around 800× lower inference cost.

YC’s QM agent: QM’s practical design principle is that an agent should not be permanently tied to one sandbox or computer. The durable thing is centrally stored conversation and state, while sandboxes, browsers, and models are disposable resources the agent can acquire as needed. That lets the agent move between environments without losing itself, while keeping the harness relatively thin and letting the model make more runtime choices.

Newsletter angle: The most useful framing is the shift from “which model is smartest?” to “what computational environment makes this model smart?” Model labs push capability upward; agent and harness engineers push capability sideways by building the runtime that lets models remember, explore, use software, recover from failure, delegate, and improve. For Bosun/Oddship-style work, the interesting artifact may be less the agent itself than the runtime in which agents retain state, acquire capabilities, and improve.