Speculative tool calls for code-as-action agents
A concrete harness-level optimization for agents that treat code execution as the main tool, and an early sign that agent runtimes may start borrowing JIT/speculative-execution ideas.
Links: Original source · Shared link · Related link 1 · Related link 2 · Related link 3
Logged at IST: 2026-08-25 12:48 IST
What it is: Alex Zhang's post and reference implementation for Speculative Programmatic Tool Calling, or sPTC.
Gist: sPTC is a latency trick for agents where the model's main action is code in a REPL. Instead of waiting for the whole generated program to finish before running tools, the harness watches the code stream, speculatively parses likely tool calls, launches expensive sub-agent or sub-LLM calls early, and then lets the real execution claim those cached futures if the calls actually happen.
The idea matters because programmatic tool calling makes the harness more like a runtime than a thin JSON-tool dispatcher. Once tools are functions in generated code, the runtime can overlap root-model decoding with sub-call execution, parallelize blocking calls the model wrote serially, and use a shadow REPL to safely evaluate enough partial state to know which calls are worth launching.
Zhang is careful about the trade-offs: the gains depend on token-generation latency, tool latency, serving-engine load, and how aggressive the speculation is. His reported RLM speedups are modest in the benchmark setting, around 1-1.2x, but the important pattern is broader: agent harnesses are starting to absorb compiler/runtime ideas like speculation, futures, purity contracts, and scheduling.
Newsletter angle: Good developer-tools item: code-as-action agents may need runtimes that optimize generated programs underneath the model, not just better prompts or bigger context windows.