Agent Seer turns MCP specs into agent evals
If agent evals can be derived from the live MCP spec, private and fast-changing tool suites get a plausible cold-start evaluation path instead of stale hand-written tests.
Links: Original source · Shared link
Logged at IST: 2026-08-30 12:51 IST
What it is: Apple's Agent Seer paper, shared by Omar Sanseviero, on synthesizing MCP agent evaluation scenarios from the tool spec itself.
Gist: Agent Seer treats an MCP specification as enough raw material to build a self-contained evaluation harness. It enriches tool descriptions, generates expected tool-call workflows, synthesizes mock outputs, and expands those into multi-turn dialogues grounded in the synthetic data. The pipeline needs no examples, no live tool access, and no domain-specific tuning.
The useful framing is the cold-start problem for agent evals. New private APIs, internal MCP servers, and rapidly changing tool suites usually have no durable benchmark. Hand-written evals also go stale as soon as the tool surface changes. Agent Seer shifts the work from manual curation to structured extraction from the live spec.
The empirical result is directionally promising, but the caveat matters. Across seven MCP specs, the paper reports strong tool-calling and coherence scores with complete coverage on small and medium specs. Quality varies more with parameter schema complexity than with tool-suite size, and the main failure mode is wrong argument values. That is exactly the kind of error coarse tool-name matching can miss.
Newsletter angle: Strong agents/evaluation item: MCP servers may be able to ship with generated eval suites that track the current spec, especially for private or fast-moving enterprise tools.