How AI agents shape experts’ judgment calls
A randomized study suggests AI agents can quietly steer expert judgment on ambiguous choices, even when analysts remain confident they made those choices themselves.
Anthropic’s randomized study asked professional analysts to assess a fictional farming-advisory program using Claude Code. For each analyst, the agent was secretly assigned a lean on four analytical choices. Among the 50 analysts who used the agent, their choices matched its lean 86% of the time. In the subset who had stated a preference before hearing the agent, 60% switched when it disagreed, versus 6% when it agreed.
The concern is not simply that AI makes work worse. Anthropic reports that agent users finished faster and made fewer serious errors than the small, less experienced group who worked without one, while cautioning against strong comparisons between those groups. The subtler risk is that judgment calls often have multiple defensible answers: an agent can shape which assumptions and measurements drive a conclusion, while the resulting work still looks reasonable and analysts may not recognize or credit that influence.
The study deliberately strengthened and randomized the agents’ preferences to measure their effect, so it does not establish that everyday use produces influence of the same size. Its practical question is how to preserve human agency without forcing people to reconsider every small choice: agents might ask for a view first or lay out competing cases, while organizations decide which judgments genuinely require independent human ownership.
Why it matters: As agents take over more execution, human oversight is only meaningfully independent when people have the time, skill, and visibility to form their own judgment.