Earendil measures code sloppiness beyond correctness

Logged at IST: 2026-09-11 17:42 IST

What it is: Sebastian Baye at Earendil asks how to measure the “sloppiness” of code when LLMs can already generate code that often passes tests.

Gist: The useful distinction is correctness versus maintainability. Agents can optimize against hidden tests, but formally correct code can still introduce unnecessary abstractions, duplicate logic, excessive volume, and complex functions that make the codebase harder for humans to steer.

Baye is skeptical of simple LLM-as-judge evaluations: scalar ratings are noisy, pairwise preferences can flip under superficial renaming, and rubrics still do not replace human taste. Human review remains the gold standard for readability, but does not scale as a benchmark across providers and harnesses.

The practical metrics he highlights are deliberately imperfect but useful. LOC delta is a crude early warning. SlopCodeBench-style verbosity combines AST-grep heuristics and clone detection to estimate duplicated or unnecessary lines. Erosion measures how much codebase “mass” is concentrated in large, complex functions using cyclomatic complexity and source lines of code.

The striking result: in the cited comparison, established repos averaged verbosity around 0.15 and erosion around 0.31, while agent-generated code averaged roughly 0.33 and 0.68. In other words, the agent code was about twice as verbose and eroded as human-maintained code by these measures.

Newsletter angle: If code generation is becoming abundant, the scarce skill is not just asking for working patches. It is building loops, metrics, and review habits that keep a codebase legible, compact, and human-governable after hundreds of agent iterations.