// HACKER NEWS — CYBERSECURITY
Measuring the sloppiness of code
Subject:If coding is solved, what now?: Measuring the sloppiness of code
LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC).
This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up.1 Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can't really deal with the slop either.
Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing.
To be frank, with the exception of a few insightful research papers, I was disappointed at how “vibes based” the industry seems at the moment. In my research and on X, I was constantly bombarded with messages such as “End-to-end coding agents”, “AI that doesn't just suggest code—it ships it” or “Human-level evaluation without human-level cost”. Which like all good tales, have a grain of truth in them.
LLMs are able to write almost perfectly correct code. This is because of the scalability and the verifiability of code. It is pretty straightforward to let LLMs generate code and then let that code be checked by hidden tests, which results in a clear reward signal. In stark contrast to that, checking the ‘sloppiness’ of this code often requires human intuition and taste, and is an extremely difficult task in general. I think the best way to illustrate why that is, is by going through possible ways of measuring slop.
AI as a judge: This is probably the most common way of evaluating code quality in the industry and from my observations it rarely works. The most naive way of doing it, namely asking the models how good the code is on a scale from 1-10, is basically equivalent to a random number generator. The more sophisticated approach, namely trying to give the judge model two solutions A and B, and then letting it decide which solution it prefers, has the downside of the model changing its preference, when you rename the solutions. I am being a bit facetious here and the effect isn’t as pronounced with larger models, but the main point still stands. Asking LLMs to judge the code they write is not a substitute for a proper evaluation. Even though there are some interesting approaches with rubrics or the LLMs writing tests, they are still a far shot from actually getting rid of the slop.
Human judges the AI: If we ignore the fact that there is huge diversity in the quality of software-engineers, this would be the best solution to assure that the code stays human readable. With the downside being that this is not scalable for training AI or having large benchmarks with multiple model providers and harnesses.2
The simplest method: In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.
The next two measures were introduced to me by the paper SlopCodeBench, and seemed promising because they were able to separate legacy code bases from LLM-slop quite well.