// HACKER NEWS — CYBERSECURITY
There's No Limit to How Bad Code Can Get
TL;DR: Metaphors like "a sinking ship" are often used to describe codebases, but are misleading.
A business will sink long before code quality reaches a hypothetical floor. Technical debt has
no bankruptcy, no clean reset, so metaphors that imply an end provide a false sense of security.
Software is in the domain of the abstract. It is not like a building, or a bridge, that is in
the physical realm where you can see and feel the nature of the thing. If you continue to add
floors and rooms to a building forever, it will collapse. Software faces no such constraint.
The code can always get worse. There can always be a new layer of indirection
or a reduction in performance.
Over a decade ago I had my first encounter with an ugly legacy codebase. I had just joined Amazon
as a "Software Development Engineer", fresh out of university, and worked on a team that owned code
related to processing orders. On the surface, what our code had to
do seemed simple. Processing an order involved writing some things to a database and calling into
services owned by other teams, either to ask validity questions or to update bookkeeping on their
end. My colleagues and I estimated that a sufficient implementation of this system shouldn't need
more than two dozen strong engineers to maintain and evolve. Yet, our organization was
hundreds of people, and the system had grown so large and complex that it had become impossible to
learn how it all worked.
It was rare to stay longer than a few years in this org, and institutional knowledge had eroded. This
resulted in code that was full of "haunted graveyards". Fear suppressed any (under-rewarded) efforts to
simplify existing systems. The business rules for what had to be done for each type of order were
decided by people long ago who weren't around anymore. These rules could sometimes be found in a
hopelessly out of date file proudly calling itself a 'living document', but often the rules simply
were not written anywhere we could find. Tracing behavior yourself wasn't easy either because much
of the system lived across team boundaries where the code was not easy to access.
When some obscure process wasn't happening with an order that should have been happening, our pagers
would angrily notify us that someone in our tangled web of service dependencies was unhappy. As a
result of this feedback mechanism, the system stayed afloat, but remained difficult to change
and had abysmal performance. Despite this, new layers were constantly added to support the latest
Amazon products and features. This felt unsustainable, and this feeling is what makes me and others
reach for a "sinking ship" as a metaphor to describe an organization that doesn't pay down their
technical debt.
To their credit, there were always ongoing attempts to fix the architecture, and these usually
went as follows: A new manager or senior engineer joins the organization and observes that "things are bad".
Leadership at the organization agrees and wants to make things better, but there are no engineers
available, so each fix attempt includes adding new engineers and teams.
The re-architectures were always a failure: The system required years of study to properly
understand and was constantly changing. It's not politically viable to take so long to design a
fix for the system, so naturally everyone attempting to fix things must work with
incomplete information. Some of this impatience is from within: If you are trying to
design a grand fix for such a prominently painful architecture, you are doing it in part because
you want a promotion (and you don't want to wait too long for a promotion).
Each cycle would end with the remains of the new attempt permanently grafted onto our architecture
and the leading engineer having departed with their requisite promotion. The increased headcount
stays because the migration plans are too painful and unpopular to actually finish. The cycle
continued as it had long before I had arrived. The sinking ship seem