// HACKER NEWS — CYBERSECURITY
ProgramBench Vetted: Reverse Engineering from a Runnable Binary
Can an agent rebuild a program from a runnable binary? That is the fundamental question behind
ProgramBench. We are releasing ProgramBench Vetted, a set of 50 tasks built around the same premise,
with improvements to the task design process that make the benchmark fairer and more reliable.
In addition to preserving measurable partial progress and deterministic test based grading, our
task design process adds a broader set of controls for failure modes involving test duplication,
environment quality, hackability, and fairness.
Tasks passing 100% of tests, pass@1 across 50 held out tasks · default ProgramBench mini-swe-agent configuration
ProgramBench matters because program reconstruction is one of the relatively few coding benchmark
formats built around very long-horizon work. An agent must preserve and revise a model of the program
across an extended trajectory, which makes these tasks useful for studying long-context coordination.
They are also valuable for RLVR: many deterministic behavioral checks can provide
dense, behavior-driven reward while the agent still has to integrate that feedback
into one coherent implementation. Because these tasks come from public repositories, some may have
appeared in the pretraining data of large models. Yet prior exposure does not imply that a model can
retrieve or reconstruct the implementation from memory when asked to move from observed behavior
back to code. The Reversal Curse offers a useful analogy
for this gap. We discuss the resulting memorization
paradox and its training implications in Appendix D.
ProgramBench, released by Meta in May 2026, gives
an agent a compiled program it can execute but not read, together with its usage
documentation. The agent must write a new codebase that reproduces the observed behavior.
Each of the 200 tasks is a real open source CLI project, from small utilities up to FFmpeg,
SQLite, and the PHP interpreter. The intended task package is a sealed container: the binary is available only for
execution, the documentation is kept, and the source and git history are removed. The
agent gets a shell, no internet, a thousand steps, and six hours. Behavioral tests grade the result.
The tests were themselves written by an agent that probed the original repository and binary.
The original construction process has three stages:
At evaluation time, the source, repository tests, and git history are absent. The agent keeps the
documentation and can execute the reference binary while building its replacement. The generated
behavioral suite then grades the reconstruction by the fraction of tests passed; a task is resolved
only when every test passes.
The retained documentation provides part of the specification, but not the complete behavioral
contract. The rest must be inferred by probing the binary, forming a hypothesis, implementing it,
comparing the result, and revising. The four frontier agents analyzed in the paper spend 23 to 34%
of their actions probing the reference, making empirical investigation a substantial part of the
task rather than a brief setup step.
ProgramBench Vetted is not designed simply to be harder. It is designed so that each point of
reward more faithfully represents the intended behavior. Our target is calibrated difficulty: a
task should leave room for meaningful partial progress, but should not award most of its score for
reproducing a shallow interface or exploiting a defect in the evaluation.