At the philosophical level, I would say this paper explores the inextricable symbiosis between form and function. The way software’s behavior is verified depends on the structure of the software. Cynically, one may conclude “oh look, when humans write the code, the quality is better!”. I myself have desired a better abstraction of defining and discussing code with agents. Ultimately, this paper may just be showing or rediscovering the truth encoded in this comic:
In my experience, I spend the most time correcting the output of agents when it comes to the structure of the program. For whatever reason, they seldom build the structure the way that I would. The structures they generate often do not follow trajectory I have in my head for the ideal overall structure of the system but rather adhere to “local” examples in the current codebase which are desirable generally or in this particular case. Or they misuse an existing pattern. Or they employ unnecessary artifices.
This paper implicitly identifies an ability gap in current models which could be improved in the future. Models today can create programs that behave as expected (generally so, to some degree) but do not create programs with structure that skilled people would consider to be “good”. This matches my experience interacting with agents every day.
The paper concludes that structure specifications improve correctness more than behavioral specifications alone. Improvement in correctness is higher when the structural specifications are written in a code-like format compared to prose.
Favorite Quotes
Ideally, this separation of concerns between the what and the how promises to free developers from the laborious task of writing code. However, we show that it can actually exponentially increase the effort required.
Developers rarely begin with a complete specifications; they adapt it to correct issues and steer the agent toward the goal. However, since prompting a model does not perform Bayesian conditioning [8], there is no guarantee that providing additional specifications will eventually converge to the goal. However, adding formally verifiable contracts [9, 10, 11] to the skeletons circumvent the issue (Theorem 3.1). Even when informal contracts are not used, an imperfect verifier can still ensure faster convergence to a valid implementation (Theorem 3.2).
Structural specifications produce less eroded code than behavioral specifications: Across all benchmarks and models, structural specifications reduce code erosion by up to 24.7%, compared to the task-only prompt. Meanwhile, behavioral specifications increase it. Verbosity varies little across conditions.
Originally encountered on X.