Member-only story
What Does the Agents on Rails AI Coding Agent Benchmark Measure?
How do you run an AI coding agent benchmark on your Rails app? Hidden tests, a restored test/ directory, and Brakeman.
I wrote a patch that deleted the two tests it broke, then ran the agent’s own suite. It was green. I ran the same patch through a scorer that restores test/ from the base commit first, and it scored zero.
That patch was hand-written, a stand-in for an agent. But an AI coding agent benchmark that trusts the suite the agent can edit can grade it as a pass. The public Rails benchmark’s methodology describes a restore for its atomic Stage 1 tasks, and your own evaluation needs a similar guard.
It has no separate security score and no review-time measure. This article builds the scorer and shows what it catches.
The Rails Foundation commissioned Evil Martians to build it. Its first AI coding agent benchmark report says it ran 8 models against 21 atomic Rails tasks, 3 runs each, 504 runs in total for $491. (The methodology at commit 2fa67edfb8dd, dated 2026–10–15, counts 19 Stage 1 tasks. It is newer than the 2026–08–13 report, so the page may simply have changed.)
Every task runs against Writebook, a real Rails application. Each is written as a…
