Claude Automates Software Safety Checks

An artificial intelligence system has successfully taken over the most tedious and expensive part of software engineering: proving that code actually works as intended.

For decades, ensuring that software runs without errors under every possible condition has been a logistical nightmare. The process, known as formal verification, is so labor-intensive that a team of human experts required twenty person-years just to verify the small operating system kernel known as seL4. Consequently, this rigorous method was largely restricted to high-stakes environments like aviation systems, where the cost of failure is catastrophic.

That dynamic is shifting. Anthropic, working alongside partner Theorem, has demonstrated that its Claude model can perform this work autonomously. The team provided Claude with test programs from 123 open-source libraries within Google’s OSS-Fuzz security project. Rather than writing new applications, the model focused on validating existing software. It translated crash tests into mathematical problems, analyzed the compiled machine code, and generated approximately 180 million lines of proof code in a matter of weeks.

The workflow is designed to minimize human intervention. An independent verification program called Rocq automatically checks the proofs produced by Claude. According to Anthropic, the process is "fully autonomous," although the underlying formal foundations and tools were originally built by human engineers at Theorem. This approach allows the AI to catch subtle bugs that traditional testing methods often miss, such as errors triggered only by unusually large input files or specific sequences of data access.

The results of this automated scrutiny have been striking. During the verification of a widely used PDF reader, the team identified as many security bugs in one week as are typically reported for that specific software in an entire year. In another instance, Claude detected a logical error that had remained undetected for over a decade. Anthropic notes that even when a complete proof fails, the attempt itself often surfaces real-world bugs, making the process valuable regardless of the final outcome.

However, the technology comes with strict practical limitations. "Formally verified" does not equate to "error-free" in all contexts. The proofs only hold under specific conditions, including a particular processor architecture, single-threaded execution, and input files smaller than 64 MB. Additionally, the compute cost is significant, with individual proofs consuming hundreds of millions of tokens. Despite these constraints, Anthropic argues that if AI can verify its own code, developers can trust AI-generated software more, reducing the time and cost spent on manual reviews.

Efficiency is also improving. A smaller test run indicated that proofs are reusable and that newer models are more cost-effective. Specifically, the Opus 5.5 model solved half the required proofs using about 40 percent fewer tokens than its predecessor, Opus 5. This trend suggests that the economic barrier to entry for formal verification is rapidly decreasing.

Looking ahead, the project strengthens the argument for broader software automation. The central question is no longer whether AI can program, but whether it can prove that its code is safe. Anthropic expects formal verification with AI to become as routine as standard testing in the near future. Theorem’s roadmap already targets critical infrastructure components such as glibc, OpenSSL, and cURL, signaling a push to apply this automated safety check to the backbone of the internet.