Agentic Coding
Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for
Claude 3 Opus. Our evaluation tests a model’s ability to understand an open source codebase and implement a
pull request, such as a bug fix or new feature, given a natural language description of the desired improvement.
For each problem, the model is evaluated based on whether all the tests of the codebase pass for the completed
code submission. The tests are not visible to the model, and include tests of the bug fix or new feature. To
ensure the evaluation mimics real world software engineering, we based the problems on real pull requests
submitted to open source codebases. The changes involve searching, viewing, and editing multiple files
(typically three or four, as many as twenty). The model is allowed to write and run code in an agentic loop
and iteratively self-correct during evaluation. We run these tests in a secure sandboxed environment without
access to the internet.
% of problems which pass all tests
Claude 3.5 Sonnet 64%
Claude 3 Opus 38%
Claude 3 Sonnet 21%
Claude 3 Haiku 17%
It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to understand what doesn’t work. If it ends up creating tests as part of that based on the original problem statement then presumably that’s allowed.
What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark.
Agentic Coding Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude 3 Opus. Our evaluation tests a model’s ability to understand an open source codebase and implement a pull request, such as a bug fix or new feature, given a natural language description of the desired improvement. For each problem, the model is evaluated based on whether all the tests of the codebase pass for the completed code submission. The tests are not visible to the model, and include tests of the bug fix or new feature. To ensure the evaluation mimics real world software engineering, we based the problems on real pull requests submitted to open source codebases. The changes involve searching, viewing, and editing multiple files (typically three or four, as many as twenty). The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation. We run these tests in a secure sandboxed environment without access to the internet.
% of problems which pass all tests Claude 3.5 Sonnet 64% Claude 3 Opus 38% Claude 3 Sonnet 21% Claude 3 Haiku 17%