It isn’t allowed to see the final evaluation test (used in calculating its pass/fail), but it can run code and see the output of its own code in order to understand what doesn’t work. If it ends up creating tests as part of that based on the original problem statement then presumably that’s allowed.
What he is describing has become the 'standard' way to run that kind of benchmark, so he is almost certainly correct. SWE Bench [1] is the best open source benchmark.
> The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation
What does this mean? How does the model iteratively self-correct without seeing the tests? Can it see the test results?