Coding-Agent Benchmarks Should Match Their Users’ Task Flows
Left: SWE-Bench Pro, where each task is a single user message, almost always a new feature or a bug fix. Right: the first five developer messages in sessions of software engineers working in JetBrains IDEs, counting only sessions with at least three messages.
JetBrains AI Blog13 min read