JetBrains has released a new tool designed to enable developers to measure their actual productivity gains from AI tools.
The company’s Developer Productivity AI Arena (DPAI Arena) is an open benchmarking platform for how well AI development tools complete real-world software engineering tasks. According to the company, current benchmarks that LLMs are run against rely on outdated datasets, cover a narrow range of technologies, and focus mainly on issue-to-patch workflows.
“As AI coding tools advance rapidly, the industry still lacks a neutral, standards-based framework to measure their real impact on developer productivity,” the company wrote in a blog post.
DPAI Arena uses a flexible, track-based architecture to enable reproducible comparisons across workflows like patching, bug fixes, PR review, test generation, static analysis, and more.
In addition to supporting multiple workflows, it also supports multiple languages and frameworks and allows for a Bring Your Own Dataset approach where contributors can create and share domain-specific benchmarks leveraging this shared infrastructure for evaluation.
JetBrains plans to contribute DPAI Arena to the Linux Foundation to ensure transparency and inclusivity in its governance. A Technical Steering Committee (TSC) will oversee the development of the platform, dataset governance, and community contributions.
The first benchmark that JetBrains created was the Spring Benchmark, which is intended to introduce the technical standard for all future contributions.
“DPAI Arena brings measurable productivity into the world of AI-assisted software development. AI tool providers can benchmark and refine their tools on real-world tasks, technology vendors keep their ecosystems first-class by contributing domain-specific benchmarks, enterprises gain a trusted way to evaluate tools before adoption, and developers get transparent insights into what truly boosts productivity,” JetBrains wrote.
