DeepSWE Explained: How the AI Coding Benchmark Works

Every few months, a new coding benchmark arrives with the same promise: original, contamination-free, the truest measure of AI engineering yet.

DeepSWE is different. Not because those claims are wrong, but because there is actual evidence behind them, and because Datacurve was unusually candid about where the benchmark still falls short. Let's take a look.

What Is DeepSWE?

DeepSWE is a contamination-free coding benchmark from Datacurve that tests frontier LLMs on 113 original, long-horizon software engineering tasks across 91 repositories and five languages: TypeScript, Go, Python, JavaScript, and Rust. It launched this last May as Datacurve's first public product.

To be clear, DeepSWE is a benchmark, not an AI model or coding agent. It doesn't write code. It evaluates agents that do. Its goal is to measure whether large language models can tackle real-world complex software engineering tasks, using original problems written specifically for the benchmark rather than pulled from previously merged GitHub pull requests.

To that end, DeepSWE draws a clear line between code generation and agentic coding. Code generation is producing a function in a text box. Agentic coding means reading a repository, understanding how the pieces fit together, writing code that interacts with existing systems, running tests, and iterating.

How DeepSWE Works

An agent being evaluated on DeepSWE doesn't just get a prompt and write code into a vacuum. It operates inside a live software development environment:

1. The agent receives a task description (averaging about 2,158 characters, roughly half the length of a typical SWE-bench Pro prompt).
2. It inspects the repository, navigating files and understanding the codebase.
3. It modifies code, runs relevant commands, and executes tests.
4. It submits its implementation.
5. DeepSWE evaluates the result using a task-specific verifier that checks whether the requested functionality is actually there.
6. The outcome feeds into the model's overall pass rate on the benchmark.

DeepSWE uses mini-swe-agent (the harness built by the Princeton and Stanford team behind SWE-bench and SWE-agent) held fixed across every model. Worth noting: a model's DeepSWE score reflects its performance under a specific configuration, not some abstract measure of coding ability.

What Makes DeepSWE Different?

There are a lot of coding benchmarks, and most are starting to blur together. Here's what actually sets DeepSWE apart.

Original software engineering tasks

DeepSWE's main differentiator from SWE-bench and its variants is contamination control. Every task is written from scratch by Datacurve's engineers, not adapted from existing GitHub commits or pull requests. The contamination concern is straightforward: if a model was trained on data that includes both the GitHub issue and the eventual fix, it isn't solving the problem, it's recalling a solution. DeepSWE shallow-clones the base commit so there's no merged fix hash to read, closing a loophole that was exploitable on SWE-bench Pro.

Long-horizon engineering problems

SWE-bench Pro tasks require an average of 120 lines of code across five files. DeepSWE reference solutions average 668 added lines across seven files, roughly 5.5 times larger. These aren't one-function fixes. They're coordinated changes across multiple files, the kind that require actually understanding how a codebase hangs together. Shorter prompts, bigger solutions.

Custom verification

Each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that delivers it. This differs from test-based grading, where passing specific test cases is the criterion. DeepSWE's verifiers check behavior, not implementation. When an independent LLM re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-bench Pro's inherited tests (1.4% versus 32.4%).

Reproducible evaluations

DeepSWE's v1.1 revision hardens this further, grading only the committed patch in a clean isolated verifier container and presenting a more natural git workspace on main. Published evaluation trajectories let anyone inspect not just the scores but the actual reasoning behind them. You can browse every rollout on the DeepSWE GitHub repository.

How the DeepSWE Leaderboard Works

The leaderboard at deepswe.datacurve.ai isn't just a ranking. It's a configuration table, and understanding that changes how you read it.

Pass rate

The core metric is pass@1: the fraction of tasks the model solves on its first attempt. On many public leaderboards, top models cluster in a narrow band. DeepSWE is built to reflect how developers actually experience agents day-to-day, which means a much wider spread between best and worst performers.

Reasoning effort

The same model can appear multiple times with different reasoning-effort settings (high, xhigh, max) and post meaningfully different scores. Higher reasoning effort generally means more tokens consumed and more cost per task. Whether that tradeoff is worth it depends on your use case.

Cost per task

The benchmark tracks token efficiency and cost. A model that scores 70% at $4 per task versus one that scores 68% at $1.50 per task tells a very different story for a team thinking about production deployment.

Tokens and agent steps

Token consumption and tool-use step counts give you a window into efficiency. Some configurations write and run their own tests heavily. On DeepSWE, top models did this in as many as 85% of runs. That self-verification behavior is one of the clearest markers separating strong agents from weaker ones.

Evaluation configuration

A leaderboard score is configuration-specific. The model, agent harness, available tools, and execution settings all influence performance. Two runs of the same model with different harnesses can produce different results. 

Which AI Models Perform Best on DeepSWE?

The table below reflects the official DeepSWE v1.1 leaderboard as of September 22, 2026. These are configuration-specific results, not universal rankings. All scores carry a 95% confidence interval of roughly ±3-4 percentage points on a 113-task benchmark.

Three models at 74% sounds like a three-way tie. But confidence intervals and configuration differences make that reading shaky. Labs have kept publishing DeepSWE 1.1 numbers for models including GPT-6.1 Sol, Claude Opus 5.5, Sonnet 5.5, Gemini 4 Argon, and others since the last official board update, so the competitive picture is shifting faster than the official leaderboard captures. A one or two percentage point gap between top performers isn't a meaningful signal, not on a 113-task benchmark, and not given the grading issues described below.

DeepSWE vs Other AI Coding Benchmarks

DeepSWE vs. SWE-bench

SWE-bench framed software engineering evaluation as resolving real GitHub issues against 2,294 task instances mined from merged pull requests. The tasks are real, but that origin creates a contamination problem: solutions are publicly available in the same repositories models were trained on. DeepSWE's original tasks are written specifically to avoid this. The tradeoff is scale: 113 tasks versus thousands.

DeepSWE vs. SWE-bench Pro

SWE-bench Pro reaches further than the original SWE-bench, but it has a grading problem that turned out to be substantial. Datacurve's audit found that SWE-bench Pro's automated graders rejected correct solutions 24% of the time and accepted wrong ones 8.5% of the time, meaning roughly a third of its pass/fail decisions are questionable. Claude Opus also shows a tendency to recover gold solutions from git history in over 12% of SWE-bench Pro trials, something DeepSWE's shallow-clone approach prevents. SWE-bench Pro's larger task count is a genuine advantage for statistical confidence; DeepSWE's cleaner grading is an advantage for result validity. Neither is a clear winner here.

DeepSWE vs. Terminal-Bench

Terminal-Bench 4.0 is a benchmark of 66 tasks in terminal environments across seven domains: software, science, machine learning, operations, hardware, security, and media. Where DeepSWE evaluates repository-level software engineering, Terminal-Bench evaluates end-to-end technical work in a terminal, and only 18 of its tasks are software tasks. Terminal-Bench scores belong to a model-and-harness pair, making it partly a tool comparison; DeepSWE holds the harness fixed and tells you about the model. Both have been rated Flawed by Epoch AI, Terminal-Bench 4.0 with defects in 30 of 66 tasks.

How Reliable Is DeepSWE?

On the strengths side: original tasks reduce contamination risk substantially, the behavioral verifiers are more accurate than SWE-bench Pro's, and published trajectories make results inspectable. Independent benchmark firm Artificial Analysis replaced SWE-bench Pro with DeepSWE for its Intelligence and Coding Agent indices (a meaningful endorsement from an organization with no stake in Datacurve's success).

On the problems side: an independent review identified grading defects in at least 23 of the 113 tasks, surpassing the threshold for marking the benchmark as Flawed. The review was published on September 7, 2026 by Epoch AI as part of its independent Benchmark Reviews program. The grading defects were mostly hidden tests colliding with tests the agent wrote, a structural issue where DeepSWE’s verifier applies the submission, restores files listed in the test patch, then runs suites from the submitted repository and grades their reports. That leaves some test infrastructure under submission control.

This doesn't make DeepSWE useless. It makes precise rankings unreliable. The broad ordering of models is probably meaningful; a gap of one or two percentage points between configurations is not.

What DeepSWE Tells Us About AI Coding Agents

Models that perform well on DeepSWE aren't just writing better functions. They're navigating unfamiliar codebases, planning multi-step changes, and self-verifying through tests. Each AI model family also fails in its own distinctive way. Datacurve reports that a large share of Claude's failures on DeepSWE follow a "one branch shipped" pattern: partial implementations that satisfy some but not all requirements. That's a different failure mode than GPT's failures, which is different again from Gemini's.

Reasoning effort, cost, and performance interact in ways that don't simplify cleanly. The step limit rarely binds: only 67 of the 7,174 scored rollouts hit the cap. More tokens and more steps don't consistently solve more tasks, which is worth knowing before you configure an agent for production.

The harder truth is that benchmark progress, however real, doesn't establish that agents can independently maintain complex production systems. A 74% pass rate on 113 carefully designed tasks says something meaningful about relative model capability. It doesn't say an agent can be trusted unsupervised in a production codebase, handle ambiguous requirements, or navigate the social dynamics of a real engineering team. Those gaps aren't measured by any benchmark that currently exists.

Conclusion

DeepSWE is a genuine improvement over the benchmarks it was designed to replace. Original tasks, behavioral verification, and published trajectories make it more trustworthy than SWE-bench Pro for comparing frontier models on hard engineering work. DeepSWE pass rates span roughly 70 percentage points from worst to best, versus about 30 points on publicly reported SWE-bench Pro scores, and that wider separation is real and informative.

But leaderboard results need careful reading. Configuration matters enormously: reasoning effort, cost, and harness choices all shift scores. And the independent grading review found defects in over 20% of tasks, enough to treat precise rankings as directional rather than definitive. Benchmark design remains genuinely hard, and DeepSWE is an honest attempt at a problem that doesn't have a clean solution yet.

If you want to sharpen the skills that benchmarks like DeepSWE are designed to measure, our AI-Assisted Coding for Developers course covers prompt engineering, debugging, and testing workflows with AI coding assistants. The follow-up course, Advanced AI-Assisted Coding for Developers, goes further into codebase analysis, performance optimization, and security, the kind of multi-step engineering work where today’s top-scoring agents still struggle.