Cognition published a benchmark table on November 4, 2026 for its new coding model, SWE-2 Flash, and one row of it makes a good lesson in how to read AI coding scores. In the table, SWE-2 Flash is listed at 89.2 on Terminal-Bench 2.1 and 20.0 on Terminal-Bench 4.0. Same model, same company, same table, a gap of nearly 70 points. Neither number is wrong. They answer different questions, and if you do not know which question a score answers, you can easily pick the wrong model for your work. (Source: Cognition)
Quick summary: A benchmark score is only meaningful next to its version, its test conditions and the other models scored the same way. In Cognition's own table, models change rank between Terminal-Bench 2.1 and 4.0, and a base model scores 64.7 on one and 0.9 on the other. Before you trust a coding score, check seven things: version, who ran it, effort setting, tooling, comparison set, spread across tests and cost per task. Then run your own small test.
The table that shows the problem
Cognition's post lists SWE-2 Flash next to its base model, Inkling-Small, and several rivals. Below are the two terminal benchmark rows. The page's text export has no column labels, so the models are matched to numbers by the order of the page's model legend and its Terminal-Bench 4.0 chart. Repeated reads of the page gave the same mapping for both rows, and a second write-up confirms the 4.0 figures for SWE-2 Flash (0.9 to 20.0 versus its base model). We found no second source for the version 2.1 row, so verify it on Cognition's page if you plan to quote it. (Sources: Cognition, explainx.ai)
| Model | Terminal-Bench 2.1 | Terminal-Bench 4.0 |
|---|---|---|
| SWE-2 Flash | 89.2 | 20.0 |
| Inkling-Small (the base model) | 64.7 | 0.9 |
| Claude Haiku 5.5 | 94.4 | 39.2 |
| GLM 5.3 Flash | 84.3 | 41.8 |
| DeepSeek V4.1 Flash 1017 | 91.0 | 42.3 |
| GPT-6 Luna | 82.0 | 16.4 |
| Muse Spark Open | 76.4 | 8.0 |
Three patterns matter. First, the scale is not comparable: an 89 on one version is not four times better than a 20 on another, because the tests are not equally hard. Second, the ordering changes. On 2.1, Claude Haiku 5.5 (94.4) sits above GLM 5.3 Flash (84.3). On 4.0, GLM 5.3 Flash (41.8) is ahead of Haiku (39.2). Third, a model that looks decent on the older test can be nearly flat on the newer one: Inkling-Small goes from 64.7 to 0.9. Cognition's post does not explain why the versions differ. Our reading, which is an inference and not a claim from Cognition, is that the newer version is much harder, so scores spread out and rankings become informative again.
Why one model can score so differently
- Difficulty and saturation: when many models score in the 80s and 90s, a test stops separating them. A harder version makes differences visible again. Gains near the ceiling mean less than gains in the middle.
- Different tasks: a new version usually swaps in new tasks, so a model trained or tuned on older-style problems may not transfer.
- Different grading: some tests grade by final state (did the tests pass), others by process. The grader changes what counts as success.
- Different tooling: the same model plugged into different agent harnesses, with different tools and time limits, can score very differently.
- Different effort settings: giving a model more reasoning budget or more attempts can move a score a lot, and not every table says which setting it used.
The same table shows a second mechanism: training emphasis. Cognition says it added a mid-training stage on a dataset of a few trillion tokens before reinforcement learning and improved the quality and variety of its reinforcement learning tasks. By its numbers that moved the base model's Terminal-Bench 4.0 score from 0.9% to 20.0%, and FrontierCode 2.0 from about 4% to 24%. Those are vendor-run figures that explainx.ai could not find independently replicated. The lesson is that a jump this large reflects targeted work on the benchmark family, which is legitimate, but it does not guarantee equal gains on your codebase. (Sources: Cognition, explainx.ai)
Seven checks before you trust a coding score
1. Which version of the benchmark?
A name is not enough. Terminal-Bench 2.1 and Terminal-Bench 4.0 produced totally different numbers for the same model above. If a chart says only "Terminal-Bench," find the version before comparing it with another chart.
2. Who ran it?
A vendor-run score on the vendor's harness is a claim, not a measurement. That does not make it false, but it is not independent. In SWE-2 Flash's case, explainx.ai classifies the headline gains as vendor-run with no independent replication found. Look for a third party that re-ran the test.
3. What was the comparison set?
Cognition says SWE-2 Flash outperforms all American open-weight models on leading evals, including models about three times larger. The qualifier matters. In its own table, GLM 5.3 Flash and DeepSeek V4.1 Flash 1017 both score above SWE-2 Flash on Terminal-Bench 4.0 (41.8 and 42.3 against 20.0), and Claude Haiku 5.5 scores 39.2. The claim can be true and still tell you little about which model to buy.
4. Is the score an average or a single test?
Averages hide splits. In Cognition's table, SWE-2 Flash roughly matches Muse Spark Open on DeepSWE 1.1 (60.1 against 60.0) but trails it on SWE-Atlas QA (50.0 against 54.0), per the full table, and trails its larger sibling SWE-2 on DeepSWE 1.1 (60.1 against 73.0) and Terminal-Bench 4.0 (20.0 against 27.3). Look at the test that resembles your work.
5. Which effort or reasoning setting?
Scores at maximum reasoning are not scores you will see in a fast, cheap agent loop. Match the setting you will really use.
6. What does a completed task cost?
Cognition says SWE-2 Flash is one-fifth the price and three times the speed of SWE-2, with a demo at a simulated 200 tokens per second against 60. Its page gives no absolute price, and says the model is free through November 30. A price ratio without a price is a hint, not a budget. Ask for cost per completed task.
7. Is it available to you?
SWE-2 Flash is available in Devin Desktop and the Devin CLI. Cognition says it is not releasing weights publicly yet, citing misuse risk, and promises details of a program for trusted partners in a few weeks. A model you cannot run in your own stack is a harder buy than the chart suggests.
How different kinds of score compare
| Kind of number | What it tells you | What it hides |
|---|---|---|
| Vendor-run score on a hard, new benchmark | Where the vendor thinks it is strong | Harness choices, effort setting, tasks tuned for |
| Score on an older, saturated benchmark | Whether a model clears a basic bar | Differences at the top, since many models bunch near 90 |
| Independent re-run | Reproducibility | Rarely matches your stack, and often lags the launch |
| Your own 10-task test | Fit for your code, tools and standards | Small sample, so keep it honest with a baseline |
If you can only do one thing, rerun your last ten closed bug tickets with the model and compare how many ended in mergeable code. Ten real tasks beat any single leaderboard number.
01Why would the same model score 89.2 and 20.0 on Terminal-Bench?
In Cognition's table the two numbers come from two versions of the benchmark. Cognition does not explain the difference. A plausible reading is that the newer version is harder, so scores spread out, but that is our inference.
02Are vendor benchmark numbers reliable?
They are usually accurate to the vendor's setup but not independent. Treat them as claims, look for third-party re-runs, and check the effort setting and tooling.
03What is a saturated benchmark?
One where most strong models score near the maximum, so it can no longer rank them. The 80s and 90s in the version 2.1 column above are an example of scores bunching up.
04Is a bigger jump over the base model always better?
No. Going from 0.9 to 20.0 shows a lot of targeted training, but the new score is still below several competitors on the same test. Look at the absolute level and your own tasks, not the size of the jump.
05How many tasks do I need for my own test?
Ten real tasks are enough to catch obvious differences in quality and cost. Use closed tickets with known good fixes so grading is quick.
The takeaway
A score is a sentence, not a verdict. "89.2 on Terminal-Bench 2.1" and "20.0 on Terminal-Bench 4.0" are both true statements about the same model, and a smart reader keeps both in view. The next time a new coding model arrives with a chart, check the version, the runner, the setting and the comparison set before you check the bar height.
If you want to test how different AI models respond to your own coding prompts without juggling accounts, LumiChats puts several in one chat so you can compare answers directly. That small, practical test tends to teach more than a leaderboard.
