Your own benchmark for choosing an LLM is a small set of tasks drawn from real company work, with predefined correct answers and evaluation rules, used to compare models before deployment. It is needed because public leaderboards quickly stop separating the top models: according to an analysis of 224 benchmarks published on October 13, 2026, half of the tests released since 2023 “die” in about 19 months, versus 38 months for older ones. A high score on someone else’s test says something about the model in general, but not whether it will handle your documents, processes, and cost constraints.
Prepared by AIrassvet October 14, 2026 based on the Habr article “The Half-Life of AI Benchmarks Has Been Cut in Half” (October 13, 2026), research on benchmark saturation on arXiv and in Nature Communications, and hands-on experience selecting models from an open AI agent deployment case study. It is aimed at executives, CIOs, and product teams choosing a language model for a chatbot, AI agent, RAG system, or document processing. The article does not rank models and does not recommend any specific vendor. AIrassvet did not conduct its own benchmark research; all figures belong to the authors of the cited works. AIrassvet is not affiliated with them; the section at the end of the article describes the publisher’s services.
Contents
- In Brief
- What a benchmark is and why it “dies”
- What the review of 224 benchmarks showed
- Why benchmarks die
- Why a high score still does not mean a model is right for you
- Public leaderboard or your own set: when to use which
- How to build your own benchmark for model selection: 7 steps
- Task card and runbook template
- Example: selecting a model for an AI agent in production
- How to keep your benchmark from going stale
- Frequently Asked Questions
- Primary sources and verification method
In Brief
- Benchmarks are aging faster: the half-life of public tests released in 2023 or later is about 19 months (95% confidence interval 15–23), versus 38 months for tests released before 2023.
- It’s not because the tests got easier: at launch, new benchmarks sit just as far from the ceiling, but models reach that ceiling about twice as fast.
- One in seven new benchmarks breaks on its own: due to task or evaluation errors. In agentic environments, this accounts for nearly half of benchmark “deaths.”
- Automated rankings can be gamed: a “zero model” that always returns the same string scored 86.5% on AlpacaEval 2.0.
- Business takeaway: a public leaderboard is useful for narrowing the candidate pool. The final choice should be made on your own set of 30–100 tasks with known answers, cost, and latency — that is this editorial team’s practical guideline, not a research standard.
What a benchmark is and why it “dies”
A language model benchmark is a fixed set of tasks with evaluation rules used to compare models against one another. For example, multiple-choice questions, coding tasks, or multi-step assignments for an AI agent in a defined environment.
Benchmark saturation is the point at which the test stops distinguishing the best models: leaders score almost the maximum, and the differences between them disappear into statistical noise. In the study discussed here, a benchmark was considered “dead” at the earliest of five events: the best result reached human level; the best result reached 90% of the main metric; the creators themselves declared the set saturated or broken; a threshold announced by the organizers was reached; or an independent audit, answer leakage, or a trivial method compromised the test.
Benchmark half-life is the time it takes for half of the tests to “die.” It is measured using survival analysis methods to account for benchmarks that are still alive. Without this, you get a biased result: for newer tests, only the short-lived ones have had time to “die,” so the simple median among dead tests understates lifespan. The study’s author demonstrated this with a zero model: even in a world with no acceleration at all, that median would have fallen from 19 months for older cohorts to two months for the newest one.
What the review of 224 benchmarks showed
The author of the wine benchmark OenoBench compiled 224 benchmarks that nine labs — OpenAI, Anthropic, Google, Meta, xAI, DeepSeek, Alibaba, Moonshot, and Mistral — reported on in 2018–2026. A test was included if the official release materials showed results from at least two of the nine labs, or from one lab in at least three releases. The dataset included 230 releases and 6,652 rows of “release × benchmark × result,” with a cutoff date of September 30, 2026. The rules for what counts as a test’s death were written into the codebook before labeling began, and the dataset and code were published (Habr, 10/13/2026).
| Benchmark release year | Benchmarks | Dead by the cutoff | Half-life, months (95% CI) |
|---|---|---|---|
| 2016–2019 | 28 | 19 | 49 (24–98) |
| 2020–2022 | 21 | 14 | 38 (25 and up) |
| 2023 | 23 | 16 | 19 (13–27) |
| 2024 | 50 | 30 | 21 (17–26) |
| 2025 | 63 | 27 | 19 (12 and up) |
| 2026 | 39 | 10 | history is too short to estimate |
Key findings from the author:
- Across the full sample, the half-life is 23 months (95% CI 19–27). One quarter of benchmarks die within the first year.
- The gap between eras is twofold: 38 months for benchmarks released before 2023 (4% died in the first year) and 19 months for those released later (about one-third died in the first year).
- With each additional release year, the risk of “death” rises by about 22%. For a typical public static test, that means a benchmark released in 2018 would have lived about 53 months, in 2022 — 31, in 2026 — 18.
- The gap holds under stricter rules: if you count only independently verified results from single models, the half-lives rise to 68 and 26 months, and the gap actually widens.
New benchmarks are not easier than old ones. The median distance to the threshold at launch is 35 percentage points for older tests and 38 for newer ones. But according to Epoch AI trajectories for 21 benchmarks in the sample, the best result on older tests increased by 0.75 logit per year, while on newer tests it increased by 1.56 — twice as fast. The road to the ceiling is the same, but models travel it faster.
Independent studies show the same pattern. A coalition study from EvalEval, published at ICML 2026, analyzed 60 language-model benchmarks: nearly half are saturated, the share rises with test age, and resistance to saturation depends on expert curation, not on keeping the test data closed (arXiv:2602.16763). Earlier, Ott and co-authors analyzed the trajectories of thousands of benchmarks from Papers with Code in Nature Communications (Ott et al., 2022).
What Causes Benchmarks to Die
Of the 116 “deaths” in the sample, 84 were the threshold being reached by models. Another 12 times, the creators themselves declared the test saturated before models reached the threshold. The remaining 20 were defects: errors in tasks or evaluation, independent auditing, or a trivial method.
| Reason | Example from the study | What this means for model selection |
|---|---|---|
| Models reached the threshold | GPQA Diamond was released in November 2023 and “died” 24 months later, when a model reached 91.9%. | The gap between leaders on such a test no longer tells you much. |
| Creators declared saturation | The first version of Finance Agent was declared saturated at a best result of 60.7%. | The “death” threshold differs from one test to another—look at the rules. |
| Task defects | During the audit of Humanity's Last Exam, only 668 of 2,500 questions were left unchanged, 1,143 were corrected, and for 689 no verdict could be reached. | Some model “errors” may actually be test errors. |
| Agent-environment defects | An audit of the airline domain in τ-bench deemed 31 of 50 tasks underdefined. | Agentic tests are especially fragile. |
| Version change | Terminal-Bench 3.0 lasted a month: in version 4.0, 8 tasks were removed from 74. | Results from different versions cannot be compared directly. |
Agent environments break most often. Among benchmarks released in 2023 and later, 45% die from models in the first two years, and 14% from defects and version changes. Two-thirds of defect-driven deaths are in agent environments: of 29 dead agent environments, 13 broke on their own, while among 81 dead static tests, there were six. According to the author, when creating SWE-bench Verified, 68% of the original tasks were filtered out as underdefined or having unfair tests.
If you are choosing a model for an AI agent—one that works in a CRM, 1C, or a browser—this matters especially: the public agent leaderboard you rely on may be outdated or may include broken tasks.
Why a high score still doesn’t mean the model is right for you
There are four reasons not to make a decision based on someone else’s leaderboard.
1. Automatic evaluations can be gamed. In Zheng et al.’s paper, a “zero model” that returns the same preselected string for any prompt achieved an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 points on Arena-Hard-Auto, and 9.55 on MT-Bench (arXiv:2410.07137). This is a proof of concept, not an accusation against any specific models, but it shows that a score assigned by another model judge is not a guarantee of quality.
2. Labs optimize models for popular tests. The author of the study himself notes that a significant part of the visible speed of progress may actually be the speed of optimization for tracked benchmarks. On his own niche wine test, the best result did not improve over the year.
3. Public tasks do not look like yours. Olympiad math and programming on public repositories say little about how a model extracts details from your invoices or responds to a customer according to your internal policy in Russian.
4. A leaderboard does not account for your constraints. Cost per thousand requests, response latency, operation inside the Russian environment, personal data requirements, reliability of tool calls—these are all things you measure yourself. Even Russian text tokenization differs across models, and so does the real cost; for more, see the article on Russian text tokenization and LLM cost.
Public leaderboard or your own set: when to use which
| Task | Is a public leaderboard enough? | What to do |
|---|---|---|
| Narrow a list of 20 models down to 3–5 candidates | Yes | Look at live, unsaturated tests from a similar class of tasks |
| Choose a model for a support chatbot or RAG over your own documents | No | Your own set of questions with ground-truth answers and source links |
| Choose a model for an AI agent with tool calling | No | Your own tasks with known answers and tracking of failures and loops |
| Decide whether to replace the current model with a new version | No | Run the old and new models on the same dataset |
| Understand the model’s overall level for an internal presentation | Yes, with caveats | State the test date and check whether it is saturated |
| Compare a cloud model and a model on your own servers | No | The same dataset, plus measuring cost, latency, and load |
How to Build Your Own Benchmark for Model Selection: 7 Steps
This is the AIrassvet editorial team’s methodology, compiled from the study author’s recommendations and practical experience evaluating AI systems. It does not require special tools at the start—just a spreadsheet and a script.
- Define one task and one success criterion. Not “smart assistant,” but “reply to the customer according to the return policy with the correct link to the relevant section.” Write down what counts as a correct answer, a partially correct answer, and an error.
- Collect tasks from real work. Take 30–100 real tickets, documents, or incidents with already known correct outcomes. De-identify personal data before an external model sees the tasks.
- Leave headroom to the ceiling. The study author showed that headroom at the start is the main driver of a test’s longevity: each logit of headroom extends life by about one and a half times. If the current model solves almost everything, add harder cases: long documents, rare exceptions, questions with no answer in the database.
- Lock in the evaluation rules before running the test. As in the study, where the codebook was written before labeling, decide in advance what counts as a correct answer. Otherwise, the criteria will subtly adapt to the model you like.
- Run multiple passes for each model. In the AI agent implementation case, the cost difference between models that looked like an order of magnitude on the first run shrank to just a few times after fifty runs. One run is not enough.
- Measure more than just quality. For each run, record: whether it was correct, how much it cost according to billing, how long it took, whether there were crashes, broken formatting, refusals, or looping.
- Separate selection from acceptance testing. It is better to keep the model-selection dataset separate from the acceptance dataset for the finished system so you do not fit the system to known tasks.
For how to evaluate search, answers, and hallucinations separately in RAG systems, see the article on RAG accuracy and hallucinations.
Task Card and Run Log Template
Copy this structure into a spreadsheet or YAML so the set can be rerun for every new model.
Task Card
| Field | What to Record | Example |
|---|---|---|
| ID and Version | Unique set number and version | refund-017, set v1.2 |
| Scenario | Which process the task belongs to | Customer response about a refund |
| Input | The question, document, or context the model receives | Customer message and an excerpt from the policy |
| Gold Standard | Correct answer or signs of a correct answer | Refund deadline and a link to section 4.2 |
| Scoring Rule | How to assign “correct / partially correct / incorrect” | Incorrect if the deadline is stated incorrectly |
| Difficulty | Normal, difficult, no answer in the database | No answer — the model should say it doesn’t know |
| Sensitivity | Whether there is personal or commercial data | Anonymized |
Run Log
| Field | What to Record |
|---|---|
| Model date and version | Exact name and version, provider, region |
| Parameters | Temperature, system prompt, prompt version |
| Task Result | Correct / partially correct / incorrect, reviewer comment |
| Cost | Based on key billing, not list price |
| Latency | Median and longest response |
| Failures | Crashes, format errors, refusals, looping |
| Summary | Share correct, share of difficult correct, cost per task |
Example: Choosing a Model for an AI Agent in Production
A good example of a small in-house benchmark is the case of a team that connected the open-source HolmesGPT agent to Kubernetes alert triage (Habr, 10/13/2026). Instead of a ranking, they ran a selection process: eight models, 52 runs in three waves, including a task with a known answer in advance.
What this selection process showed:
- Half of the candidates failed the known-answer task in the same way: confidently claimed that the needed component was not in the cluster, even though it was there — it just wasn’t visible among the usual objects.
- The winner was not the cheapest model: DeepSeek V4 Flash completed 16 runs without a single failure and solved the task correctly in all four attempts; both cheaper models failed to solve it.
- Production added new criteria: in one environment, about every fifth analysis went over 15 minutes, while in another the model got stuck in a loop. The team switched to a more expensive model by list price, but the analyses started finishing.
The practical takeaway: your set should include not only “correct / incorrect,” but also reliability — timeouts, crashes, and repeated calls. If a contractor will implement the model, ask them to show such a log on your tasks; for choosing a contractor, see the article How to Choose an LLM Integrator.
How to Keep Your Benchmark from Going Stale
Your own set also “dies,” just for different reasons: processes, documents, and models change. The recommendations from the research author for benchmark creators translate well to a corporate set:
- Plan for a lifespan of a year and a half to two years, not five. If you need a tool for years, you need a mechanism for updating tasks.
- Define saturation in advance. For example: if all candidates solve more than 95% of tasks, it’s time to make the set harder.
- Track versions. A fixed version of the set, a change log, and separate results for each version. If more than a noticeable share of tasks is replaced, old and new results can no longer be compared directly.
- Review the tasks themselves. Auditing tasks before launch is cheaper than a “corrected” version six months later: some “model errors” may turn out to be errors in the gold standard.
- Closed access does not extend lifespan. In both the Habr study and the EvalEval paper, a closed test set did not protect against saturation. It protects against leakage, but not against model progress. Keeping a corporate set off the internet is still worthwhile — for confidentiality.
Frequently Asked Questions
What is a benchmark for choosing an LLM?
It’s a set of tasks with known correct answers and scoring rules used to compare language models before deployment. You build your own benchmark from real company tasks to test the model’s quality, cost, and reliability in your specific scenario.
Why do public model rankings become outdated so quickly?
Models reach the ceiling of a test about twice as fast as before, and some benchmarks break because of errors in the tasks. Based on an analysis of 224 benchmarks, half of the tests released since 2023 stop distinguishing the best models after about 19 months.
How many tasks should an in-house benchmark have?
There is no single standard. A practical editorial guideline from AIrassvet is 30–100 real tasks with known answers for one scenario, including difficult cases and questions with no answer. What matters most is that the tasks resemble real work and are checked against predefined rules.
Can you trust an evaluation score assigned by another model?
Only with verification. A study on arXiv showed that even a model that always returns the same string can score highly on automated rankings with an LLM judge. Some answers should be checked by people and their ratings compared with the judge’s ratings.
How often should you re-evaluate the model you chose?
With every model or provider update, when you change the prompt or data, and on a schedule — for example, once a quarter. Running the old and new versions on the same dataset shows whether performance has improved on your tasks.
What matters more when choosing a model: quality or price?
What matters is the ratio: the cost of one correctly solved task, taking failures into account. A cheap model that often makes mistakes, crashes, or gets stuck in loops can end up costing more.
Original sources and verification method
Method: we read the original study in full on October 14, 2026, recorded the metrics along with the authors’ caveats, and cross-checked the descriptions of independent studies against their abstracts on arXiv and the journal page. The steps, template, and the 30–100 task benchmark are AIrassvet’s editorial method, not the studies’ conclusions.
- Habr, 10/13/2026: “The half-life of AI benchmarks has been cut in half: 19 months instead of 38” — sample of 224 benchmarks, method, results, recommendations.
- arXiv: Akhtar et al., “When AI Benchmarks Plateau”, ICML 2026 — saturation of 60 benchmarks.
- arXiv: Zheng et al., “Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates” — null model on automatic rankings.
- Nature Communications: Ott et al., 2022 — dynamics of benchmark creation and saturation.
- Habr, 10/13/2026: “AIOps: our experience deploying HolmesGPT on three projects” — an example of model selection for a task with a known answer.
Limitations: the study’s conclusions apply to benchmarks reported by major labs; the half-life estimates have wide confidence intervals; model and test names are given as the authors present them.
How AIrassvet helps you choose and validate a model
If you are choosing a model for a specific process, the main risk is relying on someone else’s ranking and only learning about problems after launch. On the AIrassvet website the directions directly relevant to this task are described:
- Process audit and data preparation: selecting real tasks, anonymizing data, and creating gold-standard answers.
- Deployment and fine-tuning of language models, including on-premises deployment on the client’s servers, when you need to compare a cloud model and an in-house model on the same dataset.
- RAG systems and enterprise knowledge bases — if the model needs to answer using your documents.
- MVP development, integration, and testing of the selected model before scaling.
The first step is to choose one process, describe the current workflow, data sources, and data-transfer restrictions, and agree on a criterion: what share of correctly solved tasks and what cost per task will be sufficient for you.
Bottom line: the ranking narrows the choice; the decision comes from your own dataset
Public benchmarks are aging faster than before: half of new tests stop distinguishing the leaders in about a year and a half, and some break because of their own errors. So a leaderboard score is a reason to add a model to your shortlist, but not a basis for deployment.
The decision should be made on your own task set: real examples, prewritten evaluation rules, multiple runs, and measurements of cost, latency, and failures. That dataset should be versioned and updated — then each new model can be tested in days instead of being chosen at random.