Open Book

A score with tools is an open-book exam: the model can search the web and run code. A score without tools is closed-book. They are different exams, so they get separate lines.

The scores

Every row links to the page where I found the number, and says who ran the test. A score on 2,500 questions carries about two points of sampling error either way, so a new high is marked record only when it clears the old one's margin, and tied when it does not. Newest scores are at the top.

The questions are public, so a model with a browser could look answers up. The blocklist column says what the linked page tells you about that. 18 of 27 rows say nothing.

Runs by someone other than the lab

Almost every score above is a lab grading its own model. Two outside groups publish their own runs. Neither page dates its rows, so they cannot go on the chart's timeline, and neither says whether tools were on.

Read together: where a lab has published a no-tools score for the same model, Scale's figure is usually a few points lower and sometimes higher. The widest gap is Fable 5.1, 46.5% at Scale against Anthropic's 60.9%, run at a lower effort setting. The one outside figure for the current top model, 74.3%, is on a text-only subset.

Not on the record line

The record line carries one model, on the full 2,500 questions. These rows are real results on a different test: a subset of the questions, or a system that combines several agents, runs or models.

Rows from The Last Line that I could not find at their source. No number, no row.

The 27 rows The Last Line took from Scale AI's table are not dropped. They are in the table of outside runs above, without dates, because Scale gives none. Eight of them had been plotted in 2024, before the exam existed.

Method

What gets a row. A score goes in when I have opened the page that reports it and found the number there. The row is dated by the day that page published the score, not by the model's release. Where a lab writes "Humanity's Last Exam" with no qualifier I count it as the full set, and the row says the page does not name the set.

What sets a record. One model, on the full 2,500 questions, with a higher score than every earlier row on the same track. When several scores are published on one day, only the highest can be a record. High-compute tiers that a lab ships as one model, such as GPT Pro and Gemini Deep Think, stay on the line and are labelled. Systems whose makers describe several agents or several runs combined come off it. That boundary is a judgement call, and the table above shows every row it removes.

Error bars. The ± on a score is the 95% sampling interval for a pass rate on 2,500 questions: 1.96 × √(p(1−p)/2500). It covers the luck of which questions were asked. It does not cover grading differences between labs.

The forecast. Two curves are fitted to the with-tools records published in the 365 days before the forecast date, 11 of them today. One is a straight line. The other is an S-curve that levels off at 90%. Both are floored at the current record. The page reports the dates each curve reaches 80% and 90%, and makes no forecast for 100%.

Why not 100%. In July 2025 FutureHouse audited the text-only chemistry and biology questions and estimated that 29% (±3.7) have answers that conflict with published research. A later review by an Alibaba team, HLE-Verified, passed 668 of the 2,500 questions unchanged, repaired 1,143 and set 689 aside as uncertain. A model at 100% would be reproducing the key's mistakes.

Updates. The sources are checked every day: the labs' own pages, Scale AI's table, Artificial Analysis, and the aggregators as leads. A new score is opened at its source, added with that day's date, and noted in the change log. The date of the last check is at the top of this page, and if 30 days ever pass without one, a notice appears there.

The whole method is one file you can run: method.js on scores.json.

What comes after this exam

The exam's makers have moved on to two successors, and as the 2025 question set fills up with scores above 70%, they are where the signal goes.

HLE-Rolling is a fork of the question set that is updated with corrections from the community. It was released on 8 October 2025; Scale's leaderboard page refers to a 17 September 2026 version.

HLE-Diamond is a cleaned set of 1,000 questions, 500 on reasoning and 500 on knowledge, released by the Center for AI Safety and Scale AI on 22 September 2026. These are the makers' own runs, so they are independent of the labs:

Source: lastexam.ai/blog/hle-diamond, checked 11 Nov 2026. All at the "reasoning high" setting. These are a different question set and are not on the chart above.

Forecast log

Each forecast is stamped with the day it was made and graded when its date arrives. Misses stay on the page.

File: forecasts.json.

Change log

• 11 Nov 2026
  First edition. Started from the 75 rows of The Last Line and opened every source. 51 dated rows are on the two tracks, each linked to a page that contains its number; 67 undated rows from Scale AI and Artificial Analysis are in the table of outside runs; 13 rows are listed as not on the record line and 9 as dropped.
• 11 Nov 2026
  Corrections to the starting data found at the sources. Grok 4's 44.4% with tools is Grok 4 Heavy, which runs several agents at once; the single model scored 38.6%. OpenAI published GPT-5 pro at 42.0% (Aug 2025) and GPT-5.2 Pro at 50.0% (Dec 2025) with tools; both were missing and both were records. Claude Opus 4.6 is 53.0%, corrected by Anthropic from 53.1%. Claude Fable 5.1 was first published at 65.0%, and at 60.9% without tools, which is the closed-book record. Qwen3.6-Plus is 50.6%, not 43.4%. Z.ai's GLM scores and Artificial Analysis's scores are on the text-only subset. Scale AI's rows carry no dates, so they left the timeline.
• 11 Nov 2026
  Two changes to the method published with the build plan: 'a year ago' is 365 days, not 356, and when several scores are published on one day only the highest can be a record. With the corrected data the six-month backtest of the straight line no longer errs in one direction: it was too low three times and too high three times.
• 11 Nov 2026
  Updates moved from manual to a scheduled daily check of every source. The page is rebuilt after each check, so the date at the top is the date of the last one.

File: changelog.json.

Sources