{
 "title": "Open Book data",
 "as_of": "2026-11-11",
 "last_verified": "2026-11-11",
 "question_count": 2500,
 "notes": [
  "Scores are percent correct on Humanity's Last Exam (HLE), finalised at 2,500 questions on 3 April 2025.",
  "date_published is the date the linked source published the score, not the model's release date.",
  "verified_on is the date the number was found at source_url.",
  "reported_by: own_lab (a lab about its own model), other_lab (a lab's table about a rival), third_party, independent.",
  "set_stated: true when the linked page says the score is on the full set. Where a lab writes 'Humanity's Last Exam' with no qualifier, the row is counted as the full set and set_stated is false.",
  "blocklist (with-tools rows): 'listed' = the linked page says what was blocked; 'mentioned' = it says a blocklist was used; 'none' = it says nothing.",
  "mode 'high-compute' marks a lab's extended-reasoning tier (GPT Pro, Gemini Deep Think). These stay on the record line; systems their makers describe as several agents or runs combined do not.",
  "moved_to_outside_runs: 27 rows The Last Line took from Scale AI's table are under independent_runs, because Scale's table gives no date for any row."
 ],
 "rows": [
  {
   "model": "Grok 4",
   "score": 38.6,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-07-09",
   "reported_by": "own_lab",
   "reporter": "xAI",
   "source_url": "https://x.ai/news/grok-4",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Chart: 'Full set (April 3, 2025) with Python and Internet tools'."
  },
  {
   "model": "GPT-5 pro",
   "score": 42.0,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-08-07",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5/",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "mode": "high-compute",
   "note": "Chart: 'Humanity's Last Exam (Full Set)', bar 'python + search with blocklist'."
  },
  {
   "model": "GPT-5",
   "score": 35.2,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-08-07",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5/",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "Same chart, 'python + search with blocklist'."
  },
  {
   "model": "o3",
   "score": 24.3,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-08-07",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Same chart, 'python + browser'. xAI's chart of 9 Jul 2025 lists o3 at 24.9. The Last Line linked OpenAI's o3 post, which has no HLE figure."
  },
  {
   "model": "Gemini 3 Pro",
   "score": 45.8,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-11-18",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://blog.google/products-and-platforms/products/gemini/gemini-3/",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "Table row 'With search and code execution'. Google's Feb 2026 pages label the same figure 'Search (blocklist) + Code' on the full set."
  },
  {
   "model": "GPT-5.2 Pro",
   "score": 50.0,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-12-11",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-2/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "mode": "high-compute",
   "note": "Row 'HLE (w/ search, Python)'."
  },
  {
   "model": "GPT-5.2 Thinking",
   "score": 45.5,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-12-11",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-2/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Row 'HLE (w/ search, Python)'."
  },
  {
   "model": "GPT-5.1 Thinking",
   "score": 42.7,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-12-11",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-2/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Row 'HLE (w/ search, Python)'."
  },
  {
   "model": "Kimi K2.5",
   "score": 50.2,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-01-27",
   "reported_by": "own_lab",
   "reporter": "Moonshot AI",
   "source_url": "https://huggingface.co/moonshotai/Kimi-K2.5",
   "verified_on": "2026-11-11",
   "blocklist": "listed",
   "note": "Row 'HLE-Full (w/ tools)'. 'Hugging Face access was blocked.' The model card shows no date; the release was reported on 27 Jan 2026."
  },
  {
   "model": "Claude Opus 4.6",
   "score": 53.0,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-05",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-opus-4-6",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "Published as 53.1; Anthropic corrected it to 53.0 on 23 Feb 2026 after flagging three more cases of cheating. Several later tables still print 53.1."
  },
  {
   "model": "Claude Opus 4.5",
   "score": 43.4,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-05",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-opus-4-6",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "Comparison column in the Opus 4.6 post. The Nov 2025 system card gives the score only in a chart; reports at the time quoted 43.2."
  },
  {
   "model": "Claude Sonnet 4.5",
   "score": 33.6,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-05",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-opus-4-6",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "Comparison column in the Opus 4.6 post."
  },
  {
   "model": "Gemini 3 Deep Think (Feb 2026)",
   "score": 53.4,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-02-12",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-deep-think/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "mode": "high-compute",
   "note": "Table row 'Search + code execution', 'full set, text + MM'."
  },
  {
   "model": "Claude Sonnet 4.6",
   "score": 49.0,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-17",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-sonnet-4-6",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "'A domain blocklist was used to decontaminate eval results.'"
  },
  {
   "model": "Gemini 3.1 Pro",
   "score": 51.4,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-02-19",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://deepmind.google/models/model-cards/gemini-3-1-pro/",
   "verified_on": "2026-11-11",
   "blocklist": "mentioned",
   "note": "Row 'Search (blocklist) + Code', 'full set, text + MM'."
  },
  {
   "model": "GPT-5.4 Pro",
   "score": 58.7,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-03-05",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-4/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "mode": "high-compute",
   "note": "Row 'Humanity's Last Exam (with tools)'."
  },
  {
   "model": "GPT-5.4",
   "score": 52.1,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-03-05",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-4/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Row 'Humanity's Last Exam (with tools)'."
  },
  {
   "model": "Claude Mythos Preview",
   "score": 64.7,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-04-07",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/glasswing",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Limited-release model. Anthropic adds that it 'still performs well on HLE at low effort, which could indicate some level of memorization'."
  },
  {
   "model": "Muse Spark (Thinking)",
   "score": 50.4,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-04-08",
   "reported_by": "own_lab",
   "reporter": "Meta",
   "source_url": "https://ai.meta.com/blog/introducing-muse-spark-msl/",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Table row 'With Tools'."
  },
  {
   "model": "Claude Fable 5.1",
   "score": 65.0,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-01",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-1",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "The page is dated 'September 2026'; the release was reported on 1 Sep. Anthropic's tables of 22 Sep and 9 Nov list 65.6 for the same model."
  },
  {
   "model": "Claude Fable 5",
   "score": 63.8,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-01",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-1",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Comparison column in the Fable 5.1 post. The model is older; this is the earliest Anthropic page I found with the figure. One aggregator lists 64.5."
  },
  {
   "model": "Claude Opus 5",
   "score": 63.6,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-01",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-1",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Comparison column in the Fable 5.1 post. The model launched on 24 Jul 2026; BenchLM cites 64.7 from its system card, which I could not read."
  },
  {
   "model": "Claude Opus 5.5",
   "score": 67.7,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-22",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-opus-5-5",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": ""
  },
  {
   "model": "GPT-6 Astra",
   "score": 57.2,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-22",
   "reported_by": "other_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-opus-5-5",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "From Anthropic's comparison table. Reports on 3 Sep 2026 give the same figure from OpenAI's launch table; I could not find OpenAI's own page."
  },
  {
   "model": "Kimi K3.1 Pro",
   "score": 70.0,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-10-23",
   "reported_by": "own_lab",
   "reporter": "Moonshot AI",
   "source_url": "https://www.kimi.ai/blog/kimi-k3-1-pro",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "Chart 'Humanity's Last Exam Full w/ tools'. The page shows no date; its chart file is dated 23 Oct 2026."
  },
  {
   "model": "GLM-5.3",
   "score": 62.5,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-10-23",
   "reported_by": "other_lab",
   "reporter": "Moonshot AI",
   "source_url": "https://www.kimi.ai/blog/kimi-k3-1-pro",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "From Moonshot's comparison chart, not from Z.ai."
  },
  {
   "model": "Claude Fable 5.2",
   "score": 74.2,
   "track": "with_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-11-09",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-2",
   "verified_on": "2026-11-11",
   "blocklist": "none",
   "note": "The model that built this page."
  },
  {
   "model": "Grok 4",
   "score": 25.4,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-07-09",
   "reported_by": "own_lab",
   "reporter": "xAI",
   "source_url": "https://x.ai/news/grok-4",
   "verified_on": "2026-11-11",
   "note": "Chart: 'Full set (April 3, 2025)'."
  },
  {
   "model": "Gemini 2.5 Pro",
   "score": 21.6,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-07-09",
   "reported_by": "other_lab",
   "reporter": "xAI",
   "source_url": "https://x.ai/news/grok-4",
   "verified_on": "2026-11-11",
   "note": "From xAI's comparison chart. Google's Nov 2025 table gives the same figure."
  },
  {
   "model": "o3",
   "score": 21.0,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-07-09",
   "reported_by": "other_lab",
   "reporter": "xAI",
   "source_url": "https://x.ai/news/grok-4",
   "verified_on": "2026-11-11",
   "note": "From xAI's comparison chart. OpenAI's own chart a month later gives 14.7."
  },
  {
   "model": "GPT-5 pro",
   "score": 30.7,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-08-07",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5/",
   "verified_on": "2026-11-11",
   "mode": "high-compute",
   "note": "Chart: 'Humanity's Last Exam (Full Set)'."
  },
  {
   "model": "GPT-5",
   "score": 24.8,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2025-08-07",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5/",
   "verified_on": "2026-11-11",
   "note": "Same chart, with thinking."
  },
  {
   "model": "Gemini 3 Pro",
   "score": 37.5,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-11-18",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://blog.google/products-and-platforms/products/gemini/gemini-3/",
   "verified_on": "2026-11-11",
   "note": "Scale AI's own run of the same model: 37.52."
  },
  {
   "model": "Gemini 3 Deep Think",
   "score": 41.0,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-11-18",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://blog.google/products-and-platforms/products/gemini/gemini-3/",
   "verified_on": "2026-11-11",
   "mode": "high-compute",
   "note": ""
  },
  {
   "model": "GPT-5.2 Pro",
   "score": 36.6,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-12-11",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-2/",
   "verified_on": "2026-11-11",
   "mode": "high-compute",
   "note": "Row 'HLE (no tools)'. The Last Line carried this figure as 'gpt-5-pro', dated Oct 2025."
  },
  {
   "model": "GPT-5.2 Thinking",
   "score": 34.5,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-12-11",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-2/",
   "verified_on": "2026-11-11",
   "note": "Row 'HLE (no tools)'."
  },
  {
   "model": "GPT-5.1 Thinking",
   "score": 25.7,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2025-12-11",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-2/",
   "verified_on": "2026-11-11",
   "note": "Row 'HLE (no tools)'."
  },
  {
   "model": "Kimi K2.5",
   "score": 30.1,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-01-27",
   "reported_by": "own_lab",
   "reporter": "Moonshot AI",
   "source_url": "https://huggingface.co/moonshotai/Kimi-K2.5",
   "verified_on": "2026-11-11",
   "note": "Row 'HLE-Full'."
  },
  {
   "model": "Claude Opus 4.6",
   "score": 40.0,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-05",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-opus-4-6",
   "verified_on": "2026-11-11",
   "note": ""
  },
  {
   "model": "Claude Opus 4.5",
   "score": 30.8,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-05",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-opus-4-6",
   "verified_on": "2026-11-11",
   "note": "Comparison column in the Opus 4.6 post."
  },
  {
   "model": "Claude Sonnet 4.5",
   "score": 17.7,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-05",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-opus-4-6",
   "verified_on": "2026-11-11",
   "note": "Comparison column in the Opus 4.6 post."
  },
  {
   "model": "Gemini 3 Deep Think (Feb 2026)",
   "score": 48.4,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-02-12",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-deep-think/",
   "verified_on": "2026-11-11",
   "mode": "high-compute",
   "note": "'full set, text + MM'."
  },
  {
   "model": "Claude Sonnet 4.6",
   "score": 33.2,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-02-17",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/news/claude-sonnet-4-6",
   "verified_on": "2026-11-11",
   "note": ""
  },
  {
   "model": "Gemini 3.1 Pro",
   "score": 44.4,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": true,
   "date_published": "2026-02-19",
   "reported_by": "own_lab",
   "reporter": "Google",
   "source_url": "https://deepmind.google/models/model-cards/gemini-3-1-pro/",
   "verified_on": "2026-11-11",
   "note": "Meta's re-run two months later: 45.4."
  },
  {
   "model": "GPT-5.4 Pro",
   "score": 42.7,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-03-05",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-4/",
   "verified_on": "2026-11-11",
   "mode": "high-compute",
   "note": "Meta's re-run: 43.9. Scale AI's run: 44.32."
  },
  {
   "model": "GPT-5.4",
   "score": 39.8,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-03-05",
   "reported_by": "own_lab",
   "reporter": "OpenAI",
   "source_url": "https://openai.com/index/introducing-gpt-5-4/",
   "verified_on": "2026-11-11",
   "note": "Meta's re-run: 43.9. Scale AI's run: 36.24."
  },
  {
   "model": "Claude Mythos Preview",
   "score": 56.8,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-04-07",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/glasswing",
   "verified_on": "2026-11-11",
   "note": "Limited-release model."
  },
  {
   "model": "Muse Spark (Thinking)",
   "score": 42.8,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-04-08",
   "reported_by": "own_lab",
   "reporter": "Meta",
   "source_url": "https://ai.meta.com/blog/introducing-muse-spark-msl/",
   "verified_on": "2026-11-11",
   "note": "Scale AI's run of 'Muse Spark': 40.56."
  },
  {
   "model": "Claude Fable 5.1",
   "score": 60.9,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-01",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-1",
   "verified_on": "2026-11-11",
   "note": "Scale AI's run at a lower effort setting (xhigh): 46.50."
  },
  {
   "model": "Claude Fable 5",
   "score": 57.8,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-01",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-1",
   "verified_on": "2026-11-11",
   "note": "Comparison column in the Fable 5.1 post."
  },
  {
   "model": "Claude Opus 5",
   "score": 56.6,
   "track": "no_tools",
   "question_set": "full_2500",
   "set_stated": false,
   "date_published": "2026-09-01",
   "reported_by": "own_lab",
   "reporter": "Anthropic",
   "source_url": "https://www.anthropic.com/claude-fable-and-mythos-5-1",
   "verified_on": "2026-11-11",
   "note": "Comparison column in the Fable 5.1 post."
  }
 ],
 "independent_runs": [
  {
   "id": "scale",
   "runner": "Scale AI",
   "who": "Co-author of the exam",
   "question_set": "Full set, 2,500 questions",
   "questions": 2500,
   "tools": "Not stated on the page. Where a lab has published a no-tools score for the same model, Scale's figure is usually within a few points of it.",
   "dated": "The table gives no date for any row. The page banner reads 'Update September 17, 2026'.",
   "source_url": "https://labs.scale.com/leaderboard/humanitys_last_exam",
   "verified_on": "2026-11-11",
   "note": "Claude Fable 5.2, Claude Opus 5.5 and Kimi K3.1 Pro are not on the table.",
   "rows": [
    {
     "model": "GPT 6 Astra",
     "score": 54.8,
     "ci95": 1.94
    },
    {
     "model": "Fable 5.1 (xhigh)",
     "score": 46.5,
     "ci95": 2.0
    },
    {
     "model": "gemini-3.1-pro-preview (thinking high)",
     "score": 46.44,
     "ci95": 1.96
    },
    {
     "model": "Sonnet 5.5 (xhigh)",
     "score": 46.12,
     "ci95": 1.95
    },
    {
     "model": "Gemini 3.8 Flash",
     "score": 44.52,
     "ci95": 1.96
    },
    {
     "model": "gpt-5.4-pro-2026-03-05",
     "score": 44.32,
     "ci95": 1.95
    },
    {
     "model": "Muse Spark",
     "score": 40.56,
     "ci95": 1.92
    },
    {
     "model": "gemini-3-pro-preview",
     "score": 37.52,
     "ci95": 1.9
    },
    {
     "model": "gpt-5.4-2026-03-05 (xhigh thinking)",
     "score": 36.24,
     "ci95": 1.88
    },
    {
     "model": "claude-opus-4-7",
     "score": 36.2,
     "ci95": 1.88
    },
    {
     "model": "claude-opus-4-6-thinking-max",
     "score": 34.44,
     "ci95": 1.86
    },
    {
     "model": "gpt-5-pro-2025-10-06",
     "score": 31.64,
     "ci95": 1.82
    },
    {
     "model": "Claude Haiku 5.5",
     "score": 30.12,
     "ci95": 1.83
    },
    {
     "model": "gpt-5.2-2025-12-11",
     "score": 27.8,
     "ci95": 1.76
    },
    {
     "model": "gpt-5-2025-08-07",
     "score": 25.32,
     "ci95": 1.7
    },
    {
     "model": "claude-opus-4-5-20251101-thinking",
     "score": 25.2,
     "ci95": 1.7
    },
    {
     "model": "kimi-k2.5",
     "score": 24.37,
     "ci95": 1.81
    },
    {
     "model": "gpt-5.1-thinking",
     "score": 23.68,
     "ci95": 1.67
    },
    {
     "model": "gemini-2.5-pro-preview-06-05",
     "score": 21.64,
     "ci95": 1.61
    },
    {
     "model": "o3 (high) (April 2025)",
     "score": 20.32,
     "ci95": 1.58
    },
    {
     "model": "gpt-5-mini-2025-08-07",
     "score": 19.44,
     "ci95": 1.55
    },
    {
     "model": "o3 (medium) (April 2025)",
     "score": 19.2,
     "ci95": 1.54
    },
    {
     "model": "claude-opus-4-6 (Non-Thinking)",
     "score": 19.0,
     "ci95": 1.54
    },
    {
     "model": "Gemini 2.5 Pro Experimental (March 2025)",
     "score": 18.16,
     "ci95": 1.51
    },
    {
     "model": "o4-mini (high) (April 2025)",
     "score": 18.08,
     "ci95": 1.51
    },
    {
     "model": "Gemini 2.5 Pro Preview (May 06 2025)",
     "score": 17.8,
     "ci95": 1.5
    },
    {
     "model": "o4-mini (medium) (April 2025)",
     "score": 14.28,
     "ci95": 1.37
    },
    {
     "model": "claude-opus-4-5-20251101",
     "score": 14.16,
     "ci95": 1.37
    },
    {
     "model": "claude-sonnet-4-5-20250929-thinking",
     "score": 13.72,
     "ci95": 1.35
    },
    {
     "model": "Gemini 2.5 Flash (April 2025)",
     "score": 12.08,
     "ci95": 1.28
    },
    {
     "model": "claude-opus-4-1-20250805-thinking",
     "score": 11.52,
     "ci95": 1.25
    },
    {
     "model": "Gemini 2.5 Flash Preview (May 2025)",
     "score": 10.96,
     "ci95": 1.22
    },
    {
     "model": "Claude Opus 4 (Thinking)",
     "score": 10.72,
     "ci95": 1.21
    },
    {
     "model": "gemini-3.1-flash-lite-preview",
     "score": 8.64,
     "ci95": 1.1
    },
    {
     "model": "glm-4p5",
     "score": 8.32,
     "ci95": 1.08
    },
    {
     "model": "glm-4p5-air",
     "score": 8.12,
     "ci95": 1.07
    },
    {
     "model": "o1 Pro",
     "score": 8.12,
     "ci95": 1.07
    },
    {
     "model": "Claude 3.7 Sonnet (Thinking)",
     "score": 8.04,
     "ci95": 1.07
    },
    {
     "model": "o1 (December 2024)",
     "score": 7.96,
     "ci95": 1.06
    },
    {
     "model": "claude-opus-4-1-20250805",
     "score": 7.92,
     "ci95": 1.06
    },
    {
     "model": "Claude Sonnet 4 (Thinking)",
     "score": 7.76,
     "ci95": 1.05
    },
    {
     "model": "claude-sonnet-4-5-20250929",
     "score": 7.52,
     "ci95": 1.03
    },
    {
     "model": "gpt-5.1-instant",
     "score": 6.8,
     "ci95": 0.99
    },
    {
     "model": "Claude Opus 4",
     "score": 6.68,
     "ci95": 0.98
    },
    {
     "model": "Gemini 2.0 Flash Thinking (January 2025)",
     "score": 6.56,
     "ci95": 0.97
    },
    {
     "model": "Llama 4 Maverick",
     "score": 5.68,
     "ci95": 0.91
    },
    {
     "model": "Claude Sonnet 4",
     "score": 5.52,
     "ci95": 0.9
    },
    {
     "model": "GPT 4.5 Preview",
     "score": 5.44,
     "ci95": 0.89
    },
    {
     "model": "GPT-4.1",
     "score": 5.4,
     "ci95": 0.89
    },
    {
     "model": "Gemini-1.5-Pro-002",
     "score": 4.6,
     "ci95": 0.82
    },
    {
     "model": "Mistral Medium 3",
     "score": 4.52,
     "ci95": 0.81
    },
    {
     "model": "Nova Pro",
     "score": 4.4,
     "ci95": 0.8
    },
    {
     "model": "Claude 3.5 Sonnet (October 2024)",
     "score": 4.08,
     "ci95": 0.78
    },
    {
     "model": "Nova Lite",
     "score": 3.64,
     "ci95": 0.73
    },
    {
     "model": "GPT-4o (November 2024)",
     "score": 2.72,
     "ci95": 0.64
    }
   ]
  },
  {
   "id": "aa",
   "runner": "Artificial Analysis",
   "who": "Independent benchmarking firm",
   "question_set": "Text-only subset, 2,158 questions",
   "questions": 2158,
   "tools": "Not stated on the page. Its page for HLE-Diamond says it runs every question without tools.",
   "dated": "The page gives no date for any row.",
   "source_url": "https://artificialanalysis.ai/evaluations/humanitys-last-exam",
   "verified_on": "2026-11-11",
   "note": "Top 12 of 27 models, read from the page's chart.",
   "rows": [
    {
     "model": "Claude Fable 5.2 (max)",
     "score": 74.3
    },
    {
     "model": "Claude Opus 5.5 (max)",
     "score": 61.4
    },
    {
     "model": "Claude Fable 5.1 (max)",
     "score": 59.1
    },
    {
     "model": "Gemini 4 Argon (high)",
     "score": 57.1
    },
    {
     "model": "Claude Fable 5",
     "score": 55.5
    },
    {
     "model": "Claude Sonnet 5.5 (max)",
     "score": 55.0
    },
    {
     "model": "Claude Opus 5 (max)",
     "score": 54.9
    },
    {
     "model": "GPT-6 Astra (max)",
     "score": 54.7
    },
    {
     "model": "Muse Wonder Full (max)",
     "score": 53.6
    },
    {
     "model": "GPT-6.1 Sol (max)",
     "score": 52.9
    },
    {
     "model": "Kimi K3.1 Pro",
     "score": 52.5
    },
    {
     "model": "Muse Wonder (max)",
     "score": 50.1
    }
   ]
  }
 ],
 "excluded": [
  {
   "model": "Grok 4 Heavy",
   "score": 44.4,
   "date": "2025-07-09",
   "source_url": "https://x.ai/news/grok-4",
   "reason": "Several agents at once. xAI: 'parallel test-time compute, which allows Grok to consider multiple hypotheses at once'.",
   "setting": "With tools, full set"
  },
  {
   "model": "Grok 4 Heavy",
   "score": 50.7,
   "date": "2025-07-09",
   "source_url": "https://x.ai/news/grok-4",
   "reason": "Text-only subset, per xAI, and several agents at once. The Last Line dated it 3 Apr 2025, which is the date of the question set.",
   "setting": "With tools"
  },
  {
   "model": "Kimi K2 Thinking (heavy)",
   "score": 51.0,
   "date": "2025-11",
   "source_url": "https://huggingface.co/moonshotai/Kimi-K2-Thinking",
   "reason": "Eight parallel runs combined, on text-only questions. Its single-run score with tools was 44.9, also text-only. The model card shows no date.",
   "setting": "With tools"
  },
  {
   "model": "Muse Spark (Contemplating)",
   "score": 58.4,
   "date": "2026-04-08",
   "source_url": "https://ai.meta.com/blog/introducing-muse-spark-msl/",
   "reason": "Meta: Contemplating mode 'orchestrates multiple agents that reason in parallel'. Without tools it scored 50.2. The single-model Thinking mode is on the record tables.",
   "setting": "With tools"
  },
  {
   "model": "Zoom Federated AI",
   "score": 48.1,
   "date": "2025-12-10",
   "source_url": "https://www.zoom.com/en/blog/humanitys-last-exam-zoom-ai-breakthrough/",
   "reason": "A system that combines several models: 'our federated multi-LLM framework'.",
   "setting": "Full set"
  },
  {
   "model": "Sup AI",
   "score": 52.15,
   "date": "2025-11-30",
   "source_url": "https://sup.ai/research/hle-white-paper-jan-9-2026",
   "reason": "Presented by its makers as a system that combines several models. The page would not load when I checked, so the number is unverified.",
   "setting": ""
  },
  {
   "model": "GLM-5",
   "score": 50.4,
   "date": "2026-02-12",
   "source_url": "https://z.ai/blog/glm-5",
   "reason": "Text-only subset. Z.ai: 'By default, we report the text-only subset; results marked with * are from the full set.'",
   "setting": "With tools"
  },
  {
   "model": "GLM-5.1",
   "score": 52.3,
   "date": "2026-04-07",
   "source_url": "https://z.ai/blog/glm-5.1",
   "reason": "Text-only subset, by the same convention. Without tools: 31.0.",
   "setting": "With tools"
  },
  {
   "model": "GLM-4.7",
   "score": 42.8,
   "date": "2025-12-22",
   "source_url": "https://z.ai/blog/glm-4.7",
   "reason": "Text-only subset: Z.ai's GLM-5 post repeats this figure without the full-set mark.",
   "setting": "With tools"
  },
  {
   "model": "GLM-4.5",
   "score": 14.4,
   "date": "2025-07-28",
   "source_url": "https://z.ai/blog/glm-4.5",
   "reason": "Text-only. Z.ai: 'only the text-based questions were evaluated'. Same for GLM-4.5-Air at 10.6.",
   "setting": "No tools"
  },
  {
   "model": "Qwen3.6-Plus",
   "score": 50.6,
   "date": "2026-04-01",
   "source_url": "https://qwen.ai/blog?id=qwen3.6",
   "reason": "The page does not say which question set was used. The Last Line listed 43.4, which is the Claude Opus 4.5 column of the same table. Without tools: 28.8.",
   "setting": "With tools"
  },
  {
   "model": "Gemini 2.5 Pro Experimental",
   "score": 18.8,
   "date": "2025-03-25",
   "source_url": "https://blog.google/innovation-and-ai/models-and-research/google-deepmind/gemini-model-thinking-updates-march-2025/",
   "reason": "Published nine days before the exam was finalised at 2,500 questions. Scale AI's table lists 18.16 for the final set.",
   "setting": "No tools"
  },
  {
   "model": "Gemini 4 Argon",
   "score": 57.1,
   "date": "",
   "source_url": "https://artificialanalysis.ai/evaluations/humanitys-last-exam",
   "reason": "An Artificial Analysis run on the 2,158 text-only questions. It is in the outside-runs table instead.",
   "setting": ""
  }
 ],
 "dropped": [
  {
   "model": "o3 (high)",
   "score": 24.9,
   "date": "2025-04-16",
   "source_url": "https://openai.com/index/introducing-o3-and-o4-mini/",
   "reason": "The linked OpenAI post has no HLE figure. xAI's chart of 9 Jul 2025 carries 24.9; OpenAI's own figure, 24.3, is on the record table.",
   "setting": ""
  },
  {
   "model": "Claude Fable 5 w/Tools",
   "score": 36.8,
   "date": "2026-07-09",
   "source_url": "https://www.anthropic.com/claude/mythos",
   "reason": "The linked page does not contain this number. Anthropic's Sept 2026 table gives Fable 5 as 63.8 with tools.",
   "setting": ""
  },
  {
   "model": "Claude Opus 4.5 w/Tools",
   "score": 43.2,
   "date": "2025-11-24",
   "source_url": "https://www.vellum.ai/blog/claude-opus-4-5-benchmarks",
   "reason": "A third-party page giving 'approximately 43.2%'. Replaced by Anthropic's own 43.4.",
   "setting": ""
  },
  {
   "model": "ERNIE 5.0",
   "score": 39.0,
   "date": "2025-01-21",
   "source_url": "https://llm-stats.com/benchmarks/hle",
   "reason": "An aggregator lists it as self-reported and undated; I found no page from Baidu with the number. The original dated it January 2025, ten months before the model existed.",
   "setting": ""
  },
  {
   "model": "MiniMax M2.1",
   "score": 22.0,
   "date": "2025-09-08",
   "source_url": "https://llm-stats.com/benchmarks/hle",
   "reason": "Not on the linked page.",
   "setting": ""
  },
  {
   "model": "DeepSeek-V3.2-Speciale",
   "score": 30.6,
   "date": "2025-12-01",
   "source_url": "https://llm-stats.com/benchmarks/hle",
   "reason": "Not on the linked page.",
   "setting": ""
  },
  {
   "model": "DeepSeek Reasoner",
   "score": 25.1,
   "date": "2025-12-01",
   "source_url": "https://llm-stats.com/benchmarks/hle",
   "reason": "Not on the linked page. Other labs' tables give 25.1 for DeepSeek V3.2 on the text-only subset.",
   "setting": ""
  },
  {
   "model": "Nemotron 3 Nano 30B",
   "score": 15.5,
   "date": "2025-12-15",
   "source_url": "https://scale.com/leaderboard/humanitys_last_exam",
   "reason": "Not on Scale AI's table.",
   "setting": ""
  },
  {
   "model": "GPT-5.3 Codex",
   "score": 39.9,
   "date": "2026-02-05",
   "source_url": "https://artificialanalysis.ai/models/gpt-5-3-codex",
   "reason": "An Artificial Analysis figure on the text-only subset. Not re-checked.",
   "setting": ""
  }
 ]
}