AI Claimed Six Perfect Scores. Only Two Were Verified · Issue #170

Opening

The International Mathematical Olympiad (IMO) wrapped up in Shanghai on July 16th. Of the 666 students who took the exam, 7 scored a perfect 42 out of 42.

But the following week made even more noise. Within seven days, six different AI systems announced, “We got a perfect 42 too.” Huawei, Xiaohongshu, and four more models from independent testing.

Reader, let me flag one thing here. Of those six, only two actually submitted their answers to the IMO organizers. The other four were graded not by humans, but by a Claude-based agent. And the repository behind those results states, in its own words, that they’re “strong but not officially certified.”

Let me give you the conclusion up front. What AI is missing right now isn’t harder problems. It’s a system that records who graded the answers, under what conditions, and how. In other words: not benchmarks, but exams.

A Season of Perfect Scores, Held in Six Different Rooms

Let’s first sort out what actually happened.

The human results are unambiguous. 7 out of 666 scored perfect. Students solved 6 problems over 9 hours across two days—same day, same time, same room, for everyone. The AI side splits into two very different tracks.

Official grading track: Huawei’s Celia and Xiaohongshu’s dots-note-3.0 received the problems after the human competition ended, submitted answers within a set time limit, and passed those answers on to the IMO organizers. Xiaohongshu stated that it “strictly prohibited any form of human intervention.”

Independent testing track: The other four came out of a testing environment one researcher built and ran independently.

Model	First Attempt	Final	Cost	Time Spent
Claude Fable 5	42/42	42/42	~$51	2.5 hours
GPT-5.6 Sol (xhigh)	39/42	42/42	~$21	3.8 hours
Kimi K3	36/42	42/42	~$31	17.4 hours
AxiomProver	Formal proof	42/42	Undisclosed	~25 hours

The numbers alone are startling. Fable 5 nailed it on the first attempt, in under a third of the 9 hours students were given. AxiomProver is a different animal—it turned all 6 problems into machine-verifiable proofs written in Lean 4¹, a formal proof language, spanning 7,722 lines total. Problem 3 alone took 4,229 lines and 869 minutes.

But what kept bothering me as I looked at this table wasn’t the scores—it was the conditions.

Bench The grader for the four models wasn’t a human medalist—it was an AI agent. Depending on how the effort tier was set, the same model’s score swung by 11 points. And here’s the most uncomfortable part: on the same test, Meta’s Muse Spark and DeepSeek V4 Pro produced the exact same wrong answer on Problem 3.

In a human exam room, this would trigger an investigation immediately. Two answers failing in the exact same way strongly suggests they weren’t solved independently. It could be a sign of overlapping training data. But benchmarks have no procedure at all for handling this. Both just get logged as “one wrong answer,” and that’s the end of it.

In 1062, Goryeo Erased Names From Exam Papers

Let’s rewind the clock quite a bit here.

When we talk about exams as an institution, we usually think about the questions—what was asked, how hard it was, who got it right. But looking at the history of examination systems, the part that received the real, painstaking development wasn’t the questions. It was the procedures for preventing cheating and verifying results.

Screenshot 2026 07 27 01.33.04 The classic example is bongmi (封彌, “sealing”), an anonymization device from the Goryeo dynasty. The test-taker’s status, name, age, and clan origin, written on the right side of the answer sheet, would be folded over and glued shut—so the grader couldn’t tell whose answer they were reading. It was established during China’s Song dynasty, and in Goryeo it was first implemented in 1062, the 16th year of King Munjong’s reign, on the recommendation of an official named Jeong Yu-san. It carried straight through into the Joseon dynasty.

And it didn’t stop there. Even with the name hidden, handwriting could still give someone away. So an additional device called yeokseo (易書, “transcription”) was added, having a third party recopy the entire answer sheet.

Why did people a thousand years ago bother building all this? It wasn’t distrust of the graders themselves. It was because giving an exam result social legitimacy required the ability to later scrutinize exactly how that result had been produced.

Passing the gwageo (科擧), the imperial civil service exam, wasn’t just a score—it was a qualification that led to actual government office. The moment an exam confers a qualification, it stops being a measurement and becomes an institution. And once it’s an institution, what it needs isn’t harder questions—it’s an appeals process, proctors, record-keeping, and someone accountable for the result.

Interestingly, the same logic shows up in how academic papers get evaluated. What’s called blind peer review hides the author’s institution or advisor precisely so those backgrounds can’t pressure or lend undue authority to the reviewer, so the paper gets judged on its own merits. And right now, AI evaluation is standing at exactly that same threshold.

On August 4th in Astana, AI Gets Proctored for the First Time

With that in mind, there’s an event starting next week that I find genuinely fascinating.

The 3rd International Olympiad in Artificial Intelligence (IOAI) runs from August 2nd to 8th in Astana, Kazakhstan. And for the first time this year, there’s a separate track called IOAI²: AI Model Track, where AI systems register as competitors and compete on a leaderboard entirely separate from human participants. Registration closes tomorrow, July 27th.

Reading through the rules, several interesting details stand out.

Fixed times: Two 6-hour sessions on August 4th and 6th. 3 problems per session.
No human intervention: Fully autonomous execution after the initial launch prompt.
Attempt limit: Up to 50 submissions per task.
Standardized grading environment: Execution happens on the participating organization’s own hardware, but grading runs on standardized GPUs.
Optional result disclosure: Percentile scores are delivered privately within 24 hours of each session; within 48 hours, organizations decide whether to publish under their name or stay anonymous.
Verification requirement: To be awarded a medal, teams must submit an execution trace² containing every model input/output, tool call, and prompt. This record is disclosed to no one outside the judging panel.
Entry fee: The first model per organization is free (two for sponsors); each additional model costs €25,000.

Compare this list against a benchmark, and the contrast becomes stark.

A benchmark can be taken anytime, any number of times, under any conditions. Only the score survives—the process leaves no trace. This track, on the other hand, fixes the testing time, limits the number of attempts, and requires submitting a process record to get results certified. And there’s a judging panel.

If bongmi was a device that hid the test-taker’s name from the grader, submitting an execution trace is a device that opens up the test-taking process to the judges. The direction is reversed, but the purpose is identical: making the result verifiable after the fact.

The €25,000 entry fee shouldn’t be read as just a revenue model either. Charging a fee means the act of taking the test now carries cost and accountability. An evaluation you can run for free, infinitely, behaves very differently from one you pay to register for and sit on a fixed date.

The Evidence That Benchmarks Are Breaking Down

You might ask, “Aren’t benchmarks still good enough?” The data from the last few years says no.

First, saturation³. GPQA Diamond climbed from 39% in late 2023 to 94% in early 2026—the usable range got exhausted in barely two years. AIME 2025 shows the same pattern: top models are all clustered in the high 80s to mid-90s. Once that happens, a difference in score stops meaning a difference in ability. GPQA Diamond has 198 questions, and at that sample size the 95% confidence interval is roughly ±3 points. Which means most marketing claims about ranking models by a 3-point gap are statistically meaningless.

Second, contamination⁴. When Scale AI built GSM1k, a new question set matched in difficulty to GSM8K, and retested the same models, some dropped by as much as 13 points. And the size of that drop correlated with how often a given model would spit out GSM8K questions verbatim. In other words: they hadn’t learned the material, they’d memorized it.

Third, the judge problem. When an LLM is used as a grader, a bias appears where longer answers score 15 to 30 points higher, regardless of quality. There’s also a bias where preference shifts depending on the order options are presented in. These aren’t random noise—they’re directional biases, and once optimization pressure is applied, models will drift straight toward exploiting them.

Put it all together and there’s one conclusion: a leaderboard number isn’t a result. It’s a hypothesis.

Oz’s Lens

I look at this problem from a slightly different angle. Not as a question of evaluation methodology, but as a question of purchasing decisions.

There’s a scene I’ve watched play out countless times while building GTM (go-to-market) strategy. A vendor shows up with a benchmark score. The person in charge copies that number into an internal approval document. Two months later, when performance doesn’t hold up in the real workflow, no one can be held accountable using that number as evidence—because no one ever knew what conditions produced that score in the first place. I’ve seen this happen repeatedly, and I keep arriving at the same conclusion: a benchmark score isn’t procurement evidence, it’s a marketing asset.

What a company actually needs when adopting AI isn’t “our model scored 94.” It’s “this result was produced under these conditions, at this time, through this process—and the record still exists.” Think of financial audits: we don’t just take a company’s earnings report at face value. We trust it because there’s an auditor, there are audit working papers, and there’s a record we can go back and examine.

AI performance claims still don’t have that infrastructure. This is exactly why something like IOAI² is interesting to me. It’s not an attempt to ask harder questions—it’s an attempt to attach auditability to the score itself. I think that direction matters far more right now.

Of course, I don’t expect this to work perfectly in its very first edition. Running execution on the organization’s own hardware while grading only on a standardized environment still leaves gaps. Allowing 50 attempts means there’s still room to cherry-pick the best result. A €25,000 entry fee favors well-resourced organizations. Even so, I think the direction is right. Institutions always start out leaky and grow by patching the holes—bongmi itself started by hiding just the name, and only later grew to hide the handwriting too.

Closing

Here’s the summary.

At IMO 2026, 7 of 666 human competitors scored perfect—and six AI systems claimed the same. Only two of them went through the official grading process.
Benchmarks are already losing their discriminating power, for three reasons: saturation, contamination, and judge bias.
The next step isn’t harder questions—it’s an exam system with defined testing conditions and verification procedures. That first experiment begins in Astana this August.

After reading this, try one thing. Next time someone brings you an AI performance number, don’t ask what the score is. Ask this instead: “Who graded that score, when, under what conditions, and does a record of it still exist?” You’ll be surprised how often the answer stalls out.

If you’ve had a moment where an AI tool’s benchmark score didn’t match its real-world performance, tell me in the comments where the gap was biggest. If enough stories come in, I’ll compile them into a “performance verification checklist for practitioners” in a future issue.

💬 Share in the comments the moment a benchmark and reality didn’t match · 📨 If a colleague is considering adopting AI, forward this issue to them

References & Further Reading

Primary sources

IOAI, “AI Model Track: Rules and Competition Format,” International Olympiad in Artificial Intelligence 2026. Link ··· The execution trace submission and verified scoring clauses are the core evidence behind today’s piece. Reading the rules themselves makes the design difference between an “exam” and a “benchmark” far clearer.
Axiom Math, “AxiomProver at IMO 2026,” GitHub, 2026. Link ··· The complete Lean 4 formal proofs for all 6 problems are published here. If you’re curious what a machine-verifiable proof actually looks like, open the Problem 3 file.
Digital Applied, “Four AIs Scored a Perfect 42/42 on IMO 2026. So What?”, 2026. Link ··· A table breaking down cost, time, retry count, and grading authority by model. Lets you distinguish the details that press coverage tends to blur together.
Acing AI, “The LLM Evaluation Crisis: Contamination, Saturation, and the Judge Problem,” 2026. Link ··· The GPQA Diamond confidence interval figures and the GSM1k replication numbers both come from here. If you work with benchmarks, the statistics section alone is worth reading.
AFP, “AI catches up with humans to score 100% at top maths contest,” July 23, 2026. Link ··· A wire-service report on the Huawei and Xiaohongshu announcements. Also a good case study in how the press translates corporate announcements.

Background

「Bongmi (封彌)」, Encyclopedia of the Annals of the Joseon Dynasty, The Academy of Korean Studies. ··· Lays out how bongmi was introduced under King Munjong of Goryeo in 1062. Good confirmation that what mattered most in exam systems was procedure, not the questions.
“China’s Imperial Examination System,” Wikipedia (Korean). Link ··· A quick way to skim the full context of the Song-dynasty examination reforms.

📝 Glossary

The author, Kwangseob Ahn, is a professor of business administration at Sejong University and lead consultant at OBF (Oswarld Boutique Consulting Firm). He teaches statistics and data analysis, including business data management and business analytics, while leading GTM and AI strategy consulting in the field, designing the seam between technology and business. He has published academic research on a memory architecture for AI dialogue systems (HEMA) and runs Daily Arxiv, a daily curation of global AI papers. He holds a master's from Korea University's Graduate School of Technology Management and a KMBA. He is the author of Homo Brainless: The People Who Outsource Their Thinking.

Footnotes

Lean 4: A programming language for writing mathematical proofs in a form a computer can check line by line. Instead of a human reading it and deciding “this looks right,” a machine automatically catches logical gaps. ↩
Execution trace: A time-ordered record of every input, output, tool call, and prompt exchanged while an AI works through a problem. Think of it less as an answer sheet and more as CCTV footage of the exam room. ↩
Saturation: A phenomenon where model performance rises to the point that a test becomes too easy, and every top model clusters near a perfect score. Once this happens, score differences can no longer distinguish ability. ↩
Contamination: A situation where the questions used in an evaluation were already present in a model’s training data. It’s the equivalent of seeing the exam questions beforehand, so the score ends up reflecting memorization rather than ability. ↩