AI’s New Ultimate Challenge: Humanity’s Last Exam Pushes Models to Their Limits

In the high-stakes world of artificial intelligence, where large language models like ChatGPT and Gemini are posting near-perfect scores on traditional benchmarks, researchers face a growing problem: how do you measure progress when the tests themselves become too easy?

Enter Humanity’s Last Exam (HLE), an audacious new benchmark developed by an international consortium that’s being called the “new SAT” for AI systems. With 2,500 meticulously crafted questions spanning everything from Roman inscriptions to hummingbird anatomy, this test represents the most challenging assessment yet for determining whether AI has truly achieved expert-level knowledge across academic disciplines.

The Benchmark Crisis: When AI Becomes Too Good at Tests

For years, researchers have relied on standardized benchmarks to track AI capabilities. These collections of questions, designed to be impossible to answer through simple web searches, have served as the primary metric for measuring whether algorithms are genuinely reasoning or merely pattern-matching.

But something unprecedented has happened: cutting-edge large language models are now regularly scoring over 90 percent on these tests. What was once a reliable way to measure progress has become obsolete, with AI systems essentially “gaming” the benchmarks by training on similar question formats or searching for information during the test itself.

“The problem has grown worse because as well as being trained on the entire internet, current AI systems can often search for information online during the test,” explain MIT researchers Katherine Collins and Joshua Tenenbaum, who were not involved in the HLE project.

Humanity’s Last Exam: A New Standard for AI Evaluation

The HLE Contributors Consortium, working with the non-profit Center for AI Safety and Scale AI, took a radically different approach to benchmark design. They enlisted thousands of experts from 50 countries to submit graduate-level questions across mathematics, humanities, and natural sciences.

The questions underwent a rigorous multi-stage selection process. Initially, roughly 70,000 submissions were tested against multiple AI models. Only those that consistently stumped the algorithms advanced to expert review, where specialists evaluated each question’s usefulness for AI assessment using strict guidelines.

What makes HLE particularly challenging is its answer format. Questions require either exact matches to correct solutions or multiple-choice responses, making automated scoring straightforward while eliminating the ambiguity that often plagues open-ended assessments.

The team has released 2,500 questions publicly while keeping the remainder private—a strategic move to prevent AI systems from training specifically on the benchmark and artificially inflating their scores.

Initial Results: AI Struggles with Expert-Level Knowledge

When HLE was first released in early 2025, the results were sobering. Leading AI models from Google, OpenAI, and Anthropic scored in the single digits, demonstrating just how far these systems still have to go to achieve true expertise.

As the benchmark gained attention, AI companies began using it to showcase their latest model improvements. While newer algorithms have shown progress, even the most advanced systems continue to struggle with expert-level questions.

OpenAI’s GPT-4o managed only a 2.7 percent success rate, while the company’s newer GPT-5 improved to 25 percent—still leaving vast room for advancement.

The Intelligence Question: What Does HLE Actually Measure?

Despite its ambitious scope, HLE has sparked intense debate about what constitutes “intelligence” in artificial systems and whether academic expertise translates to genuine understanding.

Critics argue that the test’s focus on expert-level academic problems doesn’t capture the messy, interdisciplinary thinking required in real-world scenarios. “Humanity is not contained in any static test, but in our ability to continually evolve both in asking and answering questions we never, in our wildest dreams, thought we would—generation after generation,” notes Subbarao Kambhampati, former president of the Association for the Advancement of Artificial Intelligence.

Others question whether HLE measures true intelligence or simply task performance. The benchmark emphasizes answering existing questions rather than evaluating whether an AI can identify meaningful problems, consider alternative interpretations, or assess its own confidence in responses—all crucial aspects of human expertise.

There’s also concern about the “study effect.” If AI companies train their models specifically on the public HLE dataset, the resulting score improvements might reflect test preparation rather than fundamental advances in capability.

Beyond Traditional Testing: The Future of AI Evaluation

The limitations of HLE have prompted researchers to explore entirely new approaches to measuring AI intelligence. Some are investigating ways to capture scientific creativity, collaborative problem-solving with humans, and real-world reasoning abilities that go beyond answering questions.

As MIT’s Collins and Tenenbaum observe, “HLE no doubt offers a useful window into today’s AI expertise. But it is by no means the last word on humanity’s thinking or AI’s capacity to contribute to it.”

The HLE team acknowledges these criticisms and continues refining the benchmark. Their goal isn’t to create a perfect test but to push the field toward more sophisticated evaluation methods that better capture the nuances of intelligence—both artificial and human.

The Race Continues: AI’s Quest for True Expertise

Despite its shortcomings, HLE represents a significant step forward in AI evaluation. It provides an objective, standardized way to measure improvement across diverse domains while highlighting the vast gap that still exists between current AI capabilities and genuine expertise.

As AI systems continue to evolve, benchmarks like HLE will likely become increasingly important for distinguishing between incremental improvements and fundamental breakthroughs. The test may eventually make itself obsolete—not because AI has mastered all knowledge, but because it will have forced the development of even more innovative paradigms for understanding artificial intelligence.

The ultimate question remains: can AI ever truly achieve the kind of flexible, creative, and self-aware intelligence that characterizes human expertise? Humanity’s Last Exam doesn’t answer that question, but it provides a crucial measuring stick for tracking progress toward that ambitious goal.

Tags: AI benchmark, artificial intelligence testing, large language models, ChatGPT evaluation, Gemini performance, machine learning assessment, AI intelligence measurement, academic AI challenges, expert-level AI, benchmark crisis, AI capabilities, language model testing, AI development, artificial general intelligence, AI progress tracking

Viral Sentences:

• “AI just got its hardest test yet—and the results are humbling”
• “The new SAT for AI is so tough even GPT-5 barely passes”
• “Humanity’s Last Exam reveals the shocking gap between AI and true expertise”
• “AI companies are scrambling as their models fail the ultimate intelligence test”
• “The benchmark crisis: When AI becomes too good at taking tests”
• “Is this the end of traditional AI evaluation as we know it?”
• “AI scored 2.7% on the hardest test ever created—here’s why that matters”
• “The intelligence question: What HLE really tells us about AI’s limitations”
• “From Roman inscriptions to hummingbird anatomy: The questions that stump AI”
• “The race to create AI that can think, not just answer questions”

,