Anthropic and independent researchers have introduced a new way to measure something AI models have historically struggled to prove they’re good at: reasoning through problems that have no clean, verifiable answer. The Conceptual Reasoning Index (CRI), unveiled this week, combines three separate benchmarks to score how well large language models handle philosophical arguments, logical consistency, and decision-theory puzzles — the kind of thinking many researchers believe will matter most as AI systems take on a bigger role in managing their own risks.
Key takeaways
The Conceptual Reasoning Index combines three benchmarks — LMCA, ACCoRD, and DTBench capabilities — into a single 0-to-100 score of AI conceptual reasoning.
The top-performing model, Opus 5, scored 73.6 on the CRI, well below an estimated ceiling of around 91.
The LMCA dataset includes 560 position texts and 1,461 expert-rated arguments covering philosophy, decision theory, and AI risk.
Scores have climbed roughly linearly since late 2024 with no sign of leveling off.
The project was built in collaboration with Anthropic and researchers Emery Cooper and Caspar Oesterheld.
Introduction to the Conceptual Reasoning Index (CRI)
The CRI exists because some of the most important work AI systems might eventually do — helping humans understand and plan for advanced AI risk — can’t be checked against a clear right answer the way math or coding problems can. Its creators argue that many of the tasks tied to AI risk mitigation require the kind of argumentation used in philosophy and AI futurism rather than empirical verification, and that current training methods, which lean heavily on data with reliable feedback, tend to leave models weaker at exactly this kind of reasoning.
To close that measurement gap, the researchers built three separate AI risk benchmarks and rolled them into one aggregate score. The Conceptual Reasoning Index (CRI) aggregates three benchmarks — LMCA, ACCoRD, and DTBench capabilities — into a single number meant to capture a model’s overall conceptual reasoning ability. The index is publicly available at conceptualreasoning.ai, where the team says it will keep updating scores and methodology as new models and new benchmarks arrive.
Purpose of CRI in AI risk management
Why build this at all? The researchers behind the project say that once AI models can do risk-reduction work at a human-expert level, AI-assisted output in that space could dwarf what humans alone produce. That means how quickly models can be trained to reason well about governance, alignment, and cooperation failures involving AI may become a decisive factor in whether risks get addressed in time. Improving conceptual reasoning in AI models is considered important precisely because it targets that gap — the tasks that lack empirical feedback loops and therefore tend to be neglected by standard training.
Benchmark aggregation and availability
The CRI isn’t a single test — it’s a weighted composite. Currently, LMCA counts for 60% of the score, with ACCoRD and DTBench capabilities each contributing 20%. The team says it plans to add new benchmarks over time, retire ones that become saturated, and potentially rebalance those weights as models improve.
Detailed Breakdown of CRI Component Benchmarks
Each of the three components targets a different flavor of reasoning that’s hard to verify empirically but still measurable through expert judgment or logical rules.
LMCA dataset: scope and expert ratings
LMCA, short for Language Model Conceptual Argumentation, is built around curated, expert-rated arguments spanning decision theory, philosophy, and risks from advanced AI. The LMCA dataset contains 560 position texts paired with 1,461 expert-rated arguments written against those positions. Nearly all of the arguments were rated by conceptual researcher Emery Cooper, with a portion independently rated by at least one other researcher, producing 2,140 ratings in total. A validation set of roughly 50 arguments was rated independently by four to six people and then debated for seven to eight hours combined, a process meant to establish how much humans agree with each other before comparing them to model outputs.
Right now, LMCA only scores how well models judge existing arguments rather than generate new ones, though the team says it hopes to add an argument-generation measure down the line.
ACCoRD: measuring logical consistency in model beliefs
ACCoRD checks something different: whether a model’s own stated beliefs and preferences hold together logically. If a model reports a probability for event A and, separately, a probability for A and B occurring together, does it respect basic rules like P(A) being greater than or equal to P(A&B)? The full ACCoRD dataset contains close to 14,000 model-generated consistency constraints across 18 constraint types, run through an automated checker. Of those, 567 validated constraints made it into the CRI after manual review — a deliberately conservative filter meant to keep only the checks the researchers are confident in.
DTBench: decision-theory reasoning evaluation
DTBench tests decision-theory reasoning with 407 multiple-choice questions, most of them handcrafted by Caspar Oesterheld, who has published academic work on decision theory, and independently validated by Emery Cooper. The questions probe scenarios involving self-prediction and interactions with near-copies of an agent — the sort of thought experiments decision theorists have long debated. An additional 130 questions in the full DTBench suite measure decision-theoretic attitudes but were left out of the CRI score.
CRI Performance Results and Trends
Even the best models tested still fall well short of what the benchmarks consider a near-perfect score, though the gap has been closing steadily.
Current performance scores and comparison to ceilings
CRI scores range from 0 to 100, with 0 representing random guessing. The researchers estimate a realistic ceiling of around 91 rather than a full 100, because human ratings themselves carry some noise — even expert raters don’t agree with each other 100% of the time. The highest score recorded so far belongs to Opus 5, at 73.6, with a 95% confidence interval of plus or minus 2.1. That leaves a meaningful gap between current top performance and the estimated ceiling, suggesting there’s still real room for models to improve at conceptual reasoning before the benchmark itself becomes the limiting factor.
Trends in model performance improvement over time
What stands out in the data isn’t just where models are now, but how fast they’ve been moving. Model performance has been improving roughly linearly since late 2024, and the researchers report no sign of that trend flattening out. That steady climb matters for anyone trying to forecast when AI systems might become genuinely useful collaborators on the kind of argument-heavy work that AI safety research depends on.
Benchmark-specific saturation estimates
Not every component is moving at the same pace. Extrapolating current trends, the researchers loosely estimate that LMCA will start saturating in about a year. DTBench, on the other hand, is already close to its ceiling — one evaluated model, Fable 5, answered 98% of DTBench questions correctly. ACCoRD is the wildcard: the team says it’s genuinely uncertain about when that benchmark will saturate, since consistency errors can persist even as raw reasoning ability improves.
Significance and Collaborative Development of the CRI
Why does any of this matter beyond a leaderboard? Because the people building the Conceptual Reasoning Index see conceptual reasoning as a bottleneck skill for AI risk work specifically — the kind of work that involves reasoning about governance, alignment, and catastrophic cooperation failures where there’s no dataset of past outcomes to train on. Improving conceptual reasoning in AI models is considered important to help reduce advanced AI risks precisely because so much of that risk-mitigation work depends on argumentation rather than empirical testing.
The project was built through a collaboration involving Anthropic alongside conceptual researchers Emery Cooper and Caspar Oesterheld, who contributed both to the dataset design and the expert ratings that anchor the LMCA and DTBench components. That combination of institutional backing and domain-specific expert input is part of what distinguishes the CRI from more conventional AI benchmarks focused on math, coding, or general knowledge — areas where correctness can usually be checked automatically.
The live scores and methodology notes are hosted at conceptualreasoning.ai, and access to the underlying LMCA dataset is available through a request form, a structure that suggests the team wants outside researchers testing and challenging the benchmark rather than treating it as a closed internal metric.
FAQ
What is the Conceptual Reasoning Index (CRI)?
The CRI aggregates three benchmarks — LMCA, ACCoRD, and DTBench — to measure AI models’ conceptual reasoning abilities, producing a single 0-to-100 score.
What types of reasoning does the LMCA dataset evaluate?
LMCA evaluates models on judging conceptual arguments related to philosophy, decision theory, and AI risks, comparing model ratings against expert human ratings.
How is logical consistency tested in AI models by ACCoRD?
ACCoRD measures whether a model’s reported beliefs and preferences satisfy logical consistency constraints, such as basic probability rules, using 567 validated constraints in the CRI.
How well are current AI models performing on the CRI?
The current top model, Opus 5, scores around 73.6 out of 100 on the CRI, with scores across evaluated models improving roughly linearly since late 2024 and no sign yet of that growth slowing down.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.