Research-driven evaluation of frontier language models — measuring what AI knows about what it knows, through multi-dimensional metacognition benchmarks, hallucination resistance testing, and confidence calibration analysis.
Introduction to Evaluations
Evaluations are structured tests of what an AI system can do, where it fails, and how well its confidence matches its accuracy. For language models, this includes factual accuracy, hallucination resistance, recognizing uncertainty, and confidence calibration—not just producing convincing answers.
Reliable evaluation uses clear criteria, appropriate baselines, and held-out data that do not leak into training or model selection. Reporting limitations and reproducible methods matters as much as reporting a score. An AI audit can uncover errors, but it does not by itself prove correctness or clinical safety.
The papers below explore two complementary perspectives: directly benchmarking LLM metacognition, and auditing an AI-assisted MEQ-30 machine-learning study. The latter shows how correcting measurement errors and data leakage can produce a more honest assessment of predictive performance.
Research
Knowing What You Don't Know: Building a Multi-Dimensional Metacognition Benchmark for Frontier Language Models
This paper describes the design, implementation, and rationale of a three-generation metacognition benchmark built for the Google DeepMind x Kaggle "Measuring Progress Toward AGI" Hackathon. The final version — Metacognition Benchmark v3 — evaluates frontier language models across five interlocking dimensions: classic hallucination resistance, near-miss misconception detection, easy factual accuracy, hard factual accuracy, and confidence calibration. The dataset comprises 150 handcrafted questions spanning 10+ domains, with novel question types including near-miss hallucination baits (subtle, almost-true false premises) and recent-event questions (2024+) designed to reduce the memorization confound. Evaluation is performed using a composite "MetaScore" informed by Expected Calibration Error (ECE), Brier score, and binary refusal accuracy.
Predicting Psychological Outcomes from Subscale Configurations in Psychedelic-Induced Mystical Experiences
An MEQ-30 Machine Learning Study, Corrected and Reproduced (August 15, 2026). This AI-assisted secondary-data study examines whether MEQ-30 questionnaire items and subscale configurations predict life satisfaction better than a binary Complete Mystical Experience threshold. Using an open survey of 700 Polish adults, including 414 who reported psychedelic use, it evaluates Ridge regression, XGBoost, and LightGBM with nested cross-validation, SHAP feature attributions, and exploratory clustering. The best model, XGBoost, achieved RMSE = 6.2673 and R² = 0.0165, indicating very limited predictive power. An accuracy audit identified and corrected an incorrect MEQ-30 subscale mapping and test-fold information leakage during hyperparameter selection. The paper includes source code and discloses AI assistance. This is a case study in evaluating AI-assisted research, not a direct LLM benchmark. Life satisfaction is used as a proxy outcome; the cross-sectional design cannot establish causation, and the findings do not establish clinical utility. Its central evaluation lesson is the importance of valid measurements, leakage-free testing, reproducibility, and transparent reporting of weak results.