Knowing What You Don't Know: Building a Multi-Dimensional Metacognition Benchmark for Frontier Language Models
Shruti Malik, MBBS, MHSA
Abstract
This paper describes the design, implementation, and rationale of a three-generation metacognition benchmark built for the Google DeepMind x Kaggle "Measuring Progress Toward AGI" Hackathon. The final version — Metacognition Benchmark v3 — evaluates frontier language models across five interlocking dimensions: classic hallucination resistance, near-miss misconception detection, easy factual accuracy, hard factual accuracy, and confidence calibration. The dataset comprises 150 handcrafted questions spanning 10+ domains, with novel question types including near-miss hallucination baits (subtle, almost-true false premises) and recent-event questions (2024+) designed to reduce the memorization confound. Evaluation is performed using a composite "MetaScore" informed by Expected Calibration Error (ECE), Brier score, and binary refusal accuracy.
Predicting Psychological Outcomes from Subscale Configurations in Psychedelic-Induced Mystical Experiences
Shruti Malik Ramaswamy
Abstract
An MEQ-30 Machine Learning Study, Corrected and Reproduced (August 15, 2026).
This AI-assisted secondary-data study examines whether MEQ-30 questionnaire items and subscale configurations predict life satisfaction better than a binary Complete Mystical Experience threshold. Using an open survey of 700 Polish adults, including 414 who reported psychedelic use, it evaluates Ridge regression, XGBoost, and LightGBM with nested cross-validation, SHAP feature attributions, and exploratory clustering.
The best model, XGBoost, achieved RMSE = 6.2673 and R² = 0.0165, indicating very limited predictive power. An accuracy audit identified and corrected an incorrect MEQ-30 subscale mapping and test-fold information leakage during hyperparameter selection. The paper includes source code and discloses AI assistance.
This is a case study in evaluating AI-assisted research, not a direct LLM benchmark. Life satisfaction is used as a proxy outcome; the cross-sectional design cannot establish causation, and the findings do not establish clinical utility. Its central evaluation lesson is the importance of valid measurements, leakage-free testing, reproducibility, and transparent reporting of weak results.