MMMU

Uses your system voice; voice and rate in settings.

Snapshot

≈ 3 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a benchmark for evaluating multimodal large language models on tasks that require college-level subject knowledge and deliberate reasoning. It was released in November 20231 by a team of 22 researchers led by Xiang Yue at Ohio State University and Wenhu Chen at the University of Waterloo, and was presented as an oral paper at CVPR 2024. The benchmark comprises 11.5 thousand multimodal questions manually collected from college exams, quizzes, and textbooks, spanning six disciplines — Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering — covering 30 subjects and 183 subfields. The questions incorporate highly heterogeneous image types, including charts, diagrams, maps, tables, music sheets, and chemical structures, and are designed to test expert-level perception, knowledge, and reasoning beyond commonsense visual understanding. At the time of release, the best-performing models achieved only around 56% accuracy, well below the human expert range of 76–89%. MMMU has since become a standard evaluation for frontier multimodal models, with scores routinely reported in the technical reports of systems such as GPT-4o, Gemini, and Qwen-VL.

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a benchmark for evaluating multimodal large language models on tasks that require college-level subject knowledge and deliberate reasoning. It was released in November 2023 by a team of 22 researchers led by Xiang Yue at Ohio State University and Wenhu Chen at the University of Waterloo, and was presented as an oral paper at CVPR 2024. The benchmark comprises 11.5 thousand multimodal questions manually collected from college exams, quizzes, and textbooks, spanning six disciplines — Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering — covering 30 subjects and 183 subfields. The questions incorporate highly heterogeneous image types, including charts, diagrams, maps, tables, music sheets, and chemical structures, and are designed to test expert-level perception, knowledge, and reasoning beyond commonsense visual understanding. At the time of release, the best-performing models achieved only around 56% accuracy, well below the human expert range of 76–89%. MMMU has since become a standard evaluation for frontier multimodal models, with scores routinely reported in the technical reports of systems such as GPT-4o, Gemini, and Qwen-VL.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378911467 under CC BY-SA 4.0; modifications have been made.

References

  1. ↩Wikipedia release date (×1) · confidence 0.95 https://en.wikipedia.org/w/index.php?oldid=1378911467

Provenance

snapshot enwiki-001378911467 · captured 1791315980.0 · version history

Cite this page as: encyclopegia.org, “MMMU”, snapshot enwiki-001378911467. https://encyclopegia.org/a/mmmu