Lesson plan

MMMU

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a benchmark for evaluating multimodal large language models on tasks that require college-level subject knowledge and deliberate reasoning. It was released in November 2023 by a team of 22 researchers led by Xiang Yue at Ohio State University and Wenhu Chen at the University of Waterloo, and was presented as an oral paper at CVPR 2024. The benchmark comprises 11.5 thousand multimodal questions manually collected from college exams, quizzes, and textbooks, spanning six disciplines — Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering — covering 30 subjects and 183 subfields. The questions incorporate highly heterogeneous image types, including charts, diagrams, maps, tables, music sheets, and chemical structures, and are designed to test expert-level perception, knowledge, and reasoning beyond commonsense visual understanding. At the time of release, the best-performing models achieved only around 56% accuracy, well below the human expert range of 76–89%. MMMU has since become a standard evaluation for frontier multimodal models, with scores routinely reported in the technical reports of systems such as GPT-4o, Gemini, and Qwen-VL.

Back to article

Article summary

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a benchmark for evaluating multimodal large language models on tasks that require college-level subject knowledge and deliberate reasoning. It was released in November 2023 by a team of 22 researchers led by Xiang Yue at Ohio State University and Wenhu Chen at the University of Waterloo, and was presented as an oral paper at CVPR 2024. The benchmark comprises 11.5 thousand multimodal questions manually collected from college exams, quizzes, and textbooks, spanning six disciplines — Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering — covering 30 subjects and 183 subfields. The questions incorporate highly heterogeneous image types, including charts, diagrams, maps, tables, music sheets, and chemical structures, and are designed to test expert-level perception, knowledge, and reasoning beyond commonsense visual understanding. At the time of release, the best-performing models achieved only around 56% accuracy, well below the human expert range of 76–89%. MMMU has since become a standard evaluation for frontier multimodal models, with scores routinely reported in the technical reports of systems such as GPT-4o, Gemini, and Qwen-VL.

Key claims

  1. release date: 2023 — source: Wikipedia, revision 1378911467, retrieved 2026-10-21 · confidence 0.95

Discussion questions

  1. Summarise the article in your own words. Which single claim would you verify first, and why?
  2. The record states that the release date is "2023". What evidence would count against this claim?
  3. Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
  4. If two sources disagreed about one of these facts, what would you need in order to trust one over the other?

Bibliography

  1. Wikipedia, “MMMU”, revision 1378911467, retrieved 2026-10-21 https://en.wikipedia.org/w/index.php?oldid=1378911467

Key claims and bibliography are generated from the same tracked claims and sources as the article page.