MMMU-Benchmark/MMMU

MMMU is a category-defining benchmark that has become the de facto standard for reporting the performance of frontier multimodal models (used by OpenAI for GPT-4V/o, Google for Gemini, and Anthropic for Claude 3). Its defensibility stems from its status as a 'gold standard' in academic and industrial leaderboards; while the code is a simple evaluation harness, the curated dataset of 11,500 college-level problems is difficult to replicate and even harder to displace once it gains industry-wide adoption. The project has strong network effects: models are compared against MMMU because everyone else uses it. However, the 'displacement horizon' is set to 1-2 years because benchmarks in AI suffer from inevitable saturation (models reaching human parity) and potential data contamination in training sets, which eventually necessitates the creation of 'MMMU-Pro' or similar successors. Frontier labs are unlikely to compete with the benchmark itself, as they rely on it for external validation of their own progress. The 555 stars and 50 forks indicate high prestige relative to the niche (benchmark repositories typically have lower star counts than the models they evaluate).