open-compass/VLMEvalKit

VLMEvalKit has established itself as an infrastructure-grade project in the vision-language model (VLM) space. Its defensibility stems from a 'maintenance moat'—the sheer engineering effort required to maintain compatibility with 220+ different model architectures and 80+ disparate benchmarks (MMMU, MathVista, AI2D, etc.). With 4,000+ stars and 600+ forks, it has high velocity and institutional backing from the OpenCompass/Shanghai AI Lab ecosystem. Frontier labs like OpenAI or Anthropic are unlikely to build this; they prefer being evaluated by neutral third parties rather than building the evaluation software themselves. The primary threat comes from platforms like Hugging Face, which could centralize evaluation via their 'Evaluate' library, but VLMEvalKit’s deep specialization in the nuances of multimodal scoring (e.g., OCR-based metrics, spatial reasoning) gives it a significant edge. The displacement horizon is long because any competitor would need to replicate thousands of hours of model-wrapper and dataset-parser development.