CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

CARV is a pure research artifact: an academic benchmark paper introducing a diagnostic dataset and evaluation methodology for testing compositional analogical reasoning in MLLMs. Key defensibility challenges: **Why defensibility is low (2/10):** - Zero adoption signals (0 stars, 0 forks, 8 days old). Purely academic release. - The contribution is a dataset + benchmark methodology, not proprietary code or infrastructure. - No novel algorithmic innovation—it applies existing MLLM evaluation patterns (inference + accuracy scoring) to a new task domain. - The benchmark itself is reproducible by anyone with access to public MLLMs and image generation tools. - Academic benchmarks are typically public goods; reimplementing or extending it requires minimal engineering effort. **Platform domination risk: HIGH** - OpenAI, Google DeepMind, Anthropic, and Meta are all actively building multimodal reasoning benchmarks and diagnostic tools. - These platforms control the inference APIs used to evaluate models and will naturally integrate compositional reasoning tests into their own evaluation suites. - CARV will likely be absorbed into broader MLLM evaluation frameworks (like MMLU extensions, vision-language leaderboards) within 12-18 months. - The dataset itself has limited moat; once published, platforms can incorporate it directly or create competing benchmarks with superior coverage. **Market consolidation risk: LOW** - No commercial benchmark provider currently dominates compositional reasoning evaluation in MLLMs. - However, this is not because of market fragmentation—it's because the space is too nascent for consolidation. - Academic institutions (CMU, Stanford, Berkeley, DeepMind) release benchmarks freely, preventing vendor lock-in. **Displacement horizon: 1-2 years** - Platforms will likely absorb compositional reasoning tests into their native evaluation tools within 12-18 months. - The benchmark could achieve modest academic adoption (citations, use in model papers) but will not become a defensible business asset. - Survival path: become a standard reference in the literature (like ImageNet for vision or GLUE for NLP), but this does not create defensibility—it creates a public good. **Why the novelty is novel_combination, not breakthrough:** - Analogical reasoning is well-established in cognitive science and AI (Hofstadter, analogy-making literature). - Compositional learning from multiple sources is an existing theme in multi-task learning and meta-learning. - The contribution is combining these with multimodal evaluation: a useful but not fundamentally new angle. - The benchmark follows standard academic methodology (data collection, annotation, baseline runs). **Composability: algorithm** - The benchmark is usable as an evaluation suite within other research projects and MLLM development pipelines. - But it is not a component, library, or framework—it is a reference standard and dataset. - Researchers integrate it by running their models and reporting scores; there is no API, no SDK, no installed dependency.