Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

Quantitative signals strongly indicate early-stage status: the repo shows ~0 stars, ~3 forks, near-zero velocity, and an age of 2 days. That typically means there is no established community, no proof of sustained usage, and no evidence that the benchmark has become a de-facto standard. Defensibility (score = 3/10): - What it is: A benchmark for evaluating multimodal copyright unlearning in LVLMs. Benchmarks can be defensible only if they become widely adopted and continuously maintained (data gravity, standardized tooling, and leaderboards). None of that exists yet given the extremely low traction and recency. - Why the moat is weak: The underlying concept (unlearning evaluation) is not inherently exclusive; evaluation harnesses are relatively easy to replicate once the benchmark definition and metrics are known. Even if the benchmark is technically strong, a platform or a top lab can reimplement it quickly. - No evidence of an ecosystem: With near-zero stars/velocity and very new age, there’s no sign of network effects (contributors adding models/metrics, third-party comparisons, or a standard leaderboard). Frontier risk (high): - This is directly aligned with what frontier labs increasingly care about: copyright/memorization safety, post-training mitigation, and evaluation protocols for multimodal models. - Frontier labs could integrate adjacent benchmark logic into their internal evaluation suites or release their own standardized harness, making this project’s relative differentiation fragile. - The “benchmarking multimodal unlearning” niche is narrow enough to survive only if it becomes the community standard; but given the current adoption signals, frontier labs are more likely to absorb or supersede it. Three-axis threat profile: 1) Platform domination risk = high - Who could replace it: OpenAI, Anthropic, Google (and large model vendors) can either (a) include similar metrics in their safety eval frameworks, or (b) publish an official benchmark suite for multimodal copyright unlearning. - Why high: The required components—test set creation, prompt/vision input design, and evaluation metrics—are not tied to proprietary data gravity. Platforms can recreate using their own copyrighted-image corpora and their own model variants. 2) Market consolidation risk = medium - While model vendors could standardize evaluation, the broader unlearning/safety evaluation space can fragment because every lab/model family may use slightly different threat models and datasets. - However, benchmarks tend to consolidate once one approach proves convenient and is endorsed by multiple stakeholders. 3) Displacement horizon = 6 months - Given (i) repo novelty is more about evaluation methodology than a hard-to-replicate algorithm, (ii) the repo is extremely new, and (iii) frontier labs can deploy internal eval suites quickly, this can be displaced on a sub-year timeline if larger labs publish competing benchmarks/metrics or incorporate it into internal tooling. Opportunities (what could increase defensibility if it succeeds): - If the benchmark ships with: (a) a stable, versioned dataset; (b) clear metrics and baselines; (c) an automated evaluation API/CLI; and (d) a leaderboard with continuous updates, it can gain adoption and become the de-facto standard. - If it captures genuinely hard multimodal nuances (e.g., cross-modal leakage beyond simple classification/regeneration tests) and demonstrates reproducible improvements across multiple unlearning methods, it becomes more valuable and harder to replace. Key risks: - Low adoption risk today: with 0 stars and negligible velocity, external validation is absent. - Replicability risk: evaluation benchmarks are typically straightforward to reproduce once disclosed. - Platform supersession risk: frontier labs have both incentives and resources to create or standardize their own unlearning evaluation suites. Overall: This looks like an important and timely research/benchmark artifact (supported by an arXiv paper reference), but current defensibility is low due to lack of traction, no visible ecosystem, and high likelihood of rapid reimplementation or absorption by frontier labs.