ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints

Quantitative signals indicate extremely low adoption and near-zero community traction: 0 stars, 7 forks, velocity reported as 0.0/hr, and the repo age is ~1 day. A benchmark repository at this stage is almost certainly in an early release/prototype phase and lacks the ecosystem signals (documentation maturity, stable interfaces, continuous updates, citations-to-code, third-party integrations) that typically create defensibility. Why defensibility is scored 3/10: - The project appears primarily benchmark-focused (DynAfford) rather than providing a durable, infrastructure-grade asset (e.g., a maintained dataset with strong licensing, standardized leaderboards, APIs, or an eval protocol used by many downstream training pipelines). Benchmarks can be defensible if they become de facto standards, but that requires time, adoption, and recurring maintenance—none of which is evidenced here. - The core idea—evaluating planning under affordance constraints in dynamic settings—sounds like an extension of existing embodied-agent evaluation paradigms. Without evidence of a novel data-collection method, unique environment generator, or specialized ground-truth affordance labeling pipeline, this is best characterized as incremental. - The code/implementation details are not provided in the prompt, so we must assume limited production readiness. With repo age of 1 day, it’s unlikely to have hardened evaluation scripts, reproducible environment generation, and stable baselines. Frontier risk assessment (high): - Frontier labs (OpenAI/Anthropic/Google) actively build embodied and agent planning benchmarks and are likely to incorporate or replicate evaluation suites, especially when the benchmark targets an important failure mode: agents that follow instructions but ignore affordance feasibility. This is exactly the kind of eval that can be added as a component to their internal eval harnesses. - Because it is a benchmark (rather than a unique underlying model/dataset that is hard to recreate), it is comparatively easy for a frontier lab to re-implement the evaluation logic and generate comparable scenarios. Three-axis threat profile: 1) Platform domination risk: high - Big platforms can absorb this by adding DynAfford-like evaluation to their existing agent evaluation suites. The competitive leverage would be internal (compute + agent development + proprietary environment simulators + standardized tooling), not the benchmark code itself. - Likely displacers: internal eval teams at OpenAI, Google DeepMind, Anthropic, or any large org already running embodied planning evaluations. They can implement comparable “unspecified affordance constraint” tests using their own simulation environments. 2) Market consolidation risk: medium - Benchmark ecosystems can consolidate around a few popular leaderboards (e.g., if DynAfford establishes a strong standard). However, consolidation is not guaranteed for small, new benchmarks. - Medium rather than high because many labs may maintain multiple benchmarks simultaneously; but if this becomes prominent, it could consolidate within embodied planning eval. 3) Displacement horizon: 6 months - Given the benchmark nature and the lack of visible adoption, the likely path is rapid replication: similar evaluation scenarios, metrics, and baselines can be built by adjacent research teams quickly. - A 6-month horizon is plausible for meaningful replication/adjacent benchmark saturation, especially if the underlying arXiv paper is already informing ongoing work. Key opportunities: - If DynAfford is paired with a truly reusable, well-specified dataset generation pipeline (or a strong reference implementation + leaderboards + baseline agents), it could gain traction and become a de facto standard. - Strong defensibility could emerge if the benchmark includes hard-to-recreate elements: expensive environment generation, unique affordance annotation/ground truth, or an evaluation protocol that becomes embedded in downstream training/evaluation tooling. Key risks: - Low momentum risk: with 0 stars and 1-day age, it may not overcome the “benchmark churn” problem where new eval suites don’t become widely adopted. - Replicability risk: other teams can implement similar constraints/metrics without relying on this repo, especially if the benchmark definition is not uniquely difficult. - Platform capture risk: frontier labs can incorporate the idea into their broader eval suite faster than smaller players can differentiate.