VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models

VSAS-BENCH addresses a critical gap in VLM evaluation: the transition from static, offline video QA to real-time streaming interaction. While current benchmarks (like Video-LLaVA or MVBench) focus on accuracy, VSAS-BENCH introduces metrics for 'proactiveness' and 'consistency,' which are essential for applications like wearable AI (Project Astra, GPT-4o) and robotics. However, the project currently has a low defensibility score (3) because it is primarily a research-oriented reference implementation with zero stars and high reproducibility. Its 9 forks suggest initial academic interest, but it lacks the industry-wide adoption needed for a moat. Frontier labs (OpenAI, Google) are the primary competitors here, as they are developing proprietary evaluation suites for their native multimodal streaming models. These labs are likely to define the 'de facto' standards for streaming latency and proactiveness, potentially sidelining independent academic benchmarks unless this gains massive community momentum quickly. The displacement horizon is set at 1-2 years, reflecting the speed at which frontier labs are moving toward 'native' streaming multimodal architectures.