From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs

The project addresses a critical gap in the LLM ecosystem: the disconnect between static benchmarks (MMLU, GSM8K) and the subjective 'feel' of model quality that drives user adoption. While currently a research artifact (0 stars, 4 forks, 2 days old), it attempts to bridge the gap between human intuition and data science. Its defensibility is low because it is primarily a methodological contribution rather than a tool with network effects or proprietary data. Competitors include established evaluation frameworks like LMSYS (Chatbot Arena), Giskard, and LangSmith. Frontier labs like OpenAI and Anthropic are already building 'vibe-aligned' evaluation suites internally to capture exactly what this project seeks to formalize. The primary risk is that as soon as these 'vibe' metrics are successfully formalized, they will be absorbed into standard evaluation platforms (Weights & Biases, Arize) or the labs' own developer dashboards, leaving a standalone research implementation with little moat beyond its initial insight.