kasahart/AudioLLMArena

AudioLLMArena is a direct implementation of the 'Chatbot Arena' (Elo-based human preference) pattern for the emerging category of native audio-language models (like GPT-4o, Gemini 1.5 Pro, and open-weights alternatives like Qwen-Audio or Salmonn). While the need for such a benchmark is significant given that traditional metrics like Word Error Rate (WER) fail to capture the nuance of native audio reasoning, the project currently has zero traction (0 stars, 0 forks, 0 days old). Its defensibility is near zero because the value of an 'Arena' is entirely dependent on the volume of human voters and the participation of top-tier model providers. If a major player like LMSYS (the creators of Chatbot Arena) or Hugging Face launches an audio-specific leaderboard, this repository will likely be rendered obsolete. Furthermore, frontier labs (OpenAI/Google) are increasingly building their own internal human-eval pipelines for audio, reducing the external market for such tools unless they achieve massive community 'Switzerland-style' neutrality and scale.