MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols

MTalk-Bench targets a critical bottleneck in the generative AI space: the lack of robust, multi-dimensional evaluation for native Speech-to-Speech (S2S) models like GPT-4o or Gemini Live. While traditional speech metrics (WER, CER) focus on transcription, MTalk-Bench attempts to quantify 'vibe' factors like paralinguistics (tone, emotion) and environmental robustness. The defensibility is low (3) because, despite the 12 forks indicating academic interest, the project currently lacks the 'data gravity' or institutional backing of an LMSYS or an MLCommons to become an industry standard. Frontier labs are the primary risk here; as they release S2S models, they typically ship proprietary or closed evaluation sets that the community defaults to. The novelty lies in the specific focus on multi-turn interactions, which is significantly more complex than single-turn audio prompting. However, without a hosted 'Arena' platform for public S2S comparison (which is cost-prohibitive due to audio inference/hosting costs), it remains a reference implementation for researchers rather than a market-shifting tool.