HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

HiL-Bench addresses a critical 'last-mile' problem in agentic workflows: the tendency of LLMs to hallucinate requirements instead of asking for missing information. While 0 stars indicates it is in its absolute infancy (4 days old), the 12 forks suggest it is likely being vetted by an academic research group or an early-stage internal dev team. Its defensibility is currently low because it is a benchmark/methodology rather than a proprietary technology; its value depends entirely on becoming a community standard. The moat is effectively zero until it achieves 'Data Gravity' (i.e., top-tier labs citing it). Frontier labs like OpenAI and Anthropic are internally optimizing for 'active clarification' (e.g., GPT-4o's interactive nature), which poses a high risk of this being absorbed into general model capabilities. However, a third-party, reproducible benchmark is necessary to verify these claims across vendors. The project competes conceptually with SWE-bench (which focus on execution) and GAIA (general assistant tasks), but fills a specific niche in collaborative AI. Its primary risk is platform domination—Microsoft or GitHub could easily integrate a similar 'clarification score' into their own agent evaluations, rendering a standalone benchmark less relevant.