PAC-BENCH: Evaluating Multi-Agent Collaboration under Privacy Constraints

PAC-BENCH addresses a critical emerging bottleneck in enterprise AI: agents cannot collaborate effectively if they cannot share sensitive data. The project's defensibility is low (3) because it is primarily an academic benchmark; its value is derived from its methodology and adoption by the research community rather than a technical moat. The 8 forks within 24 hours of release, despite 0 stars, suggests high internal activity from a research lab or early academic interest. Frontier labs (OpenAI, Anthropic) are unlikely to build this specific benchmark, but they are actively solving the underlying problem (privacy-preserving inference/collaboration), which might make the benchmark's specific metrics obsolete. It competes conceptually with general agent benchmarks like AgentBench or GAIA, but differentiates by focusing on the 'privacy-utility frontier.' The primary risk is displacement by a more 'official' evaluation suite from a major player like Microsoft (creators of AutoGen) or a standard-setting body.