Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

This project addresses a critical bottleneck in the 'System 2' reasoning paradigm (e.g., OpenAI o1, DeepSeek-R1): the tendency of Process Reward Models (PRMs) to be 'hacked' or 'Goodharted' during training or inference-time search. While the project introduces a novel three-tiered diagnostic framework for quantifying these vulnerabilities—such as the 'fluency-logic dissociation'—its defensibility is low (3) because it functions primarily as a research reference implementation accompanying an arXiv paper. With 0 stars but 8 forks, it shows early academic interest but lacks the community density or data gravity of a production-grade tool. Frontier labs like OpenAI and Anthropic are internally building nearly identical robustness-checking suites to ensure their reasoning models don't collapse under adversarial optimization. The risk of platform domination is high, as these diagnostic techniques are likely to be absorbed directly into the RLAIF (Reinforcement Learning from AI Feedback) pipelines of major model providers. Competitors include academic benchmarks like PRM8K and specialized evaluation frameworks from organizations like the UK AI Safety Institute or Scale AI.