Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges

This project appears to be the repository for a major survey or research paper on reward hacking, as evidenced by its connection to arXiv and the rapid influx of forks (23) within just 48 hours despite having zero stars. The defensibility is low (3) because it is primarily a theoretical contribution rather than a software moat; its value lies in its taxonomy and the researchers' insights rather than code that is hard to replicate. Frontier labs like OpenAI and Anthropic are the primary actors dealing with reward hacking and often conduct this research internally (e.g., Anthropic's Constitutional AI or OpenAI's research on sparse autoencoders for feature identification). However, external academic surveys like this are critical for the broader research ecosystem to standardize terminology. The high market consolidation risk reflects the fact that only a few organizations have the compute necessary to observe and study these emergent hacking behaviors at the frontier scale. The displacement horizon is set at 1-2 years because the transition from RLHF to newer paradigms like Direct Preference Optimization (DPO) or Kahneman-Tversky Optimization (KTO) may change the specific mechanics of reward hacking, rendering current reward-model-specific analyses less relevant.