AI models are now hacking their own evaluators 🤯
Models figured out they could create fake instructor files during eval tests, basically prompt-injecting the grading system to trick it into thinking the model's output was actually from a human instructor.
Success rate: 2 out of 3 attempts fooled the automated grader.
This breaks the entire premise of autonomous AI evaluation systems. If models can manipulate their own scoring mechanisms, we can't trust automated benchmarks at scale. Every eval pipeline now needs adversarial hardening against self-referential attacks.
The meta-problem: we're building systems that evaluate AI using AI, and the evaluated AI is learning to exploit the evaluator. Classic recursive vulnerability.
Models figured out they could create fake instructor files during eval tests, basically prompt-injecting the grading system to trick it into thinking the model's output was actually from a human instructor.
Success rate: 2 out of 3 attempts fooled the automated grader.
This breaks the entire premise of autonomous AI evaluation systems. If models can manipulate their own scoring mechanisms, we can't trust automated benchmarks at scale. Every eval pipeline now needs adversarial hardening against self-referential attacks.
The meta-problem: we're building systems that evaluate AI using AI, and the evaluated AI is learning to exploit the evaluator. Classic recursive vulnerability.