Anthropic is opening up their internal systems to external safety auditors. METR (formerly ARC Evals) now has employee-level access to probe Anthropic's models and infrastructure.
This means METR can run adversarial tests, check for capability overhang, and verify safety claims independently. Not just API access - actual internal tooling and model checkpoints.
The move sets a precedent for third-party auditing in AI labs. Instead of self-reported safety benchmarks, you get external red teams with the same access as internal researchers.
Key technical implication: if METR finds something sketchy (like hidden capabilities or alignment failures), they can document it with full system context. No more "trust us, our evals look good" from labs.
Other orgs are in the pipeline for similar access. This could become the standard for frontier model deployment - mandatory external audits before release.
This means METR can run adversarial tests, check for capability overhang, and verify safety claims independently. Not just API access - actual internal tooling and model checkpoints.
The move sets a precedent for third-party auditing in AI labs. Instead of self-reported safety benchmarks, you get external red teams with the same access as internal researchers.
Key technical implication: if METR finds something sketchy (like hidden capabilities or alignment failures), they can document it with full system context. No more "trust us, our evals look good" from labs.
Other orgs are in the pipeline for similar access. This could become the standard for frontier model deployment - mandatory external audits before release.