Nathan Labenz (AI:AM host) points out a critical security gap in open-weight models: there's zero proof that knowledge can be reliably unlearned from trained weights, and refusal mechanisms are trivially bypassed once you have local access.
The core issue: weight modification is fundamentally different from never training on data in the first place. Once knowledge is encoded in the parameter space, it's essentially permanent - you can't just "delete" it without catastrophic forgetting or architectural changes.
This creates a massive tension between open-weight releases and safety constraints. Every refusal layer, every alignment fine-tune, every RLHF guardrail becomes reversible when users can directly manipulate the model's weights.
Labenz is calling for serious research into the difference between:
1. Pre-training exclusion (never learning X)
2. Post-training removal (unlearning X from weights)
Right now we're treating these as equivalent problems when they're architecturally distinct. The unlearning challenge needs way more rigorous study before we can make any claims about controllable open-weight models.
The core issue: weight modification is fundamentally different from never training on data in the first place. Once knowledge is encoded in the parameter space, it's essentially permanent - you can't just "delete" it without catastrophic forgetting or architectural changes.
This creates a massive tension between open-weight releases and safety constraints. Every refusal layer, every alignment fine-tune, every RLHF guardrail becomes reversible when users can directly manipulate the model's weights.
Labenz is calling for serious research into the difference between:
1. Pre-training exclusion (never learning X)
2. Post-training removal (unlearning X from weights)
Right now we're treating these as equivalent problems when they're architecturally distinct. The unlearning challenge needs way more rigorous study before we can make any claims about controllable open-weight models.