Union's CEO Samvit Ramadurgam calls out a blind spot in current AI alignment research: we're obsessing over intent but ignoring outcomes.
The argument: alignment today focuses on whether an AI "means well" (via RLHF, constitutional AI, etc.), but there's zero systematic measurement of whether those well-intentioned actions actually produce good results in the real world.
Good intentions ≠ good outcomes. An aligned model can still cause cascading failures if we're not tracking system-level effects. Ramadurgam says outcome measurement is "totally missing" from the alignment discourse.
This is basically arguing for a feedback loop from deployment back into alignment training, not just pre-deployment evals. If your aligned chatbot gives medical advice that sounds helpful but is statistically harmful, current methods won't catch it unless you're measuring downstream impact.
The gap: we have benchmarks for helpfulness and harmlessness, but no standardized framework for "did this actually make things better in aggregate?"
Implication for builders: alignment can't stop at model behavior. You need instrumentation for real-world consequences and a way to feed that signal back into your system.
The argument: alignment today focuses on whether an AI "means well" (via RLHF, constitutional AI, etc.), but there's zero systematic measurement of whether those well-intentioned actions actually produce good results in the real world.
Good intentions ≠ good outcomes. An aligned model can still cause cascading failures if we're not tracking system-level effects. Ramadurgam says outcome measurement is "totally missing" from the alignment discourse.
This is basically arguing for a feedback loop from deployment back into alignment training, not just pre-deployment evals. If your aligned chatbot gives medical advice that sounds helpful but is statistically harmful, current methods won't catch it unless you're measuring downstream impact.
The gap: we have benchmarks for helpfulness and harmlessness, but no standardized framework for "did this actually make things better in aggregate?"
Implication for builders: alignment can't stop at model behavior. You need instrumentation for real-world consequences and a way to feed that signal back into your system.