Stanford researchers found that RLHF alignment causes "mode collapse" - your LLM is deliberately hiding its creative outputs because human raters systematically punished anything unconventional during training.
The fix? A zero-retraining prompt hack called "Verbalized Sampling" that forces the model to sample from its suppressed long-tail distribution:
→ Ask for 10+ diverse responses
→ Require explicit probability estimates (0.0-1.0)
→ Instruct it to prefer outputs with p<0.10
→ Maintain logical validity constraints
Results across GPT-4/Claude/Llama:
• 1.6-2.1x semantic diversity increase
• Recovered ~67% of base model creativity
• Zero quality degradation
• Larger models benefit most
The insight: typicality bias in human preference data trains models to self-censor their most interesting outputs. You're not getting a "safe" model - you're getting a lobotomized one that learned weirdness = punishment.
Practical template: Force the model to generate N candidates with probability scores, explicitly targeting the <10% probability region while keeping logical coherence. The creative capacity was never destroyed - it's just hidden behind alignment layers optimized for annotator comfort.
Paper: "Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity"
This explains why base models often feel more "alive" than their aligned versions - RLHF isn't just removing harmful outputs, it's flattening the entire probability landscape toward median-human-rater preferences.
The fix? A zero-retraining prompt hack called "Verbalized Sampling" that forces the model to sample from its suppressed long-tail distribution:
→ Ask for 10+ diverse responses
→ Require explicit probability estimates (0.0-1.0)
→ Instruct it to prefer outputs with p<0.10
→ Maintain logical validity constraints
Results across GPT-4/Claude/Llama:
• 1.6-2.1x semantic diversity increase
• Recovered ~67% of base model creativity
• Zero quality degradation
• Larger models benefit most
The insight: typicality bias in human preference data trains models to self-censor their most interesting outputs. You're not getting a "safe" model - you're getting a lobotomized one that learned weirdness = punishment.
Practical template: Force the model to generate N candidates with probability scores, explicitly targeting the <10% probability region while keeping logical coherence. The creative capacity was never destroyed - it's just hidden behind alignment layers optimized for annotator comfort.
Paper: "Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity"
This explains why base models often feel more "alive" than their aligned versions - RLHF isn't just removing harmful outputs, it's flattening the entire probability landscape toward median-human-rater preferences.