Peter Gostev compared Fable 5.2 and Opus round by round across more than 100 generation tasks.
Crypto Briefing’s report only names the two models, gives the scale of the comparison, and says there were “more than 100 generations.” It does not disclose the specific scoring criteria, number of wins, or types of test tasks.
· Treating the number of generations as a sample size tells us how many tests were run; it is not a quantitative conclusion about model capabilities.
· Without item-by-item results, claims about which model is stronger are merely the host’s take on this comparison, not a conclusion.
If you were only shown the overall win rate, would you want to see the item-by-item scores or the raw generation samples first?
Crypto Briefing’s report only names the two models, gives the scale of the comparison, and says there were “more than 100 generations.” It does not disclose the specific scoring criteria, number of wins, or types of test tasks.
· Treating the number of generations as a sample size tells us how many tests were run; it is not a quantitative conclusion about model capabilities.
· Without item-by-item results, claims about which model is stronger are merely the host’s take on this comparison, not a conclusion.
If you were only shown the overall win rate, would you want to see the item-by-item scores or the raw generation samples first?
