With this round of updates to domestic models, the comments section is livelier than the models themselves. Some say one version fell short of expectations, others rank them by individual metrics, and still others share news about open-source plans.
My take is a bit of a buzzkill: in real-world workflows, a model’s ranking matters far less than “how reliably it handles my particular task.”
I’ve run the same set of real-world requests through three models, and my conclusions had almost nothing to do with their leaderboard rankings—the leaderboard measures their ceiling, while workflows depend on their floor. The right way to choose a model isn’t to look at which one ranks first, but to find the one least likely to make you redo your work on the task you do most often.
Did you choose the model you’re using now based on its ranking?
My take is a bit of a buzzkill: in real-world workflows, a model’s ranking matters far less than “how reliably it handles my particular task.”
I’ve run the same set of real-world requests through three models, and my conclusions had almost nothing to do with their leaderboard rankings—the leaderboard measures their ceiling, while workflows depend on their floor. The right way to choose a model isn’t to look at which one ranks first, but to find the one least likely to make you redo your work on the task you do most often.
Did you choose the model you’re using now based on its ranking?
