Someone conducted an "StarCraft" AI head-to-head test (Brood War Bench), pitting today’s mainstream large language models against each other in real-time strategy games. The result: none of the models performed above a beginner.
StarCraft: Brood War is the classic 1998 real-time strategy game and a longtime friend of AI research. In 2019, DeepMind’s AlphaStar even defeated professional players in this game. But back then it was a reinforcement-learning AI specifically trained for the task. This time the test is different: it lets general-purpose large models jump in directly as AI agents and see whether they can figure things out on their own—build a base, produce units, and fight.
The author, Ben Swerdlow, originally made a StarCraft version that could only be controlled through agents, just to play with friends. He didn’t expect that a few friends who had barely played the game would still do pretty well. They said they simply issued a command like "go attack," and the agent built a small squad and marched off on its own. That made him curious: if you let the AI play completely by itself, how good could it get?
The answer is: pretty bad—but really interesting.
Codex Astra, ranked #1, went 18-for-18. But what it’s best at isn’t front-line combat—it’s "harassment": sending a mining worker (a Probe) to the opponent’s base to cause trouble. This tactic works especially well against AI opponents, because when the other agent sees a worker coming, it will spend dozens of seconds figuring out what to do, doing nothing in the meantime. In terms of serious economic development and large-scale battles, though, Codex is relatively weak: it often makes one or two units and throws them at the opponent, instead of saving up forces before launching an attack.
Claude Fable ranked third with an 83.3% win rate, and was the most "serious about playing" among all the participating models. It would faithfully develop its economy, climb the tech tree, and even produced Mutalisks in one match and researched the Templar Archives (Templar tech) in another. Sometimes it would over-research and still lag on unit production, but at least in the sense of "trying to understand the rules of the game," Fable was more diligent than anyone.
Grok performed worst. Grok 4.6 generated over 11,000 reasoning tokens in a 43-minute match, yet it only issued 6 batches of action commands, and never produced a single combat unit throughout the entire game. In essence, it treats real-time strategy like a turn-based game: it keeps thinking and forgetting it needs to actually act.
The core issue this test reveals is that current large models are still far from being sufficient for real-time environments that require continuous observation, fast decision-making, and multi-thread coordination. Even the best model can be beaten in every match by a human beginner who knows how to pull off a "Photon Cannon rush" (one of the most basic early-attack tactics). But on the flip side, these models already understand the basic concepts of building, mining, and attacking—they just fall far short in pacing execution and coordinating multiple tasks.
The test code and battle platform have already been released. Anyone can bring their own agent to play a match. The address is http://bw.swerdlow.dev.
StarCraft: Brood War is the classic 1998 real-time strategy game and a longtime friend of AI research. In 2019, DeepMind’s AlphaStar even defeated professional players in this game. But back then it was a reinforcement-learning AI specifically trained for the task. This time the test is different: it lets general-purpose large models jump in directly as AI agents and see whether they can figure things out on their own—build a base, produce units, and fight.
The author, Ben Swerdlow, originally made a StarCraft version that could only be controlled through agents, just to play with friends. He didn’t expect that a few friends who had barely played the game would still do pretty well. They said they simply issued a command like "go attack," and the agent built a small squad and marched off on its own. That made him curious: if you let the AI play completely by itself, how good could it get?
The answer is: pretty bad—but really interesting.
Codex Astra, ranked #1, went 18-for-18. But what it’s best at isn’t front-line combat—it’s "harassment": sending a mining worker (a Probe) to the opponent’s base to cause trouble. This tactic works especially well against AI opponents, because when the other agent sees a worker coming, it will spend dozens of seconds figuring out what to do, doing nothing in the meantime. In terms of serious economic development and large-scale battles, though, Codex is relatively weak: it often makes one or two units and throws them at the opponent, instead of saving up forces before launching an attack.
Claude Fable ranked third with an 83.3% win rate, and was the most "serious about playing" among all the participating models. It would faithfully develop its economy, climb the tech tree, and even produced Mutalisks in one match and researched the Templar Archives (Templar tech) in another. Sometimes it would over-research and still lag on unit production, but at least in the sense of "trying to understand the rules of the game," Fable was more diligent than anyone.
Grok performed worst. Grok 4.6 generated over 11,000 reasoning tokens in a 43-minute match, yet it only issued 6 batches of action commands, and never produced a single combat unit throughout the entire game. In essence, it treats real-time strategy like a turn-based game: it keeps thinking and forgetting it needs to actually act.
The core issue this test reveals is that current large models are still far from being sufficient for real-time environments that require continuous observation, fast decision-making, and multi-thread coordination. Even the best model can be beaten in every match by a human beginner who knows how to pull off a "Photon Cannon rush" (one of the most basic early-attack tactics). But on the flip side, these models already understand the basic concepts of building, mining, and attacking—they just fall far short in pacing execution and coordinating multiple tasks.
The test code and battle platform have already been released. Anyone can bring their own agent to play a match. The address is http://bw.swerdlow.dev.
