AI researchers have always been super thrilled about building Artificial Intelligence bots that can play games as smartly as a human. Back in June, OpenAI Five, the artificial intelligence bot team, had smashed amateur humans in the video game Dota 2. In August, OpenAI Five bots beat a team of former pros at Dota 2. Now, researchers from Chinese tech giant Tencent have recently developed a pair of AI agents that were successful in beating the Cheating Level Built-in AI in StarCraft II in the full game.
Starcraft II is widely considered as the most challenging Real Time Strategy game, due to large observation space, huge action space, partial observation, multi-player simultaneous game model, and long time horizon decision.
The two AI Agents
Their research paper describes two AI agents – TSTARBOT 1 and TSTARBOT 2. The first is a macro-level controller agent based on deep reinforcement learning over flat action structure. It oversees several specific algorithms designed to handle lower level functions.
TSTARBOT 1: Overview of macro actions and reinforcement learning
TSTARBOT2, the more robust of the two, is a macro-micro controller consisting of several modules that handle entire facets of the gameplay independently.
TSTARBOT 2: Overview of macro-micro hierarchical actions
The gameplay
Tencent’s AI played the game using methods similar to a mouse click and macros and played exactly the same way as a human player would. The AI saw the game by interpreting video output in a frame-to-frame basis, and translating the information into data it can work with. Tencent’s AI played StarCraft II with the “fog-of-war” turned on. This means that the AI can’t see the enemy AI’s units and base until it scouts the map. The TSTARBOTs were designed to imitate the human thought process.
The agents were tested in a 1v1 Zerg-vs-Zerg full game. They played against built-in AI ranging from level 1 (the easiest) to level 10 (the hardest). The training used the Abyssal Reef, a map known to have thwarted neural network AIs from winning against StarCraft II’s built-in AIs.
Interestingly, Tencent trained the agents using only a single CPU. This, however, took a large number of processors to process the data it takes to train the bots on billions of frames of video. The researchers took 1920 parallel actors (with 3840 CPUs across 80 machines) to generate the replay transitions, at the speed of about 16,000 frames per second.
Results
Win-rate (in %) of TSTARBOT 1 and TSTARBOT 2 agents, against built-in AIs of various difficulty levels.
Each reported win-rate is obtained by taking the mean of 200 games with different random seeds, where a tie is counted as 0.5 when calculating the win-rate.
TSTARBOT1 and TSTARBOT2 also played against several human players ranging from Platinum to Diamond level players in the ranking system of SCII Battle.net League.
Each entry means how many games TStarBot1/TStarBot2 wins and loses. The agent is able to consistently defeat built-in AIs in all levels, showing the effectiveness of the hierarchical action modeling.
In another informal test, the researchers also let the two TSTARBOT play against each other. TSTARBOT 1 always defeated TSTARBOT 2. This is because TSTARBOT 1 tends to use the Zergling Rush strategy. In StarCraft, a Zerg rush is a strategy where a player using the Zerg race tries to overwhelm the opponent through large numbers of smaller units before the enemy is fully prepared for battle. TSTARBOT 2 lacks anti-rush strategy and henceforth always loses.
In the future, the team plans to build a more carefully hand-tuned action hierarchy to enable the reinforcement learning algorithms to develop better strategies for full StarCraft II games.
If you want to dive a little deeper into how the bots work, you can read the research paper.
Read Next
AI beats human again – this time in a team-based strategy game.
OpenAI Five loses against humans in Dota 2 at The International 2018.
OpenAI set their eyes to beat Professional Dota 2 team at The International.