Gamebridge · Super Mario Land · research note
AVG3
One network plays Super Mario Land from power-on to the end screen.
AVG3 is a single neural network distilled from twelve reinforcement-learning experts, one per level, then improved with success-filtered self-imitation and weight averaging. It finishes the whole game in about 1 of every 11 games we test, with no search, save states or human input while it plays.
One run on a game never used for training or for choosing the model. It is one of the better runs, not a typical one: most games still end in a game over.
How it was built
Reinforcement-learning experts, one per level
For each of the twelve levels we trained an expert network with reinforcement learning (PPO). A save-state search first found a route through the level, by trying inputs, reloading and keeping what got furthest. Training episodes then started from points along that route, and an imitation term on the route kept the expert near it. Search and save states are training-time tools only.
One network distilled from all twelve
The player is a single network that learns to make the experts' decisions on the states it reaches itself, a form of policy distillation. There is no per-level switching code: the level is one of its inputs. At every step it sees the last few Game Boy screens plus a handful of values read from game memory (how far into the level it is, Mario's position, the level, the game mode, the power-up state) and its own previous button press. Enemies, the boss and the timer it only sees on screen; lives, coins and score are hidden from it.
A wall at the 3-3 boss
The World 3-3 expert never beats its boss: 0 of 64 tries. The student passed about 1 attempt in 4, and every further round of ordinary training made that worse. One round dropped it to 1 pass in 50, because the student was learning the expert's losing moves at the boss.
Success-filtered self-imitation
We removed every expert label inside the boss section and mixed in about 150,000 frames of the student's own successful boss passes, found by replaying many attempts. With the expert's boss labels still in, those passes helped little (18%). With them removed, one training run reached 65%.
Average, because one run is luck
A second run with a different seed reached only 35%, and succeeded on mostly different games. Averaging the weights of the two runs, and then with the starting model, removed most of that luck. That average is AVG3.
Checked on games it never saw
We confirmed the gain on a separate set of games that no model trained on or was chosen with. Some "different" test games turned out to be exact replays of each other, since both the game and the model are deterministic, so each is counted once.
What the boss taught us
Across more than 100 logged boss deaths, every one is a hit from an enemy, never the clock running out. Almost all of them land while the in-game timer reads between 191 and 206, whenever Mario arrived. Runs that win tend to hang back further left and enter the boss room later.
We tested the obvious shortcut: script Mario to wait on the left until the timer drops, then hand control back. It passed 1 time in 77, against 13 for the model left alone, because waiting ran out the clock or dropped Mario into pits. The model's own timing is what gets it through.
Where it still loses lives
| Place | Share of lives lost | Note |
|---|---|---|
| 3-1, first stretch | 16% | Three of this run's five lost lives |
| 3-3, right at the start | 15% | The experts fail here too |
| 2-3, near the end | 7% | Often repeated within one game |
| 3-3 boss | 4% | Was the main wall |
The next round applies the same recipe to these places: drop the expert labels where the expert fails, add the model's own successful passes, train two seeds, and average them.
What this is, and isn't
- Reinforcement learning, search and imitation together. The experts are PPO agents; the final network learns from them by distillation and from its own boss passes by self-imitation. It is not a scripted bot, and none of its play is hand-written.
- The final network itself was not optimized with reinforcement learning directly. Its reinforcement learning comes through the experts it was distilled from.
- It plays autonomously, but it reads a few values from game memory, so its view is not identical to a human player's.
- Samples are small: a few dozen games per test set and five finishing games. A final set of games held back for the last check has not been used yet.