Playtest bots that prove every level can be beaten, and that the core rule is not optional

Every level designer has shipped a level that turned out to be impossible under some condition nobody tried, and every designer has shipped a level that could be beaten by ignoring the clever rule the game was built around. Human playtesting finds some of these, but people get tired, learn the levels and stop seeing them. In our two big 3D browser games we let bots play the real rules over and over on every change, and the tests fail if a level cannot be finished, if a player who ignores the core rule can still win, or if the difficulty curve bends the wrong way. This article shows what those bots are, how they are built and what they have caught.

#Three questions a bot can answer

1. Can every level be finished? A strong bot plays each level with the starting equipment and must complete it within a time limit. If it cannot, either the level is broken or the bot needs to get smarter, and both are worth knowing before a player finds out.
2. Does the core rule matter? A bot that ignores the game's main idea must fail. If it wins, the idea is decoration, and players will discover the shortcut within minutes.
3. Does the difficulty rise sensibly? Bots of different skill play every level with several seeds, and a table of wins, times and losses shows where the curve has a wall or a dip.

None of this works without a simulation that runs headless, fast and deterministically: the rules must not depend on the screen's refresh rate or on unseeded random numbers. Our fixed-timestep game loop article explains that foundation. In both games the rules are pure TypeScript with no access to the renderer, so a test can run a full level in a fraction of a second of real time.

#Bots must play by the player's rules

A bot that reads hidden state or presses impossible inputs proves nothing. Ours see only what a player sees and press only the inputs a player has. In Billowpost, a calm game where you deliver mail by hot-air balloon and can only rise or sink, the bot reads the wind of every height on the on-screen ladder, the ground, the drop marker and the fuel gauge, and it presses burner, vent, drop and sandbag, nothing else. In Faultwarden, where you cut a floating island so that pieces fall with the enemies on them, the bot aims with the same predict function that draws the player's aiming preview.

Human limits are modelled too. Each skill level has a reaction time, an aiming error and a habit of being late:

// Faultwarden: how often a bot rethinks, and how many directions it tries
const REACT = { novice: 0.55, casual: 0.32, careful: 0.22, reckless: 0.25, idle: 9 };
const DIRS  = { novice: 16,   casual: 16,   careful: 32,   reckless: 16,   idle: 1 };

const noise = skill === 'novice' ? 0.28 : skill === 'casual' ? 0.12 : 0.03; // radians
const angle = Math.atan2(best.dz, best.dx) + rng.range(-noise, noise);

The novice tries fewer options, waits longer between decisions and misses by up to 16 degrees. It also never plans two moves ahead, which the casual and careful bots do when one cut alone would not drop anything.

#A planner that steers a balloon

The Billowpost bot is the more interesting one, because a balloon cannot be steered directly. The only control is height, and each height has its own wind. For every open address the planner rolls the balloon forward in the simulation for each candidate height, climbing or sinking through the winds in between, and follows where the drop marker would travel. The height whose marker passes closest and soonest wins. When no single height reaches, it searches two-leg plans: ride height A for some seconds, then height B. It replans as it flies, refines the height on the final approach and releases the mail at the marker's closest pass. Paths that would meet a hillside, a tree or a roof are rejected.

To make that check cheap, the bot precomputes once per map the lowest safe basket height on a 4-metre grid, including the width of the envelope above the basket and every tree and roof. Each rollout then reads that grid with bilinear interpolation instead of testing geometry.

#The tests that hold it together

The solvability tests in Billowpost are three short rules repeated for each of its five regions and six rounds per region:

test(`${dale}-${i}: the careful pilot delivers everything within 900 s`, () => {
  const res = runBot(new Sim(makeRoute(dale, i), baseParams()), 'careful', 900);
  expect(res).toMatchObject({ done: true, failed: null });
  expect(res.delivered).toBe(res.total);
});

test('the rule-ignoring pilot never completes a route', () => {
  for (let i = 0; i < ROUTES_PER_DALE; i++)
    expect(runBot(new Sim(makeRoute(dale, i), baseParams()), 'ignore', 900).done).toBe(false);
});

test('an idle player never delivers anything', () => {
  for (let i = 0; i < ROUTES_PER_DALE; i++)
    expect(runBot(new Sim(makeRoute(dale, i), baseParams()), 'idle', 900).delivered).toBe(0);
});

"Base balloon" matters: the careful pilot has no upgrades, so no round depends on something the player may not have bought yet. The rule-ignoring pilot picks one height and drops whenever any address is below it. If it ever finishes a round, the winds of that round are too uniform and height choice no longer decides the route, so the round is changed before anything else.

#Win tables for difficulty

Faultwarden prints a table instead of passing or failing, because difficulty is a judgement call. On 3 November 2026, with six seeds per island, it gave this:

Read it as a designer would. An idle player loses every island within half a minute except the first, which is a gentle teaching island where nothing attacks hard enough to end the run, though no wave is ever cleared. Casual and careful players win everything, in about three to four and a half minutes per island. The novice shows the real curve, and the third island stands out: it is the ice island, where plates are slippery and enemies slide, and the novice wins once in six. That is a wall for beginners, and the table makes it visible long before any player review would. Whether to soften it or to teach the ice better is a design decision; the bots only make sure it is made on purpose.

#From bot times to par times

Billowpost awards stars for delivering under a par time. Bot times are too good for that, because a bot reacts instantly. So the careful pilot's plan is also flown through a human model: the same plan, but with the height-hold assist a player uses, reaction lag and imprecise burner presses. On the first round the careful bot needed 48 seconds and the human model 66 to 70. Par times are set from the human model, and where that model is unreliable they fall back to the careful bot's time multiplied by 1.3.

#What bots cannot tell you

Bots measure whether something can be done, not whether it is fun, readable or fair-feeling. They do not notice that a warning is hard to see on a phone, that a tutorial line is confusing or that a sound is annoying. We pair them with recorded play by literal newcomer scripts that only do what the screen tells them, with checks on every screen size, and with human review before release. Bots remove the obvious failures so that people can spend their attention on feel. For that side, see making a browser game feel good and what to build first in a game prototype.

Both games are free and listed as Coming soon in the PN Scripts games catalogue.

#Frequently asked questions

Do I need machine learning to build a playtest bot?

No. Our bots are hand-written: they try a set of options with the game's own prediction functions, score them and act with human-like delays and errors. That is enough to prove solvability and compare difficulty, and it is easy to debug.

How long do bot tests take?

Because the rules run headless without rendering, a full level takes from a fraction of a second to a few seconds. Our solvability tests run per region in parallel, so the whole set fits into the normal test run.

What is a rule-ignoring bot?

A bot that plays without the game's central idea, for example by never choosing a height in a balloon game. If it can win, the idea is optional and players will skip it, so the test expects it to fail.

Can bots replace human playtesting?

No. They find impossible levels, shortcuts and difficulty walls quickly, but they cannot judge clarity, comfort or fun. Use them so that human testers spend their time on those questions.

#Sources

• Vitest documentation
• Receding horizon (model predictive) control
• Game Accessibility Guidelines