Reinforcement Learning Arena

About

Nobody tells this agent where the goal is. It starts knowing nothing, wanders, collects rewards and penalties, and slowly builds a table of how good each move is in each cell. That table is the heatmap. Watch reward spread outwards from the goal, one episode at a time, until the arrows form a route.

This is tabular reinforcement learning: the same ideas behind game-playing agents, robot controllers and the reinforcement learning stages used to train today's language and reasoning models, shrunk to a world small enough to see every number. Deep RL replaces the table with a neural network, because real worlds have far too many states to list.

Paint walls, pits, goals and bonuses onto the board, tune the learning rate, discount and exploration, and compare Q-learning with SARSA. The Reinforcement Learning lesson covers the theory in depth.

How it learns

The world is a Markov decision process. In state s (a cell) the agent picks an action a (up, right, down, left), receives a reward r and lands in state s'. Walls and edges bounce it back. Pits and goals end the episode. With slip turned on, the move goes sideways some of the time, so the world is no longer deterministic.

Q-values

Q(s, a) estimates the return: the sum of future rewards, each discounted by γ per step. With γ = 0.95, a +10 goal five steps away is worth 10 × 0.95⁵ ≈ 7.7 now. That is why the heatmap fades with distance from the goal, and why a small γ makes the agent short-sighted.

Learning from each step

After every step the agent nudges Q(s, a) towards a better-informed target: the reward it just got plus the discounted value of where it landed. The learning rate α sets the size of the nudge. This is bootstrapping: estimates are learned from other estimates, which is why value creeps backwards from the goal one cell per visit rather than appearing everywhere at once.

Q-learning versus SARSA

The only difference is the value of the next state. Q-learning uses the best next action, max Q(s', ·), even if the agent will actually explore: it learns the value of the optimal policy (off-policy). SARSA uses the action the agent really takes next, exploration included: it learns the value of the policy it is actually following (on-policy). Near danger, that makes SARSA cautious, because it knows it will sometimes stumble.

Exploration vs exploitation

An agent that only exploits never discovers anything better than its first lucky route. An agent that only explores never uses what it learned. Every learning agent needs a rule for balancing the two.

Here it is epsilon-greedy: with probability ε take a random action, otherwise take the action with the highest Q-value (ties broken at random). Start with a large ε and multiply it by the decay factor after every episode, so early episodes explore widely and later ones commit. Q-learning is guaranteed to converge to optimal values provided every state-action pair keeps being tried and the learning rate shrinks appropriately; a floor on ε keeps some exploration alive.

Exploration has a price. In the cliff walk, a random step next to the edge is fatal. That is exactly the situation where on-policy SARSA and off-policy Q-learning disagree.

Try this

1. 01The cliff: Q-learning versus SARSALoad Cliff walk (constant ε = 0.1, γ = 1, as in Sutton and Barto). Train Q-learning for 500 episodes (T five times) and replay the greedy route; reset, switch to SARSA, repeat. Q-learning’s route hugs the cliff; SARSA’s keeps a row or two away, and its average return while training is higher.
2. 02Reward hackingLoad Bonus trap and train. Stepping onto the +1 bonus pays every time, so the agent learns to shuttle on and off it until the episode times out instead of finishing. Lower the bonus reward below 0.2, where a round trip no longer pays for its two steps, and keep training. The agent is doing exactly what the reward says, not what we meant.
3. 03No explorationOn the Lava field, set ε to 0 and train. Ties are broken at random, so the first episodes still wander, but once any route is found the agent stops looking for a better one.
4. 04Slippery floorSet slip to 0.2 on the Lava field. The greedy route moves away from the pits, because the value of a cell next to a pit now includes the chance of sliding in.
5. 05Credit travels slowlyLoad Four rooms, set speed to watch, and train. The first reward reaches only the cell next to the goal. Each later episode pushes value one step further back.
6. 06Change the worldAfter training, paint a wall across the learned route. Keep training without resetting and watch the old policy break and relearn, a small taste of non-stationary environments.