diff options
Diffstat (limited to 'docs/training.md')
| -rw-r--r-- | docs/training.md | 260 |
1 files changed, 260 insertions, 0 deletions
diff --git a/docs/training.md b/docs/training.md new file mode 100644 index 0000000..79da8f2 --- /dev/null +++ b/docs/training.md @@ -0,0 +1,260 @@ +# The neural network and training + +`NeuralPlayer` uses the same [minimax search](search.md) as `ClaudePlayer`, but scores positions with +a small neural network instead of hand-written rules. The network starts out random and learns only +by playing against itself. No human idea of a good position goes into it, and `ClaudePlayer` is +only ever used to measure progress, never as something to learn from. + +The network is in [`src/network.rs`](../src/network.rs) and the training loop in +[`src/train.rs`](../src/train.rs). Neither uses a machine-learning library: the forward pass, the +backpropagation and the optimiser are all written by hand. + +## The network + +A multilayer perceptron: + +```text +52 inputs ──▶ 64 ReLU ──▶ 32 ReLU ──▶ 1 tanh +``` + +The output is in `-1..1` and, like every score, positive favours the tigers. + +### Inputs + +Only the raw position, nothing derived from it: + +| Inputs | Meaning | +|--------|---------| +| 0..25 | 1 if a tiger is on that point (bit `x + 5y`) | +| 25..50 | 1 if a sheep is on that point | +| 50 (`HAND`) | Sheep in hand / 20 | +| 51 (`TIGER_TO_MOVE`) | 1 if tigers are to move | + +The network has to work out for itself that dead sheep are bad for the sheep, that a sheep with an +empty point behind it can be jumped, and so on. + +### Parameters + +All the weights and biases live in one flat `Vec<f32>` (`params`), at fixed offsets `W1`, `B1`, +`W2`, `B2`, `W3`, `B3`, `PARAMS` in all (about 5,500). That lets the optimiser, the gradient and the +file format treat them as one list of numbers. + +- `W1` is **input-major**: input `i`'s weights to the 64 hidden units are contiguous, at + `W1 + i*64 .. W1 + (i+1)*64`. +- `W2` is **unit-major**: hidden-2 unit `k`'s 64 incoming weights are contiguous. + +### Forward pass + +Most inputs are 0: at most 4 tiger bits and 20 sheep bits are set out of 50. So instead of a full +matrix multiply, `forward_full` starts layer 1 at its biases and **adds the weight row of each set +input**, then adds `hand × W1[HAND]` and applies ReLU. That's why `W1` is input-major. `for_active` +walks the set bits of the board masks with `trailing_zeros`. Layers 2 and 3 are ordinary dot +products, then `tanh`. + +`forward_matches_a_plain_matrix_multiply` checks this against the textbook calculation. + +### Initialisation + +`Network::random(seed)` uses He initialisation: normal weights scaled by `sqrt(2 / fan_in)`, with +zero biases. For layer 1 the fan-in is taken as 20 rather than 52, since only about a third of the +inputs are set at once. + +### Backpropagation + +The loss for one sample is the squared error `(out - target)² / 2`. `add_gradient` runs the forward +pass, keeping each layer's activations, then works backwards: + +1. Through `tanh`: `d_out = (out - target) × (1 - out²)`. +2. Layer 3's weight gradients are `d_out × h2`. The gradient flowing into each hidden-2 unit is + `d_out × W3[k]`, or 0 where that ReLU was off. +3. The same again through layer 2 into hidden layer 1, skipping units with no gradient. +4. Layer 1: only the set inputs have non-zero gradients, so again only their rows are touched. + +`gradient_matches_finite_differences` checks every layer's gradients against numerical estimates. + +### Parallel and deterministic + +`gradient(batch)` splits a batch into chunks of `GRAD_CHUNK` = 32 samples, computes each chunk's +gradient on rayon's thread pool, then **adds the chunks up in order**. `loss` does the same in chunks +of `LOSS_CHUNK` = 1024. Floating-point addition isn't associative, so summing in a fixed order is +what makes the result identical whatever the number of threads +(`training_is_the_same_on_any_number_of_threads`). Training is therefore reproducible: the same +seed gives the same network on any machine with any `RAYON_NUM_THREADS`. + +### The optimiser: Adam + +`train_batch` averages the gradient over the batch and takes one `Adam` step. Adam keeps running +averages of each parameter's gradient (`m`, decay `β1` = 0.9) and squared gradient (`v`, decay +`β2` = 0.999), corrects their bias towards their zero start, and moves each parameter by +`learning_rate × m̂ / (√v̂ + ε)`. In effect every parameter gets its own step size, which suits a +network where some inputs (a sheep on a rarely used point) are seen far less often than others. + +### File format + +`save` and `load` use a small binary format, all little-endian: + +| Bytes | Contents | +|-------|----------| +| 0..4 | `TSNN` | +| 4..8 | version (`u32`, currently 1) | +| 8..20 | inputs, hidden 1, hidden 2 (`u32` each) | +| 20.. | every parameter as `f32`, in `params` order | + +`load` rejects files of the wrong size, magic, version or layer sizes with `InvalidData`. So +changing the layer sizes makes old weights fail to load, and changing what an input *means* without +changing the sizes calls for bumping `VERSION`. + +## Self-play training + +```bash +cargo run --release -- train [generations] [games] [out-dir] [--from <weights>] [options] +``` + +Training runs in **generations**. Each one plays a batch of games, turns them into training samples, +trains the network on them, and saves it. + +### 1. Starting point + +`run` starts from `--from` if given, otherwise the newest `gen-NNN.bin` in the out dir, otherwise +random weights (seed `0x5EED`). Generation numbers carry on from the newest file in the out dir +either way, so a run can be stopped and restarted. The Adam state and the sample buffer aren't saved, +so they start empty on a restart. + +### 2. Self-play + +Each generation plays `games_per_generation` (500) games of the network against itself, in parallel +across all cores (`parallel`, on rayon's pool). Each game: + +- uses a `NeuralPlayer` that searches to exactly `depth` (3) plies. A depth limit rather than a time + limit keeps runs reproducible, and the game's seed is derived from the generation and game number; +- plays random moves for the first `random_plies` (10) plies, so games start from different places; +- after that, plays a random move instead of the searched one with probability `epsilon` (0.05), + to explore positions the network would never choose; +- is a draw on a threefold repetition (`play` passes the earlier boards to the search, so the + search sees repetitions too), or after `ply_cap` (200) plies, a backstop for games that wander + without repeating. + +Every position is recorded as a `Ply` with the search's score for it (clamped to `-1..1`, so forced +wins become ±1), and whether a random move was played there. + +### 3. Labelling positions: TD(λ) + +Each position needs a target, the score the network should learn to give it. Two obvious choices +both have problems: + +- **The game's result** (+1, -1 or 0) is the truth, but noisy: a position early in the game had many + moves, some of them random, between it and the result. +- **The search score** of the next position is less noisy but biased, since it comes from the + network itself. + +TD(λ) blends them, working backwards from the end of the game: + +```text +target[last] = result +target[t] = (1 - λ) × score[t+1] + λ × target[t+1] +``` + +With `λ = 1` every position is labelled with the result; with `λ = 0` each is labelled with the next +position's search score. The default `lambda` is 0.7. + +**Random moves cut the chain.** A position where a random move was played is labelled with its own +search score instead, and the positions before it then build on that, so they don't learn from the +random move's consequences. This matters because random moves aren't equally bad for both sides: a +random sheep move usually leaves a sheep to be jumped, while a random tiger move costs little. +Without the cut, the network learnt that tigers nearly always win. + +### 4. The sample buffer + +The samples of each generation are kept, split by outcome (tiger win, sheep win, draw), and the +network trains on the last `buffer_generations` (5) generations together. Training on more than the +latest games smooths out the network's swings from one generation to the next. + +### 5. Balancing outcomes + +Self-play ends mostly in tiger wins, largely because of those random sheep moves. Trained on +everything, the network would mostly learn what tiger wins look like. So with `balance_outcomes` +(on by default; `--no-balance` turns it off), `balanced` builds each epoch from: + +- as many positions from sheep-win games as from tiger-win games: their mean, so the epoch keeps the + same size. The bigger pool gives a random subset (every position at most once), and the smaller one + gives every position plus random repeats; +- every position from drawn games, once. + +If either kind of win has no positions at all there is nothing to balance, and everything is used. + +### 6. Training + +For each of `epochs` (2) epochs: build the epoch's samples, shuffle them, show each in a **random one +of the 8 symmetric orientations** (`GameBoard::transformed`), and train in batches of `batch_size` +(256) with Adam at `learning_rate` (1e-3). The random orientations are data augmentation: they teach +the network that symmetric positions are equally good, and multiply the variety of positions it sees. + +### 7. Matches + +On every generation whose number is a multiple of `match_every` (10), the network plays two matches +of `match_games` (100) games each, alternating sides, at the same depth: + +- against the generation that last played matches (N − 10), loaded from its `gen-NNN.bin`, or the + network the run started from if that file is missing; +- against `ClaudePlayer`. + +Each pair of games (one as each side) shares a seed, so both sides get the same 4 random opening +moves. The score is the share of points won, with a draw worth half. Matches are slow and only +measure progress, so other generations skip them. + +### 8. Saving + +Each generation is saved as `out_dir/gen-NNN.bin` and copied to `out_dir/latest.bin`. + +## Reading the output + +```text + gen trained tigers sheep draws T before S before T after S after vs prev vs Claude train s time s + 499 128657 460 22 18 0.0161 0.0211 0.0194 0.0165 - - 0.26 1.3 + 500 128335 461 17 22 0.0198 0.0185 0.0161 0.0229 45% 82% 0.28 2.2 +``` + +| Column | Meaning | +|--------|---------| +| `gen` | Generation number | +| `trained` | Positions in each epoch | +| `tigers`, `sheep`, `draws` | How this generation's self-play games ended | +| `T before`, `S before` | Mean error on positions from tiger-win and sheep-win games in the buffer, before training | +| `T after`, `S after` | The same after training | +| `vs prev`, `vs Claude` | Match scores (only every `match_every` generations; `-` otherwise) | +| `train s`, `time s` | Seconds spent training, and for the whole generation | + +**Judge progress by `vs Claude`.** Self-play results say little, since the random moves skew them +towards tigers, and the errors measure how well the network fits its own targets, not how well it +plays. `vs prev` above 50% means the network beats its older self. + +## Settings + +Every `train::Config` field is a command-line option (`train --help`): + +| Option | Default | Meaning | +|--------|---------|---------| +| `generations` (positional) | 50 | Generations to train | +| `games` (positional) | 500 | Self-play games per generation | +| `out_dir` (positional) | `weights` | Where weights are saved and resumed from | +| `--from` | – | Start from this weights file | +| `--depth` | 3 | Search depth in self-play and matches | +| `--ply-cap` | 200 | A game this long is a draw | +| `--random-plies` | 10 | Random opening plies in self-play | +| `--epsilon` | 0.05 | Chance of a random move after that | +| `--lambda` | 0.7 | TD(λ) blend | +| `--buffer-generations` | 5 | Generations of samples to train on | +| `--no-balance` | off | Train on all positions without balancing outcomes | +| `--epochs` | 2 | Passes over the samples per generation | +| `--batch-size` | 256 | Positions per optimiser step | +| `--learning-rate` | 1e-3 | Adam's learning rate | +| `--match-games` | 100 | Games per match | +| `--match-every` | 10 | Play matches every this many generations | + +`RAYON_NUM_THREADS` limits the number of threads; the results don't depend on it. + +Then play the result: + +```bash +cargo run --release -- -S human -T weights/latest.bin +``` |