The neural network and training

NeuralPlayer uses the same minimax search as ClaudePlayer, but scores positions with a small neural network instead of hand-written rules. The network starts out random and learns only by playing against itself. No human idea of a good position goes into it, and ClaudePlayer is only ever used to measure progress, never as something to learn from.

The network is in src/network.rs and the training loop in src/train.rs. Neither uses a machine-learning library: the forward pass, the backpropagation and the optimiser are all written by hand.

The network

A multilayer perceptron:

52 inputs ──▶ 64 ReLU ──▶ 32 ReLU ──▶ 1 tanh

The output is in -1..1 and, like every score, positive favours the tigers.

Inputs

Only the raw position, nothing derived from it:

The network has to work out for itself that dead sheep are bad for the sheep, that a sheep with an empty point behind it can be jumped, and so on.

Parameters

All the weights and biases live in one flat Vec<f32> (params), at fixed offsets W1, B1, W2, B2, W3, B3, PARAMS in all (about 5,500). That lets the optimiser, the gradient and the file format treat them as one list of numbers.

• W1 is input-major: input i's weights to the 64 hidden units are contiguous, at W1 + i*64 .. W1 + (i+1)*64.
• W2 is unit-major: hidden-2 unit k's 64 incoming weights are contiguous.

Forward pass

Most inputs are 0: at most 4 tiger bits and 20 sheep bits are set out of 50. So instead of a full matrix multiply, forward_full starts layer 1 at its biases and adds the weight row of each set input, then adds hand × W1[HAND] and applies ReLU. That's why W1 is input-major. for_active walks the set bits of the board masks with trailing_zeros. Layers 2 and 3 are ordinary dot products, then tanh.

forward_matches_a_plain_matrix_multiply checks this against the textbook calculation.

Initialisation

Network::random(seed) uses He initialisation: normal weights scaled by sqrt(2 / fan_in), with zero biases. For layer 1 the fan-in is taken as 20 rather than 52, since only about a third of the inputs are set at once.

Backpropagation

The loss for one sample is the squared error (out - target)² / 2. add_gradient runs the forward pass, keeping each layer's activations, then works backwards:

1. Through tanh: d_out = (out - target) × (1 - out²).
2. Layer 3's weight gradients are d_out × h2. The gradient flowing into each hidden-2 unit is d_out × W3[k], or 0 where that ReLU was off.
3. The same again through layer 2 into hidden layer 1, skipping units with no gradient.
4. Layer 1: only the set inputs have non-zero gradients, so again only their rows are touched.

gradient_matches_finite_differences checks every layer's gradients against numerical estimates.

Parallel and deterministic

gradient(batch) splits a batch into chunks of GRAD_CHUNK = 32 samples, computes each chunk's gradient on rayon's thread pool, then adds the chunks up in order. loss does the same in chunks of LOSS_CHUNK = 1024. Floating-point addition isn't associative, so summing in a fixed order is what makes the result identical whatever the number of threads (training_is_the_same_on_any_number_of_threads). Training is therefore reproducible: the same seed gives the same network on any machine with any RAYON_NUM_THREADS.

The optimiser: Adam

train_batch averages the gradient over the batch and takes one Adam step. Adam keeps running averages of each parameter's gradient (m, decay β1 = 0.9) and squared gradient (v, decay β2 = 0.999), corrects their bias towards their zero start, and moves each parameter by learning_rate × m̂ / (√v̂ + ε). In effect every parameter gets its own step size, which suits a network where some inputs (a sheep on a rarely used point) are seen far less often than others.

File format

save and load use a small binary format, all little-endian:

load rejects files of the wrong size, magic, version or layer sizes with InvalidData. So changing the layer sizes makes old weights fail to load, and changing what an input means without changing the sizes calls for bumping VERSION.

Self-play training

cargo run --release -- train [generations] [games] [out-dir] [--from <weights>] [options]

Training runs in generations. Each one plays a batch of games, turns them into training samples, trains the network on them, and saves it.

1. Starting point

run starts from --from if given, otherwise the newest gen-NNN.bin in the out dir, otherwise random weights (seed 0x5EED). Generation numbers carry on from the newest file in the out dir either way, so a run can be stopped and restarted. The Adam state and the sample buffer aren't saved, so they start empty on a restart.

2. Self-play

Each generation plays games_per_generation (500) games of the network against itself, in parallel across all cores (parallel, on rayon's pool). Each game:

• uses a NeuralPlayer that searches to exactly depth (3) plies. A depth limit rather than a time limit keeps runs reproducible, and the game's seed is derived from the generation and game number;
• plays random moves for the first random_plies (10) plies, so games start from different places;
• after that, plays a random move instead of the searched one with probability epsilon (0.05), to explore positions the network would never choose;
• is a draw on a threefold repetition (play passes the earlier boards to the search, so the search sees repetitions too), or after ply_cap (200) plies, a backstop for games that wander without repeating.

Every position is recorded as a Ply with the search's score for it (clamped to -1..1, so forced wins become ±1), and whether a random move was played there.

3. Labelling positions: TD(λ)

Each position needs a target, the score the network should learn to give it. Two obvious choices both have problems:

• The game's result (+1, -1 or 0) is the truth, but noisy: a position early in the game had many moves, some of them random, between it and the result.
• The search score of the next position is less noisy but biased, since it comes from the network itself.

TD(λ) blends them, working backwards from the end of the game:

target[last] = result
target[t]    = (1 - λ) × score[t+1] + λ × target[t+1]

With λ = 1 every position is labelled with the result; with λ = 0 each is labelled with the next position's search score. The default lambda is 0.7.

Random moves cut the chain. A position where a random move was played is labelled with its own search score instead, and the positions before it then build on that, so they don't learn from the random move's consequences. This matters because random moves aren't equally bad for both sides: a random sheep move usually leaves a sheep to be jumped, while a random tiger move costs little. Without the cut, the network learnt that tigers nearly always win.

4. The sample buffer

The samples of each generation are kept, split by outcome (tiger win, sheep win, draw), and the network trains on the last buffer_generations (5) generations together. Training on more than the latest games smooths out the network's swings from one generation to the next.

5. Balancing outcomes

Self-play ends mostly in tiger wins, largely because of those random sheep moves. Trained on everything, the network would mostly learn what tiger wins look like. So with balance_outcomes (on by default; --no-balance turns it off), balanced builds each epoch from:

• as many positions from sheep-win games as from tiger-win games: their mean, so the epoch keeps the same size. The bigger pool gives a random subset (every position at most once), and the smaller one gives every position plus random repeats;
• every position from drawn games, once.

If either kind of win has no positions at all there is nothing to balance, and everything is used.

6. Training

For each of epochs (2) epochs: build the epoch's samples, shuffle them, show each in a random one of the 8 symmetric orientations (GameBoard::transformed), and train in batches of batch_size (256) with Adam at learning_rate (1e-3). The random orientations are data augmentation: they teach the network that symmetric positions are equally good, and multiply the variety of positions it sees.

7. Matches

On every generation whose number is a multiple of match_every (10), the network plays two matches of match_games (100) games each, alternating sides, at the same depth:

• against the generation that last played matches (N − 10), loaded from its gen-NNN.bin, or the network the run started from if that file is missing;
• against ClaudePlayer.

Each pair of games (one as each side) shares a seed, so both sides get the same 4 random opening moves. The score is the share of points won, with a draw worth half. Matches are slow and only measure progress, so other generations skip them.

8. Saving

Each generation is saved as out_dir/gen-NNN.bin and copied to out_dir/latest.bin.

Reading the output

 gen trained tigers  sheep  draws T before S before  T after  S after vs prev vs Claude train s  time s
 499  128657    460     22     18   0.0161   0.0211   0.0194   0.0165       -         -    0.26     1.3
 500  128335    461     17     22   0.0198   0.0185   0.0161   0.0229     45%       82%    0.28     2.2

Judge progress by vs Claude. Self-play results say little, since the random moves skew them towards tigers, and the errors measure how well the network fits its own targets, not how well it plays. vs prev above 50% means the network beats its older self.

Settings

Every train::Config field is a command-line option (train --help):

RAYON_NUM_THREADS limits the number of threads; the results don't depend on it.

Then play the result:

cargo run --release -- -S human -T weights/latest.bin