What each training word does to pint

The exception word pint is the test case on this page. Train the network one word at a time, on pint itself or on neighbors like mint, bind, and punt, and step through each training exposure: how the word is encoded, how activation flows forward, how error is measured, and how every weight changes. The panel below tracks how the network reads pint after every exposure, so you can see which words help it and which hurt it. A companion to the reading network demo.

Tracking pint The network reads it as–
Error score for pint –
Word for the next training exposure

1.0 Exposures so far: 0

Error score for pint after each exposure

Dots: black after training on pint, green when another word lowered pint's error, red when it raised it.

    1. Encode the input
    2. Hidden units
    3. Output units
    4. Target and error
    5. Output error signals
    6. Hidden error signals
    7. Weight changes
    8. Weights updated

    Ready to train

    The network

    Input layer: 105 grapheme units

    Output layer: 59 phoneme units

    Things to try

    1. Step through one exposure to pint from the start. At step 7, notice how many weights change and which ones don't.
    2. Train on pint until it reads correctly, then train on mint a few times. Watch the /aɪ/ and /ɪ/ units in the top panel. What happens to pint, and why? Then train on bind and watch again.
    3. Train on punt. Which of pint's units does it help, and which does it leave alone?
    4. Train on home, which shares no letters with pint. Does pint still change? Where in the network does that influence travel?
    5. Use Restart from the same starting weights to try two different training orders from an identical start, such as pint ×5 then mint ×5, versus alternating the two. Does the order matter?
    6. Early on, training on almost any word lowers pint's error. Look at which output units change and explain why.

    How this page relates to the main demo

    The arithmetic is the same kind the main demo does hundreds of thousands of times per run, with three changes to keep every number visible. The network has 6 hidden units instead of 100, so each output unit's net input is a sum of 6 terms you can read. It trains on only a handful of words, one exposure at a time, with no frequency weighting. And it uses a much larger learning rate, so a single exposure makes a visible difference; in the main demo, any one exposure barely moves the weights.

    The error score is the sum of squared errors, as in Seidenberg and McClelland (1989), and training reduces that same quantity. Strictly, the derivative of a squared error carries a factor of 2; here it is folded into the learning rate, which is common practice. The main demo trains with a different error measure (cross-entropy) but reports the same squared-error score.

    With only 6 hidden units, words share more of the network, so interference between them is stronger than in the main demo. That makes it easy to see, which is the point. Given enough rounds of interleaved training on all seven words, even this small network eventually reads them all correctly; training on one word at a time in long blocks produces much more back-and-forth.