A small connectionist model in the tradition of Seidenberg and McClelland (1989). It learns to turn spellings into pronunciations with no rules and no word list, only weighted connections adjusted by practice. It uses the input and output coding of Plaut, McClelland, Seidenberg, and Patterson (1996). For a closer look at how individual nodes in the model get updated, see the demo of What each training word does to "pint"
Spelling units that switch on (105 possible)
hidden units; darker means more active, red outline means lesioned
Most active sound units (59 possible). Consonants count as present above the dashed 0.5 line; the strongest vowel wins.
Error is the summed squared difference between the network's output and the correct pronunciation. Seidenberg and McClelland used this score as a stand-in for naming time: more error, slower naming.
Each point is the mean error for 12 test words. Regular words are pronounced the way their spelling pattern is pronounced in other words (need, like feed and seed). Exception words break the pattern (have, unlike gave and save).
Lines that are not parallel are an interaction: the cost of being an exception depends on frequency. Drag the epoch slider to see how the picture changes with training.
The percentages below the plot are the share of each of the four groups the network pronounces exactly right at the epoch shown.
Mean error for each test group.
Share of each test group pronounced exactly right.
Nonwords are never trained. Dotted line: all 0 training words.
| Word | Group | Correct | Network says | Error |
|---|
| Nonword | Word body | Accepted | Network says | Pattern used |
|---|
The network sees about 430 one-syllable English words. Each spelling switches on a handful of letter units; activation flows through a layer of hidden units to a set of sound units; the result is compared with the correct pronunciation, and every connection weight is nudged to reduce the difference (backpropagation). That's one presentation. An epoch is as many presentations as there are words, chosen at random, with high-frequency words chosen more often.
Nothing in the network stores a rule like "final e makes the vowel long," and nothing stores a list of words. Whatever it knows about English spelling is spread across the same set of weights, and the same weights produce the pronunciation of mint, pint, and a nonword like bint. That's the central claim of Seidenberg and McClelland's model: one mechanism can handle regular words, exception words, and new strings.
The 48 test words are in the style of the experiments the model was built to explain (Seidenberg et al., 1984; Taraban & McClelland, 1987): 12 each of high- and low-frequency regular and exception words. "Exception" is defined by this corpus. Have is an exception because gave, save, wave, cave, brave, and grave are all in the training set. The pronunciations are General American English. Where American speakers differ, the targets follow dictionary pronunciations: saw, ball, and straw have /ɔ/, as for speakers who distinguish cot from caught, and what has /ʌ/.
Each reset starts from new random weights, so runs differ a little. Compare two or three runs before drawing a conclusion.
Seidenberg and McClelland's error scores showed the frequency-by-regularity interaction found in human naming times: exception words were slower than regular words mainly at low frequency, because frequent exposure lets the network learn high-frequency exceptions well. Early in training, error is high for everything; with more training, high-frequency exceptions converge on regular words well before low-frequency exceptions do.
Their model read nonwords poorly compared with skilled readers (Besner et al., 1990). Plaut et al. (1996) showed that a network with their input and output coding reads nonwords about as well as people do. People also sometimes give nonwords with inconsistent bodies, like heaf, an exception-like pronunciation, and name them more slowly than consistent nonwords (Glushko, 1979).
Plaut et al. found that damaging a network like this one, which has no semantic pathway, did not produce a deficit as selective for exception words as the one seen in patients with surface dyslexia. Their account of surface dyslexia relied on the division of labor with a semantic pathway, which this demo doesn't include.
The learning principles and the test design follow SM89. Almost everything about the scale and the representations is different.
| SM89 | This demo | |
|---|---|---|
| Training words | 2,897 one-syllable words | About 430 one-syllable words, including the 48 test words |
| Frequency | Presentation probability based on log-compressed Kučera–Francis word frequency | Two levels, high and low, with an adjustable ratio |
| Spelling input | 400 units, each responding to a large, randomly chosen set of letter triples | 105 grapheme units in onset, vowel, and coda positions (Plaut et al., 1996) |
| Sound output | 460 Wickelfeature units, each coding a triple of phonetic features across neighboring phonemes | 59 phoneme units in onset, vowel, and coda positions (Plaut et al. used 61) |
| Hidden units | 200 (100 in a comparison run) | 100 by default, adjustable |
| Other outputs | Also trained to reproduce the spelling, used to model lexical decision | Pronunciation only |
| Error measure in training | Squared error | Cross-entropy, as in Plaut et al.; squared error is still reported as the naming-time measure |
| Reading out a pronunciation | No procedure for reading a pronunciation off the output; performance was judged from error scores, including comparisons of the correct pronunciation with alternatives | Read directly from the output units |
| Meaning | Part of the framework (the "triangle"), but not implemented | Not implemented |
SM89 borrowed its coding scheme from Rumelhart and McClelland's (1986) past-tense model, which built on Wickelgren (1969). A word is treated as a string of overlapping triples: make becomes _ma, mak, ake, ke_. On the spelling side, each of 400 units responds to many such letter triples. On the sound side, each unit codes a feature of one phoneme together with features of the phonemes on either side. Nothing marks a letter's position in the word or syllable. Position is implicit in the neighbors.
Plaut et al. (1996) split every spelling into an onset (initial consonants), a vowel, and a coda (final consonants, including a silent e). There is one unit for each grapheme in each slot, and multi-letter graphemes such as th, ea, and ck are single units. Plaid is onset p + onset l + vowel ai + coda d. Sounds are coded the same way, one unit per phoneme per slot. Order inside a consonant cluster is not coded, because English sound patterns fix it (/s/ comes before /t/ at the start of a syllable, /n/ before /t/ at the end), and the readout applies that order. Plaut et al. added units for /ps/, /ks/, and /ts/ where order would otherwise be ambiguous; this demo keeps only /ks/, spelled with x.
Yes, though not for the central claim. The change leaves intact what the theory says reading is: the learned mapping of spelling onto sound by a single network of weighted connections, shaped by how often each word is encountered, with no rules and no separate store of words. It changes whether the model succeeds, and the model succeeds partly because of structure the modelers put into the representation format. That makes the encoding a substantive part of the theory, not a technical detail.
What doesn't depend on the encoding. The core claim of SM89 is that a single network, with no rules and no separate word store, learning from exposure weighted by frequency, can read both regular and exception words and will show the frequency-by-regularity interaction. Networks with either encoding produce that result, so the claim doesn't rest on Wickelfeatures or on the slot scheme.
What does: generalization. In a Wickelfeature scheme, the same letter activates different units in different words. The p in pint (_pi, pin) and the p in spade (spa) appear in completely different letter triples, and no unit stands for p by itself. What the network learns about p in one word therefore carries over to another only indirectly. Plaut et al. called this the dispersion problem, and it is the main reason SM89 read nonwords much worse than skilled readers. That failure was the strongest argument against SM89 and for dual-route models with a separate rule system (Besner et al., 1990; Coltheart et al., 1993). With the slot scheme, onset p is the same unit in every word, and nonword reading reached human levels without any change in architecture. The lesson is that the nonword problem was about representations, not about single-mechanism networks as such. A network generalizes along whatever similarity structure its representations encode, so any claim about what "a connectionist network" can do is really a claim about a network with particular representations.
What does: what is built in. The slot scheme hands the network knowledge a beginning reader has to acquire: that th and ea act as units, that syllables have onsets and codas, and which order consonants go in. Critics argued that this does some of the work the model is supposed to explain. Plaut et al. replied that these representations plausibly come from earlier learning, since spoken language supplies syllable structure and exposure to print supplies common letter groups. Later models learned more of this structure rather than having it supplied, and how much has to be built in remains an open question.
Smaller differences. Slot coding as used here only handles one-syllable words; Wickelfeatures were not limited to one syllable, although SM89 was trained only on monosyllables. And the slot scheme lets a pronunciation be read straight off the output units, which SM89's scheme did not.