What is an RNN?

Softmax β†’ next word hβ‚€=0 Wβ‚•β‚• Β· carries the memory β†’ h₁ hβ‚‚ h₃ hβ‚„ Wβ‚“β‚• Wₕᡧ w₁ wβ‚‚ w₃ wβ‚„ Read left β†’ right: the SAME weights (Wβ‚“β‚•, Wβ‚•β‚•, Wₕᡧ) are reused at every step.

A Recurrent Neural Network reads a sequence one word at a time, keeping a small memory (a hidden vector h) that it updates at every step. Here it reads your 4 words left β†’ right; after each word it folds that word into its memory, and the final memory predicts the next word.

ht = tanh( WxhΒ·xt  +  WhhΒ·htβˆ’1  +  b )

The same weights are reused at every step β€” that weight-sharing is the whole idea, and it's why an RNN can read a sequence of any length (5 words, 50 words…), not a fixed number.

vs. #6 Word Predictor: that model glued 4 word-vectors side-by-side and always needed exactly 4 inputs. The RNN reuses one tiny cell across the sequence, carrying earlier words forward inside its memory.

How it learns: backpropagation through time β€” the error at the last step flows backward through all 4 steps to nudge the weights and the word embeddings.

Watch the 🧠 memory rows below: each row is the memory after reading that word. Change the last word and the bottom row shifts the most.

Training data

Epoch
0
Loss
β€”
Perplexity
β€”
Log-likelihood
β€”
Loss (cross-entropy) per epoch

Predict the next word β€” read one word at a time

Your 4 words
β†’
Embeddings
β†’
Model's next word
🧠 The RNN's memory (hidden state) after reading each word

Word embedding map

Each word is a point. As the model learns, words it uses the same way move together.

πŸ§ͺ Parameters & things to try

Embedding dims

Each word's vector size. Keep at 2 to watch words cluster on the map; try 4 or 8 for more capacity (harder to picture).

Memory size

How much the RNN can remember (hidden units). Try 6 vs 16 β€” does a bigger memory reach a lower loss? Too big can overfit tiny data.

Learning rate

0.03 is slow & steady; 0.3 is fast but RNNs can get unstable β€” watch the loss curve bounce.

Your 4 words

Change them and watch the prediction and the memory rows update β€” the RNN responds to the whole sequence, not just the last word.

Upload .txt

Train on your own sentences. Short, repetitive text with a small vocabulary learns fastest.

Compare

Open #6 Word Predictor with the same text β€” similar predictions, very different architecture.

  1. Watch the memory: Train, then change word 4 β€” the bottom memory row changes the most.
  2. Bigger brain? Set Memory size to 16, press Reset, then Train β€” compare final perplexity with 6.
  3. Break it on purpose: Set Learning rate to 0.3 and Train β€” watch the loss bounce (instability).
  4. Your turn: Upload a short .txt of your own and see what it predicts.