What is an RNN?

Softmax → next word h₀=0 Wₕₕ · carries the memory → h₁ h₂ h₃ h₄ Wₓₕ Wₕᵧ w₁ w₂ w₃ w₄ Read left → right: the SAME weights (Wₓₕ, Wₕₕ, Wₕᵧ) are reused at every step.

A Recurrent Neural Network reads a sequence one word at a time, keeping a small memory (a hidden vector h) that it updates at every step. Here it reads your 4 words left → right; after each word it folds that word into its memory, and the final memory predicts the next word.

ht = tanh( Wxh·xt  +  Whh·ht−1  +  b )

The same weights are reused at every step — that weight-sharing is the whole idea, and it's why an RNN can read a sequence of any length (5 words, 50 words…), not a fixed number.

vs. #6 Word Predictor: that model glued 4 word-vectors side-by-side and always needed exactly 4 inputs. The RNN reuses one tiny cell across the sequence, carrying earlier words forward inside its memory.

How it learns: backpropagation through time — the error at the last step flows backward through all 4 steps to nudge the weights and the word embeddings.

Watch the 🧠 memory rows below: each row is the memory after reading that word. Change the last word and the bottom row shifts the most.

Training data

Epoch
0
Loss
—
Perplexity
—
Log-likelihood
—
Loss (cross-entropy) per epoch

Predict the next word — read one word at a time

Your 4 words
→
Embeddings
→
Model's next word
🧠 The RNN's memory (hidden state) after reading each word

Word embedding map

Each word is a point. As the model learns, words it uses the same way move together.

🧪 Parameters & things to try

Embedding dims

Each word's vector size. Keep at 2 to watch words cluster on the map; try 4 or 8 for more capacity (harder to picture).

Memory size

How much the RNN can remember (hidden units). Try 6 vs 16 — does a bigger memory reach a lower loss? Too big can overfit tiny data.

Learning rate

0.03 is slow & steady; 0.3 is fast but RNNs can get unstable — watch the loss curve bounce.

Your 4 words

Change them and watch the prediction and the memory rows update — the RNN responds to the whole sequence, not just the last word.

Upload .txt

Train on your own sentences. Short, repetitive text with a small vocabulary learns fastest.

Compare

Open #6 Word Predictor with the same text — similar predictions, very different architecture.

  1. Watch the memory: Train, then change word 4 — the bottom memory row changes the most.
  2. Bigger brain? Set Memory size to 16, press Reset, then Train — compare final perplexity with 6.
  3. Break it on purpose: Set Learning rate to 0.3 and Train — watch the loss bounce (instability).
  4. Your turn: Upload a short .txt of your own and see what it predicts.