Tiny Transformer — Attention for Next-Word Prediction

Training data

⬆ Upload .txt

Embedding dims

Head neurons

Attention heads

Learning rate

Speed

Epoch

Loss

—

Perplexity

—

Log-likelihood

—

Loss (cross-entropy) per epoch

Predict the next word

Your 4 words

→

Model's next word

Test a prediction — actual next word:

Attention

When predicting, the last word looks back at all 4 words. Thicker green = more attention. Watch it change as the model learns (and when you change the words).