Case study

LLM From Scratch

Hand-written decoder-only Transformer with causal attention and a Gradio teaching UI. Toy Shakespeare metrics only. Not a production LLM.

At a glance

Plain summary for recruiters and visitors. Technical detail follows below.

What it is: I built a tiny character-level language model by hand. You can see attention and next-character prediction without a library Transformer wrapper.
What I owned: Solo end to end. Tokenizer, attention, decoder LM, LSTM baseline, training loop, tests, and the Gradio teaching UI.
Why it matters: Makes the decoder rules inspectable. Causal mask, scaled attention, and next-token softmax on a toy corpus anyone can re-run.
Try it on this page vs full project: This page is the architecture story and published toy metrics. The interactive Gradio UI runs locally for hiring managers on request. A free Cloudflare playable demo is planned next.

Quick comparison

Toy Transformer val loss
About 2.54 after 400 steps on the short Shakespeare excerpt.
LSTM baseline
About 2.71 val loss on the same data and step budget.

Problem

Most LLM demos wrap a library Transformer or a hosted API. That hides the causal mask and attention math you need to debug and explain a decoder model.

Method

Implemented scaled dot-product attention and a Pre-LN decoder LM by hand. Trained a toy char model on a short Shakespeare excerpt. Added an LSTM baseline and a Gradio UI that shows the causal map and next-character bars.

Result

On the published toy holdout, Transformer val loss about 2.54 versus LSTM about 2.71 after 400 steps. The UI proves the mask and next-character softmax. Full training repo stays on request.

Architecture

How the system is shaped. Full implementation stays private.

Step 1
Tokens
Character tokenizer over the training text. Small vocab so the UI can show each symbol, including spaces as spc.
Step 2
Attention
Scaled dot-product Q Kᵀ / √d_k with a lower-triangular causal mask before softmax. Multi-head merge and a linear output projection.
Step 3
Decoder LM
Pre-LN Transformer blocks with token and position embeddings, residual skips, and a tied output head for next-character logits.
Step 4
Teach and compare
Gradio UI walks prompt → causal map → next-character bars. An LSTM baseline trains on the same toy data for a lineage comparison.

Toy holdout loss (same corpus)

Character LM on a short public-domain Shakespeare excerpt. Both models train for 400 steps with block size 64 and seed 42. Numbers are educational. They are not a production LLM claim.

Model	Val loss	Note
Decoder-only Transformer	2.54	Published toy
LSTM baseline	2.71

Algorithm

Causal scaled dot-product attention

Score every past key for each query, hide the future with −∞, softmax, then mix values. That is the decoder-only LM rule.

scores = (Q @ K.T) / sqrt(d_k)
scores = mask_future(scores, fill=-inf)  # lower triangle only
weights = softmax(scores, dim=-1)
out = weights @ V

Key logic

Compact illustrative snippet (Causal mask before softmax (illustrative)). Not the full codebase.

def causal_mask(seq_len, device=None):
    return torch.tril(torch.ones(seq_len, seq_len, dtype=torch.bool, device=device))

scores = (q @ k.transpose(-2, -1)) / math.sqrt(d_k)
scores = scores.masked_fill(~mask, float("-inf"))
weights = F.softmax(scores, dim=-1)

Stack

PyTorchTransformersAttentionCausal LMNLPGradioPython

Source code

Full source is available to hiring managers on request. The public page shows architecture, algorithms, and compact proofs only.