A recent tutorial shows that building a language model no longer requires a data centre. The authors train a model from scratch—including the tokenizer—using a single consumer GPU, demonstrating that a practice once reserved for large‑scale facilities is now accessible to hobbyists and small research groups.

The training pipeline uses a byte‑level BPE tokenizer and a compact six‑layer Transformer with a hidden dimension of 384. Training runs in fp16 with the AdamW optimiser. Hyperparameters such as learning rate and dropout are chosen by a grid search. Each document is split into 64‑token windows; an end‑of‑sequence token is added and padding applied.

Two generation modes are implemented on the same network. In the autoregressive mode the model predicts each token from previous ones using causal attention. In a masked diffusion mode the model predicts masked tokens from visible ones with bidirectional attention, following recent diffusion‑based language models.

On validation data the masked diffusion mode surpasses the autoregressive likelihood by 0.06 bits per byte, which meets a preregistered minimum effect size. Despite this advantage, both modes tie a 6‑gram language‑model floor on held‑out legal texts, showing that the model retains baseline performance.

However, the autoregressive mode exhibits significant over‑fitting on a protagonist corpus. After 256 epochs it reaches 0.036 bits per byte on training data but 2.851 on validation, indicating near‑perfect memorisation. The diffusion model is less prone to over‑fitting, achieving 1.550 on training versus 1.825 on validation.

These results illustrate that, with careful design and validation, a single consumer GPU can yield a functional language model, though attention must be paid to over‑fitting when using autoregressive training.