Nanochat: full-stack LLM training in 8,000 lines for $72

Andrej Karpathy's nanochat is the most transparent, minimal, and complete open-source pipeline for training a ChatGPT-class model from absolute scratch — tokenization through web UI — in roughly 8,300 lines of Python and Rust. A depth-24 model matching GPT-2's capability now trains in 2.76 hours on 8×H100 GPUs for approximately $72, a 600× cost reduction from OpenAI's original $43,000. For someone building sovereign AI infrastructure on constrained hardware like a Jetson Orin Nano, nanochat provides both a trainable baseline model and an architectural blueprint. The resulting ~561M–768M parameter models fit comfortably in 8GB of unified memory at INT4 quantization (~280 MB), achieving an estimated 40–70 tokens/sec — fast enough for real-time local triage in a multi-model relay system.

This report covers nanochat's architecture, training pipeline, code structure, edge deployment feasibility, and how it could anchor a hybrid local-cloud AI relay with air-gapped security requirements.

One dial controls everything: the depth-first design philosophy

Nanochat's most distinctive design decision is its single complexity dial: --depth. This integer — the number of transformer layers — automatically determines every other hyperparameter: model width (depth × 64), number of attention heads (width ÷ 128), learning rate scaling (∝ 1/√(dim/768)), weight decay schedule, training token budget, and iteration count. A user never configures anything else. They simply request a smaller or bigger model, and the scaling laws produce a compute-optimal result.

This approach emerged from Karpathy's explicit goal of creating a "maximally forkable" repository — not a framework with configuration abstractions, but a single coherent codebase where any modification must work across all depth settings. The project serves as the capstone for his LLM101n course and doubles as a community research harness with a competitive speedrun leaderboard. The current record stands at 2.76 hours to exceed GPT-2's DCLM CORE score of 0.256525, achieved with a d24 model (~768M parameters, 4.33×10¹⁹ total FLOPs).

Model sizes scale predictably with depth:

DepthParametersTraining cost (8×H100)Comparable capability
d4~32MFree (CPU/MPS)Educational toy
d20~561M~$100 / 3–4 hoursConversational baseline
d24~768M~$73 / 3 hoursGPT-2 grade (CORE >0.256)
d30~1.8B~24 hoursGPT-3 Small (MMLU 40s, ARC-Easy 70s)
d32~1.88B~$1,000 / 41 hoursProduction quality

Karpathy discovered through 320 hyperparameter sweep experiments that nanochat's optimal token-to-parameter ratio is approximately 8:1 — far below Chinchilla's canonical 20:1 — meaning nanochat prefers bigger models trained shorter, likely due to the Muon optimizer's convergence properties.

A Llama-class transformer with ten modern departures

The model architecture in nanochat/gpt.py (~291 lines) implements a Llama-type decoder-only transformer with several carefully chosen innovations. Each was validated across depth settings, and features that didn't generalize were ruthlessly cut — multi-token prediction, asymmetric softcap, and bigram embeddings all failed this test.

Rotary Position Embeddings (RoPE) replace learned positional embeddings, enabling better length generalization with a base theta of 10,000. RMSNorm without learnable parameters is applied everywhere — after embedding, before each attention and MLP block, and before the output head — using bare F.rms_norm(). QK Normalization bounds attention scores to [-1, 1] by normalizing Q and K after RoPE, eliminating the need for softcapping within attention. Logit softcapping at 15 (computed in float32 via 15 * tanh(logits/15)) bounds output logits to [-15, 15].

The MLP uses ReLU² activation (F.relu(x).square()) instead of GELU — sparse and computationally cheaper with a 4× expansion ratio. Sliding window attention follows an "SSSL" tiling pattern: three short-window layers (1,024 tokens) followed by one long-window layer (full 2,048 context), with the final layer always using full context. This leverages Flash Attention 3's native window_size parameter on Hopper GPUs.

The most parameter-efficient innovation is Value Embeddings: at alternating layers, a gated embedding lookup adds directly to the V tensor, contributing ~150M parameters at d24 with near-zero additional FLOPs. Per-layer residual scalars (resid_lambdas initialized to 1.0, x0_lambdas to 0.1) create learnable skip connections that mix each layer's residual stream with the original normalized embedding — yielding 0.003–0.01 bits-per-byte improvement. Input and output embeddings are untied, with different initializations and learning rates.

The optimizer system is equally distinctive. MuonAdamW is a dual optimizer: Muon (with Polar Express orthogonalization and Adafactor-style variance reduction) handles all weight matrices, while AdamW handles embeddings and scalars. The distributed variant (DistMuonAdamW) implements ZeRO-2 style sharding with three-phase async gradient sync, eliminating the need for PyTorch DDP entirely.

Five training stages from raw text to chat assistant

The full pipeline executes sequentially through runs/speedrun.sh, which runs start-to-finish on a blank cloud box.

Stage 0 — Tokenizer training uses a custom Rust BPE implementation (rustbpe/src/lib.rs) compiled via pyo3/maturin. Python's minbpe was too slow; HuggingFace tokenizers too bloated. The tokenizer trains on 2–4 billion characters from FineWeb-EDU in under one minute, producing a 32,768-token vocabulary with ~4.8 characters per token. Nine special tokens define the chat protocol: <|bos|>, <|user_start|>, <|user_end|>, <|assistant_start|>, <|assistant_end|>, <|python_start|>, <|python_end|>, <|output_start|>, <|output_end|>. At inference time, OpenAI's tiktoken handles encoding for speed. Stage 1 — Base pretraining consumes ~75% of total compute. The dataset is FineWeb-EDU-100B-shuffle, re-packaged by Karpathy into 1,822 pre-shuffled Parquet shards (~100MB each, gzip compressed). A d24 speedrun requires ~240 shards for 8.8 billion tokens. The BOS-aligned dataloader uses a best-fit bin-packing algorithm achieving ~100% token utilization — a key innovation that made the separate midtraining stage unnecessary as of January 2026. Batch size is 524,288 tokens per step, yielding ~1.07M tokens/sec throughput at ~47–48% model FLOPs utilization on 8×H100. Validation uses bits-per-byte (BPB), a tokenizer-invariant metric, with periodic CORE evaluation against a 22-task benchmark ensemble. Stage 2 — Midtraining (now deprecated, merged into SFT) originally bridged document completion and conversation by exposing the model to SmolTalk conversations (460K rows), MMLU multiple-choice (100K rows), and GSM8K math problems (8K rows). Stage 3 — Supervised Fine-Tuning runs for ~7 minutes, processing conversations individually (not packed) to eliminate the domain mismatch between packed training and padded inference. Identity customization is supported via dev/gen_synthetic_data.py, which generates synthetic personality conversations that can be mixed into SFT data. Stage 4 — Reinforcement Learning (optional) implements a stripped-down GRPO focused on GSM8K. Karpathy deleted the trust region, reference model, KL regularization, PPO ratios, and clip — reducing it to essentially REINFORCE with group-relative, token-level advantages. GSM8K accuracy improves from ~4.5% to ~7.6% at speedrun scale, and up to ~20% at d32.

Code structure maps cleanly to the pipeline

The repository's 44 files organize into five clear areas:

The core library (nanochat/) contains the model (gpt.py), inference engine (engine.py), optimizer (optim.py), data loading (dataloader.py, dataset.py), evaluation (core_eval.py, loss_eval.py), checkpoint management (checkpoint_manager.py), Python sandbox for tool use (execution.py), tokenizer wrapper (tokenizer.py), Flash Attention dispatch (flash_attention.py), distributed training utilities (common.py), report generation (report.py), and the web UI (ui.html).

The scripts directory provides entry points for each pipeline stage: tok_train.py, base_train.py, chat_sft.py, chat_rl.py, plus evaluation scripts (base_eval.py, base_loss.py, chat_eval.py) and interfaces (chat_web.py, chat_cli.py).

The tasks directory defines training/evaluation datasets as clean abstractions: gsm8k.py, mmlu.py, arc.py, humaneval.py, spellingbee.py, smoltalk.py, and customjson.py for arbitrary JSONL conversation data. TaskMixture and TaskSequence in tasks/common.py compose these for mixed training.

The web server (chat_web.py) uses FastAPI with a WorkerPool managing multiple GPU workers, an OpenAI-compatible /chat/completions endpoint, and Server-Sent Events for streaming. The inference engine implements two-stage generation (prefill → decode) with KV caching. Tool use follows a state machine: the model emits <|python_start|>, the engine collects expression tokens until <|python_end|>, executes via the sandbox, then force-injects the result tokens back into the stream.

Conversation context is stateless per request — the entire history is re-encoded each time, bounded by the 2,048-token sequence length. There is no built-in persistent memory, session store, or context compression. This is by design: nanochat targets educational clarity over production features.

A 561M model runs comfortably on a Jetson Orin Nano

The Jetson Orin Nano (8GB variant, $249) provides 1,024 CUDA cores, 32 Tensor cores, and 8GB unified RAM at 102 GB/s bandwidth — up to 67 TOPS in the Super configuration. A nanochat d20 model (561M parameters) at FP16 occupies ~1.1 GB; at INT4 quantization, just ~280 MB. This leaves over 7 GB for the OS, inference framework, and application logic.

Community benchmarks on the Jetson Orin Nano Super show Qwen2 0.5B achieving ~45 tokens/sec via Ollama, and DeepSeek R1 1.5B hitting ~27.5 tokens/sec at MAXN power mode. A 561M nanochat model at INT4 would likely deliver 40–70+ tokens/sec — well above the interactive threshold. NVIDIA's official guidance positions the Orin Nano as suitable for models up to ~4B parameters.

The most practical inference path is Ollama installed natively (not containerized, which can fall back to CPU). Alternatively, llama.cpp provides direct CUDA inference, MLC-LLM offers TVM-compiled optimization (used in NVIDIA's own Jetson benchmarks), and TensorRT-LLM delivers the highest performance ceiling but requires cross-compilation from an x86 host. Nanochat doesn't natively export to GGUF, but community conversions exist: karpathy/nanochat-d32 on HuggingFace, onnx-community/nanochat-d32-ONNX with q4 quantization, and standard llama.cpp conversion tooling works on the HuggingFace export.

For air-gapped deployment, models must be pre-downloaded and packaged into Docker images or local NVMe storage. No ollama pull in disconnected mode — instead, models are imported from pre-built archives distributed via secure USB or internal transfer. Power modes (7W battery, 15W balanced, 25W MAXN) allow tuning the performance-power tradeoff for field conditions.

Anchoring a multi-model relay on local hardware

A nanochat-trained model on a Jetson could serve as the local triage layer in a three-tier architecture: edge device → optional near-edge → cloud. The small model classifies incoming queries by complexity and sensitivity, handles simple requests directly (~50ms latency), and routes complex reasoning tasks to cloud models when network and policy permit.

Several production-tested frameworks enable this pattern. RouteLLM (Berkeley/LM-Sys) routes between strong and weak models, achieving 85% cost reduction while maintaining 95% of GPT-4 quality on MT Bench. LiteLLM provides a unified OpenAI-compatible gateway with declarative routing policies: "if prompt contains PII, use local model." Fallback chains (local → cloud on failure) and three-tier routing (small/medium/large) are built-in. NVIDIA's LLM Router v2 Blueprint uses a Qwen 1.75B intent classifier or CLIP embeddings to match queries to optimal models. For Kubernetes environments, Red Hat's vLLM Semantic Router uses a Rust-based ModernBERT classifier with Envoy integration.

A practical sovereign AI stack would look like:

The critical limitation is that a 561M model will struggle with multi-step reasoning, nuanced coding, or complex math — but that's precisely why the relay exists. The local model's job is fast triage and privacy enforcement, not deep reasoning.

For context continuity across model handoffs, the local model maintains a JSON conversation log and entity store. When routing to cloud, it transmits a compressed context bundle: conversation summary, extracted entities, task classification, and the raw last few turns. The cloud model receives this as an enriched system prompt. Responses flow back through the local session, maintaining a unified conversation record.

Conclusion: transparent training meets practical edge deployment

Nanochat occupies a unique position in the LLM landscape — it is neither an inference runtime (llama.cpp, Ollama) nor a production serving system (vLLM), but the only complete, minimal, hackable training-through-deployment pipeline for ChatGPT-class models. Its ~8,300-line codebase is essentially hand-written (Karpathy noted that Claude/Codex agents were "net unhelpful" because the code is too far off their training distribution).

Three insights matter most for sovereign AI infrastructure builders. First, the single-dial depth architecture makes it trivial to train models precisely sized for target hardware — a d12 (~124M) for extremely constrained devices, d20 (~561M) for Jetson-class edge, or d32 (~1.88B) for beefier edge servers. Second, the complete training pipeline means you can retrain from scratch on domain-specific data in a fully auditable, air-gappable process — the entire dataset, code, and training procedure is transparent and reproducible. Third, the ~561M model at INT4 quantization fits in under 300 MB of memory, making it viable as an always-on local triage agent even on $249 hardware, while the relay architecture delegates complex work to more capable systems when connectivity and policy allow.

The gap between nanochat's educational focus and production deployment is real but bridgeable: community exports to GGUF and ONNX already exist, Ollama can serve the converted models, and frameworks like LiteLLM and RouteLLM provide the routing logic. The missing piece — which nanochat's design invites community contribution on — is persistent context management, conversation memory, and automatic query complexity classification tuned for relay routing decisions.