Yesterday, It Fit: Dividing a 70B Model Across 64 GPUs
A worked example in training memory, long context, and choosing a parallel layout you can actually evaluate.
Latest articles
Self-Host vs. API for LLMs, Actually Costed
The decision as a cost calculation, not an ideology and why the idle accelerator is the cost nobody puts on the slide.
Making LLM Decode Cheaper: Quantization and More
Move fewer bytes per token, quantization (FP8/FP4), KV-cache compression, and speculative decoding and what each one costs.
Batching and the Idle GPU: Serving LLMs Faster
Static vs. continuous batching, prefill-decode disaggregation, and the throughput-latency tradeoff you're really choosing.
The Inference Wall: Where LLM Latency and Cost Go
Why the model that felt fast in the demo gets slow and expensive in production and the memory bottleneck nobody budgets for.
Beyond 1D Data Parallelism: ZeRO, TP, PP, and CP
When data parallelism hits its ceiling: sharding model state, splitting layers and sequences, and why the interconnect picks your strategy
Scaling the Batch and the Cluster: Gradient Accumulation and 1D Data Parallelism
A from-scratch data-parallel wrapper that overlaps gradient sync with the backward pass and buckets the collectives — and how to verify it on a profiler trace.
Activation Recomputation and the HFU/MFU Trap
How trading FLOPs for HBM works and the metric that makes a less efficient run look more efficient.
The Memory Wall: Where GPU Memory Actually Goes in LLM Training
Why activations, not parameters, are what really push you out of HBM and what to do about it.
