Reading Notes: “Training Compute-Optimal Large Language Models”

Reading Notes: “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”

Reading Notes: GPT Series

Reading Notes: “Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning”

Reading Notes: “GPipe: Easy Scaling with Micro-Batch Pipeline”

Distributed Training Basics

Reading Note: Megatron-LM v1

© Lifan Sun 2023 - 2025