distributed-llm-pretraining-torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Links

• Roles: llm-engineer
• Source repo: Orchestra-Research/AI-Research-SKILLs
• Same repo: axolotl, deepspeed, fine-tuning-openvla-oft, fine-tuning-serving-openpi, awq-quantization, gguf-quantization, gptq, hqq-quantization
• Registry seed skills for this role: llm-engineer seed skills