emerging-techniques-knowledge-distillation

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

Links

• Roles: llm-engineer
• Source repo: davila7/claude-code-templates
• Same repo: eval-genius, evaluation-bigcode-evaluation-harness, evaluation-lm-evaluation-harness, evaluation-nemo-evaluator, prompt-engineering-dspy, distributed-training-deepspeed, distributed-training-megatron-core, distributed-training-pytorch-fsdp
• Registry seed skills for this role: llm-engineer seed skills