type skillowner Thomas Cherickalcurator Thomas Cherickalrole local-ai-engineercategory ai-ml/llm-systemsversion 0.1.0status seedupdated 2026-11-15
gguf-quant-selection
Steps
- Record RAM, VRAM, CPU and target context length.
- Estimate weight memory: params x bits / 8, plus KV cache and overhead.
- Shortlist quantisations (e.g. Q4_K_M, Q5_K_M, Q8_0) that fit.
- Benchmark tokens per second and quality on a small task set.
- Recommend one model and quant with the numbers.
Done When
- [ ] Recommendation fits memory with headroom
Registry
| id |
role |
version |
status |
owner |
last_tested |
| SK-LOCAL-AI-ENGINEER-gguf-quant-selection |
local-ai-engineer |
0.1.0 |
seed |
Thomas Cherickal |
- |
Links
