type skillowner Thomas Cherickalcurator Thomas Cherickalrole ai-engineercategory ai-ml/applied-aiversion 0.1.0status seedupdated 2026-11-15
model-selection-eval
Steps
- Collect the task definition, volume, latency budget and cost ceiling.
- Build or reuse a golden set of at least 30 real inputs.
- Run every candidate on the same set with the same prompt.
- Score with deterministic checks plus a versioned judge rubric.
- Report a table: quality, p95 latency, cost per 1k requests.
Done When
- [ ] Same prompt and temperature across models
- [ ] Results include variance across at least 3 runs
Registry
| id |
role |
version |
status |
owner |
last_tested |
| SK-AI-ENGINEER-model-selection-eval |
ai-engineer |
0.1.0 |
seed |
Thomas Cherickal |
- |
Links
