Testing Guide
A prompt or skill is code. It gets tests.
Three Layers of Test
- Golden inputs - at least 3 realistic inputs per entry, including one edge case and one adversarial case.
- Assertions - deterministic checks: required sections present, valid JSON, word limit met, no banned phrases, citations present.
- Judge rubric - an LLM-as-judge score from 1 to 5 against a written rubric, with the judge prompt itself versioned in the registry.
Pass Criteria
- All deterministic assertions pass on all golden inputs.
- Mean judge score of 4.0 or higher, no single score below 3.
- Results recorded for each model the entry claims to support.
Tooling Options
- Plain Markdown records (see
example-test.md) are enough to start. - Promptfoo or a small Python harness can automate the assertions in CI once you have 20+ entries.
- Run the suite on every pull request touching an entry and on every model change.
The Judge Prompt (versioned as JUDGE-01)
You are grading the output of a registered prompt.
<rubric>{{rubric}}</rubric>
<input>{{golden_input}}</input>
<output>{{model_output}}</output>
Score 1 to 5 against each rubric line. Quote the evidence for each score.
Return JSON: {"scores": {...}, "mean": n, "fail_reasons": [...]}
Links
- Up: Registry home