RUNNABLE TOOLKIT · VERSION 1.0.0
Score recorded retrieval and chatbot outputs, compare a baseline with a regression and inspect release gates.
For AI developers and evaluation leads.
USD · one-time purchase · AI design category: $6.99. This category includes its guided skills and this separate toolkit ZIP. Any applicable tax is shown at checkout.
Python 3.10+ · Standard library only · Runs locally · No API key
INSPECT BEFORE YOU BUY
All examples below use the shipped synthetic fixtures. They demonstrate operation and limitations, not customer outcomes.
01 / SUPPLIED INPUT
corpus.jsonl: 11 fictional source documents
cases.jsonl: 8 synthetic support cases
predictions-baseline.jsonl
predictions-regression.jsonl
gates.json: authored release thresholdsRecorded predictions are supplied. The evaluator does not call a model or generate answers.
02 / RUN LOCALLY
python scripts/evaluate.py --corpus examples/corpus.jsonl --cases examples/cases.jsonl --predictions examples/predictions-regression.jsonl --gates examples/gates.json --output reports/regressionExtract the toolkit ZIP and run from its folder. Use a new output directory. On Windows, py -3 can replace python; on macOS/Linux, use your Python 3 command.
03 / READ THE REPORT
Baseline: exit 0 · 0 automated gate failures
Deliberate regression: exit 1 · 14 gate failures
Recall at 3: 71.43% · Citation coverage: 71.43%
Decision-label match: 75%
Semantic review: not_performed04 / FOLLOW AN EXCEPTION
Case C05 cites an existing document but makes an unsupported compliance claim. Valid citation IDs alone cannot establish answer correctness. The worked human rubric blocks that answer; the evaluator does not automate this semantic judgment.
# RAG evidence-metadata evaluation
Automated status: **automated_fail**
Semantic accuracy, grounding and usefulness: **human review required**.
Passing these gates is not a production release approval or automated hallucination check.
## Macro metrics
| Metric | Value | Eligible cases |
| --- | ---: | ---: |
| retrieval_precision_at_k | 0.380952 | 7 |
| retrieval_recall_at_k | 0.714286 | 7 |
| citation_validity | 0.714286 | 7 |
| required_citation_coverage | 0.714286 | 7 |
| decision_accuracy | 0.750000 | 8 |
## Gate failures
- `{"actual": 0.38095238095238093, "metric": "retrieval_precision_at_k", "minimum": 0.5, "scope": "aggregate"}`
- `{"actual": 0.7142857142857143, "metric": "retrieval_recall_at_k", "minimum": 0.8, "scope": "aggregate"}`
- `{"actual": 0.7142857142857143, "metric": "citation_validity", "minimum": 1.0, "scope": "aggregate"}`
- `{"actual": 0.7142857142857143, "metric": "required_citation_coverage", "minimum": 1.0, "scope": "aggregate"}`
- `{"actual": 0.75, "metric": "decision_accuracy", "minimum": 1.0, "scope": "aggregate"}`
- `{"actual": 0.0, "metric": "retrieval_recall_at_k", "minimum": 1.0, "scope": "C02"}`
- `{"actual": 0.0, "metric": "required_citation_coverage", "minimum": 1.0, "scope": "C02"}`
- `{"actual": 1, "maximum": 0, "metric": "invalid_citations", "scope": "C02"}`
- `{"metric": "retrieval_document_integrity", "scope": "C04", "unknown_document_ids": ["D99"]}`
- `{"actual": 0.0, "metric": "retrieval_recall_at_k", "minimum": 1.0, "scope": "C04"}`
- `{"actual": 0.0, "metric": "required_citation_coverage", "minimum": 1.0, "scope": "C04"}`
- `{"actual": 1, "maximum": 0, "metric": "invalid_citations", "scope": "C04"}`
- `{"actual": "answer", "expected": "refuse", "metric": "expected_decision", "scope": "C07"}`
- `{"actual": "answer", "expected": "refuse", "metric": "expected_decision", "scope": "C08"}`
## Case review queue
| Case | Critical | Decision match | Invalid citations | Missing required citations | Human review |
| --- | --- | --- | ---: | ---: | --- |
| C01 | False | True | 0 | 0 | not performed |
| C02 | True | True | 1 | 2 | not performed |
| C03 | False | True | 0 | 0 | not performed |
| C04 | True | True | 1 | 1 | not performed |
| C05 | True | True | 0 | 0 | not performed |
| C06 | False | True | 0 | 0 | not performed |
| C07 | True | False | 0 | 0 | not performed |
| C08 | True | False | 0 | 0 | not performed |
See report.json for answers, ordered retrieval, citation IDs, input SHA-256 hashes and source line numbers.
WHAT ARRIVES IN YOUR DOWNLOAD
The supplied results describe eight fictional cases. They are metadata and label checks, not a benchmark of a real chatbot. Human assessment of meaning, support and factual grounding remains necessary even when the automated gates pass.
Standalone toolkit files are MIT licensed. The category’s premium guided runbooks have separate single-user terms. Ongoing consulting, hosted execution and model subscriptions are not included.
CHOOSE YOUR SCOPE
The AI design category is $6.99. The complete 108-skill collection is $19.99 and includes all 3 runnable toolkits as separate ZIPs. Individual $0.99 skills are guided instructional packs.