agent-trace-evals
Evaluate tool-using LLM agents and RAG pipelines the way they fail: stochastically and through their traces. Python + strict TypeScript, MIT.
What is in it
- Trace-based agent evals (standard-library Python): assertions on tools called, order, retrieval before answer, citations, latency/cost budgets; pass rate with a Wilson 95% lower bound per case.
- RAG module (numpy + fastembed, CPU only): BM25 and dense retrieval, hybrid fusion (RRF), cross-encoder rerank with
bge-reranker-base (MIT). jina-reranker-v2 is shown as a comparison only: it scores higher here but its licence is non-commercial (CC-BY-NC-4.0).
- Pipeline checks: a refusal gate on the rerank score, citations must point to retrieved documents, a lexical faithfulness check, one JSONL trace per query.
- TypeScript port (strict): retrieval and metrics only (BM25, dense, RRF, recall/MRR/nDCG, citation check), with tests asserting the same numbers as Python. Plus a TS tool router with budgets and fallbacks.
Results (64 synthetic IT-helpdesk docs, Swedish/English, 30 queries)
| retrieval (26 answerable) | recall@1 | MRR | nDCG@5 |
| BM25 | 0.615 | 0.699 | 0.736 |
| dense (multilingual MiniLM) | 0.885 | 0.936 | 0.952 |
| hybrid + bge rerank (default) | 0.846 | 0.917 | 0.938 |
| hybrid + jina rerank (comparison, non-commercial) | 0.962 | 0.981 | 0.986 |
| end to end, free LLM, bge rerank | answer accuracy | refused unanswerable | false refusals | faithfulness | latency p50 / p95 |
| with refusal gate | 1.00 | 4 / 4 | 0 | 0.885 | 4.9 s / 7.9 s |
| without gate | 1.00 | 4 / 4 | 0 | 0.846 | 5.3 s / 13.4 s |
Caveats: one run per configuration on 30 queries; the gate threshold and other settings were tuned on the same queries (no held-out set). The bge reranker does not beat dense retrieval alone on this small set. Faithfulness is a conservative lexical check, and answer accuracy is a lenient key-fact match. This is a demonstration of the engineering pattern, not a benchmark.
How it was built
Built by Markus Isaksson with AI assistance: Grok Bot wrote specs and code, other models proposed patches, and the module went through several rounds of read-only AI review (scored on correctness, honesty of claims, eval methodology and licensing). Fixes were integrated until the reviewer rated it publishable. Tests re-check the published results against the recorded traces.
Repo: github.com/MarkusIsaksson1982/agent-trace-evals · release v0.2.0