agent-trace-evals

Evaluate tool-using LLM agents and RAG pipelines the way they fail: stochastically and through their traces. Python + strict TypeScript, MIT.

What is in it

Results (64 synthetic IT-helpdesk docs, Swedish/English, 30 queries)

retrieval (26 answerable)recall@1MRRnDCG@5
BM250.6150.6990.736
dense (multilingual MiniLM)0.8850.9360.952
hybrid + bge rerank (default)0.8460.9170.938
hybrid + jina rerank (comparison, non-commercial)0.9620.9810.986

end to end, free LLM, bge rerankanswer accuracyrefused unanswerablefalse refusalsfaithfulnesslatency p50 / p95
with refusal gate1.004 / 400.8854.9 s / 7.9 s
without gate1.004 / 400.8465.3 s / 13.4 s

Caveats: one run per configuration on 30 queries; the gate threshold and other settings were tuned on the same queries (no held-out set). The bge reranker does not beat dense retrieval alone on this small set. Faithfulness is a conservative lexical check, and answer accuracy is a lenient key-fact match. This is a demonstration of the engineering pattern, not a benchmark.

How it was built

Built by Markus Isaksson with AI assistance: Grok Bot wrote specs and code, other models proposed patches, and the module went through several rounds of read-only AI review (scored on correctness, honesty of claims, eval methodology and licensing). Fixes were integrated until the reviewer rated it publishable. Tests re-check the published results against the recorded traces.

Repo: github.com/MarkusIsaksson1982/agent-trace-evals · release v0.2.0