ragbench-lite
BM25 + LiteLLM RAG pipeline scored by recall@k, faithfulness, and relevance (LLM-as-judge) — rendered as a self-contained HTML report.
Pipeline
ingest → retrieve → answer → evaluate → report
Five small modules, one Makefile, one command (make demo) reproduces
the full run end to end, including the report.
| module | responsibility |
|---|---|
ragbench/ingest.py | chunks a markdown corpus with LangChain's RecursiveCharacterTextSplitter |
ragbench/retrieve.py | BM25 index (rank_bm25) over chunks, top-k search |
ragbench/answer.py | grounded answer generation via LiteLLM |
ragbench/evaluate.py | recall@k + LLM-as-judge scoring (faithfulness, relevance) |
ragbench/report.py | self-contained HTML report (Jinja2, no external assets) |
Metrics
| metric | definition |
|---|---|
| recall@k | gold source document present in the top-k retrieved chunks, per question |
| faithfulness | judge scores every claim in the answer against the retrieved context (0–1) |
| relevance | judge scores the answer against a human-written reference answer (0–1) |
Current results
10 golden questions (eval/questions.jsonl: question, reference answer,
gold source doc) over a real corpus — Agent Barn architecture decision records and
Klikt docs.
| metric | score |
|---|---|
| recall@5 | 0.90 |
| faithfulness | 1.00 |
| relevance | 0.95 |
The single recall@5 miss is known: one gold ADR was absent from the ingested corpus. This is the intended behavior of the metric — it catches corpus and chunking gaps before users do.
Reproduce
python3 -m venv .venv && .venv/bin/pip install langchain-text-splitters litellm rank_bm25 jinja2 export OPENROUTER_API_KEY=sk-... # or any LiteLLM provider key make demo # ingest → evaluate → report.html
Model strings are configurable via RAGBENCH_MODEL and
RAGBENCH_JUDGE_MODEL (any LiteLLM model string). The published build used
openrouter/google/gemini-3-flash-preview for both roles.