Benchmark results
Accuracy by tier, latency, and cost for the strategies recorded in each run.
Plan a run
Pick a corpus and strategies, then copy the command to run in your terminal.
kb-arena benchmark --corpus aws-compute --strategy naive_vector,contextual_vector,qna_pairs,knowledge_graph,hybrid,raptor,pageindex,bm25,qisskb-arena retriever-lab --corpus aws-compute --strategies naive_vector,contextual_vector,qna_pairs,knowledge_graph,hybrid,raptor,pageindex,bm25,qissMethodology
Each question is sent to the strategies recorded in the run. Answers are evaluated through a 4-pass pipeline: structural checks (must_mention / must_not_claim), entity coverage against source documentation, source attribution, and LLM-as-judge scoring for accuracy, completeness, and faithfulness.
Composite ranking: 0.5 * accuracy + 0.3 * reliability + 0.2 * latency_score. Latency score inverts p95 so lower is better.
Tiers: 1 = lookup (single fact retrieval), 2 = how-to (procedure within one topic), 3 = comparison (option A vs B), 4 = integration (cross-topic dependencies), 5 = architecture (3+ topics, system design).