1Cademy - Unbiased GraphRAG Evaluation Framework (Zeng et al., 2025)

Learn Before

Research

Concept

Unbiased GraphRAG Evaluation Framework (Zeng et al., 2025)

Unbiased GraphRAG evaluation framework is a methodological proposal by Zeng et al. (2025) targeting answer-evaluation pipelines for Graph Retrieval-Augmented Generation (GraphRAG). The authors identify two critical flaws in canonical GraphRAG answer evaluation: (i) unrelated questions, where the evaluation questions are not actually grounded in the underlying graph or text corpus, and (ii) evaluation biases, where LLM-as-judge assessments systematically favor certain answer styles independent of correctness. The framework's corrective has two parts: graph-text-grounded question generation, which constructs evaluation questions from anchor subgraphs together with their supporting text so the questions are answerable from the corpus, and an unbiased judging procedure that mitigates LLM-judge biases. Applied to three representative GraphRAG systems, the framework reports that previously claimed performance gains are substantially more moderate than originally stated, supporting the broader observation that graph-aware RAG comparisons are highly sensitive to evaluation-protocol choices.