Evaluation datasets for the graph scorer

Question: is there a public, relevance-labelled dataset that can measure whether graph proximity improves nDCG? Milestone 4 does not exist without one.

Answer: yes. No single dataset works off the shelf — it has to be human-made labels plus a link graph joined from a separate source (§1). Milestone 2’s scope is unaffected (§4).

1. Disqualification criterion

The graph scorer converts hop distance from a seed into 1/(1+hops). A dataset whose labels were derived from the same link structure is unusable: the scorer reproduces the label-generation rule, so a higher nDCG is guaranteed and means nothing. Same failure mode as the seed echo in FINDINGS §2.3 — there one text vote was counted twice, here the answer key would be visible during the exam.

2. Candidates assessed

NFCorpus — the dangerous near miss

3,633 documents and 323 queries make it the first candidate anyone reaches for, because it fits an in-memory engine exactly. Its label construction:

That is very nearly the function the graph scorer computes, and all 169,756 judgments in the original release are automatically extracted rather than human-assessed. Recorded explicitly because the attractive size makes it likely to be picked up again.

3. Recommendation — TREC-COVID joined with Semantic Scholar citations

Join on document id. The point is that labels and graph come from different places.

Why TREC-COVID specifically:

• 493.5 qrels per query on average — the highest in BEIR, where most are under 5.
• That depth is decisive here. Documents absent from qrels count as irrelevant, so shallow judgments penalize a system for surfacing relevant-but-unjudged documents — exactly what the graph scorer exists to do.
• 171K documents and 50 queries, within reach of an in-memory index.
• nDCG@10 is BEIR’s primary metric, matching milestone 4’s directly.

Measurement requirements

1. Include the neighbourhood when restricting to the judged pool. Re-ranking over judged documents only is standard and sufficient for an A/B, but keeping only the pool turns every citation edge to an unjudged document into a dangling edge, disabling the graph scorer. Index the pool plus its one- and two-hop neighbours; neighbours participate in traversal and count as unjudged when scoring.
2. Three arms, not two — required by FINDINGS §2.3: text + vector as baseline, + graph.New for the real contribution, + graph.NewIncludingSeeds to quantify how much double counting inflated it. Without the third, an improvement cannot be attributed to the graph rather than to doubled text weight.
3. Expect no link structure from the tooling. In the ir_datasets catalogue, links and citations mean documentation links and bibliographic entries, not inter-document edges; its entity types are docs, queries, qrels, docpairs and scoreddocs. BEIR’s corpus.jsonl carries only _id, title and text. The graph must be joined in.
4. Sweep RRF k alongside. Damping is stronger than expected (FINDINGS §3.2), so measuring the graph contribution at a fixed k risks measuring k = 60 instead.

4. Effect on milestone 2

This survey ran ahead of milestone 2 because its outcome could have changed that scope. It does not, and it validates two existing design points:

• Keep Document.Links keyed by document key. The recommended path is exactly “join an external citation graph by document id”, and many targets fall outside the corpus, remaining dangling — already handled by design (TestDanglingLinksAreIgnored). A DocID adjacency list would have blocked this path.
• Persist the graph as a first-class scorer. No case for dropping it; milestone 4 is executable.

Sources

BEIR corpus · BEIR statistics · NFCorpus · NFCorpus in ir_datasets · mteb/nfcorpus · DBpedia-Entity v2 · SIGIR'17 paper · ClueWeb09 · TREC 2013 Web Track · TREC 2014 Web Track · HotpotQA · ir_datasets · Semantic Scholar Open Data