This dataset contains the retrieval run files (top 1,000 documents per query) of the ECIR 2027 reproducibility paper "Revisiting Dense Information Retrieval for Large Index Sizes". They were produced by BM25, SPLADE and three uncompressed dense retrievers on the full (not subsampled) corpora used in the paper. The code that produced and consumes them is the code repository linked in the paper.
The folder structure mirrors the RUNS_FOLDER of the code repository exactly:
<dataset>/full/<num_docs>/<system>.run.gz<dataset> is the ir-datasets id with / replaced by _, <num_docs> is the number of documents in the corpus, and
<system> is one of bm25, bm25-elasticsearch, splade, dense-sbert, dense-gemma, dense-jinav5.
SHA256SUMS lists a checksum for every run file (sha256sum -c SHA256SUMS).
Each file is a gzipped TREC run: one line per retrieved document, six whitespace-separated columns
qid Q0 docno rank score tagwith at most 1,000 documents per query (rank 1..1000). Sparse (BM25) runs can contain fewer than 1,000 lines for a query if fewer documents match.
| Dataset (directory) | Corpus size | Queries | Systems |
|---|---|---|---|
beir_msmarco_dev (MS MARCO v1 dev) | 8,841,823 | 6,980 | all six |
core_passage (CoRE Passage) | 100,000,000 | 65 | all six |
core_document (CoRE Document) | 10,000,000 | 55 | all six |
random-noise_beir_msmarco_dev_random-noise-reimers-no-query-words (Random characters, seed 42) | 100,000,000 | 6,980 | all six |
random-noise_beir_msmarco_dev_random-noise-reimers-no-query-words-43 (Random characters, seed 43) | 100,000,000 | 6,980 | all six |
random-noise_beir_msmarco_dev_random-noise-reimers-no-query-words-44 (Random characters, seed 44) | 100,000,000 | 6,980 | all six |
random-noise_beir_msmarco_dev_random-noise-faker (Random words, seed 42) | 100,000,000 | 6,980 | all except dense-gemma |
random-noise_beir_msmarco_dev_random-noise-faker-43 (Random words, seed 43) | 100,000,000 | 6,980 | all six |
random-noise_beir_msmarco_dev_random-noise-faker-44 (Random words, seed 44) | 100,000,000 | 6,980 | all six |
random-noise_beir_msmarco_dev_random-noise-reimers (verbatim noise) | 100,000,000 | 6,980 | all six |
Notes:
dense-gemma was not run on random-noise_beir_msmarco_dev_random-noise-faker (Random words, seed 42).bm25-elasticsearch contains 6,979 of the 6,980 queries on the Random characters and Random words corpora
(one query has no entry); all other files contain every query.results/results.csv of the code repository.Download directly into the RUNS_FOLDER of the code repository:
hf download PaDaS-Lab/ecir2027-curse-dense-retrieval-run-files --repo-type dataset --local-dir runs/The download is about 3.2 GB.
Unpacked into RUNS_FOLDER, these files let
repro pool rebuild the MS MARCO v1 pool used for pooled corpus subsampling, andrepro significance recompute the significance tests of Table 4.bm25: BM25 with Terrier in Anserini mode (k1=0.9, b=0.4).bm25-elasticsearch: BM25 with Elasticsearch (Lucene 8.11, default analyzer, k1=1.2, b=0.75).splade: SPLADE v3 (naver/splade-v3).dense-sbert: sentence-transformers/msmarco-distilroberta-base-v2.dense-gemma: google/embeddinggemma-300m.dense-jinav5: jinaai/jina-embeddings-v5-text-nano.All dense models use uncompressed 768-dimensional embeddings with exact (brute-force) cosine search.
[TODO: citation, to be added after the double-blind review]