CellFishing.jl: an ultrafast and scalable cell search method for single-cell RNA sequencing.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> faithful 1:1. CellFishing.jl (authors' own code, repo @ fd9e437) reproduced on the assigned dataset GSE84133 (Baron2016-human, 8569 cells) by running the authors' own benchmark scripts (Additional file 3) at paper-default params (128-bit, 4 LSH hashes, seed 1234) on «our HPC». The headline accuracy claim (Fig 4 self-mapping 5-fold-CV consistency and Cohen's kappa) reproduced to SIX decimal places against the authors' own computed values (MOESM2 notebook): consistency ftt 0.984852->0.9848524, kappa 0.981135->0.9811348, log1p consistency 0.983685->0.9836854. Data integrity exact (GSE84133_RAW.tar md5 matches the paper manifest; 8569 cells, 20125 genes, 2190 default features). DB-size claims (file 2.933 MiB, RAM 3.082 MiB) reproduced within <0.3% (benign Blosc/Zstd codec-version drift; size even varies ~0.3% between our own reps). Shipped unit tests pass. Overall 7/9 exact, 2/9 within-tol, 0 mismatch, no fabrication concern. NOT attempted (out-of-scope hard 20%): scmap-cell/SC3 baselines (need authors' R Docker image), other datasets (Shekhar2016/Plass2018/TabulaMuris/1M_neurons), cross-batch/species/protocol experiments (Figs 6-8), and absolute wall-clock speed (hardware-dependent, not a 1:1 target).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 97assessed: 2026-06-16 ⛓ 891ccd91f233
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan similar cells be searched from atlas-scale single-cell RNA-seq datasets with high accuracy and throughput without requiring dedicated hardware accelerators or cluster annotations? The authors propose that locality-sensitive hashing with bit-vector indexing can achieve this scalably to over one million cells.
- ★ CellFishing.jl searches prebuilt databases for cells with similar expression patterns with high accuracy and throughput using locality-sensitive hashing. resource
- ★ CellFishing.jl achieves accuracy comparable to and search speed markedly faster than the state-of-the-art software (scmap-cell). finding
- ★ CellFishing.jl is scalable to more than one million cells with a search throughput of approximately 1600 cells per second. finding
- ★ Using multiple mutually independent bit indexes (hashes) and aggregating by total Hamming distance reduces the risk of overlooking neighboring cells and significantly improves accuracy. method
- ★ Randomized SVD remarkably accelerates low-rank approximation for scRNA-seq data without introducing large errors in the largest singular value components. method
- ★ Orthogonalizing the normal vectors of random hyperplanes before hashing reduces the variance of the angular distance estimator. method
- A query cell can be compared with its neighbors to prioritize differentially expressed genes, enabling quick single-cell-resolution DEG analysis. method
- Hashed expression profiles preserve the original cell-type structure as shown by UMAP embeddings derived from Hamming distances. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq (inDrop) | Baron2016: pancreas cells of human and mouse | none | transcriptome expression / cluster (cell type) assignment consistency | inDrop |
| scRNA-seq (Drop-seq) | Shekhar2016: retinal bipolar neurons of mouse | none | transcriptome expression / nearest-neighbor cell-type label | Drop-seq |
| scRNA-seq (Drop-seq) | Plass2018: mature and progenitor planarian cells | none | cluster (cell type) assignment consistency | Drop-seq |
| scRNA-seq (Chromium) | TabulaMuris: mouse cell atlas | none | cluster (cell type) assignment consistency | 10x Chromium |
| scRNA-seq (Chromium) | 1M_neurons: brain cells of mouse (1,306,127 cells) | none | search scalability and throughput | 10x Chromium |
| Benchmarking of SVD algorithms (full, truncated, randomized) | first four scRNA-seq datasets in Table 1 (feature-selected matrices) | none | elapsed time and relative error of top 50 singular values | Julia svd; Arpack.jl svds; randomized SVD in CellFishing.jl |
| Self-mapping fivefold cross-validation (cell search benchmark) | five scRNA-seq datasets; comparison of CellFishing.jl vs scmap-cell | none | consistency score and Cohen's kappa of nearest-neighbor cluster label | CellFishing.jl; scmap-cell (scater normalize) |
| Locality-sensitive hashing similarity estimation / UMAP embedding | Shekhar2016 (100 randomly sampled cells; 64/128/256/512-bit vectors) | orthogonalization vs no orthogonalization of hyperplanes | angular estimation error, Hamming vs cosine distance, 2D embedding | UMAP |
- – Search throughput of CellFishing.jl on the largest dataset approximately 1600 cells per second
- – CellFishing.jl scales to the 1M_neurons dataset 1,306,127 cells
- – Fraction of random cells finding true nearest neighbor in top ten candidates with 64-bit vectors (Shekhar2016) 28%
- ▲ Fraction finding true nearest neighbor with 128-, 256-, and 512-bit vectors (Shekhar2016) 42%, 72%, and 85% respectively
- – Overall consistency and Cohen's kappa scores for both CellFishing.jl and scmap-cell across datasets (except Plass2018) >0.94
- ▲ CellFishing.jl consistently outperformed scmap-cell in consistency and Cohen's kappa with default parameters across all datasets
- ▲ Randomized SVD accelerates low-rank approximation without large errors in largest singular values
- ▲ Using multiple independent hashes significantly improved self-mapping scores
- other approximately 1600 cells per second (search throughput)
- count 1,306,127 (#cells in 1M_neurons dataset (mouse brain, Chromium))
- count 10,455 (#cells Baron2016 (inDrop, 14 clusters))
- count 27,499 (#cells Shekhar2016 (Drop-seq, 19 clusters))
- count 21,612 (#cells Plass2018 (Drop-seq, 51 clusters))
- count 54,967 (#cells TabulaMuris (Chromium, 57 clusters))
- other 28% / 42% / 72% / 85% (top-ten true nearest neighbor recovery for 64/128/256/512-bit vectors (Shekhar2016, 100 random cells))
- other >0.94 (consistency and Cohen's kappa scores for both methods (except Plass2018))
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a methods/software paper that evaluates a new cell-search tool (CellFishing.jl) primarily through computational benchmarking rather than classical hypothesis testing. Accuracy was assessed with fivefold cross-validation using consistency and Cohen's kappa agreement scores against the existing tool scmap-cell, while runtime and approximation accuracy of the randomized SVD were timed and quantified via relative singular-value errors. Variability of benchmark measurements was reported as the standard deviation over repeated trials (e.g., ten trials), and results were presented mainly as plots and descriptive summaries.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| fivefold cross-validation (self-mapping) with consistency score | Fig 4a, comparison of CellFishing.jl vs scmap-cell across five data sets | one fifth of cells mapped to remaining four fifths per fold; data set sizes in Table 1 | na |
| Cohen's kappa agreement score | Fig 4b, label agreement between query cell and nearest neighbor | nearest-neighbor labels across cross-validation folds | na |
| relative error of singular values (descriptive accuracy metric) | Fig 2b, randomized SVD vs reference singular values | standard deviation over ten trials | na |
| elapsed-time benchmarking of full/truncated/randomized SVD | Fig 2a, four data sets | data set sizes after feature selection as stated | na |
| estimation-error distributions for angular distance / Hamming-vs-cosine comparison | Fig 3a, b (Shekhar2016) | 100 randomly sampled cells; five independent hash values | na |
-
Accuracy was compared between two methods using consistency and Cohen's kappa scores reported as point values across data sets.↳ Could also: Repeating the fivefold cross-validation multiple times (or bootstrapping) and reporting confidence intervals or distributions of the scores. — Adding interval estimates would convey the run-to-run variability of the agreement metrics and is often preferred when comparing methods.
-
Method comparisons (CellFishing.jl vs scmap-cell) were described descriptively via score differences.↳ Could also: A paired test across data sets/folds (e.g., paired Wilcoxon signed-rank) or a mixed-effects model treating data set as a factor. — A formal paired analysis would summarize whether observed differences are consistent across data sets while accounting for data set as a repeated unit.
-
Benchmark variability was summarized with the standard deviation of ten trials.↳ Could also: Reporting a 95% confidence interval or the interquartile range/range in addition to SD. — A CI or IQR can complement SD in conveying spread and is sometimes favored, particularly for small numbers of trials.
-
Relative error of randomized SVD was used to quantify approximation accuracy of singular values.↳ Could also: Also reporting subspace-distance metrics (e.g., principal angles between approximate and exact singular subspaces). — Since the paper notes the subspace itself matters for hashing, a subspace-based metric would directly characterize the quantity of interest alongside singular-value error.
-
Nearest-neighbor recovery was summarized as the percentage of cells finding their true neighbor in the top ten across bit lengths.↳ Could also: Reporting recall/precision at k curves or mean average precision with uncertainty across sampled cells. — Ranking-based curves with uncertainty would summarize retrieval performance across a range of k and convey sampling variability.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Randomized SVD accelerates computation of the top-50 singular-value low-rank approximation without introducing large errors relative to full or truncated SVD.other multiple-datasets 2019×1papers★ This paper is the founder (earliest)
-
CellFishing.jl scales to a reference database of 1,306,127 mouse brain cells without degradation in search capability.scRNA-seq mouse brain 2019×1papers★ This paper is the founder (earliest)
-
CellFishing.jl achieves approximately 1600 cells per second search throughput on the 1.3M mouse brain neuron dataset.scRNA-seq mouse brain 2019×1papers★ This paper is the founder (earliest)
-
With 64-bit LSH vectors only 28% of randomly sampled cells find their true nearest neighbor in the top-10 candidates in the Shekhar2016 retinal dataset.scRNA-seq mouse retina 2019×1papers★ This paper is the founder (earliest)
-
True nearest-neighbor recall in the top-10 candidates increases from 42% to 72% to 85% as LSH bit-vector length increases from 128 to 256 to 512 bits in the Shekhar2016 retinal dataset.scRNA-seq mouse retina up 2019×1papers★ This paper is the founder (earliest)
-
Cell-type label consistency and Cohen's kappa exceed 0.94 for both CellFishing.jl and scmap-cell across pancreas, retina, and mouse-atlas datasets.scRNA-seq multiple-datasets 2019×1papers★ This paper is the founder (earliest)
-
CellFishing.jl consistently outperforms scmap-cell in cell-type label consistency and Cohen's kappa across all benchmarked scRNA-seq datasets with default parameters.scRNA-seq multiple-datasets up 2019×1papers★ This paper is the founder (earliest)
-
Using multiple independent locality-sensitive hash tables significantly improves self-mapping consistency scores in CellFishing.jl.scRNA-seq multiple-datasets up 2019×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-30744683 (CellFishing.jl, Sato et al., Genome Biol 2019)
Paper
CellFishing.jl: an ultrafast and scalable cell search method for scRNA-seq.
A locality-sensitive-hashing (LSH) cell-search engine. Repo (authors' own code,
P16 not needed): https://github.com/bicycle1885/CellFishing.jl @ fd9e437 (v0.3.0).
Exact benchmark code shipped as Additional file 3 (13059_2019_1639_MOESM3_ESM.tar
= cellfishing-experiments/: Snakefile + Julia/Python/R scripts + Analysis.ipynb).
The rendered notebook (Additional file 2, MOESM2 HTML) contains the authors'
own computed result tables — used here as the ground-truth "reported value".
Assigned dataset
Baron2016 = GSE84133 (inDrop human/mouse pancreas). The benchmark dataset
Baron2016-human = the 4 human donors (GSM2230757–60) → 8569 cells, 20125 genes,
14 cell-type clusters (assigned_cluster column of the GEO count CSVs).
In scope (pipeline-derived, deterministic — seed RANDSEED=1234)
All from running the authors' own scripts on GSE84133 human data:
- Data integrity — loomify(GSE84133 human) → n-cells=8569, n-genes=20125.
- Default feature selection —
selectfeaturesdefault (≥10% genes) → number of selected features (reported 2190 in MOESM2 knn-cv-features "default"). - Self-mapping accuracy (HEADLINE, Fig. 4) —
knn-cv.jl: 5-fold CV, k=10 NN, default n-bits=128, n-lshashes=4; consistency =accuracy_score(true_label, top1-NN label), Cohen's kappa =cohen_kappa_score(...), mean over 5 reps. Reported (MOESM2 table): ftt consistency 0.984852, kappa 0.981135; log1p consistency 0.983685, kappa 0.979683. - Database footprint (Table 3 family) —
save-load.jl, 128-bit, keep-counts= false: file-size 2.933306 MiB, ram-size 3.082169 MiB (8569 cells). - Algorithm correctness — shipped unit test suite
Pkg.test("CellFishing")(LSH search recovers exact NN: totaldist≤1, n_ok/n_all≈1.0; CMatrix/Features/DEG).
Out of scope / not attempted (the hard ~20%)
- scmap-cell / SC3 comparisons (need their Docker image
ksato/scmap; R tooling). - Other datasets (Shekhar2016, Plass2018, TabulaMuris, 1M_neurons) — not assigned; 1M_neurons needs 10x downloads + large RAM.
- Cross-batch / cross-species / cross-protocol mapping experiments (Figs 6–8).
- Absolute wall-clock speed claims (index/query seconds) — hardware-dependent, recorded as context but NOT graded 1:1 (different CPU than the authors').
Approach
Run the authors' exact scripts (Additional file 3) at default parameters on the GSE84133 human data, on «our HPC» (SLURM «job»). Compare reproduced consistency/kappa/DB-size/feature-count to the authors' own MOESM2 values. Self-mapping consistency is fully deterministic (fixed seed) → a clean 1:1 test.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
A faithful 1:1 reproduction of CellFishing.jl on its assigned GSE84133 (Baron2016-human, 8569 cells). Running the authors' own benchmark code at default params reproduced the headline self-mapping consistency 0.984852 and kappa 0.981135 to six decimal places, with exact data-integrity (8569 cells / 20125 genes / 2190 features) and passing unit tests. The only deviations are serialized DB sizes within <0.3%, attributable to Blosc/Zstd codec-version drift — a technical, expected difference. No fabrication concern; out-of-scope items (R-Docker baselines, other datasets, wall-clock speed) honestly disclosed.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.