CellFishing.jl: an ultrafast and scalable cell search method for single-cell RNA sequencing.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> faithful 1:1. CellFishing.jl (authors' own code, repo @ fd9e437) reproduced on the assigned dataset GSE84133 (Baron2016-human, 8569 cells) by running the authors' own benchmark scripts (Additional file 3) at paper-default params (128-bit, 4 LSH hashes, seed 1234) on «our HPC». The headline accuracy claim (Fig 4 self-mapping 5-fold-CV consistency and Cohen's kappa) reproduced to SIX decimal places against the authors' own computed values (MOESM2 notebook): consistency ftt 0.984852->0.9848524, kappa 0.981135->0.9811348, log1p consistency 0.983685->0.9836854. Data integrity exact (GSE84133_RAW.tar md5 matches the paper manifest; 8569 cells, 20125 genes, 2190 default features). DB-size claims (file 2.933 MiB, RAM 3.082 MiB) reproduced within <0.3% (benign Blosc/Zstd codec-version drift; size even varies ~0.3% between our own reps). Shipped unit tests pass. Overall 7/9 exact, 2/9 within-tol, 0 mismatch, no fabrication concern. NOT attempted (out-of-scope hard 20%): scmap-cell/SC3 baselines (need authors' R Docker image), other datasets (Shekhar2016/Plass2018/TabulaMuris/1M_neurons), cross-batch/species/protocol experiments (Figs 6-8), and absolute wall-clock speed (hardware-dependent, not a 1:1 target).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 97assessed: 2026-06-16 ⛓ 891ccd91f233
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan a locality-sensitive-hashing-based method (CellFishing.jl) search atlas-scale single-cell RNA-seq datasets for similar cells and detect noteworthy differentially expressed genes with accuracy comparable to state-of-the-art tools (e.g., scmap-cell) while being markedly faster and scalable to millions of cells?
- ★ CellFishing.jl achieves accuracy comparable to state-of-the-art software (scmap-cell) but is markedly faster finding
- ★ CellFishing.jl scales to more than one million cells with search throughput of approximately 1600 cells per second finding
- ★ Orthogonalizing random hyperplane normal vectors before hashing reduces variance in Hamming-distance-based angle estimation method
- ★ Randomized SVD substantially accelerates dimensionality reduction of scRNA-seq expression matrices without introducing large errors in top singular values method
- Using multiple independent bit indexes reduces the risk of overlooking neighboring cells compared to a single bit index method
- ★ CellFishing.jl outperformed scmap-cell in consistency and Cohen's kappa scores across benchmark datasets under default parameters finding
- CellFishing.jl does not require dedicated hardware accelerators (e.g., GPUs), unlike CellAtlasSearch resource
- CellFishing.jl is an open-source software package implemented in the Julia programming language resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| SVD algorithm benchmarking (full, truncated, randomized) | 4 scRNA-seq datasets (Baron2016 human subset, Shekhar2016, Plass2018, TabulaMuris) | none (algorithm comparison) | elapsed time; relative error of top 50 singular values | Julia svd function; Arpack.jl svds; custom randomized SVD implementation |
| LSH bit-vector hashing angle estimation | Shekhar2016 scRNA-seq dataset, 100 randomly sampled cells | with vs without orthogonalization of random hyperplanes | estimation error of angle (radians) across 5 independent hash values | — |
| Hamming distance vs cosine distance comparison across bit lengths | Shekhar2016 scRNA-seq dataset, 100 random cells | bit vector length (64, 128, 256, 512 bits) | Hamming distance vs cosine distance; percentage recovering true nearest neighbor in top 10 | — |
| UMAP 2D embedding of hashed expression profiles | Shekhar2016 scRNA-seq dataset | hashing bit length (64, 128, 256 bits) vs original cosine distance | preservation of cell-type cluster structure in low-dimensional embedding | UMAP |
| Self-mapping fivefold cross-validation cell search benchmark | 5 scRNA-seq datasets (Baron2016, Shekhar2016, Plass2018, TabulaMuris, 1M_neurons) | CellFishing.jl: number of bits/indexes varied; scmap-cell: number of centroids/features varied | consistency score and Cohen's kappa score of nearest-neighbor cluster label agreement | CellFishing.jl vs scmap-cell (scater package for normalization/log-transform) |
- – Randomized SVD markedly accelerated low-rank approximation computation without introducing large errors in top singular values
- ▼ Orthogonalization technique reduced variance of angle estimation error compared to non-orthogonalized hyperplanes
- ▲ Percentage of cells finding true nearest neighbor among top 10 candidates increased with bit vector length 28% (64-bit), 42% (128-bit), 72% (256-bit), 85% (512-bit)
- ▲ CellFishing.jl consistently outperformed scmap-cell in consistency and Cohen's kappa scores with default parameters across all data sets
- – Overall consistency/kappa scores were high for both methods except on the Plass2018 data set >0.94
- – CellFishing.jl scaled to the 1M_neurons data set (>1.3 million cells) with a search throughput of ~1600 cells/second ~1600 cells/second
- count 1,306,127 cells (size of 1M_neurons benchmark dataset, largest publicly available at the time)
- other ~1600 cells/second (search throughput of CellFishing.jl)
- other 28%, 42%, 72%, 85% (percentage of randomly sampled cells finding true nearest neighbor in top 10 with 64/128/256/512-bit vectors, Shekhar2016)
- other >0.94 (consistency and Cohen's kappa scores for both CellFishing.jl and scmap-cell across most datasets)
- count 10,455 cells / 14 clusters (Baron2016 dataset size (inDrop, human/mouse pancreas))
- count 27,499 cells / 19 clusters (Shekhar2016 dataset size (Drop-seq, mouse retinal bipolar neurons))
- count 21,612 cells / 51 clusters (Plass2018 dataset size (Drop-seq, planarian cells))
- count 54,967 cells / 57 clusters (TabulaMuris dataset size (Chromium, mouse cell atlas, excluding Smart-Seq2))
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a methods/software paper that evaluates a new cell-search tool (CellFishing.jl) primarily through computational benchmarking rather than classical hypothesis testing. Accuracy was assessed with fivefold cross-validation using consistency and Cohen's kappa agreement scores against the existing tool scmap-cell, while runtime and approximation accuracy of the randomized SVD were timed and quantified via relative singular-value errors. Variability of benchmark measurements was reported as the standard deviation over repeated trials (e.g., ten trials), and results were presented mainly as plots and descriptive summaries.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| fivefold cross-validation (self-mapping) with consistency score | Fig 4a, comparison of CellFishing.jl vs scmap-cell across five data sets | one fifth of cells mapped to remaining four fifths per fold; data set sizes in Table 1 | na |
| Cohen's kappa agreement score | Fig 4b, label agreement between query cell and nearest neighbor | nearest-neighbor labels across cross-validation folds | na |
| relative error of singular values (descriptive accuracy metric) | Fig 2b, randomized SVD vs reference singular values | standard deviation over ten trials | na |
| elapsed-time benchmarking of full/truncated/randomized SVD | Fig 2a, four data sets | data set sizes after feature selection as stated | na |
| estimation-error distributions for angular distance / Hamming-vs-cosine comparison | Fig 3a, b (Shekhar2016) | 100 randomly sampled cells; five independent hash values | na |
-
Accuracy was compared between two methods using consistency and Cohen's kappa scores reported as point values across data sets.↳ Could also: Repeating the fivefold cross-validation multiple times (or bootstrapping) and reporting confidence intervals or distributions of the scores. — Adding interval estimates would convey the run-to-run variability of the agreement metrics and is often preferred when comparing methods.
-
Method comparisons (CellFishing.jl vs scmap-cell) were described descriptively via score differences.↳ Could also: A paired test across data sets/folds (e.g., paired Wilcoxon signed-rank) or a mixed-effects model treating data set as a factor. — A formal paired analysis would summarize whether observed differences are consistent across data sets while accounting for data set as a repeated unit.
-
Benchmark variability was summarized with the standard deviation of ten trials.↳ Could also: Reporting a 95% confidence interval or the interquartile range/range in addition to SD. — A CI or IQR can complement SD in conveying spread and is sometimes favored, particularly for small numbers of trials.
-
Relative error of randomized SVD was used to quantify approximation accuracy of singular values.↳ Could also: Also reporting subspace-distance metrics (e.g., principal angles between approximate and exact singular subspaces). — Since the paper notes the subspace itself matters for hashing, a subspace-based metric would directly characterize the quantity of interest alongside singular-value error.
-
Nearest-neighbor recovery was summarized as the percentage of cells finding their true neighbor in the top ten across bit lengths.↳ Could also: Reporting recall/precision at k curves or mean average precision with uncertainty across sampled cells. — Ranking-based curves with uncertainty would summarize retrieval performance across a range of k and convey sampling variability.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Randomized SVD accelerates computation of the top-50 singular-value low-rank approximation without introducing large errors relative to full or truncated SVD.other multiple-datasets 2019×1papers★ This paper is the founder (earliest)
-
CellFishing.jl scales to a reference database of 1,306,127 mouse brain cells without degradation in search capability.scRNA-seq mouse brain 2019×1papers★ This paper is the founder (earliest)
-
CellFishing.jl achieves approximately 1600 cells per second search throughput on the 1.3M mouse brain neuron dataset.scRNA-seq mouse brain 2019×1papers★ This paper is the founder (earliest)
-
With 64-bit LSH vectors only 28% of randomly sampled cells find their true nearest neighbor in the top-10 candidates in the Shekhar2016 retinal dataset.scRNA-seq mouse retina 2019×1papers★ This paper is the founder (earliest)
-
True nearest-neighbor recall in the top-10 candidates increases from 42% to 72% to 85% as LSH bit-vector length increases from 128 to 256 to 512 bits in the Shekhar2016 retinal dataset.scRNA-seq mouse retina up 2019×1papers★ This paper is the founder (earliest)
-
Cell-type label consistency and Cohen's kappa exceed 0.94 for both CellFishing.jl and scmap-cell across pancreas, retina, and mouse-atlas datasets.scRNA-seq multiple-datasets 2019×1papers★ This paper is the founder (earliest)
-
CellFishing.jl consistently outperforms scmap-cell in cell-type label consistency and Cohen's kappa across all benchmarked scRNA-seq datasets with default parameters.scRNA-seq multiple-datasets up 2019×1papers★ This paper is the founder (earliest)
-
Using multiple independent locality-sensitive hash tables significantly improves self-mapping consistency scores in CellFishing.jl.scRNA-seq multiple-datasets up 2019×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-30744683 (CellFishing.jl, Sato et al., Genome Biol 2019)
Paper
CellFishing.jl: an ultrafast and scalable cell search method for scRNA-seq.
A locality-sensitive-hashing (LSH) cell-search engine. Repo (authors' own code,
P16 not needed): https://github.com/bicycle1885/CellFishing.jl @ fd9e437 (v0.3.0).
Exact benchmark code shipped as Additional file 3 (13059_2019_1639_MOESM3_ESM.tar
= cellfishing-experiments/: Snakefile + Julia/Python/R scripts + Analysis.ipynb).
The rendered notebook (Additional file 2, MOESM2 HTML) contains the authors'
own computed result tables — used here as the ground-truth "reported value".
Assigned dataset
Baron2016 = GSE84133 (inDrop human/mouse pancreas). The benchmark dataset
Baron2016-human = the 4 human donors (GSM2230757–60) → 8569 cells, 20125 genes,
14 cell-type clusters (assigned_cluster column of the GEO count CSVs).
In scope (pipeline-derived, deterministic — seed RANDSEED=1234)
All from running the authors' own scripts on GSE84133 human data:
- Data integrity — loomify(GSE84133 human) → n-cells=8569, n-genes=20125.
- Default feature selection —
selectfeaturesdefault (≥10% genes) → number of selected features (reported 2190 in MOESM2 knn-cv-features "default"). - Self-mapping accuracy (HEADLINE, Fig. 4) —
knn-cv.jl: 5-fold CV, k=10 NN, default n-bits=128, n-lshashes=4; consistency =accuracy_score(true_label, top1-NN label), Cohen's kappa =cohen_kappa_score(...), mean over 5 reps. Reported (MOESM2 table): ftt consistency 0.984852, kappa 0.981135; log1p consistency 0.983685, kappa 0.979683. - Database footprint (Table 3 family) —
save-load.jl, 128-bit, keep-counts= false: file-size 2.933306 MiB, ram-size 3.082169 MiB (8569 cells). - Algorithm correctness — shipped unit test suite
Pkg.test("CellFishing")(LSH search recovers exact NN: totaldist≤1, n_ok/n_all≈1.0; CMatrix/Features/DEG).
Out of scope / not attempted (the hard ~20%)
- scmap-cell / SC3 comparisons (need their Docker image
ksato/scmap; R tooling). - Other datasets (Shekhar2016, Plass2018, TabulaMuris, 1M_neurons) — not assigned; 1M_neurons needs 10x downloads + large RAM.
- Cross-batch / cross-species / cross-protocol mapping experiments (Figs 6–8).
- Absolute wall-clock speed claims (index/query seconds) — hardware-dependent, recorded as context but NOT graded 1:1 (different CPU than the authors').
Approach
Run the authors' exact scripts (Additional file 3) at default parameters on the GSE84133 human data, on «our HPC» (SLURM «job»). Compare reproduced consistency/kappa/DB-size/feature-count to the authors' own MOESM2 values. Self-mapping consistency is fully deterministic (fixed seed) → a clean 1:1 test.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
A faithful 1:1 reproduction of CellFishing.jl on its assigned GSE84133 (Baron2016-human, 8569 cells). Running the authors' own benchmark code at default params reproduced the headline self-mapping consistency 0.984852 and kappa 0.981135 to six decimal places, with exact data-integrity (8569 cells / 20125 genes / 2190 features) and passing unit tests. The only deviations are serialized DB sizes within <0.3%, attributable to Blosc/Zstd codec-version drift — a technical, expected difference. No fabrication concern; out-of-scope items (R-Docker baselines, other datasets, wall-clock speed) honestly disclosed.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.