Community assessment of methods to deconvolve cellular composition from bulk gene expression.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH to run the tool, but the headline benchmark is NOT 1:1 reproducible from public data. The published winning method (Team Aginome-XMU PyTorch deconvolution) was fully obtained (code + 3.4GB pretrained weights from Mendeley) and runs correctly on the paper's own public validation data GSE199324: a purified-cell positive control passes cleanly (dominant population predicted at 0.82-1.03, predictions sum ~1), so the TOOL reproduces (C4=exact). However the reported per-cell-type Pearson scores (coarse r=0.85, fine r=0.76, rho=0.64; Fig 2) CANNOT be reproduced from public artifacts: the per-sample link between the public GEO expression (GSE199324 BM*/RM* samples) and the public ground-truth proportions (Nat Commun Supplementary Data 3-6 BM*/RM* rows) is broken/anonymized. Three model-free checks prove this independently: naive same-label self-match 1/96 (chance); canonical marker genes (incl. cancer EPCAM/KRT19) do NOT track their same-named proportions (all |r|<0.3); Hungarian reconstruction-matching is ambiguous (mean r 0.80-0.84, collisions) so the permutation is not cleanly recoverable. The DREAM challenge anonymized validation IDs; the gold-standard sample<->proportion key is on gated Synapse (syn21574276 family) = the hard ~20%, deliberately not attempted. NOT attempted: Synapse gold-standard linkage, full multi-dataset challenge aggregate/ranking, CIBERSORTx (closed web tool), ensemble results. Fabrication: NONE — values are real; this is a data-linkage gap. Human review flagged.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 63assessed: 2026-06-14 ⛓ c8f2be84be50
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether published and newly developed (community-contributed) computational deconvolution methods can accurately infer the proportions of coarse-grained and fine-grained immune/stromal cell types from bulk gene expression profiles, using controlled admixtures with known ground-truth composition.
- ★ Most deconvolution methods accurately predict coarse-grained immune/stromal cell populations from bulk expression. finding
- ★ Published methods either were not trained to evaluate all functional CD8+ T cell states or do so with low accuracy. finding
- ★ Several community-contributed methods, including a deep learning-based approach, improve prediction of fine-grained cell states such as memory and naive CD8+ T cells. finding
- ★ The strong performance of the deep learning-based method establishes the applicability of deep learning as a paradigm for deconvolution. finding
- ★ Deconvolution methods trained largely on healthy-tissue immune cell profiles nonetheless predict levels of tumor-derived (cancer-associated) immune cells well. finding
- ★ An ensemble approach combining all methods exploits their individual strengths, since no single method performs best across all cell types. finding
- ★ Sensitive prediction of CD4+ T cell functional states remains a common, pervasive challenge shared across most methods. finding
- ★ The generated purified and in vitro/in silico admixed expression profiles are released as a resource for developing and training future deconvolution methods. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq | in vitro admixtures of cancer cell lines with healthy immune and stromal cells | controlled admixture at known, tumor-representative mixing proportions | bulk gene expression used to benchmark predicted vs true cell type proportions | — |
| in silico admixture simulation from purified expression profiles | purified cancer, immune, and stromal cell expression profiles | computational mixing at known proportions | predicted vs ground-truth cell type proportions (correlation) | — |
| in silico admixture from single-cell RNA-seq (scRNA-seq) profiles | tumor tissue samples (cancer-associated immune cells) | computational admixture derived from scRNA-seq profiles | deconvolution accuracy for tumor-derived/cancer-associated immune cell types | — |
| crowdsourced deconvolution algorithm benchmarking (DREAM Challenge) | bulk expression data from in vitro and in silico admixtures | none (methods applied to pre-generated admixtures) | correlation between method-predicted and known cell type proportions across coarse- and fine-grained sub-Challenges | — |
- – Most of the 6 published and 22 community-contributed methods predict coarse-grained cell populations (e.g., B cells, CD8+ T cells, NK cells, fibroblasts) well.
- – Published methods show gaps in evaluating fine-grained functional CD8+ T cell states.
- – Community-contributed methods, including a deep learning approach, improve prediction of memory and naive CD8+ T cell sub-populations.
- – Deep learning-based method performs strongly relative to other approaches in the Challenge.
- – Methods trained on healthy-tissue-derived immune profiles still accurately deconvolve tumor-derived immune cells in scRNA-seq-based in silico admixtures.
- – No single method is best across all cell types, but an ensemble of methods leverages complementary strengths.
- – Difficulty in sensitively predicting CD4+ T cell functional states persists across diverse method types.
- count 8 (number of major immune and stromal cell populations predicted in the coarse-grained sub-Challenge)
- count 14 (number of fine-grained sub-populations predicted (e.g., memory, naive, regulatory CD4+ T cells))
- count 6 (number of published deconvolution methods benchmarked)
- count 22 (number of community-contributed deconvolution methods benchmarked)
- count >60 (bioinformatic algorithms benchmarked historically across DREAM Challenges)
- count >30,000 (cumulative community of participants across DREAM Challenges)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper presents a community-wide DREAM Challenge benchmarking study that evaluates 28 deconvolution methods (6 published, 22 community-contributed) for inferring cellular composition from bulk gene expression. Performance was assessed primarily via correlation between method-predicted and known ground-truth cell-type proportions derived from in vitro and in silico admixtures of cancer, immune, and stromal cells. Evaluation was organized across two sub-challenges: coarse-grained (8 major cell populations) and fine-grained (14 sub-populations). An ensemble approach combining all methods was also assessed alongside individual methods.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Correlation coefficient (Pearson or Spearman; specific type not stated in provided text) | Primary performance metric across all methods and cell types, comparing predicted proportions to known admixture ground truth | — | not stated |
-
Correlation was stated as the primary metric for assessing how well predicted proportions matched known ground-truth proportions↳ Could also: Root mean squared error (RMSE) or mean absolute error (MAE) could also be used as complementary metrics — Correlation measures rank-order agreement but is insensitive to systematic bias (e.g., consistent over- or underestimation); absolute-error metrics additionally quantify the magnitude of deviation, which may differentiate methods that have similar correlation but differ in calibration
-
In vitro admixtures were constructed from healthy donor immune cells mixed with cancer cell lines at defined proportions↳ Could also: Validation against patient-derived tumor samples with orthogonally measured cell-type proportions (e.g., flow cytometry or CyTOF on matched biopsies) could also be incorporated — Admixtures of healthy immune cells provide well-controlled ground truth; paired orthogonal measurements from actual tumor specimens would additionally assess method performance in the presence of the biological complexity and cell-state alterations characteristic of the tumor microenvironment
-
Methods were ranked and compared by their correlation performance within each cell type and sub-challenge separately↳ Could also: Bootstrap resampling of samples or permutation-based confidence intervals on performance differences between methods could also be applied — Point estimates of correlation across a finite number of admixture samples carry sampling uncertainty; bootstrap or permutation intervals would quantify whether observed performance differences between methods exceed what is expected by chance, supporting more nuanced interpretation of method rankings
-
A simple ensemble combining all submitted methods was evaluated as an additional predictor↳ Could also: A cross-validated weighted or stacked ensemble, with per-cell-type weights optimized on a held-out validation partition, could also be constructed — Equal-weight averaging treats all methods symmetrically regardless of per-cell-type accuracy; a weighted ensemble that up-weights better-performing methods for each cell type could exploit individual method strengths more efficiently
-
In silico admixtures were generated from scRNA-seq profiles of tumor samples to test generalization to cancer-associated immune cells↳ Could also: Cross-validation across multiple independent scRNA-seq tumor datasets from different cancer types or cohorts could also be used to assess generalizability — Validation on a single in silico dataset derived from one set of scRNA-seq profiles may not capture variability across tumor types, patients, or sequencing platforms; multi-dataset cross-validation would provide a broader assessment of how well findings generalize
-
Performance was summarized and compared across methods at the level of individual cell types within each sub-challenge↳ Could also: A mixed-effects or hierarchical model treating method and cell type as factors could also jointly summarize performance and estimate method-by-cell-type interaction effects — Separate per-cell-type analyses do not account for the correlation structure of errors across cell types within a method; a joint model could more formally characterize whether a method's strengths and weaknesses are cell-type-specific or systematic
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Deconvolution of CD4+ T cell functional states from bulk RNA-seq remained a shared failure mode across all benchmarked methods.RNA-seq in-vitro-admixture down 2024×1papers★ This paper is the founder (earliest)
-
Community-developed methods improved fine-grained deconvolution of memory and naive CD8+ T cell subsets relative to previously published methods in bulk RNA-seq admixtures.RNA-seq in-vitro-admixture up 2024×1papers★ This paper is the founder (earliest)
-
An ensemble of all deconvolution methods outperformed any single method for estimating cell-type proportions from bulk RNA-seq admixtures.RNA-seq in-vitro-admixture up 2024×1papers★ This paper is the founder (earliest)
-
A deep learning deconvolution method achieved strong performance relative to other benchmarked approaches on bulk RNA-seq admixtures.RNA-seq in-vitro-admixture 2024×1papers★ This paper is the founder (earliest)
-
Most deconvolution methods accurately estimated coarse-grained immune and stromal cell proportions (CD8+ T cells, B cells, NK cells, fibroblasts) from bulk RNA-seq admixtures.RNA-seq in-vitro-admixture 2024×1papers★ This paper is the founder (earliest)
-
Deconvolution methods trained on healthy-tissue immune reference profiles accurately predicted cancer-associated immune cell levels in tumor bulk RNA-seq admixtures.RNA-seq tumor-in-vitro-admixture 2024×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
99 downstream papers · 1 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- The GENCODE v7 catalog of human long noncoding RNAs:... 2012 · 4,084 cites
- GENCODE: the reference human genome annotation for T... 2012 · 3,759 cites
- Integrative annotation of human large intergenic non... 2011 · 2,858 cites
- Simultaneous enumeration of cancer and immune cell t... 2017 · 1,078 cites
- Ensembl 2013. 2013 · 827 cites
- De novo mutations in histone-modifying genes in cong... 2013 · 760 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-39191725
Paper: White BS et al. Community assessment of methods to deconvolve cellular composition from bulk gene expression. Nat Commun 2024. PMID 39191725 / PMC11350143 / DOI 10.1038/s41467-024-50618-0. ("Tumor Deconvolution DREAM Challenge")
This is a community benchmark (DREAM Challenge). Many reported results are meta-analytic (rankings, ensembles, cross-method comparisons across many teams and external datasets) and depend on the full challenge harness + gold standards hosted on Synapse (registration-gated). Per the brief (P16), reproducing a third-party / participant tool on the paper's own data is equally valid.
In scope (pipeline-derived, attempted)
Run the published winning participant method, Team Aginome-XMU (deep-learning deconvolution, repo github.com/xmuyulab/DCTD_Team_Aginome-XMU, pretrained weights on Mendeley DOI 10.17632/yt79wsksg9.1) on the paper's published validation expression data (GEO GSE199324, the in-vitro/in-silico admixture RNA-seq), producing predicted cell-type proportions, and comparing accuracy against the paper's reported performance for that method.
Pipeline: run_DCTD.py {coarse,fine} — PyTorch MLP ensemble, 5080-gene signature,
input = HUGO-symbol expression matrix (TPM, -scale Linear), output = long-format
predicted proportions for 8 (coarse) / 14 (fine) cell types.
Target reported values (paper Fig. 2; metric = cross-sample within-cell-type Pearson)
| id | claim | reported | location |
|---|---|---|---|
| C1 | Aginome-XMU coarse-grained aggregate Pearson r | 0.85 | Fig. 2A / Results |
| C2 | Aginome-XMU fine-grained aggregate Pearson r | 0.76 | Fig. 2B / Results |
| C3 | Aginome-XMU fine-grained Spearman rho | 0.64 | Fig. 2B / Results |
Ground-truth admixture proportions: paper Supplementary Data 3–8 (open access) and /or GEO series-matrix sample characteristics — resolved at run time.
Out of scope (not attempted, why)
- Full challenge ranking / aggregate scoring across all validation datasets and all teams (needs Synapse gold standards + bootstrap-rank harness, syn21574276) — the hard ~20%; we use GSE199324 (the paper's own admixture data) as the validation set.
- Ensemble-of-methods results, runtime comparisons, training-data curation, wet-lab admixture generation — external/manual, not a single reproducible pipeline.
- CIBERSORTx (r=0.90) — closed web tool, registration + license gated.
Honesty notes
- We reproduce one method on one (the paper's own) validation dataset, not the full benchmark; the reproduced aggregate r is expected to be close in magnitude to the reported value but need not match exactly (different/!subset of validation data, possible scale/normalization differences). Grade accordingly (partial/within-tol).
- Every comparison is provisional; a human reviewer decides the final grade.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The published winning method (Team Aginome-XMU) was fully obtained and reproduces: it runs on the paper's own GSE199324 data and passes clean purified-cell positive controls (dominant population 0.82–1.03, preds sum ~1), so C4 is exact. However the Fig-2 headline scores (coarse r=0.85, fine r=0.76, ρ=0.64) are not reproducible from public artifacts because the per-sample link between public expression and public ground-truth proportions is anonymized/broken — confirmed by three independent model-free checks (self-match 1/96, all markers |r|<0.3, ambiguous Hungarian recovery). This sits on the data-availability/challenge-anonymization side (gold-standard key gated on Synapse), not a methodological error or fabrication — the values are real, just not gradable from shared data. Net: a solid reproduction of the tool with an explainable, externally-caused gap on the benchmark, hence overall yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.