An NMF-Based Methodology for Selecting Biomarkers in the Landscape of Genes of Heterogeneous Cancer-Associated Fibroblast Populations.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the per-dataset pipeline. The repo (flaespo/NMF-for-GSE @ba1ab74) ships the data + two TEMPLATED R scripts; instantiated for GSE51257 (colon-CAF, GPL6244) on «our HPC» with R-NMF/GEOquery conda env. 1:1 EXACT on the two clearly-specified, deterministic preprocessing outputs: 20304 genes x 4 samples after GEOquery AnnotGPL + MAD-dedup, matching Table 1 to the digit (C1 is sample-choice-independent => a clean fabrication checkpoint). NMF rank=2 (C3) reproduced via nmfEstimateRank: dispersion (0.50>0.325) and consensus-silhouette (0.657>0.125) favor r=2 as the paper states, though cophenetic alone favors r=3 (within-tol; note the repo's NMF script hard-codes r=3, a template leftover). NOT attempted/obtained: C4 biomarker extractFeatures list (final nmf nrun=200 OOM'd, AND not 1:1-gradeable anyway since NMF uses random init and the paper publishes no per-dataset gene list); and the paper's HEADLINE cross-dataset intersections (108/9/ENPP2 across colon/breast/ovary/MM) which require running all 6 datasets and are outside this single-RU scope. No fabrication signals: every in-scope reported number is regenerable from shipped data+code and C1 matched exactly. Verdict: faithful PARTIAL reproduction of the preprocessing + rank-selection stage; biomarker-selection stage and cross-dataset claims not reproduced.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 84assessed: 2026-06-15 ⛓ 37e0cf0f8c67
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan an NMF-based gene extraction methodology, capable of analyzing microarray matrices that differ in both gene (row) and patient (column) dimensions, identify common biomarker genes that characterize the heterogeneity of cancer-associated fibroblasts (CAFs) across solid tumors of different embryonic origin (and bone marrow of MM/MGUS patients)?
- ★ An ensemble methodology integrating nonnegative matrix factorization, a gene.score-based gene landscape extraction procedure, and the WebGestalt functional enrichment tool can automatically mine biologically meaningful biomarkers from microarray datasets of different tumor histotypes. method
- ★ The proposed Gene Landscape Extraction procedure allows comparison of microarray matrices that differ in both number of rows (genes) and columns (patients) via intersection set operations on the most representative metagenes. method
- ★ NMF mines metagenes (linear combinations of samples weighted by nonnegative coefficients) that are most representative of information embedded in tumor datasets, from which interpretable common genes can be extracted. mechanism
- ★ The methodology reveals common biomarkers among CAFs of different embryonic origin (colon/endodermal, breast/ectodermal, ovary/mesodermal) that uniquely identify the activated fibroblast phenotype. finding
- The approach can investigate common genes between CAF profiles of solid tumors and bone marrow of multiple myeloma (MM) and MGUS patients, since bone marrow is a CAF recruitment source. finding
- Code and dataset for the data preparation and pre-processing module are made publicly available on GitHub. resource
- The gene.score scoring method selects genes whose score exceeds threshold tau = median + 3*MAD of the gene score vector, identifying the most representative metagene. method
- Generalized Kullback-Leibler divergence (maximum likelihood under independent Poisson assumption) with multiplicative update rules is used to solve the NMF optimization. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Microarray gene expression analysis | CAFs from primary cultures of colon carcinoma (endodermal origin) | none | gene expression levels (metagenes / extracted common genes) | R/BioConductor with NMF library; WebGestalt |
| Microarray gene expression analysis | CAFs from primary cultures of breast carcinoma (ectodermal origin) | none | gene expression levels (metagenes / extracted common genes) | R/BioConductor with NMF library; WebGestalt |
| Microarray gene expression analysis | CAFs from primary cultures of ovary carcinoma (mesodermal origin) | none | gene expression levels (metagenes / extracted common genes) | R/BioConductor with NMF library; WebGestalt |
| Microarray gene expression analysis | Bone marrow of patients with multiple myeloma (MM) and MGUS | none | gene expression levels / common genes with CAF profiles | R/BioConductor with NMF library; WebGestalt |
- – Common biomarker genes characterizing CAF heterogeneity were extracted across the three solid tumor histotypes of different embryonic origin via intersection of most-representative metagenes.
- – Rank values r_i used to factorize each gene expression matrix were automatically determined (reported in Table 2) using cophenetic coefficient, RSS, dispersion, and consensus matrices.
- other tau = median + 3*MAD (gene.score threshold) (score threshold for selecting genes within a metagene)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper presents a computational pipeline that applies nonnegative matrix factorization (NMF) with Kullback-Leibler divergence minimization separately to microarray gene expression matrices from three CAF tumor histotypes (colon, breast, ovary) and bone marrow samples from multiple myeloma and MGUS patients. Optimal NMF rank is selected per dataset using cophenetic coefficient, residual sum of squares, and dispersion curves; representative genes within each leading metagene are selected by a threshold of median + 3×MAD of gene scores; and common genes across datasets are identified by set intersection. The resulting gene set is biologically characterized via functional and pathway enrichment analysis in WebGestalt.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Nonnegative matrix factorization with Kullback-Leibler divergence minimization (multiplicative update rules) | Dimensionality reduction of each tumor microarray gene expression matrix | — | not stated |
| Cophenetic coefficient, residual sum of squares, and dispersion curve analysis | Selection of optimal NMF rank r for each gene expression matrix (values reported in Table 2, not reproduced in available text) | — | not stated |
| Gene score threshold (τ = median + 3×MAD of gene scores) | Selection of representative genes within the leading metagene column of each W matrix | — | not stated |
| Set intersection operation | Identification of gene labels common across all tumor histotype metagenes to form the common gene set C | — | na |
| Functional and pathway enrichment analysis (WebGestalt) | Biological characterization of the extracted common gene set C | — | not stated |
-
Common genes across tumor types were identified by strict set intersection of the genes passing the gene.score threshold in each dataset↳ Could also: A probabilistic overlap approach—such as a hypergeometric test or rank-aggregation (e.g., Borda count, RRA)—applied to ranked gene score lists could also quantify the degree and significance of cross-dataset sharing — Strict intersection is sensitive to the choice of threshold: a gene scoring just below τ in one dataset is excluded entirely; ranked or probabilistic methods recover near-miss genes and provide a confidence measure for the overlap
-
Gene selection within each metagene used a fixed empirical rule τ = median + 3×MAD of gene scores↳ Could also: A permutation-based null distribution (randomly shuffling gene labels and recomputing scores many times) could also be used to derive a dataset-specific significance threshold — Permutation thresholds directly model the null distribution of gene scores for each specific dataset and noise level, complementing the fixed-rule approach especially when datasets differ substantially in size or variance structure
-
NMF was run with KL divergence (Poisson log-likelihood) as the reconstruction loss↳ Could also: Frobenius-norm NMF (Gaussian/least-squares loss) is also widely applied to normalized microarray data — Frobenius-norm NMF is computationally straightforward and may be more appropriate when data have been pre-normalized to approximately Gaussian distributions; KL divergence is often preferred when data retain count-like or non-negative Poisson structure
-
NMF rank r was selected per dataset using cophenetic coefficient, RSS, and dispersion curves↳ Could also: Cross-validation on held-out matrix entries, or information-theoretic criteria (AIC/BIC applied to the Poisson log-likelihood), could also guide rank selection — Cross-validation estimates predictive generalizability of the chosen rank directly, complementing the stability-oriented cophenetic approach and providing a criterion that scales more consistently across datasets of different sizes
-
Functional enrichment analysis was performed with a single tool (WebGestalt) on the binarized common gene set C↳ Could also: Gene Set Enrichment Analysis (GSEA) applied to the full ranked gene score list, or parallel enrichment with tools such as clusterProfiler or g:Profiler, could also be used — GSEA uses the full ranked list rather than a binarized threshold and can detect enrichment distributed across the ranking; using multiple tools reduces sensitivity to annotation database differences specific to any single tool
-
Pre-processing selected the single probeset with the highest MAD to represent each gene when multiple probesets mapped to the same symbol↳ Could also: Summarizing across all probesets mapping to a gene (e.g., RMA median polish or arithmetic mean) could also be applied — Averaging or RMA-based summarization retains information from all probesets and may reduce the influence of a single high-variability probeset that reflects noise rather than true biological signal
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Common biomarker genes characterizing CAF heterogeneity are shared across colon, breast, and ovary carcinoma-derived CAF populations, identified via NMF metagene intersection.microarray human caf 2020×1papers★ This paper is the founder (earliest)
-
Optimal NMF factorization rank for CAF gene expression matrices was automatically determined using cophenetic coefficient, RSS, dispersion, and consensus matrices.microarray human caf 2020×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
- CoINcIDE: A framework for discovery of patient... L1 87/100
- A curated collection of transcriptome datasets... L1 62/100
- Unveiling prognostics biomarkers of tyrosine m...⚑ L1 51/100 ⚑
- Meta-analysis of gene expression profiles of l... L1 78/100
- Colorectal Cancer Prediction Based on Weighted...⚑ L1 80/100 ⚑
- Curation of over 10 000 transcriptomic studies... L1 80/100
- Construction and Validation of an Immune Infil...⚑ L1 51/100 ⚑
- Identification of a novel 10 immune-related ge...
- Exploration of the shared diagnostic genes and... L1 76/100
- IRSN-23 gene diagnosis enhances breast cancer... L1 71/100
- Molecular Classification Models for Triple Neg... L1 86/100
- Predicting Bone Metastasis Using Gene Expressi... L1 62/100
- Autoencoder Networks Decipher the Association... L1 74/100
- Comprehensive analysis of a novel RNA modifica... L1 71/100
- Discovery and validation of molecular patterns... L1 83/100
- Comparative profiling of skeletal muscle model... L1 64/100
- A curated collection of transcriptome datasets... L1 62/100
- PulmonDB: a curated lung disease gene expressi...⚑ L1 53/100 ⚑
- Meta-analysis of gene expression profiles of l... L1 78/100
- Curation of over 10 000 transcriptomic studies... L1 80/100
- Predicting Bone Metastasis Using Gene Expressi... L1 62/100
- Comprehensive analysis of a novel RNA modifica... L1 71/100
- Comparative profiling of skeletal muscle model... L1 64/100
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-32425511
Paper: Esposito, Boccarelli, Del Buono (2020). An NMF-Based Methodology for Selecting Biomarkers in the Landscape of Genes of Heterogeneous Cancer-Associated Fibroblast Populations. Bioinform Biol Insights. PMID 32425511 / PMC7218276.
Code: https://github.com/flaespo/NMF-for-GSE @ ba1ab74 (2019-10-03).
Repo ships 6 GEO *_series_matrix.txt.gz + two templated R scripts (gse import,
NMF) + README. Scripts are skeletons with placeholders ("series name",
"name to identify gene symbol", "number of samples") — must be instantiated per
dataset from the Methods.
Pipeline (R / Bioconductor):
gse import—GEOquery::getGEO(GSE, AnnotGPL=TRUE)→fDatagene symbols +exprsmatrix; subset samples; drop probes with empty gene symbol; collapse duplicate gene symbols keeping the probe with the highest MAD; matrixX(genes × samples).NMF—nmfEstimateRank(X, 2:3, nrun=10)to pick rank;nmf(X, r, nrun=200)(KL divergence, multiplicative updates, random init, RNMFpkg by Gaujoux);featureScore+extractFeatures→ biomarker genes (score > τ = median+3·MAD).
In scope (this RU = GSE51257 only)
GSE51257 = colon-carcinoma CAFs, platform GPL6244, 4 CAF samples (GSM1241296–99 = Low/Med/High/Extra-High migratory; the 2 normal fibroblasts GSM1241300–01 are excluded → "4 samples" in Table 1).
| id | reported (Table 1, GSE51257 row) | pipeline | gradeability |
|---|---|---|---|
| C1 | #genes after MAD-dedup = 20,304 | gse import |
deterministic — independent of sample choice (= #unique non-empty gene symbols on GPL6244) → cleanest 1:1 target |
| C2 | #samples used = 4 | gse import |
deterministic |
| C3 | NMF rank rᵢ = 2 | nmfEstimateRank(X,2:3) cophenetic |
robust (n=4 ⇒ small rank range) |
| C4 | biomarker gene set via extractFeatures (r=2, τ=med+3·MAD) |
NMF |
provisional only — random init + no per-dataset list reported in paper ⇒ not byte-reproducible; reported as derived output, not graded 1:1 |
Out of scope (the hard 20% / cross-dataset)
- The headline intersections — 108 genes common to colon/breast/ovary, 9 genes across colon/breast/ovary/MM, ENPP2 across all 4 tumor types — require running the pipeline on all 6 datasets and set intersection. This RU is scoped to GSE51257; not attempted here (would need the other 5 RUs' outputs).
- WebGestalt functional/pathway enrichment (Tables 2–3): external GUI tool, non-pipeline / not deterministically reproducible.
- The
NMFscript's hard-codedr=3is a leftover template value; the paper's Table-1 rank for GSE51257 is 2 — we follow the paper.
Compute
All on «our HPC» («infra» SLURM) via «host»; conda prefix-env on «infra» with
r-nmf, bioconductor-geoquery, bioconductor-biobase. Data fetched by GEOquery
inside the compute job (nodes have internet). See kartei «our HPC»-conda-r.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Deterministic preprocessing reproduces exactly — 20304 genes × 4 samples matches Table 1 to the digit (C1 is sample-choice-independent, a clean fabrication checkpoint that passed), and the rank-2 choice (C3) is recoverable from the documented multi-criterion analysis. The deviations are on our side / availability: the 4-of-6 sample selection was inferred (not deposited), and the actual contribution — biomarker selection (C4) and cross-dataset intersections (108/9/ENPP2) — was not reproduced (OOM-kill + out-of-scope), and is inherently un-gradeable given random NMF init and no per-dataset reference list. No fabrication signals; this is a solid partial where the central biomarker conclusion remains untested rather than confirmed or refuted.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.