SMAC, a computational system to link literature, biomedical and expression data.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
SMAC is a SOFTWARE/TOOL paper that reports its computational results ONLY as illustrative figures with no printed numeric values (Fig 2 bioinformatics example, Fig 3 benchmark, Fig 4 speed). It IS reproducible as a pipeline-on-data exercise, so this is a valid PARTIAL (figure-level / qualitative) reproduction, not a numeric 1:1. WHAT WE DID: ran SMAC's three deterministic, cancer-specific analyses -- PAM50 molecular subtyping (genefu 2.34.0), ESTIMATE tumour purity (1.0.13), and mclust receptor status -- exactly as in scripts/mClass.R, tPurity.R, rStatus.R, on the paper's own case-study dataset GSE102484 (683 invasive breast-cancer samples, GPL570) on «our HPC». Data prep mirrors functions.py::DownloadGEODataset (probe->gene-symbol via GPL, make.unique) and the verbatim countCancer sample selection (selected all 683/683). RESULTS (all biologically sensible): PAM50 Luminal-dominated (LumB 30.2% + LumA 25.3% = 55.5%, Her2 19.6%, Basal 17.3%, Normal 7.6%); ESTIMATE purity median 0.575; receptor status ER+ 62.1% > PR+ 54.9% > HER2+ 28.3%. These match the KIND of output in Fig 2D/2E. WHY ONLY PARTIAL: the paper prints no value and does not even state which dataset Fig 2 used, so no exact comparison is possible. FIDELITY CAVEAT: SMAC's PAM50 uses do.mapping=FALSE with src/pam50.txt which ships only inside the Docker image (not the GitHub repo) and «our HPC» has no docker/apptainer; we used standard genefu do.mapping=TRUE with an org.Hs.eg.db annot -- method-equivalent, not byte-identical. NOT ATTEMPTED (hard 20%, out of scope): Fig 3 semantic-similarity benchmark (no benchmark code/inputs in repo; Polysearch2 gold-standard defunct; SMAC MeSH list depends on live PubMed), Fig 4 speed (hardware-specific), and the literature/MeSH-RRA + gene-network layers (live PubMed / Entrez-elink / Mentha, non-deterministic). NO fabrication signs: all reproduced values derive solely from the public GSE102484 data + the methods named in the paper.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-14 ⛓ ddf6e06b09a2
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a single, automated, locally-deployable tool simultaneously mine, prioritise and integrate biomedical literature (PubMed/MeSH) with the molecular expression data (GEO) generated by those studies, overcoming the limitation of existing tools that do not link literature to -omics data?
- ★ SMAC is a tool that extracts, prioritises, integrates and analyses biomedical literature and molecular data according to user-defined terms, linking PubMed and GEO. resource
- ★ SMAC is the only available tool that links and integrates PubMed literature/biomedical information with -omics data stored in GEO. finding
- ★ SMAC ranks/prioritises MeSH terms using Robust Ranking Aggregation based on hierarchical specificity, topic abundance and global PubMed abundance. method
- ★ SMAC performs core (PCA, gene expression heatmap, gene interaction network) and cancer-specific (tumour purity, molecular classification) bioinformatics analyses on GEO data. method
- ★ Benchmarking against Polysearch2 demonstrates SMAC reliably captures correct biomedical insights from literature. finding
- SMAC is distributed as a Docker container runnable on Windows, Mac and Linux and is open-source/modular. resource
- SMAC has been integrated into the Breast Cancer Now Tissue Bank bioinformatics platform and the Pancreatic Expression Database. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Semantic similarity benchmarking of MeSH terms (meshes R package) | PubMed/MeSH terms for diabetes, multiple sclerosis and metformin | none | semantic similarity scores (Shortest-Path, Weighted-Link, Wu & Palmer, Leacock & Chodorow, Li, Lord) between SMAC and Polysearch2 terms | meshes R package |
| Computational speed / Time of Execution benchmarking | Invasive breast cancer samples from GEO dataset GSE102484 | none (increasing number of samples) | download and analysis execution time vs number of samples | Local machine with 2 Xeon 5600 processors and 6GB RAM |
| Gene expression analysis (PCA, heatmap z-scores) | GEO expression datasets (any biological context) | none | principal components / normalised expression levels of most variable genes | plotly, R/Bioconductor GEOquery |
| Gene interaction network construction | GEO/PubMed-linked gene sets | none | gene-gene association/co-citation network overlapped with Mentha interactome | visNetwork R package, Entrez elink |
| Tumour purity estimation | Cancer expression datasets from GEO | none | Stromal, Immune and ESTIMATE scores / purity percentage | ESTIMATE algorithm |
| Molecular classification | Breast cancer samples from GEO | none | PAM50 molecular subtype (LumA, LumB, Basal, Her2, Normal) and ER/PR/Her2 receptor status | PAM50 single sample predictor, mclust |
- ▲ All path-based semantic similarity measures exceeded the 0.5 significance threshold in the Polysearch2 benchmark >0.5
- ▲ Shortest-Path method peaked when comparing diabetes-related terms 0.84
- ▲ Lord's information-based metric showed high similarity across all tests 0.89-0.97
- ▲ Positive correlation between Time of Execution and number of samples for both downloading and analysing data <1 s per sample overall
- ▲ Both download and analysis time curves followed a polynomial quadratic trend
- – SMAC supports PCA, DEGs, tumour purity and molecular classification, more analysis types than compared GEO tools
- other 0.84 (Shortest-Path, Diabetes terms) (peak semantic similarity in Polysearch2 benchmark)
- other 0.89-0.97 (Lord's metric) (similarity range across all three benchmark tests)
- other 0.5 (minimum threshold (50% probability) for statistically significant MeSH similarity comparisons)
- other <1 s per sample (overall computational speed for download and analysis)
- count over 27 million citations (biomedical literature citations available in PubMed for querying)
- count more than 60 unique combinations (searches enabled by Polysearch2 interface)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics software/methods paper describing SMAC, a literature- and expression-data mining tool; it does not report a hypothesis-driven experimental study with replicate-based statistics. Statistical elements are confined to (i) a Robust Rank Aggregation procedure used to prioritise MeSH terms relative to a null hypothesis of random item allocation, (ii) a benchmark of semantic-similarity measures comparing SMAC and Polysearch2 terms against a fixed 0.5 threshold, and (iii) a runtime/scalability evaluation fitted with a quadratic (polynomial) trend. Results are reported as single similarity scores per metric/topic and as timing curves rather than as group comparisons with dispersion or p-values.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Robust Rank Aggregation (detecting items ranked better than expected under a null of random allocation) | prioritisation of MeSH terms across specificity, topic-related abundance, and global PubMed abundance rankings | — | stated |
| Semantic-similarity measures (Shortest-Path, Weighted-Link, Wu & Palmer, Leacock & Chodorow, Li, Lord) thresholded at 0.5 | benchmark comparison of SMAC vs Polysearch2 terms for diabetes, multiple sclerosis, metformin (Fig. 3) | three biomedical test topics | stated |
| Polynomial (quadratic) trend/curve fitting | Time of Execution vs number of breast cancer samples for download and analysis tasks (Fig. 4) | increasing number of GSE102484 samples (count not stated) | not stated |
-
Semantic-similarity agreement with Polysearch2 is summarised as single point scores per metric and topic.↳ Could also: Reporting each score with a measure of uncertainty (e.g., bootstrap confidence intervals over terms, or distributions across many query topics) would also be possible. — Interval estimates convey the precision and stability of the agreement, which complements a single reported value, especially with only three test topics.
-
Statistical significance of comparisons is assessed against a fixed 0.5 similarity threshold.↳ Could also: A permutation/randomisation test that derives an empirical null distribution of similarity scores could also be used to assign significance. — An empirical null lets significance adapt to each metric's score distribution rather than a single shared cutoff, which some workflows prefer for calibrating thresholds.
-
The benchmark uses Polysearch2 as the single gold-standard reference across three topics.↳ Could also: Evaluating against multiple reference resources or a larger panel of query topics would also be an option. — Broader references and more topics would characterise generalisability of the agreement across biomedical domains.
-
Runtime scaling is described by fitting a quadratic polynomial trend to Time-of-Execution versus sample count.↳ Could also: Reporting goodness-of-fit (e.g., R²) or comparing candidate models (linear, polynomial, log-linear) with an information criterion would also be informative. — Model-comparison statistics and fit metrics would quantify how well the quadratic form captures the scaling relative to alternatives.
-
MeSH-term prioritisation relies on Robust Rank Aggregation against a random-allocation null.↳ Could also: Alternative rank-aggregation or meta-ranking approaches (e.g., Borda count, weighted Stuart/Kru aggregation, or a stability-selection scheme) could also combine the three ranking criteria. — Comparing aggregation methods would show the sensitivity of the prioritised term list to the chosen aggregation rule.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
SMAC supports more analysis types than compared GEO tools, including PCA, DEGs, tumour purity estimation, and PAM50 molecular classificationother geo 2019×1papers★ This paper is the founder (earliest)
-
SMAC download and analysis time scales with a polynomial quadratic trend as sample count increasesother human-invasive-breast-cancer 2019×1papers★ This paper is the founder (earliest)
-
SMAC execution time increases with number of samples but remains under 1 second per sample for both download and analysisother human-invasive-breast-cancer up 2019×1papers★ This paper is the founder (earliest)
-
Shortest-Path semantic similarity method peaks at 0.84 when comparing diabetes-related MeSH termsother pubmed-mesh-diabetes up 2019×1papers★ This paper is the founder (earliest)
-
Lord information-based similarity metric scores 0.89–0.97 across all MeSH term comparisons, the highest of all tested methodsother pubmed-mesh up 2019×1papers★ This paper is the founder (earliest)
-
All path-based semantic similarity measures exceed the 0.5 significance threshold in the Polysearch2 benchmarkother pubmed-mesh up 2019×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
19 downstream papers · 1 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Sphingomyelin synthase 2 promotes an aggressive brea... 2019 · 81 cites
- Identification of five hub genes as monitoring bioma... 2019 · 48 cites
- Fucosyltransferase 8 regulation and breast cancer su... 2021 · 29 cites
- Potent Anticancer Effect of the Natural Steroidal Sa... 2020 · 24 cites
- BRMS1 coordinates with LSD1 and suppresses breast ca... 2018 · 23 cites
- Identification of Hub Genes Using Co-Expression Netw... 2019 · 23 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-31324861 (SMAC, Pirro et al., Sci Rep 2019)
Paper: "SMAC, a computational system to link literature, biomedical and expression data." DOI 10.1038/s41598-019-47046-2 · PMCID PMC6642118 · Code https://github.com/wynstep/SMAC (master, last push 2020-07-03) · Data: GEO GSE102484 (case-study breast-cancer cohort).
What SMAC is
A software/tool paper. SMAC is a Python+R pipeline distributed as a Docker image
(hfx320/smac). It (i) mines PubMed via Entrez for a user query, (ii) ranks MeSH terms
with Robust Rank Aggregation, (iii) builds gene co-citation networks, (iv) pulls GEO
expression data via GEOquery/GEOparse, and (v) runs bioinformatics analyses on it.
Pipeline-derived results in the paper (classification)
| Reported item | Where | Pipeline | In scope? | Why |
|---|---|---|---|---|
| Fig 2A PCA | Fig 2 | scripts/pca.R |
partial | illustrative, no numeric claim |
| Fig 2B gene-expression heatmap | Fig 2 | scripts/gExpr.R |
partial | illustrative, no numeric claim |
| Fig 2C gene network | Fig 2 | scripts/gNet.R |
out | needs live Entrez elink + Mentha interactome (non-deterministic) |
| Fig 2D tumour purity (ESTIMATE) | Fig 2 | scripts/tPurity.R |
IN | deterministic fn of GEO expression |
| Fig 2E molecular subtypes (PAM50) | Fig 2 | scripts/mClass.R |
IN | deterministic fn of GEO expression |
| Fig 2E receptor status (mclust) | Fig 2 | scripts/rStatus.R |
IN | deterministic fn of GEO expression |
| Fig 3 semantic similarity (0.84 shortest-path; Lord 0.89–0.97) | Results | meshes R pkg vs Polysearch2 |
out | no benchmark script/inputs shipped in repo; Polysearch2 gold-standard service defunct; SMAC MeSH list depends on live PubMed (drifts) → not reproducible from shipped artifacts |
| Fig 4 speed (<1 s/sample, quadratic) | Results | whole pipeline | out | hardware-specific (their 2×Xeon 5600/6 GB); not a reproducible value |
| Literature/MeSH RRA ranking | Methods | main-*.py,rra.R |
out | depends on live PubMed (grows daily) → non-deterministic, no pinned snapshot |
In-scope target (the 80%)
Run SMAC's three deterministic, cancer-specific analyses — exactly the outputs the paper showcases in Fig 2D/2E — on the case-study dataset GSE102484:
- PAM50 molecular subtyping (genefu
molecular.subtyping) → subtype distribution. - Tumour purity (ESTIMATE) → stromal/immune/ESTIMATE scores + purity.
- Receptor status (mclust 2-Gaussian on ESR1/PGR/ERBB2) → %positive.
Faithfulness: data prep mirrors functions.py::DownloadGEODataset (probe→gene-symbol via
GPL annotation, keep all probes, make.unique) and scripts/vars.R cancer-sample
selection (countCancer). Analyses are the authors' own mClass.R/tPurity.R/rStatus.R
computational core (plotly visualization dropped). The PAM50 annot is rebuilt from
org.Hs.eg.db (Symbol→Entrez) because the authors' src/pam50.txt ships only inside the
Docker image, not the GitHub repo — this is a standard genefu-equivalent (do.mapping=TRUE).
The honest catch (no_expected_result risk)
The paper prints no numeric value for the Fig 2 GSE102484 outputs (bar heights only),
and does not even state which dataset Fig 2 used (GSE102484 is named only for the Fig 4
speed test). So this is necessarily a figure-level / qualitative reproduction: we show
the SMAC pipeline runs on the case-study data and yields a biologically sensible breast-cancer
profile (Luminal-dominated PAM50, ESTIMATE purity scores, ER/PR/HER2 +/- splits) consistent
with Fig 2D/2E — not a numeric 1:1. Graded partial accordingly.
Not attempted (hard 20%, explicitly skipped)
Fig 3 semantic-similarity benchmark, Fig 4 speed curve, the literature/MeSH RRA layer, the gene-network layer, PCA/heatmap figures — see "out" rows above for the controlled reasons.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
SMAC is a software/tool paper whose computational results appear only as illustrative figures with no printed numeric values, and the paper never states that Fig 2 used GSE102484 — so a numeric 1:1 is structurally impossible (q2 red, blocker on the authors'/paper-completeness side). The reproduction ran the three deterministic Fig 2 analyses on the public GSE102484 cohort and got biologically sensible, Luminal-dominated outputs (PAM50 Luminal 55.5%, purity 0.575, ER>PR>HER2), all derivable from the shared data with no fabrication signs. Remaining deviations are mild and explainable (assumed dataset; method-equivalent PAM50 since the authors' src/pam50.txt ships only in Docker), and Fig 3/Fig 4 benchmark and speed claims were out of scope — leaving the core tool claim confirmed only qualitatively/limited. Overall a fair figure-level partial, not a defect or discrepancy.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.