A scalable, open-source implementation of a large-scale mechanistic model for single cell proliferation and death signaling.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1). SPARCED is a large-scale single-cell mechanistic ODE model (914 species, 2434 reactions) built from tab-delimited inputs via Antimony->SBML->AMICI(SUNDIALS), with a hybrid stochastic-deterministic gene module exchanging every 30 s. Pinned to release v1.0.0 (Zenodo 6482266; master has drifted). Built + ran end-to-end on «our HPC» (amici 0.11.12 compiled WITHOUT HDF5; numpy/pandas updated from pins; 2 faithful forward-compat patches: np.float->float and dtype=object on ragged array reads; antimony 2.11.0->2.12.0.1). CORE RESULT: a fresh deterministic single-cell simulation (EGF+INS growth stim, 12 h) reproduces the authors' shipped reference trajectory across ALL 914 species x 1441 timepoints to global relative difference 1.1e-13 (machine precision) -- a clean bit-for-bit reproduction (C4 exact). Structural: genes=141 EXACT (C1). The two headline counts are marginally larger in the paper than in the released artifact: 774 reported vs 773 non-mRNA species (C2, within-tol, off by 1) and 2449 reported vs 2434 compiled reactions (C3, partial, ~99.4%) -- a sub-percent manuscript-vs-release skew flagged for human audit, NOT a reproduction failure or fabrication. NOT attempted (out of scope): wet-lab assay figures, SPARCED-I/SOCS1 extensions, nf/Kubernetes scaling, and the non-seeded stochastic runs.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 84assessed: 2026-06-21 ⛓ 7430225f3c56
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-21
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a putative crosstalk mechanism could be consistent with experimental observations from the LINCS MCF10A Data Cube that IFNγ acts as an anti-proliferative factor when combined with EGF stimulation.
- ★ Developed a python-based, scalable model creation and simulation pipeline that converts structured text files into an SBML-standard model and is high-performance/cloud-computing ready. method
- ★ Applied the pipeline to rebuild the lab's large-scale pan-cancer signaling model (Bouhaddou2018) as SPARCED, adding a new IFNγ pathway submodel. resource
- ★ SPARCED model simulations reproduced the expected experimental and simulation observations of the original Bouhaddou2018 model across all unit tests performed. finding
- ★ IFNγ-induced SOCS1 sequestering of activated EGF receptors can explain the anti-proliferative effect of IFNγ observed in the LINCS MCF10A Data Cube. mechanism
- ★ IFNγ could inhibit cell proliferation through SOCS1 induction and consequent reduced AKT and MAPK (ERK) activity. mechanism
- All model species were annotated with HGNC gene identifiers to enable programmatic filtering and curation. resource
- The pipeline and SPARCED model are released as open-source on GitHub (github.com/birtwistlelab/SPARCED). resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Deterministic ODE simulation | SPARCED model (MCF10A breast epithelial cell line-based) | ligand stimulation (EGF, FGF, IGF, INS, HGF, NRG1, PDGF) | ligand-receptor cooperativity coefficients, receptor internalization dynamics, ppERK/ppAKT activation | AMICI (Python) |
| Hybrid (deterministic + stochastic) simulation | SPARCED model (MCF10A) | DNA damage, TRAIL, etoposide treatment | p53 activation peaks, time to cell death, fraction of surviving cells, cell cycle arrest | AMICI plus custom Python stochastic gene expression submodule |
| Deterministic ODE simulation | SPARCED model (MCF10A) | Cyclin D mRNA level variation; AKT/ERK pathway inhibition | cyclin-CDK complex oscillations, cell cycle entry, synergistic cell death | AMICI (Python) |
| Comparison to LINCS MCF10A Data Cube experimental data | MCF10A cells | IFNγ combined with EGF stimulation | cell proliferation (anti-proliferative response) | — |
| Deterministic simulation | SPARCED model (MCF10A) | none (baseline growth) | ribosome number over time | AMICI (Python) |
- – The SPARCED model passed every unit test and recapitulated experimental and simulation observations reported by the original Bouhaddou2018 model.
- ▲ Activated EGF receptors internalize and peak approximately 30 minutes after ligand treatment. ~30 min peak
- ▲ Dual EGF and insulin stimulation induces prolonged AKT activation compared to single stimulation.
- ▼ Increasing simulated TRAIL dose decreases time to death and decreases the fraction of surviving cells.
- – Increasing ERK and AKT activity prolongs TRAIL-induced time to death, while increasing PUMA/NOXA expression decreases time to death.
- ▲ The number of ribosomes within the cell doubles within 24 hours in simulation. 2-fold in 24 h
- ▼ IFNγ-induced SOCS1 sequestration of activated EGF receptors is proposed to reduce AKT and MAPK activity, consistent with anti-proliferative observations.
- other deterministic module updated every 30 s with mRNA numbers from stochastic module (hybrid simulation functional test)
- other ~30 min (time to peak EGF receptor internalization after ligand treatment)
- fold_change doubles within 24 h (ribosome number increase in deterministic simulation)
- other cell cycle non-traversal and no apoptosis within 48 h under serum/growth-factor starvation (Initialization calibration criterion for new cell line models)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes construction of a large-scale mechanistic (ODE/stochastic) signaling model (SPARCED) built from structured input files, converted to SBML, and simulated with the AMICI Python package. Validation is presented primarily as a series of qualitative 'unit tests' comparing simulated model behaviors (e.g., receptor dynamics, pathway activation, cell-cycle and death outcomes) against previously published simulation results and general experimental trends from the Bouhaddou2018 model and the LINCS MCF10A Data Cube, rather than through classical inferential hypothesis testing. The provided text does not describe formal statistical test statistics, p-values, or error-bar-based comparisons for these unit-test figures.
-
Model validation ('unit tests') is presented as qualitative visual/pattern agreement between simulated outputs and prior published simulation or experimental results (Table 1, Supplementary Figs. 2-11).↳ Could also: Quantitative goodness-of-fit metrics (e.g., root-mean-square error, correlation coefficients, or formal distributional tests such as Kolmogorov-Smirnov) between simulated and reference trajectories — Quantitative fit statistics would allow readers to assess the degree of numerical agreement between the new SPARCED implementation and the original Bouhaddou2018 model beyond visual/qualitative pattern matching.
-
Comparisons across simulated conditions (e.g., cycling vs. non-cycling cells' death fractions under etoposide, TRAIL dose-response survival curves) are described narratively (e.g., 'increased percentage of death') without an accompanying inferential statistic in the provided text.↳ Could also: A formal two-group or dose-response test (e.g., Fisher's exact test or logistic/dose-response regression for survival fractions, or a t-test/Mann-Whitney U for time-to-death comparisons across many simulated single cells) — Such tests would provide a p-value or confidence interval quantifying how distinguishable the simulated conditions are, complementing the descriptive comparison of percentages or trends.
-
Single-cell stochastic simulations (hybrid mode) generate distributions of outcomes (e.g., time-to-death, number of p53 peaks) across simulated cell populations.↳ Could also: Reporting summary statistics with a measure of spread (e.g., median and interquartile range, or mean ± SD/CI) and/or a nonparametric test (Mann-Whitney U, Kruskal-Wallis) across simulated-cell distributions — Because single-cell stochastic outputs are often non-normally distributed, nonparametric summaries and tests can characterize population heterogeneity without assuming normality.
-
The IFNγ/EGF crosstalk hypothesis is evaluated by checking whether model simulations are 'consistent with' experimental observations from the LINCS MCF10A Data Cube, as described in the introduction/abstract.↳ Could also: A formal model-selection or hypothesis-testing framework (e.g., likelihood-ratio test, Akaike Information Criterion comparison between competing mechanistic hypotheses, or parameter estimation with confidence intervals) — This would let a competing set of candidate mechanisms be ranked statistically rather than assessed only by qualitative consistency with the data.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35729113 (SPARCED)
Paper: Erdem et al. 2022, Nat Commun 13:3555. "A scalable, open-source implementation of a large-scale mechanistic model for single cell proliferation and death signaling." PMID 35729113 · PMCID PMC9213456 · DOI 10.1038/s41467-022-31138-1.
Code: https://github.com/birtwistlelab/SPARCED — pinned to release v1.0.0
(the exact version archived for the paper at Zenodo 10.5281/zenodo.6482266,
birtwistlelab/SPARCED-v1.0.0.zip, published 2022-04-25). NOTE: master has
drifted; v1.0.0 is the paper artifact and is what we reproduce.
Data: Zenodo 10.5281/zenodo.6482266 = a snapshot of the GitHub repo at tag v1.0.0 (code + tab-delimited model input files). There is no external sequencing/omics accession; the "data" is the model definition (input_files/) plus MCF10A-derived parameters shipped in the repo. This is a mechanistic ODE model, not a sequencing pipeline.
What kind of paper this is
SPARCED is a Python pipeline that builds a large-scale single-cell mechanistic
model: tab-delimited input files (Species.txt, Ratelaws.txt, StoicMat.txt,
OmicsData.txt, GeneReg.txt, Compartments.txt) → Antimony → SBML
(SPARCED.xml) → compiled to C++ via AMICI (SUNDIALS CVODES solver). It runs
a deterministic ODE module and a hybrid stochastic-deterministic gene-
expression module (modules exchange every 30 s).
In scope (pipeline-derived, reproducible)
The reproduction is fully deterministic for the structural + deterministic- simulation claims (given identical code+inputs, AMICI ODE integration is reproducible to solver tolerance — a clean 1:1).
| # | Result | Where in paper | Pipeline | How we reproduce |
|---|---|---|---|---|
| C1 | Stochastic module = 141 genes | Methods (gene-expression module) | createModel input parse | count rows of OmicsData.txt/GeneReg.txt; count m_* mRNA species in compiled model |
| C2 | Deterministic module = 774 proteins, protein complexes & PTM species | Methods ("The deterministic module describes the concentration dynamics of 774 … species through 2449 reactions") | createModel → SBML → AMICI | build model; count non-mRNA species (and total species) |
| C3 | Deterministic module = 2449 reactions | Methods (same sentence) | createModel → SBML → AMICI | build model; model.getNumReactions() / StoicMat columns |
| C4 | SPARCED deterministic ≈ Bouhaddou2018 (indistinguishable; EGF+insulin) | Fig. 3a/3b | deterministic AMICI sim of SPARCED vs SPARCED_Brep (Bouhaddou replicate) | run both deterministic single-cell sims, compare trajectories / AUC of key species (e.g. ppERK, ppAKT) |
C1–C3 are structural (cheap, high-confidence). C4 is the core scientific deterministic reproduction. Stretch (if feasible): hybrid stochastic single-cell runs (Fig. 2e / 3c) — stochastic, compared distributionally not 1:1.
Out of scope (not attempted, why)
- Wet-lab / experimental data and figures: EdU proliferation assays, etoposide death % (Fig. 3d/e experimental bars), IFNγ EdU p-values (Fig. 4a), Western blots — measured in the lab, not pipeline-derived.
- SPARCED-I / SPARCED-I-SOCS1 model extensions (954/1302 species, Fig. 4–5): IFNγ pathway additions; secondary to the base-model reproduction. Out of the ~80% floor; may revisit only if base model fully reproduces and time allows.
- SPARCED-nf Kubernetes/cloud scaling claims: infrastructure, not a numeric result.
Compute plan («our HPC»/«infra» only)
- front1 (internet):
git clonev1.0.0 + build conda env (python3, amici, libsbml, python-libsbml, antimony, sympy, numpy, pandas, scipy) into «infra» workdir. - SLURM (compute only,
--partition=std --nodes=1 --cpus-per-task=N, no--mem): AMICI model compilation (C++ build, ~15–25 min) + deterministic simulation; emit species/reaction/parameter counts and trajectory CSVs to «infra»; pull back small summaries to «host».
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.