spliceJAC: transition genes and state-specific gene regulation from single-cell transcriptome data.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce 1:1. The brief's harvested code link (BoolODE) is only the third-party synthetic-data simulator; the paper's actual tool is spliceJAC (first author Bocci), pip splicejac==0.0.1, repo federicobocci/spliceJAC @4939ca9 (MIT). It ships the input (scVelo pancreas) AND the author reference output (results/export_output/transition_score.csv), making the pancreas transition-gene result (Fig 3) a clean target. On «our HPC» we built the pinned env (numpy1.23.2/scanpy1.9.1/scvelo0.2.4/anndata0.8.0/sklearn1.1.2), replicated the repo's Transitions.ipynb pipeline, and regenerated the 7-transition per-gene transition-score table. RESULT: the spliceJAC computation reproduces essentially EXACTLY (per-transition Pearson 0.998-1.000, Spearman 0.985; deterministic seed=100). The endocrine biology reproduces: Iapp->Beta, Gcg->Alpha, Ghrl->Epsilon are exact top-1 matches and Sst->Delta is top-5. The ONLY divergence is the upstream 50-highly-variable-gene SELECTION: 39/50 overlap, and 4 of the reference's top genes (Spp1, Neurog3, Chga, Chgb) fell outside our HVG set, flipping 4/7 top-1 calls; the shipped CSV was most likely generated by the repo's now-removed Pancreas.ipynb (referenced in .ipynb_checkpoints) rather than the current Transitions.ipynb. No fabrication signal: near-perfect numeric agreement on every commonly-selected gene confirms the reported values are genuine pipeline outputs. NOT attempted (the hard 20%): BoolODE synthetic-circuit GRN-inference benchmarking (AUPRC/AUROC, Fig 2), the A549 EMT/GSE147405 analysis, and grn_statistics network stats (its nx.from_numpy_matrix is gone in networkx>=3 and is not needed for transition scores).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 53assessed: 2026-06-15 ⛓ b1e4bb75b1c5
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCell state-specific gene-gene regulatory interactions and the stability/instability of cell states can be inferred from single-cell unspliced and spliced mRNA count data by fitting a multivariate, state-specific linearized mRNA splicing model, whose Jacobian eigen-analysis can predict the genes that drive transitions between cell states.
- ★ spliceJAC uses unspliced and spliced mRNA count matrices to construct cell state-specific gene-gene regulatory interaction (Jacobian) matrices from scRNA-seq data method
- ★ Spectral analysis of the inferred Jacobian matrix predicts putative driver genes ('transition genes') critical to transitions between cell states via an instability score bound between 0 and 1 mechanism
- ★ spliceJAC accurately reconstructs ground-truth interaction matrices for in silico multistable circuits (toggle switch, 3-gene activation loop, EMT circuit) finding
- ★ spliceJAC outperforms several existing GRN inference methods (Beeline pipeline) on synthetic circuit benchmarks based on AUPRC ratio and matrix element sign accuracy finding
- ★ spliceJAC correctly discriminates stable versus unstable fixed points, predicting a positive eigenvalue at unstable fixed points finding
- When cell states are erroneously mixed, spliceJAC predicts the largest eigenvalue to be positive or close to zero, providing self-consistent validation of the stable-state assumption finding
- spliceJAC inference remains robust when genes are removed one at a time from the input count matrix finding
- ★ Applied to pancreas endothelium development and EMT in A549 lung cancer cells, spliceJAC recovers known differentially expressed genes and predicts new transition genes exclusive or shared between cell state transitions finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| in silico simulation / synthetic scRNA-seq-like count generation | bistable toggle switch circuit (genes X, Y) | stochastic perturbation around stable fixed points | inferred vs ground-truth gene-gene interaction matrix | — |
| in silico simulation | three-gene mutual activation loop circuit | stochastic perturbation around single fixed point | inferred vs ground-truth gene-gene interaction matrix | — |
| in silico simulation | EMT gene circuit (Tian et al 2013): miR-34, miR-200, ZEB, SNAIL, TGF-beta, E-cadherin, N-cadherin | TGF-beta inducer level tuned to tristability; perturbation around fixed points | inferred vs ground-truth interaction matrices for epithelial, hybrid E/M, mesenchymal states; eigenvalue spectra | — |
| in silico mixed-state simulation | EMT circuit and toggle switch circuit | artificial mixing of cells from multiple simulated cell states | sign/magnitude of largest Jacobian eigenvalue | — |
| GRN inference benchmarking (AUPRC comparison) | in silico circuits (toggle switch, 3-gene loop, EMT circuit) | none | AUPRC ratio and absolute difference between ground truth and predicted interaction matrix, fraction of correct matrix element signs | Beeline pipeline |
| scRNA-seq (unspliced/spliced mRNA quantification) | mouse pancreas endothelium development | none (developmental time course) | cell state-specific gene regulatory networks and predicted transition driver genes | — |
| scRNA-seq (unspliced/spliced mRNA quantification) | A549 lung carcinoma cells | epithelial-mesenchymal transition (EMT) induction | cell state-specific gene regulatory networks, differentially expressed genes, and predicted transition genes | — |
| gene knockout robustness test (leave-one-gene-out) | in silico circuits | sequential removal of individual genes from count matrix input | stability/consistency of inferred interaction matrix on remaining gene subset | — |
- – spliceJAC reconstructs interaction matrices for both stable fixed points of the toggle switch with high precision, correctly capturing state-specific reversal of dominant inhibition direction
- – spliceJAC correctly recovers the interaction matrix of the monostable three-gene mutual activation loop
- – spliceJAC robustly recovers interaction matrices for epithelial, hybrid E/M, and mesenchymal states of the EMT circuit at a tristable TGF-beta level
- ▲ At the unstable fixed point of the toggle switch, spliceJAC correctly predicts a positive eigenvalue indicating instability
- – When cells from multiple EMT circuit states are mixed, spliceJAC's predicted largest eigenvalue is always positive or extremely close to zero
- – spliceJAC inference remains robust when individual genes are removed one at a time from the input data
- ▲ spliceJAC outperforms other GRN inference methods in the Beeline pipeline based on normalized AUPRC ratio across circuit states
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational methods paper presenting spliceJAC, a tool that infers cell state-specific gene–gene (Jacobian) interaction matrices from unspliced/spliced scRNA-seq counts and uses linear stability (eigenvalue) analysis to predict transition driver genes. Rather than classical hypothesis testing, the statistical approach centers on regression-based parameter inference and benchmarking: performance is evaluated against analytically derived ground-truth circuits using metrics such as absolute matrix-element difference, fraction of correctly signed matrix elements, and AUPRC (via the Beeline pipeline), and applied to real pancreas-endothelium and A549 EMT datasets. Results are reported largely as inferred matrices, eigenvalue spectra, instability/signaling scores, and comparisons with prior differentially expressed gene analyses.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Linear regression to infer attractor-specific gene–gene interaction (Jacobian) matrices from steady-state assumption | Inference step within each cell state (Eqs. 5a–5b; Fig 1B–C, Fig 2) | number of cells within a given cell state (stated as upper limit on number of inferable interactions); exact n not given in visible text | stated |
| Eigenvalue (linear stability / spectral) analysis of the inferred Jacobian | Stability of cell states and identification of transition driver genes (Fig 1E, Fig EV1) | — | stated |
| Benchmark metric: absolute difference between ground-truth and inferred matrices | In silico EMT circuit comparison across epithelial/hybrid/mesenchymal states (Fig 2I) | — | na |
| Benchmark metric: fraction of correctly signed matrix elements | In silico circuit comparison (Fig 2J) | — | na |
| AUPRC (area under precision–recall curve) ratio, normalized per state | Comparison against GRN inference methods in the Beeline pipeline (Fig 2K) | — | na |
-
Method performance was summarized with point metrics (absolute matrix difference, fraction of correctly signed elements, AUPRC ratio) across in silico circuits.↳ Could also: Reporting these metrics with dispersion across multiple simulation replicates (e.g., mean ± SD/95% CI or full distributions via boxplots) and bootstrap confidence intervals. — Adding variability estimates would convey how stable the benchmark rankings are across stochastic simulations and would complement the central-tendency comparison.
-
spliceJAC was compared against other GRN inference methods using normalized AUPRC ratios.↳ Could also: Complementing AUPRC with AUROC, early-precision, and a paired statistical comparison (e.g., Wilcoxon signed-rank across circuits) of per-circuit scores. — Multiple complementary metrics and a paired test across shared circuits would give a more complete picture of relative performance under class imbalance.
-
Cell annotations (clusters) were assumed to correspond to stable cell states, with stability analysis offered as self-consistent validation.↳ Could also: Sensitivity analyses across alternative clustering resolutions or annotation schemes, reported as agreement statistics. — Quantifying how inference outputs vary with upstream clustering choices would characterize robustness to the state-definition step.
-
Interaction matrices were inferred via linear regression under a steady-state, near-attractor linearization with a unique-solution constraint (genes ≤ cells).↳ Could also: Regularized regression (e.g., ridge/LASSO/elastic-net) with cross-validated penalty selection. — Regularization would allow stable estimation when gene count approaches or exceeds cell number and could improve generalization in high-dimensional, noisy settings.
-
Robustness was assessed by removing genes one at a time and re-running inference.↳ Could also: Bootstrap or subsampling of cells and repeated resampling, reporting the distribution of inferred edges/eigenvalues. — Resampling-based stability assessment would quantify uncertainty in individual inferred interactions and driver-gene rankings.
-
Transition driver genes were prioritized by an instability score derived from the unstable manifold projection.↳ Could also: Attaching empirical significance or confidence to driver-gene scores via permutation/null-model comparison. — A null distribution would help distinguish genuinely high-instability genes from those expected under randomized interaction structure.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
spliceJAC robustly recovers distinct gene-gene interaction matrices for epithelial, hybrid, and mesenchymal states from a tristable EMT regulatory circuit.other in-silico emt-circuit 2022×1papers★ This paper is the founder (earliest)
-
spliceJAC outperforms existing GRN inference methods on absolute matrix element difference, fraction of correctly signed interactions, and AUPRC ratio across epithelial, hybrid, and mesenchymal EMT states.other in-silico emt-circuit up 2022×1papers★ This paper is the founder (earliest)
-
spliceJAC GRN inference is robust to single-gene removal from the input count matrix, with no significant degradation in inferred interactions.other in-silico emt-circuit none 2022×1papers★ This paper is the founder (earliest)
-
spliceJAC predicts a positive or near-zero largest eigenvalue for mixed-state EMT cell populations, flagging state instability from co-mingled epithelial and mesenchymal cells.other in-silico emt-circuit up 2022×1papers★ This paper is the founder (earliest)
-
spliceJAC correctly recovers the interaction matrix of a three-gene mutual-activation circuit at its monostable fixed point.other in-silico three-gene-loop 2022×1papers★ This paper is the founder (earliest)
-
spliceJAC predicts a positive leading eigenvalue at the unstable fixed point of a bistable toggle switch, correctly indicating state instability.other in-silico toggle-switch up 2022×1papers★ This paper is the founder (earliest)
-
spliceJAC reconstructs state-specific gene-gene interaction matrices for both stable fixed points of a bistable toggle switch with high precision, capturing reversed interaction amplitudes between states.other in-silico toggle-switch 2022×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36321549 (spliceJAC)
Paper: Bocci F, Zhou P, Nie Q. spliceJAC: transition genes and state-specific gene regulation from single-cell transcriptome data. Mol Syst Biol 2022. PMID 36321549 · PMCID PMC9627675 · DOI 10.15252/msb.202211176.
What spliceJAC is
A Python tool that uses spliced + unspliced scRNA-seq counts (RNA velocity inputs) to fit a cell-state-specific linear Jacobian by regression, then does stability analysis: the leading eigenvector of the Jacobian at an unstable steady state ranks transition (driver) genes for each cell-state transition.
Code situation (P16 note)
- The link harvested into the brief (
github.com/Murali-group/BoolODE) is not the paper's analysis code — BoolODE is the third-party synthetic-data simulator the paper used only for its benchmarking circuits (toggle switch, EMT circuit…). - The paper's actual analysis tool spliceJAC is released by the first author:
github.com/federicobocci/spliceJAC(MIT, default branchmain, pipsplicejac==0.0.1). This is what we reproduce — running the paper's own tool on the paper's own bundled data is the most faithful reproduction.
In scope (pipeline-derived, attempted)
| result | pipeline | data | reproducible? |
|---|---|---|---|
| Pancreas transition genes / per-gene transition scores (Fig 3) | spliceJAC estimate_jacobian → transition_genes |
bundled datasets/Pancreas/pancreas.h5ad (scVelo pancreas endocrinogenesis) |
YES — repo ships both the input pancreas.h5ad AND the reference output results/export_output/transition_score.csv (7 transitions × ~50 genes). We regenerate the table and compare 1:1. |
The shipped transition_score.csv is the concrete, machine-checkable reference.
Top-scoring genes per transition are the canonical lineage markers (Ins1/Ins2/Iapp
→ Beta, Gcg → Alpha, Sst → Delta, Ghrl → Epsilon), i.e. the biology the paper
reports for the pancreas in Fig 3.
Out of scope (not attempted — 80/20)
- BoolODE synthetic-circuit benchmarking (AUPRC/AUROC vs other GRN methods,
Fig 2 / EV figs): would require regenerating BoolODE simulations + the separate
jacobian-inference-benchmarkingrepo. Larger effort, deferred (the hard 20%). - A549 EMT / TGFβ (GSE147405) analysis: not bundled; would require fetching + full RNA-velocity preprocessing. Deferred.
- All wet-lab / manual / schematic figure content — out of scope by definition.
Reproduction strategy
- «our HPC» SLURM job, conda env pinned to spliceJAC
requirements.txt(numpy 1.23.2, scanpy 1.9.1, scvelo 0.2.4, anndata 0.8.0, sklearn 1.1.2 …). - Load bundled
pancreas.h5ad, follow the repo's Pancreas/Transitions notebook steps, runestimate_jacobian+transition_genes. - Export our
transition_scoretable; compare to repotransition_score.csv(correlation + top-gene overlap per transition).
All heavy compute on «our HPC»/«infra»; «host» holds only small result artifacts.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The spliceJAC algorithm reproduces essentially exactly — per-transition Pearson 0.998–1.000 and mean Pearson 0.9997 across the 39/50 shared genes, with no fabrication signal. The sole deviation is upstream HVG selection: 4 of the reference's top genes (Spp1, Neurog3, Chga, Chgb) fell outside our 50-gene set, flipping 4/7 top-1 calls (C1–C3, C6), most plausibly because the shipped CSV came from the repo's now-removed Pancreas.ipynb rather than the current Transitions.ipynb. This is a preprocessing/code-version issue on a mix of our pipeline choice and an author-side code mismatch — explainable and non-severe; the well-separated endocrine branches (Iapp→Beta, Gcg→Alpha, Ghrl→Epsilon, Sst→Delta top-5) confirm the central biology.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.