Forseti: a mechanistic and predictive model of the splicing status of scRNA-seq reads.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Forseti (COMBINE-lab). The paper is described and shipped well enough to reproduce its fragment-length spline model 1:1 WITHOUT the ~3 TB read download: the experiments repo ships the exact aggregated input (workflow_output/models/spline_model/df_tlen.pkl) that the heavy process_data step produces, so the deterministic (seed=10) spline fit + RMSE from the authors' own fit_spline.py reruns in seconds on «our HPC». Result is byte-exact: both held-out test RMSEs reproduce the shipped test_rmse.json exactly (GSE122357 0.0004024, GSE125188 0.0001822), satisfying the paper's mean<3e-4 (got 2.923e-4); the 2-test-set std is 1.101e-4, marginally over the paper's <1e-4 which most plausibly describes the 5-fold CV error bars (Fig 2b). As an extra check, the independently refit spline equals the shipped spline_model.pkl exactly (knot/coeff max-diff 0.0). NOT ATTEMPTED (the hard ~20%, all gated on the 3 TB Nextflow data download, deliberately skipped): C2 binding-affinity MLP accuracy (needs per-SRR polya_seq.txt, not shipped; only trained mlp.pkl is shipped), C3 splicing-status AUC 0.85-0.93 (needs full aligned + simulated reads), C4 multi-gene origin resolution. Verdict: faithful 1:1 partial reproduction of the cleanly-specified, shipped-data result; honest non-attempt of the data-heavy claims.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 87assessed: 2026-06-14 ⛓ 4fd991d5d43a
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan the splicing status (spliced vs unspliced) of ambiguous exonic scRNA-seq reads be probabilistically recovered by modeling oligo(dT) priming-site binding affinity together with the cDNA fragment length distribution?
- ★ Forseti is the first probabilistic model for resolving the splicing status of exonic scRNA-seq reads by scoring putative fragments linking read alignments to proximate priming sites method
- ★ A trained oligo(dT) binding affinity (MLP) model assigns the probability that a given transcriptomic A-SNR/polyA site is used in fragment generation method
- ★ A robust cDNA fragment length distribution model (cubic spline) generalizes across datasets from different species and tissue types method
- ★ Combining the binding affinity and fragment length models predicts the splicing status of reads' molecule of origin and identifies the true gene origin of multi-gene mapped reads finding
- Internal A-SNR/polyA stretches predominantly on unspliced transcripts are frequently primed by oligo(dT), explaining the high fraction of intronic reads mechanism
- Fundamentally ambiguous reads (e.g. those primed at the shared terminal exon/3'UTR/polyA region) are not targeted and are correctly assigned ambiguous method
- Forseti and the code are available at github.com/COMBINE-lab/forseti under a BSD 3-clause license resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| paired-end 3' scRNA-seq (alternative sequencing format, read1 same cycles as read2) | human and mouse nucleus and cell samples across multiple tissue types | none | cDNA fragment length and priming-window sequences; splicing status of reads | 10x Chromium v2 and v3 |
| cubic spline fragment length distribution fitting | 8 training scRNA-seq datasets (human GRCh38 2020-A, mouse mm10 2020-A) | none | empirical cDNA fragment length distribution (1-1000 bp) | — |
| MLP oligo(dT) binding affinity model training/testing | 8 training and 2 test scRNA-seq datasets | none | binding-affinity prediction / classification accuracy of priming windows | — |
| simulated data evaluation | in silico simulated reads using A-SNR annotations | none | AUC of splicing-status prediction | — |
| experimental evaluation (read2-only single-end prediction vs paired-end ground truth) | 2 held-out test scRNA-seq datasets | none | AUC / accuracy of splicing-status prediction | — |
- – Forseti achieves a mean AUC on simulated data for predicting splicing status AUC=0.92
- – Forseti achieves a mean AUC on experimental data for predicting splicing status AUC=0.88
- – Trained MLP predicts hold-out and test datasets consistently well, indicating no overfitting mean accuracy 0.84 and 0.89
- – Intronic reads account for a substantial fraction of total gene counts (UMIs) in scRNA-seq ~20% to ~40%
- – Applying ambiguous-status strategy assigns over half of gene counts as ambiguous across eight datasets ~46% to ~62%, mean ~53%
- other AUC=0.92 (mean AUC on simulated data)
- other AUC=0.88 (mean AUC on experimental data)
- other 0.84 and 0.89 (mean MLP prediction accuracy on training and hold-out/test sets)
- count ~20% to ~40% (intronic reads' share of total gene count (UMIs))
- count ~46% to ~62% (mean ~53%) (fraction of gene counts assigned ambiguous across eight datasets)
- other 190-290 bp (preferred cDNA insert/fragment length)
- other ~300 to ~400 bp (fragmentation length preference of synthesized cDNAs)
- count 10 (publicly available scRNA-seq datasets collected (8 train, 2 test))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
Forseti is a computational methods paper introducing a probabilistic model that combines a multilayer perceptron (MLP) for oligo(dT) binding affinity with a cubic spline–fitted fragment length distribution to predict the splicing status of ambiguous exonic scRNA-seq reads. Ten publicly available datasets were split into 8 training and 2 test sets; model components were trained on the former and evaluated on the latter. Performance is reported as mean AUC on simulated and experimental data and as mean classification accuracy for the MLP; no classical inferential statistical tests are applied.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Area Under the ROC Curve (AUC) | Overall Forseti model performance on simulated data (mean AUC 0.92) and on experimental test datasets (mean AUC 0.88) | 2 test datasets for experimental evaluation; simulation sample count not stated in provided text | not stated |
| Mean classification accuracy | MLP binding affinity model evaluated on 8 training sets, hold-out sets (1 000 examples per training dataset), and 2 test datasets (mean accuracy 0.84 and 0.89, respectively) | 8 training datasets; 1 000 hold-out examples per training dataset; 2 test datasets | not stated |
-
Model performance is summarized as a single mean AUC across test datasets with no measure of variability reported↳ Could also: Report per-dataset AUC values alongside the mean, or compute a bootstrap 95% confidence interval around the mean AUC — With only two test datasets, variance across datasets cannot be formally estimated, but per-dataset values would show consistency of performance; bootstrap CIs on pooled predictions would quantify uncertainty in the aggregate estimate and are standard practice in predictive-model evaluation
-
The fragment length distribution was fitted with a cubic spline whose smoothing parameter was chosen by visual inspection of the training data↳ Could also: Select the smoothing parameter via generalized cross-validation (GCV) or leave-one-out cross-validation, or fit a parametric distribution (e.g., log-normal or gamma) — Data-driven smoothing selection is reproducible and reduces subjectivity; a parametric fit additionally allows the model to extrapolate gracefully and facilitates direct comparison of fragment length distributions across datasets
-
The train/test partition assigns 8 of 10 datasets to training and 2 to testing in a single split↳ Could also: Use leave-one-dataset-out (LODO) or repeated random cross-validation across all 10 datasets — With a small total number of datasets, a single held-out partition yields a highly variable generalization estimate; LODO cross-validation uses all datasets for evaluation and produces a less variable performance estimate
-
The MLP is evaluated primarily with mean classification accuracy on hold-out and test sets↳ Could also: Additionally report precision, recall, F1 score, or precision-recall AUC — Accuracy can be misleading when classes are imbalanced; since the ratio of genuine priming-site sequences to background sequences is unknown, precision-recall metrics are more informative about model behavior on the minority class
-
Splicing status probability is assigned using the maximum over all compatible transcripts and priming sites (Equations 1–3, a Viterbi/max-product approach)↳ Could also: Marginalize (sum) over all compatible transcripts and priming sites rather than taking the maximum — Summing is the Bayesian marginalizing approach and incorporates probability mass from all compatible explanations; taking the max is computationally simpler but discards information from near-optimal alternatives, which may matter when several priming sites have similar scores
-
Experimental evaluation assesses reads where read 2 is exonically ambiguous but the paired biological read 1 resolves the status; results are not stratified by read category↳ Could also: Report performance stratified by the type of biological read 1 evidence used to assign ground truth (intronic, intron-exon junction, exon-exon junction, distal exon) — Stratified reporting would reveal which subsets of ambiguous reads are most and least tractable for the model, providing more actionable guidance to users and clearer targets for future improvement
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-38940130 (Forseti)
Paper: He D, Gao Y, Chan SS, Quintana-Parrilla N, Patro R. Forseti: a mechanistic and predictive model of the splicing status of scRNA-seq reads. Bioinformatics 2024; 40(7):btae207. PMID 38940130 · PMCID PMC11256924 · DOI 10.1093/bioinformatics/btae207.
Code: https://github.com/COMBINE-lab/forseti (tool: pretrained models + predictor) and the experiments repo https://github.com/COMBINE-lab/forseti-experiments (Nextflow pipeline that regenerates the figures). The experiments repo is the authors' own (P16 own-code).
Data: the experiments repo's Nextflow pipeline downloads ~10 paired-end 10x Chromium
scRNA-seq datasets (GEO/SRA) and needs ~3 TB disk. It also ships precomputed
intermediates and trained models under workflow_output/.
What the paper reports (candidate claims, pipeline-derived)
| # | result | reported value | location |
|---|---|---|---|
| C1 | Fragment-length spline model, RMSE on the 2 held-out test datasets | GSE122357 = 4.024e-4, GSE125188 = 1.822e-4 (mean < 3e-4, std < 1e-4) | Sec 3.1 / Fig 2; repo workflow_output/models/spline_model/test_rmse.json |
| C2 | Binding-affinity MLP accuracy | train 0.83, hold-out 0.84, test 0.89 | Sec 3.1 / Fig 2 |
| C3 | Splicing-status prediction AUC (sim + experimental) | 0.85–0.93 (sim 0.92/0.92, mouse 0.89, human 0.85) | Sec 3.2 / Fig 3 |
| C4 | Multi-gene origin resolution | single best gene for 65–93%; correct for 69–75% | Sec 3.3 |
In scope (attempted) — reproducible from SHIPPED data, no 3 TB download
- C1 — spline RMSE (PRIMARY).
fit_spline.pyrebuilds a per-dataset fragment-length frequency tabledf_tlen_countfrom the 3 TBprocess_datastep, but that exact table is shipped asworkflow_output/models/spline_model/df_tlen.pkl(80 kB). From it the spline fit + RMSE are a deterministic (seed=10), seconds-long computation. We rerun the authors'fit_spline.pylogic on the shippeddf_tlen.pkland compare to theirtest_rmse.jsonand to the paper. We additionally check our refit spline against the shippedspline_model.pkl.
Out of scope (NOT attempted) — the hard last ~20%
- C2 — MLP retraining.
fit_mlp.pyneeds per-SRRprocess_data/.../polya_seq.txt(the oligo-dT priming windows), which come from the 3 TB read-processing step and are not shipped. Only the trainedmlp.pklis shipped, so accuracy cannot be independently recomputed without the full download. Not attempted (would not meet the 80/20 / no-3-TB constraint). - C3 — AUC evaluation. The evaluation notebooks need the full aligned read sets + simulated reads (3 TB). Reported AUCs are embedded in the shipped notebooks but cannot be independently recomputed here. Not attempted.
- C4 — multi-gene resolution. Same dependency on full data. Not attempted.
Pipelines named: C1 = scipy cubic-spline (interpolate.splrep, k=3, s=1e-6) over an
empirical fragment-length distribution; C2 = sklearn MLPClassifier; C3 = the Forseti
scoring model + sklearn roc_auc. Only C1 is reproducible from shipped artifacts.
Heavy compute for C1 runs as a SLURM job on «our HPC» per the hard rules, even though it is light, with all data on «infra».
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The attempted in-scope claim — the fragment-length cubic-spline model (C1) — reproduces byte-exact: both held-out RMSEs (0.0004024, 0.0001822) match the shipped test_rmse.json and the independently refit spline equals spline_model.pkl with max|diff|=0.0, all fully derivable from the authors' shipped df_tlen.pkl. The only deviation is a marginal std RMSE (0.0001101 vs <1e-4), which is on our side — a summary-statistic definition over 2 test sets vs the paper's likely 5-fold CV — and is negligible. The reproduction is honest and faithful but partial: the paper's headline predictive claims (C2 MLP accuracy, C3 splicing-status AUC, C4 multi-gene resolution) were not attempted because they require the ~3 TB read download, leaving the central conclusion only component-confirmed.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.