Generative and integrative modeling for transcriptomics with formalin fixed paraffin embedded material.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values are derivable from the shared data
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce: YES for the shipped software pipeline. This is a pipeline-execution reproduction (brief P16) - the authors' own PREFFECT repo (github.com/hallettmiket/preffect @ 35b5496) run on the authors' own shipped simulated data (vignettes/simple, which carry a ground-truth 'truth' layer) using the exact tutorial-notebook config (simple NB model, 2000 epochs, mini_batch=50, r=20, lr=1e-3, batch correction), on «our HPC» CPU. RESULT = 1:1 on the pipeline, partial on the numbers. The simple VAE trains and converges (loss 6036->124); inference clearly DENOISES - inferred counts correlate with ground truth far better than raw observed counts (Pearson 0.42->0.79, Spearman 0.46->0.85); imputation of MCAR-masked entries works (MRE0.40); Leiden clustering + batch adjustment run; all 21 documented output figures are produced and visually match the shipped expected outputs (gene-scatterplot reproduced side-by-side). NOT a numeric 1:1 against any paper TABLE because the paper's headline numbers are out of scope: Fig 1 distribution params (mu710, alpha~1.47, zero-frac 0.46) are compendium-wide over partly-private data; Fig 6 silhouettes (0.85/0.45/0.63) use pseudo-synthetic subtype data; Sunnybrook cohorts are explicitly not public. Also: authors used seed=None (non-deterministic) so pixel-exact figure match is impossible by design, and the shipped CPU yml omits scanpy (needed for clustering) - documented env gaps. WHAT WE DID NOT ATTEMPT: Fig1/Fig6 numbers, the full multi-tissue model (C5), and NB-vs-ZINB AIC on GSE209998 (feasible but the deprioritized 20%). No fabrication signal: every reproduced value is derivable from the shipped data+code.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 67assessed: 2026-06-14 ⛓ 20102044cd11
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusFFPE RNA-seq (fRNA-seq) data is noisy, incomplete, and heterogeneous; the authors hypothesize that its transcript counts are best modeled by the negative binomial distribution and that a generative, integrative probabilistic framework leveraging matched tissues and sample-sample adjacency networks can impute missing counts, correct technical effects, and improve downstream analyses better than generic bulk RNA-seq tools.
- ★ fRNA-seq transcript counts are best fit by the negative binomial distribution, with little evidence supporting zero-inflated extensions finding
- ★ PREFFECT is a probabilistic generative framework (conditional VAEs with graph attention) for cleaning, normalizing, and imputing fRNA-seq data method
- ★ PREFFECT accurately imputes missing values from fRNA-seq count matrices and adjusts for batch effects finding
- ★ Inclusion of sample-sample adjacency networks and multiple matched tissues enhances sample clustering finding
- ★ More advanced PREFFECT models that include matched profiles give better, more specific model fits than generic bulk RNA-seq tools finding
- PREFFECT-transformed data can be used directly with established downstream tools (differential expression, survival analysis, classification) resource
- A compendium of public fRNA-seq datasets was assembled to characterize statistical properties and benchmark PREFFECT resource
- Formalin fixation degrades nucleic acids, producing high transcript dropout, high variance, and extreme counts in fRNA-seq mechanism
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| FFPE RNA-sequencing (fRNA-seq) distributional analysis | compendium of public FFPE datasets (GEO, ENA, SRA, dbGaP; TMBC via cBioPortal; Sunnybrook cohorts) across varied tissue/cell type | none | goodness-of-fit of transcript counts to six candidate distributions (NB vs ZINB) | — |
| Distribution fitting / model selection | fRNA-seq compendium transcripts | none | AIC (2k-2ln(L)) and Kolmogorov–Smirnov D statistic | StatsModels, scipy minimize (Python 3.9) |
| Synthetic count matrix simulation | in silico NB-generated counts (N=1000 transcripts, M=1000 samples) | varied mean/dispersion (mu, theta) and dropout pi~U(0..0.8) | model technical correctness, parameter estimation, dropout role | PREFFECT generative models |
| Pseudo-synthetic sample generation and imputation benchmarking | breast cancer FFPE datasets with PAM50-estimated subtypes | constructed/permuted (null) sample-sample adjacency matrices | imputation accuracy contribution of adjacency matrix | PAM50 classifier; PREFFECT |
| Sample clustering / classification evaluation | publicly available fRNA-seq datasets | PREFFECT count adjustment vs unadjusted | cluster mixing (k-BET), cluster purity (ARI), UMAP/hierarchical clustering | scib, scikit-learn, SCANPY, umap-learn, seaborn |
- – Negative binomial distribution best fits fRNA-seq transcript counts across the compendium
- – Little evidence supports zero-inflated (ZINB) extensions over plain NB
- – PREFFECT accurately imputes missing values and adjusts for batch effects
- ▲ Adding sample-sample adjacency networks and multiple tissues enhanced sample clustering
- ▼ RNA from FFPE tissue had a median RNA integrity number of 2.5 vs 8.1 for fresh frozen ~2-fold lower
- ▼ FFPE RNA DV200 was 48% vs 97% for fresh frozen, representing nearly two-fold degradation ~2-fold
- other RIN 2.5 (FFPE) vs 8.1 (fresh frozen) (median RNA integrity number, Jacobsen et al.)
- other DV200 48% (FFPE) vs 97% (fresh frozen) (percent RNA fragments >200 nt)
- count ~1500 fRNA-seq datasets in international repositories (estimated number in ENA/SRA/dbGaP)
- count ~10^2 samples per fRNA-seq dataset vs 10^4–10^6 for scRNA-seq (typical study sizes)
- count >1 billion archival FFPE samples worldwide (available FFPE resource)
- other 1% trimmed mean (mild outlier mitigation in distribution fitting)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This methods paper introduces PREFFECT, a probabilistic framework for FFPE RNA-seq data using conditional variational autoencoders (cVAEs) with negative binomial (NB) or zero-inflated NB generative models. Distribution selection was driven by AIC and Kolmogorov–Smirnov (KS) D statistics across a compendium of public fRNA-seq datasets. Model performance was evaluated using clustering quality metrics (k-BET for batch mixing, ARI for cluster purity) on both synthetic and real datasets; no traditional group-comparison hypothesis tests were applied to biological outcomes.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Akaike Information Criterion (AIC) — model selection | Comparison of six candidate count distributions (NB, ZINB, and others) for each transcript across fRNA-seq compendium | Compendium of public fRNA-seq datasets; exact n not stated in visible text | not stated |
| Kolmogorov–Smirnov (KS) D statistic | Secondary goodness-of-fit check comparing empirical CDF per transcript to each reference distribution CDF, to avoid bias from parameter-count differences between distributions | Same compendium datasets; exact per-transcript n not stated | not stated |
| k-BET (kernel-based test for batch effects / cluster mixing measure) | Evaluation of batch mixing after PREFFECT adjustment, computed via Python scib package | null | not stated |
| Adjusted Rand Index (ARI) | Evaluation of cluster purity after PREFFECT adjustment, computed via scikit-learn | null | not stated |
| PAM50 classifier | Subtype estimation for breast cancer FFPE samples lacking provided subtypes, and for pseudo-synthetic samples | null | na |
-
The AIC was used as the primary criterion for selecting among six count distributions↳ Could also: The Bayesian Information Criterion (BIC) could also be used for distribution selection — BIC applies a stronger penalty for the number of parameters that scales with sample size (ln(n)·k vs 2k in AIC), which tends to favor more parsimonious models in large-n settings; reporting both AIC and BIC would allow readers to assess sensitivity of the distribution preference to the penalty term
-
The KS D statistic was used as a secondary goodness-of-fit measure for each transcript against each candidate distribution↳ Could also: The Anderson–Darling (AD) test statistic could also be used alongside or instead of the KS D statistic — The AD test assigns greater weight to discrepancies in the tails of the distribution, which may be particularly relevant for count data with overdispersion or extreme values characteristic of FFPE RNA-seq; it can therefore be more sensitive to departures that matter most for the NB vs ZINB distinction
-
A 1% trimmed mean was applied to mitigate outlier influence during distribution fitting (except for the NB vs ZINB comparison)↳ Could also: Robust parameter estimation methods such as M-estimators or median-based moment matching could also be used to reduce outlier influence — Trimming discards data points entirely, which may interact with the zero structure of fRNA-seq counts; robust estimation retains all observations while down-weighting extremes, potentially providing more stable parameter estimates without selective removal
-
Hierarchical clustering used average linkage with Euclidean distance↳ Could also: Ward linkage with Euclidean distance, or average linkage with a count-appropriate dissimilarity (e.g., Bray–Curtis or Jensen–Shannon divergence) could also be used — Ward linkage minimizes within-cluster variance and often produces more compact, well-separated clusters; count-specific dissimilarities may better reflect the compositional structure of RNA-seq data than Euclidean distance, which is sensitive to library-size differences
-
UMAP (min_dist=0.3) was used for dimensionality reduction and visualization↳ Could also: t-SNE or PCA could also be used for low-dimensional visualization of corrected count matrices — t-SNE preserves local neighborhood structure and is widely used for RNA-seq visualization, enabling direct comparison with published scRNA-seq workflows; PCA provides a linear projection that is fully reproducible and interpretable in terms of explained variance, complementing the nonlinear UMAP embedding
-
Model quality was summarized with k-BET (batch mixing) and ARI (cluster purity) as the primary evaluation metrics↳ Could also: Silhouette score and Normalized Mutual Information (NMI) could also be reported as complementary clustering evaluation metrics — Silhouette score quantifies cohesion and separation in a geometry-agnostic way across the full embedding; NMI measures the information shared between predicted and known cluster labels and is less sensitive than ARI to cluster-size imbalance, providing a more complete picture of clustering performance especially for unequal-size subtypes
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-41029822 (PREFFECT)
Paper: Mucaki et al. 2025, Generative and integrative modeling for transcriptomics
with FFPE material, J Transl Med. PMID 41029822 / PMC12486589 / DOI 10.1186/s12967-025-07031-y.
Code: https://github.com/hallettmiket/preffect (default branch main, license CC0-1.0).
Data: GEO GSE209998 (public, used for Fig 1B histogram); Sunnybrook cohorts NOT public.
What PREFFECT is
A PyTorch probabilistic/generative framework (conditional VAE + graph-attention
networks, GATv2) that denoises/imputes FFPE RNA-seq count matrices and integrates
multiple matched tissues + sample-sample adjacency networks. Three model types:
simple (plain CVAE), single (1 tissue + adjacency GAT), full (multi-tissue + adjacency).
The repo ships: the full package, a pinned CPU conda env (yml/preffect_cpu_LATEST.yml),
a tutorial notebook (vignettes/example_script_simple.ipynb) with saved cell outputs,
simulated example datasets for all three model types (vignettes/{simple,single,full}/
as AnnData .h5ad, each carries a ground-truth truth layer), and expected output
figures in vignettes/vignette_output/ (losses, gene scatterplot, batch adjustment,
UMAP, lib-size, dispersion PDFs).
IN SCOPE — pipeline-derived, self-contained on shipped code+data (the 80%)
The reproduction is a software/pipeline reproduction (brief P16: applying the shipped tool to the shipped data per the documented parameters is fully valid). Concrete, mostly seed-robust outputs we measure:
- C1 — Training converges. Train the
simpleNB model onvignettes/simple/exactly as notebook cell [5] (epochs=2000, mini_batch_size=50, r=20, lr=1e-3, correct batch). Pipeline: PREFFECT VAE training. Output: 8-panellosses.pdf. Metric: train/val average loss first vs last epoch (must decrease/converge). - C2 — Denoising. After inference on the held-out test set, the inferred mean
counts
X_hat_mushould correlate with observed counts and (using the shippedtruthlayer) recover the noise-free signal: corr(truth, fitted) ≥ corr(truth, observed). Pipeline: PREFFECT inference. Maps to paper's core "denoise fRNAseq" claim. - C3 — Imputation error. Default training masks 10% of entries (MCAR, lambda_counts=0.1).
calculate_imputation_errorreturns MRE/MSE on masked vs unmasked entries. Maps to paper Fig 4 ("near-perfect MRE for parameter/value recovery"). - C4 — Leiden clustering + batch adjustment run and produce the documented output PDFs; batch-adjusted inference (batch 1→0) executes. Maps to Figs 5–6 (clustering) and the batch-correction claim.
- C5 (secondary) —
full2-tissue integrative model trains end-to-end onvignettes/full/(the novel GAT/adjacency contribution), if time permits.
OUT OF SCOPE — the hard ~20% / non-public / compendium-wide (NOT attempted, why)
- Fig 1 distribution params (median zero-fraction 0.46, mean NB μ≈710, mean α≈1.47): computed across the whole ~compendium (Table S2), many datasets incl. non-public ones; not pinnable to a single shipped artifact.
- Fig 6 silhouettes (0.85 / 0.45 / 0.63) for 5-subtype clustering under adjacency manipulation: uses pseudo-synthetic subtype data (1000/250 samples per subtype) whose generator is not clearly shipped + the full adjacency pipeline → high-effort tail.
- NB-vs-ZINB AIC on GSE209998 specifically: feasible (data public,
_distributions.pyships a ZINBEstimator) but requires GEO download + matching their exact fitting protocol; deprioritized as 20%. - Sunnybrook tumor/stroma cohorts: explicitly "not currently publicly available …
pending publication" →
data_restrictedfor those analyses.
Environment note (honesty)
_config.py default seed: None → the authors' runs were non-deterministic;
byte-exact reproduction of the shipped output PDFs is impossible by design. We therefore
grade on pipeline execution + seed-robust quantitative metrics, not pixel ma
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Running the authors' own PREFFECT tool on their shipped simulated vignette data reproduces the documented pipeline 1:1 — the simple VAE trains and converges (loss 6036->124), inference clearly denoises (Pearson 0.417->0.793, Spearman 0.464->0.849), imputation and Leiden+batch-adjust run, and all 21 output PDFs are produced. Every reproduced value is derivable from the shipped data+code with no fabrication signal, so q5 is green. Deviations are explainable and on our/scope side: MRE~0.40 was measured on a different sim than Fig4's 'near perfect' sweep, while the clinical headline numbers (Fig6 silhouettes, Sunnybrook cohorts) rest on partly-private data and the full integrative model (C5) was not attempted — leaving the central conclusion only partially confirmed (q7/q8 yellow).
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.