Correcting scale distortion in RNA sequencing data.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH TO REPRODUCE; ran the authors' own Python code (commit a596299) VERBATIM on the authors' own shipped TCGA-LAML RSEM test dataset (173x20,531, matching the paper's 'TCGA AML' Datasets row, csv sha256 94ba19e...) on «our HPC» SLURM («job», COMPLETED, node n094). P16-valid reproduction (own code + own shipped data). TWO TIERS: (1) DETERMINISTIC results reproduce EXACTLY and are VERSION-INDEPENDENT -- this re-run used a newer dependency stack (numpy 2.5.0/scipy 1.18.0/pandas 3.0.3/matplotlib 3.11.0 via a fresh «infra» pip venv) yet returned byte-identical values to the earlier run («job»): dataset dims (173 patients, 20,531 genes; 13,592 after filter), the mean-centering sanity check (TPM-shifted std=2.459e-14), and the between-patient std reduction (TPM 0.16130 -> LL 0.00607, NL 0.01823), demonstrating the scale-distortion diagnostic and variance reduction the paper claims. (2) The STOCHASTIC spiking-simulation detection metrics (ROC/ccdf/detection-rate) are NOT byte-reproducible because the shipped code uses 5 unseeded random patient divisions; they reproduce in MAGNITUDE (~few %) and the paper's own single-patient 'negligible difference' statement MATCHES, but the population-level superiority of LL/NL over TPM is small and falls within run-to-run noise in a single 5-division run. NOT ATTEMPTED: GSE47774 (open, SEQC/MAQC-III, 3,396 samples) and the TCGA-bladder/prostate, SU2C, GTEx cohorts -- named in the paper but not shipped and not wired into the public code. No fabrication detected: every reproduced value is derivable from the shipped code+data, and the modest detection effects are presented by the paper itself as approximate ranges with wide error bars. Overall grade: PARTIAL (deterministic core exact + version-independent; stochastic detection claims partial/within-noise).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 76assessed: 2026-06-16 ⛓ 01bdf3c17f39
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe authors hypothesize that RNA-seq expression measurements contain per-patient, expression-level-dependent multiplicative biases (uncorrected by conventional normalization like TPM/FPKM) that corrupt gene-gene correlation estimates and t-tests between subpopulations, and that these biases can be identified and corrected via nonlinear transforms.
- ★ Local averaging reveals expression-level-dependent biases that differ from sample to sample across all RNA-seq datasets studied, and are not corrected by conventional normalization (TPM/FPKM) finding
- ★ Simulations show these per-sample biases corrupt gene-gene correlation estimations and t-tests between subpopulations finding
- ★ Two nonlinear transforms based on statistical considerations (using higher-order Taylor expansions to quantify bias and construct an inverse correction function) are introduced to correct the observed biases method
- ★ The proposed transforms remove observed per-sample biases, reduce sample-to-sample variance, and improve gene-gene correlation distribution characteristics finding
- ★ Using a novel simulation methodology creating controlled subpopulation differences, the transforms reduce variability and increase sensitivity/specificity of two population tests by roughly 3-5% in most instances finding
- ★ The log2(x+c) transform converts multiplicative measurement bias into an additive shift in log-transformed expression values, provided c is small relative to expression level mechanism
- c=0.25 is chosen as the additive constant in the log transform because integer-rounded true expression values between 0 and 0.5 (assumed uniformly distributed) round to 0 with a mean of 0.25 method
- ★ Per-patient bias shifts are systematically related to a gene's mean expression level, as previously found by the authors finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq (RSEM, raw counts and TPM) | TCGA bladder cancer firehose cohort (408 patients, 20216 genes) | none | expression-level dependent bias via local averaging; gene-gene correlation; t-statistics | RSEM |
| bulk RNA-seq (raw counts and TPM) | TCGA prostate cancer firehose cohort (498 patients, all male, 20198 genes) | none | expression-level dependent bias; correlation/t-test corruption | — |
| bulk RNA-seq (FPKM and TPM) | SU2C metastatic prostate cancer cohort (208 patients, all male, 19158 genes) | none | expression-level dependent bias; correlation/t-test corruption | — |
| bulk RNA-seq (raw counts) | GSE47774 cohort (1710 patients, 39376 genes) | none | expression-level dependent bias | — |
| bulk RNA-seq (TPM) | GTEx prostate V8 (245 patients, all male, 56200 genes) | none | expression-level dependent bias | — |
| bulk RNA-seq (raw counts) | TCGA PanCan Acute Myeloid Leukemia cohort (173 patients, 20531 genes) | none | expression-level dependent bias | — |
| computational simulation of RNA-seq subpopulation differences | simulated gene expression data derived from above datasets | controlled artificial subpopulation differences introduced | sensitivity and specificity of population statistical tests before/after bias correction | — |
| d'Agostino normality test on log-transformed expression data | all datasets (example figure shown for TCGA bladder firehose TPM) | none | per-gene p-values for normality across patients | — |
- ▲ Sensitivity and specificity of two population tests improved after bias correction 3-5%
- – Local averaging detected expression-level dependent biases differing sample-to-sample in all datasets studied
- – Simulations demonstrated that uncorrected biases corrupt gene-gene correlation estimates and t-tests between subpopulations
- ▼ Transforms reduced sample-to-sample variance and improved gene-gene correlation distribution characteristics
- ▼ For a gene with mean expression level X=8, the distortion introduced by the additive constant c in the log transform is less than 1/32 (about 3%) <1/32 (~3%)
- – Prior work found large shifts in the overall distribution of t-statistics when comparing random populations within a cohort, indicating per-patient biases affecting all/most genes
- other 3-5% (improvement in sensitivity and specificity of population tests after bias correction, most instances)
- other c=0.25 (chosen additive constant in log2(x+c) transform for expression data)
- other <1/32 (~3%) (distortion from constant c for a gene with mean expression level X=8)
- count 408 patients, 20216 genes (TCGA bladder firehose dataset size)
- count 498 patients, 20198 genes (TCGA prostate firehose dataset size)
- count 208 patients, 19158 genes (SU2C prostate dataset size)
- count 1710 patients, 39376 genes (GSE47774 dataset size)
- count 245 patients, 56200 genes (GTEx prostate V8 dataset size)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper develops and validates two nonlinear bias-correction transforms for RNA-seq expression data by analyzing expression-level-dependent per-sample biases across multiple large public cohorts (TCGA, SU2C, GTEx). Bias detection relies on local/block averaging of sorted log2-transformed expression levels; correction efficacy is evaluated via gene-gene correlation distributions, t-statistics between random subpopulations, and ROC-based sensitivity/specificity metrics under a novel simulation framework that inserts controlled subpopulation differences. Improvements after correction are reported as approximate percentage changes in sensitivity and specificity (3–5%), with normality of the preprocessed data assessed using per-gene d'Agostino tests.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| d'Agostino normality test | Per-gene log2-transformed expression levels across patients, to verify normality assumption | Varies by dataset: 173–1710 patients per gene; results shown for TCGA bladder firehose TPM (408 patients) | not stated |
| Two-sample t-test (t-statistic distributions) | Comparison of random subpopulations within each cohort, before and after bias correction; used both in real data and in simulations | Varies by dataset; exact subgroup sizes not stated in provided text | not stated |
| Local/block averaging of sorted data | Bias detection: genes sorted by mean expression level, averaged in blocks to characterize expression-level-dependent bias per sample | All genes with mean expression > 8 in each cohort; gene counts 19158–56200 depending on dataset | na |
| Pearson or Spearman gene-gene correlation (type not specified in provided text) | Gene-gene correlation distributions examined before and after correction to assess bias impact | null | not stated |
| ROC curve analysis (sensitivity/specificity) | Simulation-based evaluation of ability to detect controlled subpopulation differences before and after correction | null | na |
-
Bias was detected and characterized using block/local averaging of genes sorted by mean expression level↳ Could also: LOESS (locally estimated scatterplot smoothing) regression of per-sample residuals against mean expression could also be used to estimate expression-level-dependent trends — LOESS is a well-established nonparametric smoother for this type of mean-variance or mean-bias relationship, is widely used in RNA-seq QC (e.g., MA-plot smoothing), and would provide a continuous bias estimate with confidence bands rather than step-wise block averages
-
Population comparisons were evaluated using t-statistics / t-tests between subgroups↳ Could also: Wilcoxon rank-sum (Mann-Whitney U) tests could also be applied for the same two-group comparisons — Rank-based tests do not assume normality of expression levels and are often used alongside t-tests in RNA-seq analysis to assess robustness of findings to distributional assumptions, especially when the normality of log-transformed values is not fully confirmed across all genes
-
A novel custom simulation methodology was developed to introduce controlled subpopulation differences for benchmarking↳ Could also: Established RNA-seq simulation frameworks such as Splatter (R/Bioconductor) or polyester could also be used to generate benchmark datasets with known differential expression — Using a community-standard simulator would make the benchmarking more directly comparable to prior methods in the field and would allow reviewers to reproduce the simulation independently; it also separates the novelty of the correction method from the novelty of the simulation design
-
Sensitivity and specificity of subpopulation detection were summarized via ROC curves↳ Could also: Precision-recall (PR) curves could also be reported alongside ROC curves — When true positive rates are evaluated against a background of thousands of genes (highly imbalanced classes), PR curves are considered more informative than ROC curves because ROC area can remain high even when precision is low; reporting both would give a fuller picture of detector performance
-
Gene-gene correlation distributions were compared before and after correction as a qualitative/distributional outcome↳ Could also: A formal test comparing correlation matrices (e.g., permutation-based comparison of distribution moments, or a two-sample KS test on correlation coefficients) could also quantify the shift in correlation distributions — A formal distributional test would provide a p-value or effect size for the improvement in correlation structure, making the claim that the transform 'improves' correlations more statistically precise and reproducible across datasets
-
The same analyses were applied to all six datasets and figures were shown for only one (TCGA bladder firehose TPM) because 'results were similar in all cases'↳ Could also: Quantitative summary statistics (e.g., mean bias magnitude, sensitivity gain) for all six datasets could also be reported in a table or supplementary figure — Showing all dataset results numerically would allow readers to assess the consistency of the method across tissue types and preprocessing schemes without requiring them to take similarity on faith, and would strengthen reproducibility
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is an honest, P16-valid reproduction running the authors' own code on their own shipped TCGA-LAML data — the deterministic core reproduces essentially exactly (dims 173x20,531; between-patient std TPM 0.1613 → LL 0.0061 / NL 0.0182; TPM-shifted 2.46e-14), and no fabrication is indicated since every value is derivable from the shipped code+data. The only soft spot is the stochastic population detection claim (C5b ~3-5% / ROC 3-4% / ccdf ~10%), which reproduces in magnitude (~1-5%) but shows no clean LL/NL superiority over TPM in a single 5-division run. The cause is squarely technical/expected — the shipped code sets no RNG seed — not an authors' defect or data-derivability issue. Overall: solid reproduction with an explainable, within-noise deviation on a secondary claim → yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.