Integration of Transcriptomics With Interpretable Artificial Intelligence for Identifying Molecular Signatures of Physiological Stress in Sleep Deprivation.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to ATTEMPT but not bit-exactly reproducible from the repo alone (it ships no GSE98582-data.Rdata/insomnia-genes.txt etc.). Faithful port of the authors' own DEG.R sections 1-4 run on «our HPC» against the paper's own GEO data (GSE98582 SD-subset + GSE37667, ComBat-merged, limma). PARTIAL 1:1: cohort within 2 samples (C1: 74 vs 72; GSE37667 exact 9/9, GSE98582 56 vs ~54 because the curated SD subset is not shipped); DEG count close under the rule the code ACTUALLY uses (C2: 27 vs 25) though the up/down split diverges (24/3 vs 16/9). KEY AUDITABLE FINDING (C2b): the paper Methods claim adjusted-p<0.05 but the shipped DEG.R selects by RAW p<0.05 (line 524); under the stated adjusted-p rule only 2 genes pass, so the reported 25-DEG count is consistent with the code but NOT with the paper's stated criteria -- a methods-misstatement / possible-fabrication flag. Reported hub gene RGS18 appears in our DEG list. NOT attempted (the ~20%): the ML 6-model AUROC table (C4), the SHAP 8-gene panel (C3), and validation single-gene ROC (C5) -- the perfect AUROC is, per the authors, a data-leakage artifact; CIBERSORT/figures are qualitative/visual and need inputs (LM22, Cibersort.R) not shipped. preprocessCore::normalize.quantiles substituted with limma::normalizeQuantiles (HPC pthread bug). No completeness claimed.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 49assessed: 2026-06-14 ⛓ ccc47562afd9
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan integrating blood transcriptomic analysis with an interpretable machine learning framework identify and validate candidate molecular signatures of sleep deprivation that are also relevant to chronic insomnia?
- ★ S100A3 is a robust candidate biomarker showing consistent discriminatory performance across the acute sleep deprivation training cohort, an independent sleep deprivation cohort, and a chronic insomnia cohort. finding
- ★ An interpretable transcriptomic–machine learning framework combining six ML algorithms with SHAP selected eight high-priority candidate genes from 25 DEGs. method
- ★ VEGFB exhibits notable diagnostic potential, particularly in insomnia, possibly reflecting chronic metabolic or inflammatory adaptation. finding
- ★ Sleep deprivation is associated with altered peripheral immune composition, including reduced resting NK cells and activated dendritic cells, with changes in regulatory and naïve immune cell populations. finding
- ★ Expression levels of S100A3 and VEGFB are significantly correlated with specific immune cell subsets, linking these signatures to stress-related immunomodulation. mechanism
- The integrated transcriptomic-ML framework provides a non-invasive strategy for discovering molecular indicators of sleep-related physiological stress. resource
- 25 DEGs (16 upregulated, 9 downregulated) were identified in the integrated training cohort. finding
- DEGs were enriched in immune- and cell death-related pathways (granzyme-mediated programmed cell death, apoptosis, NK cell-mediated cytotoxicity); GSEA showed upregulation of oxidative phosphorylation and Alzheimer's disease pathways in sleep deprivation. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk blood transcriptomic microarray (training cohort, DEG analysis) | human peripheral blood, acute sleep deprivation vs control (GSE98582 + GSE37667) | acute sleep deprivation | differentially expressed gene expression (|log2FC|≥0.3, adj p<0.05) | — |
| bulk transcriptomic microarray (independent SD validation) | human blood, sleep deprivation cohort (GSE56931) | sleep deprivation | candidate gene diagnostic performance (ROC/AUC) | — |
| bulk transcriptomic microarray (insomnia validation) | human blood, elderly chronic insomnia patients vs age-matched controls (GSE208668) | chronic insomnia | candidate gene diagnostic performance (ROC/AUC) | — |
| bulk transcriptomic microarray (negative control) | human, normal circadian rhythm dataset (GSE48113) | none (circadian states) | expression stability of biomarkers and immune profiles (ANOVA) | — |
| interpretable machine learning feature selection (SVM, XGBoost, RF, NB, Elastic Net, MLP with SHAP) | integrated training cohort gene expression | none (computational) | normalised SHAP importance scores; candidate gene selection | fastshap R package |
| immune cell deconvolution (CIBERSORT) | bulk blood gene expression, SD/insomnia vs control | none (computational) | relative abundance of 22 immune cell subpopulations | immunedeconv R package, LM22 signature matrix |
| functional enrichment analysis (DAVID, GSEA) | 25 DEGs / ranked gene list from training cohort | none (computational) | enriched GO-BP, KEGG, Reactome, WikiPathways terms | DAVID; clusterProfiler R package |
| single-gene ROC curve analysis | training and validation cohorts | none (computational) | AUC for discriminating experimental vs control | pROC R package |
- – 25 significant DEGs identified in the training cohort (16 up, 9 down in SD vs control) 25 DEGs
- – Eight high-priority candidate genes selected by interpretable ML framework (genes with total normalised SHAP ≥ 2) 8 genes
- – S100A3 showed consistent discriminatory performance across training, independent SD, and insomnia cohorts
- – VEGFB showed notable diagnostic potential, particularly in insomnia
- ▼ Sleep deprivation associated with reduced resting NK cells and activated dendritic cells
- – S100A3 and VEGFB expression significantly correlated with specific immune cell subsets
- – DEGs enriched in immune/cell death pathways; GSEA showed oxidative phosphorylation and Alzheimer's disease upregulated in SD, phenylalanine metabolism and graft-vs-host disease enriched in control
- count 25 DEGs (16 upregulated, 9 downregulated) (DEGs in integrated training cohort)
- count 8 candidate genes (high-priority genes selected via SHAP (normalised SHAP ≥ 2))
- count 72 blood samples (SD n=36, control n=36) (final training cohort)
- count 27 samples (independent sleep deprivation validation cohort (GSE56931))
- count 42 samples (chronic insomnia validation cohort (GSE208668))
- other AUC > 0.7 interpreted as good predictive performance (ROC analysis threshold for candidate genes)
- other 1.61-fold (greater risk of hypertension in individuals sleeping <5 h vs 7 h (cited background meta-analysis))
- other 43.2% (adults reporting insufficient sleep in a Netherlands epidemiological study (background))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This retrospective bioinformatics study combined four publicly available GEO blood microarray datasets into a training cohort (n=72), an independent sleep deprivation validation cohort (n=27), and a chronic insomnia cohort (n=42). Differentially expressed genes were identified with limma using Benjamini–Hochberg FDR correction, followed by feature selection across six machine learning algorithms with SHAP-based composite importance scoring. Diagnostic performance of candidate genes was evaluated by single-gene ROC/AUC analysis across cohorts, and peripheral immune composition was inferred via CIBERSORT with group differences tested by Wilcoxon signed-rank tests and gene–immune associations quantified by Spearman correlation.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma linear model with empirical Bayes moderation (eBayes) | Differential expression between sleep deprivation and control groups in the combined training cohort | 72 (36 sleep deprivation, 36 control) | not stated |
| One-way ANOVA | Expression stability of candidate biomarkers across physiological states in the negative-control circadian rhythm dataset (GSE48113) | — | not stated |
| Wilcoxon signed-rank test | Differences in CIBERSORT-estimated proportions of 22 immune cell subpopulations between sleep-deprived and control groups, and between insomnia and control groups | 72 (training cohort); 27 (SD validation cohort); 42 (insomnia cohort) | stated (nonparametric nature of CIBERSORT output cited as rationale for test selection) |
| Spearman correlation | Association between expression levels of candidate genes and abundance of immune cell subsets | — | not stated |
| Student's t-test or Wilcoxon signed-rank test (normality-dependent selection) | General comparisons of continuous variables between two groups (section 2.7 general framework) | — | stated (normality of data distribution cited as the criterion for choosing between the two tests) |
| ROC curve analysis (AUC) via pROC | Diagnostic efficacy of eight candidate genes in training cohort, independent SD validation cohort, and insomnia cohort; AUC > 0.7 used as performance threshold | 72 (training); 27 (SD validation); 42 (insomnia cohort) | na |
| Gene set enrichment analysis (GSEA) via clusterProfiler, ranked by log2FC against KEGG gene sets | Pathway-level transcriptomic alterations across all detected genes in the training cohort | 72 (training cohort) | not stated |
| CIBERSORT deconvolution with 1000 permutations (permutation-based sample-level significance) | Inference of relative abundance of 22 immune cell subpopulations from bulk gene expression using LM22 signature matrix | 72 (training); 27 (SD validation); 42 (insomnia cohort) | na |
-
Differences in 22 immune cell subset proportions were each assessed with a separate Wilcoxon test; the paper does not describe a multiplicity correction across these concurrent comparisons↳ Could also: Apply Benjamini-Hochberg FDR or Bonferroni correction across the family of 22 simultaneous tests — Correcting for the number of concurrent comparisons is a standard step when testing multiple related hypotheses simultaneously; it reduces the expected number of false discoveries while still allowing meaningful findings to be identified
-
Diagnostic performance was summarized using AUC from ROC curve analysis only↳ Could also: Supplement with calibration assessment (e.g., Hosmer-Lemeshow test or calibration plot) and decision curve analysis (DCA) — AUC captures overall discrimination but does not assess whether predicted probabilities are well-calibrated to observed outcome rates; DCA additionally quantifies net clinical benefit at different decision thresholds, together providing a more complete evaluation of a marker's potential clinical utility
-
Feature importance across six ML models was aggregated by summing min-max-normalized absolute SHAP values, with a fixed cutoff of ≥ 2 to define candidate genes↳ Could also: Use stability selection (e.g., randomized Lasso over bootstrap resamples) or report selection frequency across cross-validation folds — Stability selection provides an explicit inclusion probability for each feature under resampling variation, making the threshold for defining a 'robust' feature less sensitive to the specific normalization scheme or the arbitrary numeric cutoff
-
Batch effects between the two training datasets were corrected using ComBat (parametric empirical Bayes)↳ Could also: Use limma's removeBatchEffect, surrogate variable analysis (SVA) to capture latent confounders, or Harmony for integration — ComBat assumes known batch labels and a parametric error model; SVA can estimate and remove variation from latent, unobserved batch-like sources without requiring explicit batch assignments, and different methods may yield different degrees of biological signal preservation
-
Immune cell deconvolution was performed with a single method, CIBERSORT with the LM22 matrix↳ Could also: Additionally apply complementary methods such as xCell, TIMER2, or MCP-counter and compare results across methods — Different deconvolution algorithms use distinct reference signatures and estimation strategies; comparing outputs across multiple methods helps distinguish findings that are robust from those that are sensitive to method-specific assumptions
-
The DEG threshold combined adjusted p < 0.05 with a relatively permissive |log2FC| ≥ 0.3 cutoff↳ Could also: Report all genes passing FDR correction without a hard fold-change cutoff, or use a shrinkage-based fold-change estimator (e.g., apeglm) to stabilize effect-size estimates — Shrinkage estimators reduce noise in fold-change estimates for high-variability or low-expression features; omitting a hard FC cutoff avoids potentially excluding genes with consistent but modest effects, which may be relevant when the underlying biological signal is subtle
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
29 downstream papers · 5 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Mistimed sleep disrupts circadian regulation of the... 2014 · 259 cites
- Universal method for robust detection of circadian s... 2018 · 140 cites
- Blood transcriptome based biomarkers for human circa... 2017 · 125 cites
- Machine learning identifies a compact gene set for m... 2017 · 55 cites
- Platform-independent estimation of human physiologic... 2024 · 14 cites
- Diurnal and circadian rhythmicity of the human blood... 2022 · 11 cites
- Identification and validation of diagnostic genes <i... 2025 · 4 cites
- The Extract of <i>Piper nigrum</i> Impro... 2025 · 2 cites
- Tryptophan metabolism-related gene CYP1B1 serves as... 2025 · 1 cites
- An integrated machine learning framework for develop... 2026 · 0 cites
- Circadian Biology and Gynecological Diseases: Geneti... 2026 · 0 cites
- Gomisin E attenuates pentobarbital-induced sleep def... 2026 · 0 cites
- Universal method for robust detection of circadian s... 2018 · 140 cites
- Blood-gene expression reveals reduced circadian rhyt... 2014 · 70 cites
- Machine learning identifies a compact gene set for m... 2017 · 55 cites
- Platform-independent estimation of human physiologic... 2024 · 14 cites
- tauFisher predicts circadian time from a single samp... 2024 · 11 cites
- New Insights into the Role of Inflammatory Pathways... 2025 · 1 cites
- Exploring gene expression biomarker candidates for n... 2018 · 23 cites
- Relationship between circadian genes and memory impa... 2022 · 12 cites
- Molecular basis identification and hypnotic drug int... 2025 · 2 cites
- New Insights into the Role of Inflammatory Pathways... 2025 · 1 cites
- An integrated machine learning framework for develop... 2026 · 0 cites
- Bioinformatics analysis of the mechanisms and effica... 2026 · 0 cites
- Potential association between COVID-19 and neurologi... 2024 · 0 cites
- Integrated transcriptomic identification and validat... 2026 · 0 cites
- Peripheral and central immune features in insomnia:... 2026 · 0 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-42216239 (BioSleep-AI, Wang K. et al., J Cell Mol Med 2026)
Repo: github.com/wangk-oj/BioSleep-AI @ 2171232015ed31bbf8bb27aba7a108a84f6de123 (authors' own code, 4 R scripts). Data: GEO GSE98582 (+GSE37667 training), GSE208668 / GSE56931 (validation). All public.
Pipeline-derived results (IN SCOPE)
| id | result | pipeline | script |
|---|---|---|---|
| C1 | Training cohort 72 samples (36 SD / 36 ctrl) = GSE98582+GSE37667 merged via ComBat | GEOquery + sva ComBat | DEG.R §1-3 |
| C2 | 25 DEGs (16 up, 9 down) at threshold | limma eBayes | DEG.R §4 |
| C3 | 8 SHAP biomarker genes (VEGFB,CLDN7,S100A3,ZNF594,GZMB,RGS18,HPS5,COX7B) | caret 6-model CV + fastshap | machine learning.R |
| C4 | ML AUROC per model (ENet/MLP=1.00, RF/XGB=0.99, SVM/NB=0.98) | caret 5-fold CV | machine learning.R |
| C5 | S100A3 single-gene ROC AUC (0.83/0.75/0.92) | pROC | verify.R |
OUT OF SCOPE (not attempted / 20%)
- Figures as rendered images (volcano, heatmap, PCA, CIBERSORT heatmaps) — visual, not numeric.
- CIBERSORT immune-cell correlation directions (Fig 6) — qualitative, version-sensitive; requires LM22 + Cibersort.R not shipped in repo.
- GO/KEGG enrichment text (if any) — not a pinned number.
Known gaps / audit flags (discovered by code reading)
- DEG threshold mismatch (code vs paper): DEG.R line 524 selects by raw P.Value<0.05 (
ifelse(deg$P.Value>0.05,'stable',...)), but the paper Methods state adjusted p<0.05. We compute BOTH and report. - Repo does NOT ship: GSE98582-data.Rdata, GSE56931-data.Rdata, insomnia-genes.txt, GSE208668-gpl.Rdata (intermediates). Reconstructed via getGEO. The GSE98582 *.Rdata could alter the exact SD-sample expression matrix — possible source of small deviation.
- ML perfect AUROC (ENet/MLP=1.00) is a data-leakage artifact (DEG selection done on full data before CV) — the authors themselves note this.
Plan
Phase 1 («job»): C1 + C2 keystone on «our HPC» (DEG.R §1-4 faithful port). Phase 2 (if Phase 1 ok): C4 + C3 (ML on the 25-gene matrix; tiny compute). C5 validation ROC = optional 20%.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The DEG phase reproduces reasonably — cohort within 2 samples (74 vs 72), DEG count 27 vs 25, and reported hub gene RGS18 recovered — with deviations explainable by a self-defined GSE98582 subset (curated input not shipped) and a quantile-norm substitution. The decisive finding is on the authors' side: the paper's stated adjusted-p<0.05 criterion yields only 2 DEGs, so the reported 25 is derivable only from the raw-p<0.05 rule actually coded (DEG.R line 524) — a methods-text-vs-code misstatement, compounded by an author-acknowledged data-leakage AUROC=1.00. Severity is moderate (DEG magnitude holds, but up/down direction diverges 24/3 vs 16/9 and the stated criterion collapses the result), so this lands at yellow overall with a clear threshold-misstatement / possible-methods-integrity flag rather than confirmed fabrication.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.