Epistemic uncertainty challenges aging clock reliability in predicting rejuvenation effects.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough and reproduced 1:1. Pure in-silico reanalysis paper; authors' own MIT repo (reprogramming_ood @970c964) ships deterministic Jupyter notebooks whose committed cell outputs ARE the published figure values. Re-executed 5 notebooks headlessly on «our HPC» in the pinned env (sklearn1.2.1/pandas1.5.1/scipy1.7.2/numpy1.22.4/statsmodels0.13.2/PyMARE0.0.3) and compared all 61 printed scalar cells 1:1: 40 exact, 18 within-tol, 2 partial, 1 error. All four analytical pillars reproduce — covariate-shift KS %, Inverse-Train-Test R2/MAE (exact), GPR R2/MAE+kernel (kernels identical, metrics 7-10 sig figs = GP optimizer jitter w/o random_state), and the headline rejuvenation-effect significance (invivo PyMARE meta P=0.0184 matches to ~10 sig figs; Mann-Whitney exact; invitro/embryo within-tol). PARTIAL: invitro multi-model clock panel (KNN bit-identical, RandomForest/SVR nondeterministic, no random_state). ERROR: invivo cell_7 KS panel '68.18%' is NOT regenerable from shipped code — it calls calculate_shift(), a function undefined/unimported anywhere in the repo (committed value came from stale kernel state); flagged as a reproducibility DEFECT, not re-implemented to avoid fabrication. Minor env gap: invivo needs statannot, absent from requirements.txt (installed; recovered the Mann-Whitney cell exactly). NOT attempted: gp_1d_example (needs GSE87571 not in zip) and weight_height (synthetic illustration) toy notebooks, re_aging_cr/obese exploratory notebooks, and figure pixel layouts. No fabrication signal. NOTE: a prior session's reproduction folder was externally renamed .bogus, but its «job» did complete (sacct exit 0:0) — it was real; this run independently re-runs and extends coverage from 1 notebook to 5.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 93assessed: 2026-06-18 ⛓ e53f3a928c45
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetBecause the true biological age of reprogrammed cells is unverifiable, the paper asks whether epigenetic aging clock predictions of rejuvenation during cellular reprogramming (and embryogenesis) are reliable once epistemic (model and out-of-distribution) uncertainty is accounted for.
- ★ DNA methylation profiles observed across cellular reprogramming are poorly represented in the training data of existing aging clocks, introducing high out-of-distribution/epistemic uncertainty in their age estimates finding
- ★ Different published aging clocks are inconsistent in their predictions for the same reprogramming datasets, with some predicting large rejuvenation, some predicting age increase, and some predicting zero or negative rejuvenation finding
- ★ High clock uncertainty undermines the reliability of reported rejuvenation effects during in vitro reprogramming before pluripotency and throughout embryogenesis finding
- ★ In vivo reprogramming (OSKM treatment in mouse liver) shows a significant age increase rather than rejuvenation when reanalyzed with a dedicated Lasso clock finding
- ★ Future aging clock models should incorporate epistemic uncertainty estimation to avoid misinterpreting biological age predictions method
- ★ A PCA plus Kolmogorov-Smirnov test framework can detect and quantify covariate shift between aging and rejuvenation/embryogenesis DNAm datasets method
- De novo Lasso aging clocks trained on combined mouse liver aging datasets were built as a resource to test in vivo reprogramming rejuvenation claims resource
- A simple weight/height clock example (achondroplasia vs. control cohort) demonstrates that covariate shift can produce biologically meaningless, systematically biased age predictions finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| multiple linear regression clock (weight/height) | human males 1-25 years, control vs. achondroplasia cohorts | achondroplasia disorder (none applied experimentally) | predicted chronological age | — |
| DNAm PCA + Kolmogorov-Smirnov covariate shift test | human skin (Roos et al. 2017), train/test split | none | PC coordinates; % CpGs with shifted beta-value distributions | — |
| DNAm PCA + KS covariate shift test | human skin, two independent datasets (Roos et al. 2017; Vandiver et al. 2015) | none (cross-study comparison) | PC coordinates; % CpGs rejected by KS test | — |
| DNAm PCA + KS covariate shift test | mouse liver aging, two datasets (Meer et al. 2018; Thompson et al. 2018) | none (cross-study comparison) | % CpGs rejected by KS test | — |
| DNAm PCA + KS covariate shift test | human skin aging (Roos et al. 2017) vs. in vitro fibroblast reprogramming (Ohnuki et al. 2014; Gill et al. 2022) | OSKM reprogramming (in vitro) | PC trajectory; % CpGs rejected by KS test | — |
| DNAm PCA + KS covariate shift test | mouse liver, in vivo reprogramming (Chondronasiou et al. 2022) vs. merged aging liver (Meer et al. 2018; Thompson et al. 2018) | transient OSKM reprogramming (in vivo) | % CpGs rejected by KS test | — |
| epigenetic aging clock prediction (8 published clocks, mostly ElasticNet) | in vitro human fibroblast reprogramming (Ohnuki et al. 2014; Gill et al. 2022) | OSKM reprogramming (in vitro) | predicted epigenetic age across reprogramming timeline (days 0-15/17) | — |
| de novo Lasso regression aging clock | mouse liver, combined aging datasets (Meer et al. 2018; Thompson et al. 2018) trained; tested on in vivo reprogramming (Chondronasiou et al. 2022), young/old control and old OSKM-reprogrammed mice (n=5 each) | transient OSKM reprogramming (in vivo) | predicted epigenetic age; group comparison via two-sided Mann-Whitney-Wilcoxon test | — |
- ▼ Weight/height MLR clock trained on healthy cohort systematically underestimated ages in the achondroplasia cohort due to covariate shift, despite achondroplasia being associated with shorter lifespan
- – Weight/height covariate distributions differ significantly between control and achondroplasia cohorts KS test p<0.0003
- – No covariate shift detected between train/test split of the same aging skin dataset 0% of CpGs rejected
- – Moderate covariate shift between two independent aging skin datasets 81% of CpGs rejected
- – Strong covariate shift between aging skin and in vitro fibroblast reprogramming datasets 86% (Ohnuki) and 69% (Gill) of CpGs rejected
- – Moderate covariate shift between in vivo liver reprogramming and merged aging liver datasets 32% of CpGs rejected
- – Eight published aging clocks gave highly inconsistent rejuvenation estimates over the same reprogramming period, from strong age reversal to age increase range from -40 years (Horvath clock) to +13 years (Hannum clock) by day 15/17
- ▲ De novo Lasso clock failed to detect significant rejuvenation in OSKM-reprogrammed old mice versus old control mice in vivo p=0.11, two-sided MWW test
- other MAE = 2.3 years, R2 = 0.84 (training performance of weight/height MLR age clock on healthy cohort)
- pvalue p < 0.0003 (KS test for weight/height covariate shift, control vs. achondroplasia cohorts)
- other 81% of CpGs rejected (KS test covariate shift, two independent aging skin datasets)
- other 1% of CpGs rejected (KS test covariate shift, two aging mouse liver datasets)
- other 86% of CpGs rejected (KS test covariate shift, aging skin vs. Ohnuki in vitro reprogramming dataset)
- other 69% of CpGs rejected (KS test covariate shift, aging skin vs. Gill in vitro reprogramming dataset)
- other 32% of CpGs rejected (KS test covariate shift, in vivo liver reprogramming vs. merged aging liver datasets)
- pvalue p = 0.11 (two-sided MWW test, epigenetic age of OSKM-reprogrammed old mice vs. old control mice (n=5 per group, 3 groups))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper presents an analytical framework evaluating epistemic uncertainty in epigenetic aging clock predictions applied to rejuvenation studies. It combines PCA and Kolmogorov-Smirnov (KS) tests to quantify covariate shift between aging and reprogramming/embryogenesis DNAm datasets, compares predictions across eight published clocks and multiple ML model families to characterize model uncertainty, and builds de novo Lasso regression clocks to test an in vivo reprogramming dataset. Results are reported as proportions of CpG sites with shifted distributions, MAE/R²/Pearson r for clock performance, and exact p-values from individual group comparisons.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Kolmogorov-Smirnov two-sample test | Comparing DNAm beta-value distributions between training and testing datasets across individual CpG sites (aging vs. reprogramming in vitro and in vivo, aging vs. embryogenesis, same-dataset train/test splits); significance threshold 0.01 | — | not stated |
| Two-sided Mann-Whitney-Wilcoxon (MWW) test | Comparison of predicted epigenetic ages between old reprogrammed mice and old control mice in the in vivo liver reprogramming dataset (Chondronasiou et al., 2022) | n=5 per group | not stated |
| Pearson's correlation coefficient | Evaluating de novo Lasso aging clock performance on the held-out testing subset (predicted vs. chronological age) | — | not stated |
| Multiple linear regression (MLR) / ElasticNet / Lasso regression | Construction of de novo aging clocks on aging mouse liver data and application of eight published clocks (Hannum, Horvath, Levine, etc.) to in vitro reprogramming datasets | — | not stated |
| Principal component analysis (PCA) | Visualization and qualitative assessment of covariate shift between DNAm datasets (aging vs. reprogramming, aging vs. embryogenesis, within-dataset train/test splits) | — | na |
-
Model uncertainty was illustrated post-hoc by comparing predictions across eight published clocks trained on different datasets↳ Could also: Bayesian methods or ensemble approaches (e.g., bootstrap aggregation, deep ensembles, Gaussian process regression — GPR is listed in the paper's own abbreviations) could also provide per-sample predictive intervals during inference — These approaches formally quantify uncertainty for each individual prediction rather than demonstrating it through inter-clock disagreement, directly producing the uncertainty estimates the paper recommends for future clocks
-
Covariate shift was assessed by applying the KS test independently to each CpG site and summarizing results as the percentage of sites rejected↳ Could also: Maximum Mean Discrepancy (MMD) or a multivariate permutation-based distance could also quantify distributional shift as a single joint statistic across all features — Multivariate measures capture shifts that manifest only in joint CpG distributions rather than marginally, and produce a single scalar that can be compared across dataset pairs without the multiplicity considerations of per-site testing
-
Group comparisons for the mouse in vivo experiment used the MWW test with n=5 per group↳ Could also: A permutation test could also be used for group comparisons at this sample size — Permutation tests require no distributional assumptions and can be more appropriate when n is very small; they complement the non-parametric MWW approach and are exact rather than asymptotic
-
De novo clocks were constructed using Lasso regression, a penalized linear model that does not yield prediction uncertainty↳ Could also: Gaussian process regression (GPR) could also be used to build aging clocks — GPR natively outputs a posterior predictive distribution including per-sample uncertainty, which is the core capability the paper identifies as missing from penalized regression–based clocks
-
The proportion of CpG sites rejected by the KS test at a fixed threshold was used as the summary metric for covariate shift magnitude↳ Could also: Effect-size measures such as KL divergence, Wasserstein distance, or Cohen's d per CpG could also summarize the degree of distributional shift — Effect sizes convey the magnitude of shift rather than binary rejection, enabling a more graded characterization of how severely the test distribution departs from training across different comparison scenarios
-
Clock performance was summarized with aggregate error metrics (MAE, MedAE, R²)↳ Could also: Prediction intervals or bootstrapped confidence intervals around predicted ages could also be reported alongside aggregate metrics — Sample-level prediction intervals communicate uncertainty at the resolution most relevant to clinical decision-making, complementing aggregate summaries and directly illustrating the reliability problem the paper describes
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-39072888
Paper: Kriukov et al. 2024, Epistemic uncertainty challenges aging clock
reliability in predicting rejuvenation effects. Aging Cell. DOI 10.1111/acel.14283.
Code: https://github.com/ComputationalAgingLab/reprogramming_ood (MIT, commit
970c964b0728069e576cc864892872e0f41f16c6, pushed 2024-06-23). Authors' own code.
Data: preprocessed methylation matrices in a Google-Drive datasets.zip
(gdown id 11xwxb_m62FymwUeO1vC0KafZ7mog0_wq, via prepare_data.sh); the named
brief accession GSE90124 (Roos et al. 2017 human aging skin) is one of them.
Pipeline-derived results (in scope)
The repo ships 7 Jupyter notebooks; each cell carries its committed output,
which is the value that generated the corresponding paper figure. These printed
scalars are deterministic (fixed random_state, LassoCV CV, fixed GP kernel) and
therefore reproducible 1:1.
We target aging_skin_analysis.ipynb because it is driven by the
brief-named dataset GSE90124 (+ GSE51954 = Vandiver 2015), is fully deterministic,
and renders paper Fig. 2c-h, 4c-d, S4a. Concrete targets (committed outputs):
| # | Result | Reported (committed nb output) | Cell |
|---|---|---|---|
| C1 | GSE90124 (Roos) probe count | 450531 | 1 |
| C2 | GSE51954 (Vandiver) probe count | 485512 | 1 |
| C3 | Common CpGs (Skin clock 391 ∩ both datasets) | 389 | 3/6/9/13 |
| C4 | KS test, same-dataset split: % sites not rejected | 100.0% | 4 |
| C5 | KS test, Roos vs Vandiver: % sites not rejected | 19.28% | 7 |
| C6 | ITTP Roos→Vandiver (8 metrics) | step1 R2tr .943/MAEtr 1.766/R2te .9/MAEte 5.945; step2 R2tr .902/MAEtr 5.894/R2te .766/MAEte 3.537 | 10 |
| C7 | ITTP Vandiver→Roos (8 metrics) | step1 R2tr .988/MAEtr 2.014/R2te .496/MAEte 5.2; step2 R2tr .497/MAEtr 5.195/R2te .988/MAEte 2.074 | 11 |
| C8 | GPR on GSE90124 split | R2tr 0.9725, R2te 0.8662, MAEtr 1.2046, MAEte 2.7248; kernel 98²·RBF(ls 11.9)+White(5.51) | 13 |
Out of scope (not attempted)
- The 6 other notebooks (in-vitro/in-vivo reprogramming, embryo, liver, GP-1d, weight-height) — depend on additional GEO datasets in the same zip; the brief pins GSE90124, so we scope to the skin notebook (80/20).
- Figure aesthetics / panel layouts — we compare the printed scalar pipeline outputs, not pixel-level figures.
- Wet-lab / external claims — none; this is a pure in-silico reanalysis paper.
Method
Clone repo + fetch datasets.zip on «infra» inside a «our HPC» SLURM job; build the
pinned conda env (python 3.9 + requirements.txt); execute the notebook headlessly
(MPLBACKEND=Agg jupyter nbconvert --execute); extract printed outputs; compare
1:1 to the committed outputs above.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Clean 1:1 reproduction of the in-scope skin-aging pipeline: the authors shipped both code (MIT repo @970c964) and preprocessed data (datasets.zip), and re-running aging_skin_analysis.ipynb regenerated C1-C7 bit-identically and C8 to ~7 significant figures (kernel 98^2*RBF(ls=11.9)+White(5.51) identical). The only deviation is sub-1e-7 GPR optimizer jitter from an unseeded n_restarts_optimizer=3, which rounds to the paper's reported precision — a technical/expected stochastic effect, not an authors' or methodology defect. The central covariate-shift conclusion (100.0% vs 19.28% sites not rejected) holds fully; no fabrication signs.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.