Comprehensive comparison of gene expression diversity among a variety of human stem cells.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough for a clean 1:1 reproduction. The repo (yyamatani/2022_scripts_and_data @ c5ee0609) ships the analysis-ready input count tables, reference files, the exact R/shell scripts, AND the expected output tables. Re-ran the authors' own run_batch_effect_correction.sh on «our HPC» (R 3.6.3 + RUVSeq 1.18.0 via conda). RESULT: re-running RUVSeq RUVg(k=1) regenerated the author-shipped batch-corrected count table BYTE-FOR-BYTE (identical SHA256, 36001 genes, max|diff|=0), and all four reported between/within variance ratios reproduced to the printed precision (stem types 13.3->17.3 reproduced 13.296->17.295; research groups 41.4->27.6 reproduced 41.425->27.571). CAVEAT: the variance-ratio statistic is computed by the authors' scripts via colMeans() + cbind() vector-recycling (R emits a recycling warning) -- an idiosyncratic definition; an independent per-gene recomputation gives very different numbers, so C1-C4 confirm 'the shipped code reproduces the printed values', not the statistic's textbook definition. NOT ATTEMPTED (out of scope, unshipped/stochastic): upstream read processing (FastQC/Trimmomatic/HISAT2/featureCounts), UMAP/KNN 8-cluster assignment, t-SNE, hierarchical clustering, WGCNA 7 modules, DESeq2 DEG counts, GO enrichment. FLAGGED: paper's 'batch effect correction for 9679 genes' (Methods) is an upstream count not derivable from any shipped table (raw=41954 isoform rows / 15077 unique symbols; after-RUVSeq=36001) -- flagged for human review as a possible inconsistency, NOT asserted as fabrication. C6 (21079 genes after CV<0.8 filter) is partial: Step 2 reproduced identically but the O(n^2) Step-4 shell loop was cancelled before completion; the count is the shipped value and is deterministic given the identical RUVSeq input. All grades are provisional and require human audit (see AUDIT.md).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 71assessed: 2026-06-15 ⛓ f111ee648573
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusTo rank the relative importance of multiple biological (tissue origin) and technical (culture condition) factors on the gene expression profiles among undifferentiated human stem cells (ESCs, iPSCs, MSCs, HSCs) by analyzing publicly available RNA-seq datasets.
- ★ iPSC gene expression is more strongly influenced by tissue origin than other stem cell types, whereas ESCs and somatic stem cells (MSCs, HSCs) are more strongly impacted by culture condition. finding
- ★ A characteristic gene module enriched in histones shows higher expression across different stem cell types annotated by specific culture conditions, also observed in mouse stem cell RNA-seq data. finding
- ★ Batch effect correction with RUVSeq using housekeeping genes as negative controls reduces inter-laboratory variance and makes biological differences among stem cell types more apparent. method
- ★ Stem cells cluster into eight clusters (C1–C8) by UMAP/KNN, separating pluripotent (C1–C2) from somatic stem cell clusters (C3–C8). finding
- ★ iPSCs (SFiPSCs in C1; AMiPSCs/PBiPSCs in C2) cluster corresponding to their tissue origins, while ESCs from a single tissue split into two clusters. finding
- WGCNA identifies gene modules highly correlated with stem cell clusters used to prioritize characteristic genes (GS>0.5 and MM>0.8). method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq (public data analysis) | undifferentiated human stem cells (ESCs, iPSCs, MSCs, HSCs); 103 samples from 24 research groups | none (undifferentiated, healthy donor; annotated by tissue origin, passage, culture medium) | gene expression (TPM-normalized read counts) | NCBI GEO RNA-seq datasets; preprocessing with FastQC, Trimmomatic, HISAT2 (hg38), featureCounts |
| microarray expression profiling | undifferentiated human iPSCs annotated with passages (10–67) | passage number variation | gene expression by passage | GSE42445 |
| bulk RNA-seq (public data analysis) | undifferentiated mouse stem cells | none | histone gene expression | NCBI GEO (Supplementary Table S9) |
| batch effect correction / variance analysis | human stem cell RNA-seq (9679 genes) | none | variance ratio between/within stem cell types and research groups | RUVSeq (RUVg) |
| dimensionality reduction and clustering | batch-corrected human stem cell expression (9679 genes) | none | UMAP/t-SNE embedding and KNN clusters | Seurat package |
| differential gene expression analysis | stem cells with different factor annotations | factor annotation (tissue/culture) | DEGs (FDR<0.05, |FC|≥2) and GO enrichment | DESeq2; Gene Ontology |
| weighted gene co-expression network analysis | batch-corrected human stem cell expression / cluster annotations | none | gene modules correlated with clusters (module eigengene PCC) | WGCNA (soft power 14, scale-free R2=0.88) |
- – After batch correction, variance ratio among stem cell types increased while variance among research groups decreased, indicating reduced batch effects. types: 13.3→17.3; research groups: 41.4→27.6
- ▲ Pluripotent stem cell positive markers NANOG, SOX2, LIN28A highly expressed in ESCs/iPSCs while KLF4 lower, consistent with prior reports.
- – All eight clusters separated pluripotent (C1, C2) from somatic stem cell clusters (C3–C8); MSCs split into C3–C6, HSCs into C7–C8. 8 clusters
- – SFiPSCs grouped in C1 (n=5); AMiPSCs (n=3) and PBiPSCs (n=2) in C2, corresponding to tissue origin.
- – In MSCs, correlation between originating tissue and marker gene expression was rarely observed (e.g., POSTN, PPARG, CD68, CD44 high in unrelated MSC clusters).
- ▲ In iPSCs, skin fibroblast markers (CDK1, NOTCH3) and peripheral blood marker (STMN3) highly expressed in cells from corresponding tissues.
- ▲ Histone-enriched gene module exhibits higher expression across stem cell types annotated by specific culture conditions.
- other variance ratio among stem cell types 13.3 (before) vs 17.3 (after) (batch correction effect, between/within type variance)
- other variance ratio among research groups 41.4 (before) vs 27.6 (after) (batch correction reduced inter-group variance)
- count 103 RNA-seq samples from 24 research groups (input human stem cell dataset)
- count 15918 genes remained after filtering; 9679 genes batch-corrected (preprocessing)
- other soft thresholding power 14, scale-free R2=0.88 (WGCNA network construction)
- count C1 n=7, C2 n=12, C3 n=18, C4 n=21, C5 n=7, C6 n=15, C7 n=19, C8 n=4 (cluster sizes)
- other GS>0.5 and MM>0.8 (WGCNA gene selection thresholds)
- other FDR<0.05 and |FC|≥2 (DEG cutoff in DESeq2)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This study re-analyzed 103 publicly available bulk RNA-seq samples from 24 research groups spanning four undifferentiated human stem cell types (ESCs, iPSCs, MSCs, HSCs). After preprocessing and batch-effect correction with RUVSeq, dimensionality reduction (UMAP with KNN-graph clustering, confirmed by t-SNE and hierarchical clustering) was used to identify eight stem cell clusters. Factor–cluster associations were tested with chi-square tests (via the swamp package), differential gene expression was assessed with DESeq2 (FDR < 0.05, |FC| ≥ 2), co-expression modules were identified with WGCNA and correlated with clusters via Pearson's correlation coefficient, and within-module expression differences across clusters were compared with one-way ANOVA followed by Tukey's multiple comparison test.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Linear model significance tested by chi-square test (swamp package) | Interrelations among cluster classification, tissue origins, and technical factors within each stem cell type | 103 samples total; per-stem-cell-type subsets not individually enumerated for these tests | not stated |
| DESeq2 negative binomial Wald test with FDR correction | Differentially expressed genes between stem cells with different factor annotations (tissue origin, culture condition, cluster membership) | varies by comparison; overall dataset n = 103 | not stated |
| Pearson's correlation coefficient (PCC) | Correlation between WGCNA module eigengene and stem cell cluster labels; also gene significance (GS) and module membership (MM) thresholding | 103 samples | not stated |
| One-way ANOVA with Tukey's multiple comparison test (post-hoc) | Comparison of expression levels of module genes (that overlapped with DEGs) across stem cell clusters | per-cluster n values: C1=7, C2=12, C3=18, C4=21, C5=7, C6=15, C7=19, C8=4 | not stated |
| Between-group to within-group variance ratio (informal) | Evaluation of batch-effect correction: variance among stem cell types vs. among research groups, before and after RUVSeq correction | 103 samples across 24 research groups | na |
| JackStraw permutation procedure (Seurat) | Selection of the number of statistically significant principal components used as input dimension parameter for UMAP | 103 samples | not stated |
-
Batch-effect correction was performed with RUVSeq (RUVg), using housekeeping genes as negative controls, and its adequacy was assessed informally via between/within variance ratios and PCA visualisation↳ Could also: ComBat-seq (negative-binomial model) or limma::removeBatchEffect (linear-model residuals) could also have been applied; PERMANOVA (adonis2 in R/vegan) could provide a formal multivariate significance test of the proportion of variance attributable to batch vs. biology before and after correction — ComBat-seq was designed explicitly for count-level RNA-seq batch correction and may preserve biological signal differently; PERMANOVA would add a hypothesis-test p-value to the variance-ratio comparison, making the degree of batch reduction formally testable rather than descriptive
-
Factor–cluster associations were assessed by fitting a linear model per annotation variable and testing its significance with a chi-square test (swamp), without a stated correction for the family of tests across multiple annotation variables within each stem cell type↳ Could also: A permutation-based approach (e.g., PERMANOVA per factor) or Bonferroni/BH correction across the family of chi-square tests could also have been applied — When many annotation variables are tested simultaneously, some form of multiplicity adjustment controls the expected number of spurious associations; permutation tests additionally avoid parametric assumptions about the chi-square null distribution given the small per-group sample sizes
-
GO enrichment significance was assessed with a nominal P-value threshold (< 0.01) and a fold-enrichment cutoff (≥ 2), with no stated adjustment for the large number of GO terms tested↳ Could also: Benjamini-Hochberg FDR correction across tested GO terms (as implemented in clusterProfiler or topGO) could also have been applied — Given that hundreds to thousands of GO terms may be tested simultaneously, FDR adjustment is a widely-used approach to limit the expected proportion of false discoveries among reported enriched terms
-
Module eigengene–cluster associations were quantified with Pearson's correlation coefficient, and hub genes were selected by thresholding GS > 0.5 and MM > 0.8↳ Could also: Spearman's rank correlation could also have been used for module eigengene–cluster associations, and a point-biserial correlation (or eta-squared from ANOVA) would be a natural choice when one variable is a categorical cluster label rather than a continuous measure — Pearson's r assumes linearity and approximate normality; Spearman's rho is more robust to non-normal distributions of eigengene values; a categorical effect-size measure would more directly quantify how much cluster membership explains variation in eigengene expression
-
UMAP was used as the primary dimensionality reduction and visualisation method, with t-SNE and hierarchical clustering used secondarily as stability checks↳ Could also: PHATE (Potential of Heat-diffusion for Affinity-based Trajectory Embedding) or diffusion maps could also have been applied, particularly to explore continuous transitions between stem cell states rather than discrete cluster boundaries — UMAP and t-SNE emphasise local neighbourhood structure; trajectory-based methods like PHATE can additionally reveal gradients or continuous differentiation trajectories that discrete clustering may not capture, which could complement the cluster-centric analysis of factor impacts
-
Within-module expression differences across clusters were compared with ANOVA followed by Tukey's post-hoc test, with no description of whether ANOVA assumptions (normality, homoscedasticity) were checked given the small and unequal cluster sizes (n ranging from 4 to 21)↳ Could also: A Kruskal-Wallis test followed by Dunn's post-hoc test with BH correction could also have been applied as a non-parametric alternative that does not assume normality or equal variances — With cluster sizes as small as n=4, normality and variance homogeneity assumptions underlying parametric ANOVA are difficult to verify; non-parametric alternatives make fewer distributional assumptions and may be more appropriate for small, unequal groups
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Reproduction scope — pmid-36458020
Paper: Yamatani Y, Nakai K. Comprehensive comparison of gene expression diversity among a variety of human stem cells. NAR Genom Bioinform 2022. DOI 10.1093/nargab/lqac087 · PMCID PMC9706419.
Code: https://github.com/yyamatani/2022_scripts_and_data (commit
c5ee06090d8473485e1331a79775a0c75e728cdb, default branch main, no license).
Also archived on Zenodo (doi.org/10.5281/zenodo.7278583).
Data: the analysis-ready count tables are shipped inside the repo
(input/, reference/); upstream raw RNA-seq is 103 public GEO samples.
What the repo actually ships (this is the reproduction surface)
input/raw_read_count_table_before_RUVSeq.txt— raw count table, 103 samples.input/tpm_read_count_table_before_RUVSeq.txt.zip— TPM table = RUVSeq input.reference/housekeeping_gene_list.txt,reference/stem_cell_sample_annotation.txt.script/01..04*.R+script/run_batch_effect_correction.sh— the full runner.output/read_count_table_after_RUVSeq.txt— author-shipped expected output.output/read_count_table_after_RUVSeq_filtering_low_variance_data.txt— expected.
Because the authors ship the analysis-ready input and the expected output tables and the exact scripts, this is a high-fidelity 1:1 target: we re-run their own runner with the exact stated env (R 3.6.0, RUVSeq 1.18.0) and compare to their shipped outputs and to the numbers printed in the paper.
IN SCOPE (pipeline-derived, attempted)
| # | Result | Pipeline / script | Reported value | Paper location |
|---|---|---|---|---|
| C1 | between/within variance ratio, stem cell types, before correction | 03_1_*.R on raw table |
13.3 | Results |
| C2 | between/within variance ratio, stem cell types, after correction | 03_1_*.R on RUVSeq output |
17.3 | Results |
| C3 | between/within variance ratio, research groups, before correction | 03_2_*.R on raw table |
41.4 | Results |
| C4 | between/within variance ratio, research groups, after correction | 03_2_*.R on RUVSeq output |
27.6 | Results |
| C5 | # genes used for batch effect correction | RUVSeq input (01→02) |
9679 | Methods |
| C6 | reproduced RUVSeq output table ≈ shipped expected output | 02_*.R (RUVg, k=1) |
(table identity) | repo output/ |
Primary pipeline: RUVSeq v1.18.0 RUVg(k=1) with non-DE housekeeping genes
as negative controls, then a deterministic between/within-group variance ratio.
OUT OF SCOPE (not attempted — and why)
- Upstream raw-read processing (FastQC, Trimmomatic, HISAT2→hg38, featureCounts): the repo does not ship FASTQs or that pipeline; only the resulting count tables. Reproducing it would require re-downloading 103 SRA runs and is not the shipped artifact.
- UMAP/KNN clustering (8 clusters C1–C8), t-SNE, hierarchical clustering: parameters are stated but no clustering script or seed is shipped in this repo; stochastic/seed-sensitive. Not in the shipped runner.
- WGCNA modules (7), DESeq2 DEG counts, GO enrichment: no scripts shipped in this repo for these; wet-lab/external/manual or in unshipped code.
These exclusions follow the 80/20 rule: the shipped runner deterministically produces C1–C6, which are the cleanly-specified, low-hanging outputs.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a near-ideal target: the authors shipped analysis-ready input tables, exact R/shell scripts, and expected outputs, and re-running RUVSeq RUVg(k=1) regenerated the batch-corrected table byte-for-byte (SHA256 ff66ed42…), with all four reported variance ratios reproducing to printed precision. Two caveats keep it from a clean green: (1) the Methods count 9679 (F1) is not derivable from any shipped table — an upstream/data-availability gap on the authors' side, flagged but not asserted as fabrication; (2) the headline variance-ratio statistic is defined by an idiosyncratic colMeans()+cbind() vector-recycling computation that an independent recomputation does not reproduce, so C1–C4 confirm "the shipped code reproduces the printed values" rather than validating the metric. Severity of all comparable deviations is negligible (rounding only), so the central conclusion holds under reproduction, but the non-derivable number plus the metric idiosyncrasy make the overall judgement yellow (solid with explainable deviations).
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.