GAUGE-Annotated Microbial Transcriptomic Data Facilitate Parallel Mining and High-Throughput Reanalysis To Form Data-Driven Hypotheses.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Any deviation was negligible
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the core pipeline 1:1 on the paper's OWN data. GAUGE (DartmouthStantonLab/GAUGE @ c113f4ab) ships two self-contained CoreScripts + the GAUGE-annotated ANOVA compendia; the algorithm (stringdist of sample titles vs Euclidean expr distance -> dynamicTreeCut groups -> ape::mantel.test p) is fully specified. On «our HPC» (R 4.3.3 conda env, edgeR/GEOquery) the RNA-seq demo (3 bundled refine.bio studies) and microarray demo (3 GPL84 studies via GEOquery) both ran end-to-end and produced Mantel p-values + sample-group assignments. The cleanest checkable REPORTED number reproduced EXACTLY: the P. aeruginosa compendium is 73 studies = 64 microarray + 9 RNA-seq (paper Results + Methods), and the RU's named accession GSE21966 is among the 73. Did NOT attempt the hard ~20%: full-corpus annotation/precision/error rates (45%/54%/88.5%/87.7%/5-8% over 368 microarray + 139 RNA-seq studies, needing a manual gold standard) and the PA3923 biology/wet-lab claims. The '1,003 samples' figure is not recomputable from the shipped compendium (samples already collapsed into ANOVA) -> uncheckable, not a discrepancy. No fabrication signal: every shipped-data check matched exactly.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 70assessed: 2026-06-15 ⛓ 0d5c6a5bac7f
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusSample titles from the same experimental group are similar to each other and parallel the clustering of expression data, so an automated algorithm can detect sample groups to annotate GEO microbial transcriptomic studies for high-throughput reanalysis.
- ★ GAUGE automatically annotates GEO microbial microarray and RNA-seq data sets, increasing the percentage amenable to analysis from 4% to 33%. method
- ★ 89% of GAUGE-annotated studies matched group assignments generated by human curators. finding
- ★ GAUGE uses three core steps: a text-string distance matrix of sample titles, a gene expression Euclidean distance matrix, and a Mantel test correlating the two to decide annotatability. method
- ★ GAPE, a Shiny Web interface, provides GAUGE-annotated P. aeruginosa and E. coli compendia, making three times more P. aeruginosa data available for reanalysis. resource
- ★ PA3923, a hypothetical-protein gene, is frequently differentially expressed and significantly coregulated with laminin-binding/biofilm-formation genes (estA, oprD, oprG). finding
- ★ PA3923 encodes a putative laminin-binding protein that promotes P. aeruginosa biofilm formation; PA3923 mutants are defective in biofilm formation. mechanism
- Increasing the Mantel test P value cutoff from 0.05 to 0.1 raises annotation rate with minimal increase in error rate. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Microarray transcriptomic reanalysis (GAUGE algorithm + Mantel test annotation) | P. aeruginosa, S. aureus, C. albicans microarray studies from GEO | none (in silico reanalysis) | sample group assignment accuracy; Mantel test P value | R (single core, Linux); GEO series matrix files |
| RNA-seq transcriptomic reanalysis (GAUGE annotation) | S. cerevisiae, P. aeruginosa, E. coli RNA-seq studies | none (in silico reanalysis) | sample group assignment accuracy; Mantel test P value | refine.bio gene-level count tables |
| ANOVA differential expression compendium analysis | P. aeruginosa compendium of 73 studies (microarray + RNA-seq), 1,003 samples | none (meta-analysis across varied treatments) | differentially expressed genes; maximum absolute log2 fold change | — |
| Pearson correlation analysis | P. aeruginosa compendium (log2 FC of estA, oprD, oprG, PA3923) | none | correlation of transcript log2 fold changes | — |
| KEGG pathway enrichment analysis (Fisher's exact test) | GAUGE-annotated P. aeruginosa ANOVA compendium | none | enrichment of biofilm formation pathway in DE genes (FDR<0.05) | KEGG |
| Biofilm formation assay (wet-bench follow-up) | P. aeruginosa PA3923 mutants vs wild type | PA3923 mutant (KO) | biofilm formation defect | — |
- ▲ GAUGE increased percentage of microbial data sets amenable to analysis from 4% to 33% 4% to 33%
- – 89% of GAUGE annotations matched human curator group assignments 89%
- – For microarray studies, GAUGE achieved 88.5% precision and 54% annotation rate at Mantel P<0.1 with 6% error rate 88.5% precision
- – For RNA-seq studies, GAUGE annotated 90/139 (65%) with 87.7% precision and 8% error rate at Mantel P<0.1 65%; 87.7% precision
- ▲ PA3923 was differentially expressed in 39 of 73 studies with median fold change >7 median FC >7
- ▲ In 49% of studies (19/39) where PA3923 was DE, estA, oprD, oprG were also differentially expressed; the four genes were highly correlated P<0.001; 19/39
- ▲ PA3923 was differentially expressed in 11 of 18 (61%) studies with enriched biofilm formation pathway, higher than whole compendium (53%) 61% vs 53%
- ▼ PA3923 mutants are defective in biofilm formation, consistent with predictions
- other 33% of microbial studies annotated (GAUGE-annotated microarray + RNA-seq fraction)
- other 89% match with human curators (overall GAUGE vs curator agreement)
- other 88.5% precision (microarray GAUGE-annotated studies at Mantel P<0.1)
- other 87.7% precision; 90/139 (65%) annotated (RNA-seq GAUGE-annotated studies at Mantel P<0.1)
- fold_change median log2 FC = 1.9 (FC = 3.7) (top 10 frequently DE genes across P. aeruginosa compendium)
- fold_change PA3923 median fold change >7 (PA3923 across studies where DE)
- pvalue P<0.001 (Pearson correlation of estA/oprD/oprG/PA3923 log2 FC)
- count 73 studies, 1,003 samples (GAUGE-annotated P. aeruginosa ANOVA compendium)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
GAUGE is a computational algorithm evaluated on 368 microarray and 139 RNA-seq microbial GEO studies; its core validation used the Mantel test to assess concordance between text-based and expression-based sample clustering, with independent human curator annotations as ground truth. Differential expression across annotated studies was identified using ANOVA (FDR < 0.05), Pearson correlation was used to assess co-regulation of candidate genes across the compendium, and Fisher's exact test was applied for KEGG pathway enrichment. Results were reported primarily as precision/annotation/error rates, median log2 fold changes, and threshold P values rather than exact statistics.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Mantel test (correlation between string distance matrix of sample titles and Euclidean distance matrix of gene expression profiles) | Assessment of GAUGE sample group annotation concordance for each study; applied to 368 microarray and 139 RNA-seq studies individually | 368 microarray studies; 139 RNA-seq studies | not stated |
| ANOVA with FDR < 0.05 | Identification of differentially expressed genes across GAUGE-annotated sample groups in the P. aeruginosa compendium (73 studies, 1,003 samples total) | 73 studies; 1,003 samples | not stated |
| Pearson correlation | Co-regulation analysis of log2 fold changes of four laminin-binding protein transcripts (PA3923, estA, oprD, oprG) across all studies in the compendium | 73 studies (compendium-wide) | not stated |
| Fisher's exact test | KEGG pathway enrichment analysis across the GAUGE-annotated P. aeruginosa ANOVA compendium (FDR < 0.05) | 73 studies | not stated |
-
ANOVA was applied uniformly to identify differential expression across all GAUGE-annotated studies, including both microarray and RNA-seq data↳ Could also: limma/voom could be applied to microarray data and DESeq2 or edgeR to RNA-seq count data as platform-specific alternatives — These methods incorporate empirical Bayes variance shrinkage (limma) or negative binomial modeling of count overdispersion (DESeq2/edgeR), which are well-established for their respective data types and can improve power and error control, particularly for studies with small sample sizes
-
The Mantel test was used to measure concordance between text-based and expression-based distance matrices↳ Could also: Procrustes analysis or the RV coefficient could also quantify agreement between two multivariate distance or configuration matrices — These methods offer complementary approaches to matrix association that handle different scaling and distributional properties, and Procrustes analysis additionally produces a visual summary of the alignment between configurations
-
Euclidean distance was used to quantify similarity between gene expression profiles in the Mantel test↳ Could also: Correlation-based distance (1 − Pearson r) or Spearman-based dissimilarity could also be used — Correlation-based distance focuses on expression pattern rather than absolute magnitude, which may be more appropriate when aggregating studies across heterogeneous microarray platforms with different normalization scales
-
Pearson correlation was used to assess co-regulation of laminin-binding gene fold changes across studies↳ Could also: Spearman rank correlation could also be applied — Spearman correlation is non-parametric and less sensitive to outlier fold-change values or deviations from normality, which may be relevant when aggregating log2 FC values across heterogeneous experimental conditions
-
Fisher's exact test with a binary differentially expressed / not differentially expressed gene list was used for KEGG pathway enrichment↳ Could also: Gene Set Enrichment Analysis (GSEA) using ranked continuous statistics could also be applied — GSEA does not require a binary significance cutoff and can detect coordinated but sub-threshold shifts across a pathway, which may increase sensitivity when pathway signals are distributed across many genes of moderate effect size
-
Estimated precision, annotation, and error rates were reported as point estimates without confidence intervals↳ Could also: Wilson or Clopper-Pearson exact binomial confidence intervals could also be reported around each rate — Confidence intervals communicate the precision of the estimated rates and facilitate comparison with future methods or datasets, which is particularly informative here given the relatively fixed validation sample sizes (368 and 139 studies)
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
GAUGE annotated 54% of microbial microarray GEO studies with 88.5% precision and 6% error rate at Mantel P<0.1.microarray geo-microbial 2021×1papers★ This paper is the founder (earliest)
-
89% of GAUGE automated sample-group annotations agreed with human curator assignments across microbial GEO datasets.other geo-microbial-compendium 2021×1papers★ This paper is the founder (earliest)
-
GAUGE increased the fraction of microbial GEO datasets amenable to differential-expression analysis from 4% to 33%.other geo-microbial-compendium up 2021×1papers★ This paper is the founder (earliest)
-
Biofilm formation KEGG pathway enrichment is associated with elevated PA3923 differential-expression frequency in P. aeruginosa (61% vs 53% compendium-wide).other pseudomonas-aeruginosa up 2021×1papers★ This paper is the founder (earliest)
-
PA3923 mutants show defective biofilm formation in P. aeruginosa, experimentally validating compendium-derived predictions.other pseudomonas-aeruginosa down 2021×1papers★ This paper is the founder (earliest)
-
PA3923 transcript fold-change is strongly correlated with estA, oprD, and oprG across P. aeruginosa compendium studies (P<0.001), indicating co-regulation.other pseudomonas-aeruginosa none 2021×1papers★ This paper is the founder (earliest)
-
PA3923 is differentially expressed in 39 of 73 P. aeruginosa compendium studies with median fold change >7.other pseudomonas-aeruginosa up 2021×1papers★ This paper is the founder (earliest)
-
GAUGE annotated 65% of microbial RNA-seq GEO studies (90/139) with 87.7% precision and 8% error rate at Mantel P<0.1.RNA-seq geo-microbial 2021×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33758032 (GAUGE, mSystems 2021)
Paper: Li Z et al. "GAUGE-Annotated Microbial Transcriptomic Data Facilitate
Parallel Mining and High-Throughput Reanalysis To Form Data-Driven Hypotheses."
mSystems 6(2):e01305-20. PMID 33758032 / PMC8547006 / DOI 10.1128/msystems.01305-20.
Code: https://github.com/DartmouthStantonLab/GAUGE (commit c113f4ab662c5b83ccf9df9e88c097e17ff0bed4, master, GPL-3.0)
What GAUGE is (the pipeline)
GAUGE = "General Annotation Using Text/data Group Ensembles." For each GEO transcriptomic study it:
- builds a string-distance matrix of sample titles (
stringdist, default optimal-string-alignment), - builds a data-distance matrix = Euclidean distance between samples on
expression (microarray: log2 expr; RNA-seq: edgeR
filterByExpr+calcNormFactors+ log-CPM), - assigns sample groups with
dynamicTreeCut::cutreeDynamic(hybrid, deepSplit=4, minClusterSize=1), - computes a Mantel test p-value (
ape::mantel.test, default 999 permutations) between the two distance matrices → a study is "annotatable" when string- and data-distance agree.
In scope (pipeline-derived, attempted)
| id | what | pipeline | source |
|---|---|---|---|
| C1 | GAUGE RNA-seq core script reproduces: runs end-to-end on the bundled demo data (3 refine.bio studies SRP228531/SRP055410/SRP090296) and emits a Mantel p-value + sample-group vector per study | GAUGE_coreScript_pseudomonas_RNAseq.R (self-contained, ships data) |
repo CoreScript/RNA-seq |
| C2 | GAUGE Microarray core script reproduces: downloads 3 GPL84 studies (GSE10030/GSE10065/GSE28429) via GEOquery and emits Mantel p-value + sample groups per study | GAUGE_coreScript_pseudomonas_Microarray.R |
repo CoreScript/Microarray |
| C3 | P. aeruginosa GPL84 ANOVA compendium size = 73 studies / 1,003 samples (paper Results/Fig) — verify against shipped Pa_GPL84_refine_ANOVA_List_unzip.rds |
annotated-compendium object | repo ANOVA_compendia |
C3 is the one checkable reported number against shipped data; C1/C2 reproduce the documented pipeline behaviour/output (the repo's stated demo purpose).
Out of scope / not attempted (the hard ~20%)
- Full-corpus annotation rates (45%/54% microarray; 65% RNA-seq), precision (88.5%/87.7%) and error rates (5–8%): require re-running GAUGE over all 368 microarray + 139 RNA-seq studies and a manual/expert "correct-annotation" gold standard — large + partly manual, out of 80/20 scope.
- PA3923 biology claims (diff-expressed in 39/73 studies; 49%/61% co-expression; wet-lab biofilm reduction 33.5%/49.8%): mix of compendium mining + wet-lab; the 39/73 mining number is a possible stretch goal only if compendium structure makes it trivial, else not attempted.
- GAPE Shiny app (separate repo DartmouthStantonLab/GAPE) — not a pipeline result.
Notes
- Mantel p-values are permutation-based → small run-to-run variation expected; we fix a seed and report it, and treat agreement as order-of-magnitude / sign of significance, not exact float match.
- Two repo bugs noted: microarray script's trailing
names(report)<-expGSEreferences an undefined var (expGSEshould beexpname); harmless 1-line fix applied in our copy and documented. Core computation untouched.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Every reported value that could be checked against the shipped GAUGE artifacts reproduced exactly (73 studies = 64 microarray + 9 RNA-seq; GSE21966 present), and both core demo pipelines ran end-to-end with no fabrication signal. The only gaps are on the data-availability/scope side, not the authors': the '1,003 samples' figure is not recomputable because per-sample data was collapsed into ANOVA tables in the deposited object, and the full-corpus efficacy metrics (precision 88.5%/87.7%, error 5–8%) were out of 80/20 scope and not attempted. Net: a solid partial reproduction — 1:1 on what was checkable, with explainable, non-suspicious uncheckable remainder.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.