treeclimbR pinpoints the data-dependent resolution of hierarchical hypotheses.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough -> 1:1 reproduced. Re-ran the authors' own pipeline (treeclimbR_article/simulation_fix_throat) for the headline benchmark Fig 2b (parametric synthetic microbiome, 25 samples/group, FDR cutoff 0.05, scenarios BS/US/SS), running treeclimbR + the BH leaf-level baseline over 100 simulations (seed 2020) on «our HPC» (SLURM 2176976, COMPLETED). treeclimbR attains high TPR (0.95-0.99) with FDR controlled near 0.05 (0.027-0.057) in all three scenarios, while BH controls FDR at ~4x lower leaf-only TPR (~0.23) - exactly the contrast the paper reports. Used the maintained Bioconductor treeclimbR 1.2.0 / R 4.4.3 (paper: v0.1.1 / R 3.6.1); same algorithm, so values may differ at the 2nd-3rd decimal but the central claim reproduces faithfully. Had to patch one R>=4.2 incompatibility in the article's DataPrep.R (length>1 '&&'). NOT attempted (the 20%): the 6 other competitor methods (incl. LEfSe/Python2), sample sizes sp10/sp50, Figs 3-5, and the real-data case studies (infant microbiome / miRNA / CyTOF / LPS). Grades provisional; human reviewer decides (see AUDIT.md + agreement.json).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 73assessed: 2026-06-15 ⛓ 90f08b417df2
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a tree-guided, data-driven algorithm identify the optimal resolution at which hierarchically structured entities (e.g., taxa, miRNAs, cell subpopulations) are differentially associated with a phenotype, improving detection power while controlling false discovery rate compared to fixed-resolution or existing tree-based methods?
- ★ treeclimbR proposes multiple candidate resolutions on a hierarchical tree and selects the optimal one in a data-driven way to pinpoint signal branches/leaves of interest. method
- ★ treeclimbR outperforms competing methods on synthetic data with high TPR and well-controlled FDR when the tree has coherent-change branches, and performs similarly to BH when the tree is uninformative. finding
- ★ The algorithm is general and applicable to differential abundance (DA) and differential state (DS) analyses across diverse data types (microbiome, miRNA, single-cell CyTOF, scRNA-seq). method
- ★ In DS testing, resolution exploration is performed separately per feature, allowing different features to stop at different tree resolutions. method
- ★ Candidates are generated via node-level U scores combining direction and strength of association, varying with a tuning parameter t in [0,1], climbing from root toward leaves. method
- By selecting non-nested nodes (no ancestor-descendant relationship), treeclimbR avoids nested nodes that are difficult to interpret. method
- treeclimbR is provided as a resource/tool for inspecting entities across resolutions in multi-resolution genomic datasets. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| differential abundance analysis on parametric synthetic microbial data (OTU counts from Dirichlet-multinomial) | synthetic OTU/microbial dataset (phylogenetic tree with 956 leaves) | simulated DA between control and treatment groups (BS, US, SS scenarios) | TPR and FDR of DA branch/OTU detection | — |
| differential abundance analysis on non-parametric synthetic microbial data | synthetic microbial dataset | simulated differential abundance | TPR and FDR | — |
| differential state analysis on semi-simulated mass cytometry (CyTOF) data | AML-sim single-cell CyTOF dataset | simulated subpopulation-specific state changes | detection performance (TPR/FDR) | CyTOF (mass cytometry) |
| differential state analysis on semi-simulated mass cytometry (CyTOF) data | BCR-XL-sim single-cell CyTOF dataset | simulated differential state | detection performance (TPR/FDR) | CyTOF (mass cytometry) |
| differential abundance analysis on microbiome survey data | microbiome / taxonomic tree | none/observational disease association | phenotype-associated taxa abundance | — |
| differential abundance analysis on microRNA survey data | miRNA tree (leaves = mature miRNA sequences; internal nodes = duplexes, transcripts, clusters) | none/observational disease association | disease-associated miRNA dysregulation | — |
| differential state analysis on single-cell RNA sequencing data | scRNA-seq cell subpopulations | none/observational condition comparison | subpopulation-specific gene expression state changes | — |
- ▲ treeclimbR outperforms competing methods across all three microbial scenarios (BS, US, SS) with high TPR and well-controlled FDR.
- ▲ Tree-based methods (treeclimbR, StructFDR, minP) achieve higher power than BH on simulated microbial data.
- – HFDR detects no changes (TPR=0, FDR=0) because it stops at the root representing sequencing depth. TPR=0, FDR=0
- – lasso and miLineage identify nested nodes and cannot pinpoint DA branches; using root-closest nodes yields mostly false positives.
- count 956 leaves (phylogenetic tree used in one of 100 microbial simulation repetitions)
- count 100 repetitions (repeated simulations per scenario and sample size combination)
- count sample sizes 10, 25, and 50 per group (simulated sample sizes per group in parametric microbial benchmarks)
- other FDR cutoffs 0.01, 0.05, 0.1 (imposed FDR thresholds evaluated per method)
- other HFDR TPR=0 and FDR=0 (HFDR performance on synthetic microbial data)
- count 10 leaves and 9 internal nodes (nodes 11-19) (example schematic tree in Fig.1)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a methods/benchmarking paper introducing treeclimbR, an algorithm that performs node-level differential analysis across a hierarchical tree of entities and then selects a data-driven resolution among multiple proposed candidate node sets. Performance is assessed primarily on synthetic and semi-simulated datasets by averaging true positive rate (TPR) and observed false discovery rate (FDR) over 100 repetitions across several scenarios and sample sizes, and by comparison against competing methods (e.g., StructFDR, HFDR, miLineage, LEfSe, lasso, and Benjamini-Hochberg). The core inferential elements are per-node hypothesis tests yielding P values, combined with multiplicity correction applied within each candidate resolution.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Per-node null-hypothesis test producing a P value (test statistic/model fit at each tree node; specific model context-dependent, e.g., differential abundance/expression) | Each node of the tree in the differential analysis step (Fig. 1b) | — | not stated |
| Benjamini-Hochberg procedure (used as a comparator method and as within-candidate multiplicity correction) | Leaf-level baseline comparison and multiplicity correction within each candidate (Fig. 1d) | — | not stated |
| Kruskal-Wallis sum-rank test (described as part of the LEfSe comparator method) | Comparator method LEfSe for biomarker discovery | — | not stated |
| Lasso-regularized logistic regression (comparator method) | Synthetic microbial data comparison | — | not stated |
-
Method performance is summarized as the average TPR and FDR over 100 simulation repetitions.↳ Could also: Reporting the spread of these metrics across repetitions (e.g., SD, IQR, or 95% confidence/percentile intervals), or showing the full distributions (boxplots). — Adding a measure of variability would convey how stable each method's TPR/FDR is across repetitions, which complements the average and is often informative for benchmark comparisons.
-
Observed FDR is compared against nominal cutoffs (0.01, 0.05, 0.1) using point estimates from the simulations.↳ Could also: Pairing observed FDR estimates with Monte Carlo standard errors or interval estimates derived from the 100 repetitions. — Interval estimates would quantify the simulation uncertainty around the observed error rates and help characterize how precisely the nominal control is achieved.
-
Node-level differential analysis yields a P value at each node, with multiplicity corrected within candidates.↳ Could also: Reporting effect-size estimates (e.g., log fold changes) alongside the P values for detected nodes. — Effect sizes give a sense of the magnitude of change at detected resolutions, complementing significance-based detection.
-
Benchmarking relies on synthetic (Dirichlet-multinomial) and semi-simulated CyTOF datasets with two-group comparisons.↳ Could also: Adding evaluation under additional generative models or real datasets with known ground truth, where available. — Broader simulation settings can illustrate how conclusions generalize across different data-generating assumptions.
-
Multiplicity is controlled via Benjamini-Hochberg within candidates plus the tree-search selection criteria.↳ Could also: Comparing against alternative FDR frameworks (e.g., Benjamini-Yekutieli for dependent tests, or q-value/local-FDR approaches). — Different multiplicity frameworks make different dependence assumptions and can offer another reference point for error control under tree-structured (correlated) hypotheses.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
HFDR detects zero differential abundance signals (TPR=0, FDR=0) by halting tree traversal at the root node that conflates signal with total sequencing depthother synthetic microbial otu none 2021×1papers★ This paper is the founder (earliest)
-
lasso and miLineage identify nested rather than most-specific differentially abundant nodes in hierarchical microbial data, yielding predominantly false positive discoveries when root-closest nodes are reportedother synthetic microbial otu 2021×1papers★ This paper is the founder (earliest)
-
tree-aware differential abundance methods (treeclimbR, StructFDR, minP) achieve higher detection power than BH correction on simulated microbial OTU dataother synthetic microbial otu up 2021×1papers★ This paper is the founder (earliest)
-
treeclimbR achieves high TPR with well-controlled FDR across all simulated differential abundance scenarios (BS, US, SS) in phylogenetic-tree-structured microbial OTU data, outperforming all competing methodsother synthetic microbial otu up 2021×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34001188 (treeclimbR)
Paper: Huang et al. 2021, treeclimbR pinpoints the data-dependent resolution
of hierarchical hypotheses. Genome Biol 22:157. PMID 34001188 / PMC8127214.
Method/package: treeclimbR (Bioconductor). Analysis code:
github.com/fionarhuang/treeclimbR_article (HEAD pinned below).
What treeclimbR does
Given hypotheses arranged on a tree (OTU phylogeny, cell-type hierarchy), it
generates candidate "cut" resolutions by tuning a parameter t, then selects the
resolution that maximizes signal under FDR control (getCand → evalCand).
Pipeline-derived results in the paper (candidate targets)
| Result | Data | Pipeline | In scope? |
|---|---|---|---|
| Fig 2 — parametric synthetic microbiome (BS/US/SS scenarios), TPR/FDR vs 8 methods | throat_v35 (HMP16S) + simData |
Snakemake: simData→edgeR(runDA)→getCand/evalCand; competitors: minP, BH, StructFDR, HFDR, miLineage, lasso, LEfSe | YES (treeclimbR + BH) |
| Fig 3 — non-parametric synthetic microbiome | correlationtree_analysis repo | separate repo | no (extra repo) |
| Fig 4/5 — AML-sim / BCR-XL-sim (CyTOF) | HDCytoData | diffcyt + treeclimbR | partial-feasible, not chosen |
| infant microbiome, miRNA, LPS real-data | curatedMetagenomicData / GEO / figshare | workflowr Rmd | no (descriptive) |
Chosen target (80/20)
Fig 2b headline: parametric synthetic microbiome, 25 samples/group, FDR
cutoff 0.05, three scenarios BS (balanced) / US (unbalanced) / SS (sporadic).
Reproduce treeclimbR and the BH baseline; compute mean leaf-level TPR
and FDR over nSIM=100 simulations (seed 2020), exactly per the article's
simulation_fix_throat scripts (DataPrep.R → treeclimbR.R / BH.R →
summary/rateFun.R + fig_tpr_fdr_table.R).
Paper's qualitative claim to test: treeclimbR attains high TPR with
well-controlled FDR (≤0.05) across all three scenarios; BH controls FDR but at
leaf-only resolution (lower TPR on branch signal). minP has high TPR but does
not always control FDR.
Out of scope / the deliberate 20% (not attempted)
- The 6 competitor methods incl. LEfSe (needs a Python2 conda env) and miLineage/StructFDR/HFDR/lasso — orthogonal to reproducing treeclimbR itself.
- Other sample sizes (sp10/sp50), Figs 3–5, real-data case studies.
- Snakemake orchestration: bypassed; R steps run directly on «our HPC».
Version-fidelity caveat (honest 1:1 limits)
Article used treeclimbR v0.1.1 + TreeSummarizedExperiment v1.3.1 under R
3.6.1 (2019 Bioconductor). We run the current Bioconductor treeclimbR
(version recorded in result JSON) under R 4.4. The algorithm is unchanged but the
shipped throat_v35.rds may reflect a different HMP16SData snapshot, so exact
TPR/FDR values may differ from the paper at the 2nd–3rd decimal; the qualitative
claim (high TPR, FDR-controlled, branch-level resolution) is what we test.
Repo pin: treeclimbR_article HEAD = (recorded in AUDIT.md); treeclimbR pkg HEAD 90a98f4232fb5577b10d9a181750d1909a1aaf48. «infra» work dir: «path»
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The headline Fig 2b benchmark reproduces faithfully: re-running the authors' own pipeline gave treeclimbR TPR 0.95–0.99 with FDR controlled near 0.05 (0.027–0.057) and a BH baseline at ~4x lower TPR (~0.23) — the exact contrast the paper claims, with values derived end-to-end from shipped code+data (no fabrication). Deviations are explainable and on the technical side (package/R version gap, one patched R≥4.2 && bug, MC noise), not the authors'. Two fair caveats keep it from a 1:1 grade: Fig 2b is published as curves so the match is qualitative (q2 yellow), and the deliberate 20% (minP + 6 competitors, sp10/sp50, Figs 3–5, real data) was not attempted (q8 yellow).
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.