A network-guided protocol to discover susceptibility genes in genome-wide association studies using stability selection.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED 1:1. STAR Protocols methods paper for the authors' own gwas-tools pipeline stable_network_gwas.nf (network-guided GWAS with stability selection). Ran the authors' exact pipeline (commit a577db2c == zenodo 7395332 == v1.1.0) on the bundled synthetic toy GWAS (250 samples, 33 SNPs) INSIDE the authors' own container hclimente/gwas-tools (nextflow 22.04.5, plink 1.90b6.21, R 4.1.3, BioNet/martini/LEANR/dmGWAS/SigMod) on «our HPC» SLURM («job», node n167, ~7 min). The headline reproducible result — the 6-gene consensus subnetwork {ADM2,CHEK2,EP300,FBLN1,MAPK1,RBX1}, each n_selected=25 (selected by all 5 network methods in all 5 subsamples) with methods string dmgwas(5),heinz(5),lean(5),scones(5),sigmod(5) — reproduced EXACTLY, byte-identical to the shipped reference test/data/stable_consensus.tsv (verified by diff). The only divergences are in the explicitly non-deterministic low-count tail (random stability-selection subsampling: heinz/scones counts in {1..5} vary run-to-run) plus one upstream gene-symbol rename (ODF3B->CIMAP1B in newer HGNC); neither affects the protocol's demonstrated outcome. Runtime adaptations (documented, non-fidelity-reducing): ran nextflow inside the authors' container because host nextflow 26.04 rejects the 2022 DSL2 implicit-script syntax; used «infra» proxy:3128 for compute-node internet; patched one dead HGNC URL (EBI 404 -> identical official Google-Cloud HGNC mirror). OUT OF SCOPE (not attempted): real-application timing/resource figures (378.1 CPU-h etc.) which describe the authors' earlier 2021 study on an external real GWAS, not this toy data.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 92assessed: 2026-06-22 ⛓ 02a4d53d0325
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-22no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetIntegrating GWAS association signals with biological network structure (gene-gene interaction networks) using stability selection across multiple network-guided methods can boost discovery of disease susceptibility genes and yield a more interpretable, replicable gene set than conventional single-SNP GWAS analysis.
- ★ The protocol identifies genes that are both statistically associated with a phenotype and functionally interconnected in a biological network finding
- ★ Statistical association is computed from GWAS data while gene relatedness is computed from a co-function/gene-gene interaction network method
- ★ The resulting gene set is highly interpretable and more likely to replicate in an independent cohort finding
- ★ Stability selection (subsampling the GWAS dataset and aggregating results across five network-based methods) addresses the known instability of network-guided discovery methods method
- ★ The entire pipeline (stable_network_gwas.nf) is packaged as a single command-line tool within gwas-tools, requiring only installation and one command to run resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| SNP-phenotype chi-squared association test (PLINK 1.9) | human case-control GWAS dataset (PLINK binary format) | none | per-SNP association p-value | PLINK 1.9 |
| Gene-level association scoring (VEGAS2) | human case-control GWAS dataset | none | gene-level association score aggregated from SNP-level results | VEGAS2 |
| Network-guided module/gene discovery (dmGWAS, heinz, LEAN, SConES, SigMod) | gene-gene interaction network (user-provided or HINT protein-protein interaction network) combined with GWAS/gene scores | none | selected genes/subnetworks per method per subsample | dmGWAS; heinz (BioNet); LEAN; SConES; SigMod (via gwas-tools/Nextflow, Docker or Singularity) |
| Pipeline validation test run | fictional test GWAS dataset (test/data/gwas) and fictional edgelist network (test/data/edgelist.tsv) | none | stable_consensus.tsv listing genes selected across methods/subsamples | Nextflow with Docker image hclimente/gwas-tools |
| Full stability-selection network-GWAS pipeline application | familial breast cancer susceptibility GWAS cohort (Climente-González et al. 2021) | none | stable consensus set of susceptibility genes/subnetwork | gwas-tools stable_network_gwas.nf on CentOS 7 Linux server |
- – Test-run pipeline on fictional data selects six genes (ADM2, CHEK2, EP300, FBLN1, MAPK1, RBX1), each chosen in all 25 method-subsample combinations (5 methods x 5 subsamples) 25/25
- – Applying the pipeline to the real breast cancer GWAS dataset required 378.1 CPU hours serially, reduced to 68.8 actual hours when run with up to 60 parallel processes 378.1 CPU h -> 68.8 actual h
- – Most of the five network methods used less than 16GB of RAM <16GB RAM
- – Gene selection frequency across runs produced a long-tailed histogram, with only a few genes selected very often
- – Final consensus subnetwork was built by taking only the top 1% most frequently selected genes and their connecting edges top 1%
- count n_selected = 25 (each of six test-run genes selected in all 5 methods x 5 subsamples)
- other 378.1 CPU hours (~15.8 days) (total compute time for breast cancer GWAS analysis on CentOS 7 server)
- other 68.8 actual hours (~2.8 days) (wall-clock time for same analysis using up to 60 parallel processes)
- other <16GB RAM (memory usage of most network methods)
- other top 1% (threshold used to select most frequently chosen genes for final consensus network)
- count 5 (default number of GWAS subsamples used in stability selection)
- count 5 (number of network-based gene discovery algorithms used (dmGWAS, heinz, LEAN, SConES, SigMod))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a step-by-step computational protocol (not a primary results paper) describing a network-guided pipeline for discovering susceptibility genes in case-control GWAS data. The pipeline performs SNP-level chi-squared association testing, converts these to gene-level association scores, and applies five network-based gene-selection algorithms across repeated data subsamples, aggregating results into a stability-based consensus set of genes. No original experimental results with p-values or effect sizes are reported here; the protocol references a companion primary study for the actual outcomes.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Chi-squared test of SNP-phenotype association (via PLINK 1.9) | SNP-level association testing on the case-control GWAS dataset | — | not stated |
| Gene-based association scoring (VEGAS2) | aggregating SNP-level association into gene-level scores | — | not stated |
| Network-based module/gene selection (dmGWAS, heinz, LEAN, SConES, SigMod) | identifying genes/subnetworks associated with phenotype within a gene-gene interaction network | — | not stated |
-
Gene/subnetwork selection stability is assessed by subsampling the dataset five times (without replacement) and tallying how often each gene is chosen across methods and subsamples, with a threshold (e.g., top 1%) used to define the final consensus set↳ Could also: Formal stability selection with an explicit error-control bound (e.g., Meinshausen & Bühlmann's stability selection framework, or bootstrap-based resampling with more than five replicates) — A larger number of resampling iterations and/or a theoretically grounded stability-selection threshold could provide a quantifiable bound on the expected number of false selections, complementing the descriptive frequency-based threshold used here
-
SNP-phenotype association is tested with a chi-squared test on genotype counts↳ Could also: Logistic regression incorporating covariates (e.g., principal components for population structure, sex, age) — Regression-based association tests allow adjustment for confounders and covariates, which can be relevant when population stratification or other covariates are present in the case-control sample
-
SNP-level p-values are combined into gene-level scores using VEGAS2↳ Could also: Alternative gene-based aggregation methods such as MAGMA or PASCAL — Different gene-based test statistics handle linkage disequilibrium and gene-size effects differently, and comparing across methods can help assess robustness of the gene-level signal
-
Five network-guided methods (dmGWAS, heinz, LEAN, SConES, SigMod) are each run independently and their outputs pooled into a consensus↳ Could also: A formal ensemble or meta-analytic weighting scheme across methods, or a single unified network-regularized model — Explicit weighting or meta-analysis could combine method outputs in a way that accounts for each method's expected sensitivity/specificity, as an alternative to a simple selection-count tally
-
No multiple-testing correction or genome-wide significance threshold is explicitly described for the SNP- or gene-level tests within this protocol↳ Could also: Applying a genome-wide significance threshold (e.g., 5x10^-8) at the SNP level or an FDR correction (e.g., Benjamini-Hochberg) at the gene level prior to network analysis — A pre-filtering significance threshold or FDR correction is a standard complementary approach in GWAS pipelines and could be used alongside the stability-selection network approach to further control false-positive gene discovery
-
The consensus gene set is determined by visually inspecting a histogram of selection frequencies and choosing a threshold (e.g., top 1%) ad hoc↳ Could also: A data-driven threshold selection method, such as permutation-based null distributions or a formal stability-selection error bound — A quantitative threshold-selection procedure could provide a reproducible, statistically motivated cutoff as an alternative or complement to visual/histogram-based judgment
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36609152
Paper: Climente-González H, Azencott CA, Yamada M. A network-guided protocol to discover susceptibility genes in genome-wide association studies using stability selection. STAR Protocols 2023. DOI 10.1016/j.xpro.2022.101998 · PMC9850185.
Nature of the paper: This is a STAR Protocols methods paper — a step-by-step
protocol, not a primary research study. It documents how to run the gwas-tools
pipeline stable_network_gwas.nf (network-guided GWAS with stability selection) and
ships a toy example to demonstrate the expected output.
Code: https://github.com/hclimente/gwas-tools — the authors' own tool.
Pinned ref = release v1.1.0, commit a577db2c06825427e8b8b174c242525b0e4f8ad9
(this is exactly what the "data" deposit contains).
"Data" accession (zenodo:10.5281/zenodo.7395332): NOT a separate dataset — it is a
code snapshot of gwas-tools v1.1.0 (hclimente/gwas-tools-v1.1.0.zip, 98.6 kB).
The actual example data lives inside the repo at test/data/ (a synthetic toy GWAS).
In scope (pipeline-derived, reproducible)
- R1 (PRIMARY): Run
stable_network_gwas.nf --bfile test/data/gwas --edgelist test/data/edgelist.tsv --sigmod_nmax 6 --sigmod_maxjump 1on the bundled toy GWAS. Expected consensus subnetwork = the 6 causal/clique genes {ADM2, CHEK2, EP300, FBLN1, MAPK1, RBX1}, each withn_selected = 25(5 methods × 5 subsamples), as in the paper's "Expected outcomes" and the shipped referencetest/data/stable_consensus.tsv. Pipeline = PLINK 1.9 (χ² SNP assoc) → VEGAS2 (gene scores) → 5 network methods (dmGWAS, heinz, LEAN, SConES, SigMod) × 5 subsamples → consensus compilation. - R2 (SECONDARY): The full 47-row
stable_consensus.tsv(the lower-count tail rows from scones/heinz). The deterministic core is R1; the tail rows depend on the random subsampling and may vary run-to-run (graded leniently).
Out of scope (not reproducible from shipped artifacts)
- Real-application figures — "378.1 CPU hours (~15.8 days)… 68.8 actual hours (~2.8 days)… <16 GB RAM": these describe the authors' earlier 2021 study on a real, external GWAS (not provided, not our toy data). Timing/resource figures, not a reproducible computational output. Not attempted.
- Wet-lab / biological interpretation: none in this protocol.
Dataset profiled (same pass)
zenodo:7395332 (code+toy data). Toy GWAS: 250 samples, 33 SNPs (6 causal "cX" + 27 null "oX"), genes via positional snp2gene; 6 causal genes form a clique in edgelist.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
1:1 reproduction. This STAR Protocols methods paper's headline output — the 6-gene consensus subnetwork (ADM2,CHEK2,EP300,FBLN1,MAPK1,RBX1, each n_selected=25, all 5 methods x 5 subsamples) — was regenerated byte-identically to the authors' shipped reference by running their own pipeline (commit a577db2c = zenodo 7395332 = v1.1.0) inside their own container on the bundled synthetic toy GWAS (250 samples, 33 SNPs). The only deviations sit in the explicitly non-deterministic low-count tail (random stability-selection subsampling, 49 vs 47 rows) and one cosmetic upstream HGNC symbol rename (ODF3B->CIMAP1B); neither touches the deterministic core or the protocol's demonstrated conclusion. No authors'-side defect and no fabrication concern — the result is fully derivable from the shared code+data.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.