snpQT: flexible, reproducible, and comprehensive quality control and imputation of genomic data.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL (3 of 4 pipeline tiers reproduced, 2 of them with 1:1 figure matches). snpQT is a well-documented Nextflow GWAS QC+imputation pipeline (authors' own code wrapping third-party tools); reproduced on the shipped synthetic toy dataset (only open input; ALS is dbGaP-restricted). On «our HPC» via SLURM + conda profile («job»): (1) --qc reproduced 1:1 -- variant QC 6515->2612, sample QC 100->48, variant_qc_log figure visually IDENTICAL to the repo reference; (2) --pop_strat reproduced 1:1 -- PCA against 1000G with toy samples overlaying EUR, PC1vsPC2_before.png visually IDENTICAL to reference (only legend label pop vs race differs); (3) --gwas association reproduced -- plink2 logistic --glm with/without 3 PC covariates on 48 samples/2612 variants/22-26 case-control, lambda 1.058/1.0, top hit rs6682769, though the R Manhattan/QQ figure step failed on forcats version-rot. NOT reproduced: imputation (headline 6,500,533) -- conda blocks it and impute5 is license-gated (the container runtime itself now works via apptainer, so this corrects the prior 'no containers' finding). Toy composition shows paper-text-vs-shipped-data drift (6515 vs 6517 variants; 48/52 vs 51/49 case/control at commit b4b570a) -- the authors' OWN reference figure also starts at 6515, so this is post-publication data regeneration, not a pipeline error. Required engineering fixes were all dependency-rot (NXF_SYNTAX_PARSER=v1, conda.enabled, «infra»-prefix envs, snpflip setuptools/natsort rebuild), not science changes. Did NOT attempt: imputation (license blocker) and ALS benchmarks (restricted data).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 65assessed: 2026-06-20 ⛓ 677859a26418
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetExisting semi-automated or automated genomic quality control pipelines lack comprehensive quality checks, flexible workflow architecture, and user control, so a scalable, containerized nextflow-based pipeline (snpQT) can provide more comprehensive, reproducible, and flexible QC and imputation of human genomic data.
- ★ snpQT is a scalable, stand-alone software pipeline using nextflow and BioContainers for comprehensive, reproducible, interactive QC of human genomic data. resource
- ★ snpQT offers some 36 discrete quality filters or correction steps in a complete standardised pipeline. method
- ★ snpQT includes human genome build conversion, population stratification against 1,000 Genomes Project data, automated population outlier removal, and built-in imputation with its own pre- and post-QC. method
- ★ snpQT is organized as nine core workflow components (A-I) implemented with nextflow, executable via Docker, Singularity, Anaconda, or Environment Modules. method
- Nextflow caching allows skipping of unchanged earlier pipeline stages when parameters are altered later in the workflow, saving time. mechanism
- Each main workflow (C, D, E, I) provides a 'Make Report' module generating interactive HTML reports of before/after threshold plots. method
- ★ snpQT is open source (MIT license) and freely available at https://github.com/nebfield/snpQT, with a synthetic dataset and online tutorial provided. resource
- snpQT supports both local imputation (shapeit4/impute5) and export for use with external imputation servers or alternative reference panels. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| SNP-array genotype QC and pipeline demonstration (snpQT synthetic dataset use case) | synthetic human genomic dataset, chromosome 1, 100 female samples (51 cases, 49 controls) | none (demonstration run of snpQT QC pipeline) | 6,517 genotypes; QC plots and reports before/after thresholds | snpQT (nextflow) with PLINK, PLINK2, picard, samtools, snpflip |
| GWAS pipeline QC, imputation, and association testing (real-world tutorial dataset) | Amyotrophic Lateral Sclerosis SNP-array dataset, 2,000 samples (1,000 cases, 1,000 controls) from a restricted-access dbGaP project | none (pipeline QC/imputation/GWAS applied) | quality-filtered samples/variants, Manhattan plot, Q-Q plot, GWAS summary statistics | snpQT with shapeit4 and impute5 |
| Population stratification / outlier removal via PCA | user dataset merged with 1,000 Genomes Project reference panel | automated outlier removal (EIGENSOFT) | principal components before and after outlier removal | EIGENSOFT |
| Human genome build conversion | user genomic dataset (VCF/PLINK format) | liftover between GRCh38 and GRCh37 | converted genomic coordinates | Picard LiftoverVcf |
- – snpQT implements approximately 36 discrete QC filters/correction steps across nine workflows. 36 steps
- – Standard QC checks require less than 16GB RAM and 4 cores on large datasets of up to 40,000 individuals. <16GB RAM, 4 cores / 40,000 individuals
- ▲ Local imputation requires substantially more computing power than standard QC. up to 50GB RAM per chromosome per core
- – Core reference database download and storage size for snpQT setup. 17.3GB compressed / 19.7GB unzipped
- – Imputation reference database download and storage size. 13GB compressed / 15GB unzipped
- – Synthetic demonstration dataset provided with defined structure for reproducible testing. 6,517 genotypes, 100 samples, 51 cases/49 controls
- count 36 QC filters/correction steps (total quality filters/correction steps implemented in snpQT)
- count 9 core workflows (A-I) (number of nextflow workflow components in snpQT architecture)
- count 6,517 randomised genotypes, 100 female samples (51 cases, 49 controls) (synthetic demonstration dataset composition)
- count 2,000 samples (1,000 cases, 1,000 controls) (real-world ALS SNP-array dataset used in online tutorial)
- other 17.3GB compressed / 19.7GB unzipped (core reference database size)
- other 13GB compressed / 15GB unzipped (imputation reference database size)
- other <16GB RAM and 4 cores (hardware requirement for QC on datasets up to 40,000 individuals)
- other up to 50GB RAM per chromosome per core (hardware requirement for imputation)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software tool article describing snpQT, a Nextflow-based pipeline for GWAS quality control and imputation; it does not report a primary study with inferential statistical comparisons between experimental groups. Statistical content is descriptive of the QC and GWAS methodology the pipeline implements (HWE tests, PCA-based population stratification, imputation, association analysis), demonstrated on a synthetic 100-sample dataset and a referenced ALS dataset. No hypothesis tests are applied by the authors to evaluate or benchmark the tool itself; performance is assessed narratively through workflow outputs and added performance metrics for two cohort sizes.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Hardy-Weinberg Equilibrium exact test (implemented via PLINK as a pipeline step) | Workflow E: Variant QC — filtering SNPs deviating from HWE | — | not stated |
| Principal Component Analysis (PCA) for population stratification and outlier removal (via EIGENSOFT) | Workflow D: Population Stratification — automated outlier removal against 1000 Genomes reference | — | not stated |
| Logistic/linear regression-based GWAS association analysis (implemented via PLINK/PLINK2.0 with PC covariates) | Workflow I: GWAS with and without covariate adjustment; outputs Manhattan plot and Q-Q plot | — | not stated |
| Identity-by-descent (IBD) estimation for cryptic relatedness (via PLINK) | Workflow C: Sample QC — removal of cryptically related individuals | — | not stated |
| Haplotype phasing (SHAPEIT4) and genotype imputation (impute5) | Workflows F–H: pre-imputation QC, local imputation, post-imputation QC | — | not stated |
-
Population stratification is handled with PCA via EIGENSOFT, with automated outlier removal relative to 1000 Genomes Project reference clusters↳ Could also: Linear mixed models (e.g. BOLT-LMM, SAIGE, REGENIE) or ADMIXTURE-based ancestry estimation could also be used to control for population structure — Mixed-model approaches jointly model population structure and relatedness in the association step itself, which can offer improved control in admixed or family-based cohorts and may be preferred when population stratification and cryptic relatedness co-occur at scale
-
Cryptic relatedness is assessed via IBD estimation in PLINK, with related individuals removed↳ Could also: KING (Kinship-based INference for GWAS) is also widely used for relatedness estimation, particularly in large or diverse datasets — KING uses a moment estimator that is more robust to population structure than PLINK's method-of-moments IBD, which can be advantageous in ethnically diverse cohorts
-
Local imputation is performed using SHAPEIT4 for phasing and impute5 for imputation against a 1000 Genomes reference panel↳ Could also: Beagle 5.x or Michigan Imputation Server (using HRC or TOPMed panels) are also commonly used; the pipeline already notes compatibility with external imputation servers — TOPMed-based imputation offers a much larger and more diverse reference panel, potentially yielding better imputation quality for rare variants and non-European ancestry populations
-
GWAS association analysis is implemented with logistic/linear regression adjusted for principal components as covariates (via PLINK/PLINK2.0)↳ Could also: Mixed-model association methods (BOLT-LMM, SAIGE, REGENIE) could also be applied, incorporating a genetic relationship matrix to control for both population stratification and sample relatedness simultaneously — Mixed-model methods are particularly well-suited to biobank-scale datasets and cohorts with population structure or relatedness, where simple PC-covariate adjustment may be insufficient
-
Post-imputation QC removes poorly imputed variants but the specific imputation quality metric threshold (e.g. INFO score) is user-modifiable without a single recommended default stated in the manuscript↳ Could also: Reporting and filtering on both INFO score (impute5) and Rsq (minimac) with thresholds stratified by minor allele frequency is also a standard approach, as recommended by the TOPMed and HRC imputation guidelines — Frequency-stratified imputation quality thresholds acknowledge that rare variants are harder to impute accurately, allowing differential stringency and potentially retaining more well-imputed common variants while excluding poorly imputed rare ones
-
The synthetic validation dataset is small (100 samples, 6,517 SNPs on chromosome 1) and artificially structured; a real-world dataset is demonstrated only in the online tutorial↳ Could also: A formal benchmarking comparison against existing pipelines (e.g. GenotypeHarmonizer, ricopili, or the Anderson et al. tutorial scripts) on a shared public dataset could also be used to evaluate tool performance — Head-to-head benchmarking on common datasets would enable users to directly compare QC outputs, runtime, and reproducibility across tools, supporting evidence-based tool selection
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.