Data-driven projections of candidate enhancer-activating SNPs in immune regulation.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH, and the headline number reproduces essentially 1:1. The paper's step-1 GAS-motif scan is, in the repo, a transparent Python regex (TTC...GAA|TTC....GAA) and the 'nearly-complete GAS' = exactly one mismatch at one of six fixed positions (variant types T1/T2/C/G/A1/A2_GAS in get_create_GAS_motif_SNPs.py). We reproduced that definition independently (no FIMO PWM) on the paper's own data (hg38 25 primary chromosomes from UCSC) on «our HPC» (SLURM 2175667, 3m). Result: ncGAS = 17,042,730 (forward strand, overlapping/FIMO-style) vs reported 17,039,967 -> diff +2,763 (+0.0162%), a 99.98% match -> within-tol. The non-overlapping count (16,992,903) and both-strand count (~34.1M) confirm the paper used a single-strand, overlapping scan. Strong evidence the headline value is genuine and pipeline-derived; no fabrication signal. NOT ATTEMPTED (hard 20%): the dbSNP155 SNP intersection (3,301,512) -- tens of GB, and the paper cites the hs1/T2T build for dbSNP while motifs are hg38 (a build inconsistency flagged for the human auditor); and the downstream funnel (50,265->...->6) which depends on many GEO ChIP/RNA-seq datasets, GO gene lists, bedtools windows, and explicit manual curation. Repo is a script-dump (no env/entrypoint, hardcoded local paths, two README-referenced FIMO scripts missing/404) but the core logic is transparent and checks out.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 57assessed: 2026-06-14 ⛓ 7285d39e64c8
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a data-driven computational protocol integrating genomic, epigenomic, and cross-species conservation data can effectively narrow millions of SNPs down to a small set of high-potential candidates capable of creating de novo GAS motifs (STAT transcription factor binding sites) that could drive gain-of-function regulation of immune genes in the JAK-STAT pathway.
- ★ A data-driven computational protocol combining motif scanning, open-chromatin filtering, gene proximity, dbSNP validation, spacing, and cross-species conservation can prioritize SNPs likely to create functional GAS motifs. method
- ★ Six candidate gain-of-function SNPs (rs560898780, rs1257658099, rs370669851, rs571421696, rs910130021, rs138606888) were identified in enhancers upstream of IRF3, IL7R, JAK2, JAK3, SOCS1, and PTPN2. finding
- ★ SNPs in cis-regulatory elements can create, destroy, or enhance GAS motifs, leading to loss-of-function, gain-of-function, or supercharged expression of nearby JAK-STAT pathway genes. mechanism
- ★ JAK2, SOCS1, and PTPN2 show significant differential expression under various cytokine stimulation conditions, consistent with potential regulatory impact via GAS motif modification. finding
- None of the 30 further-investigated candidate SNPs are reported in ClinVar, underscoring their novelty and the value of this discovery approach. finding
- Independent ChIP-seq data from IL4-stimulated PBMCs confirm accessible/acetylated chromatin at the identified SNP sites. finding
- ★ The computational protocol and its stepwise filtering logic constitute a reusable resource for prioritizing regulatory SNPs in other pathways. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Motif scanning (FIMO, MEME suite) | Human genome (hg38), computational, genome-wide | none | Nearly complete GAS motifs (ncGAS) and overlapping SNPs | FIMO/MEME Suite |
| ChIP-seq (H3K27ac, active histone modification) | PBMC and T-cells | none | Open chromatin regions overlapping candidate SNPs | — |
| Bulk RNA-seq | Primary human lung cells | Cytokine stimulation (IFNα, IFNβ, IFNγ, IL-6, IL-7) vs control | DESeq2 normalized gene counts for IRF3, IL7R, JAK2, JAK3, SOCS1, PTPN2 | DESeq2 |
| ChIP-seq (acetylation) | IL4-stimulated PBMC cells | IL4 stimulation | Acetylation/accessibility at identified SNP loci | — |
| Cross-species sequence conservation analysis | Mouse genome (mm10) vs human genome (hg38) | none | Conservation of GAS motif 10 kb upstream of orthologous target genes | — |
| Variant database screening | Human population variant data (dbSNP) | none | SNP occurrence in ≥2 independent subjects at ncGAS motif positions | dbSNP |
| Clinical variant database screening | Human variant data (ClinVar) | none | Disease association status of candidate SNPs | ClinVar |
- – Genome-wide scan identified 17,039,967 nearly complete GAS motifs and 3,301,512 overlapping SNPs. 17,039,967 motifs; 3,301,512 SNPs
- ▼ Open-chromatin (H3K27ac) filtering reduced candidates to 50,265 SNPs in PBMC/T-cells. 50,265 SNPs
- ▼ Proximity filtering to immune genes (GO database) reduced candidates to 16,017 SNPs. 16,017 SNPs
- ▼ Restricting to 10 kb upstream of TSS reduced candidates to 4,391 SNPs. 4,391 SNPs
- ▼ dbSNP validation (≥2 subjects) and spacing/conservation filters sequentially reduced candidates to 416, then 273, then 214 SNPs. 416 → 273 → 214 SNPs
- – Focus on JAK-STAT pathway yielded 30 candidate SNPs, none found in ClinVar, from which 6 final SNPs were selected in IRF3, IL7R, JAK2, JAK3, SOCS1, and PTPN2 enhancers. 30 → 6 SNPs
- – JAK2, SOCS1, and PTPN2 exhibited significant differential expression across cytokine stimulation conditions (IFNα, IFNβ, IFNγ, IL-6, IL-7) versus control.
- ▲ Acetylation/accessible chromatin confirmed at identified SNP sites in independent IL4-stimulated PBMC ChIP-seq data.
- count 17,039,967 (Nearly complete GAS motifs (ncGAS) found genome-wide by FIMO)
- count 3,301,512 (SNPs identified within ncGAS motif positions (dbSNP))
- count 50,265 (SNPs remaining after open-chromatin (H3K27ac) filter in PBMC/T-cells)
- count 16,017 (SNPs remaining after filtering for proximity to immune genes)
- count 4,391 (SNPs remaining after restricting to 10 kb upstream of TSS)
- count 416 (SNPs remaining after requiring presence in ≥2 independent subjects in dbSNP)
- count 273 then 214 (SNPs remaining after spacing filter, then after mouse conservation filter)
- pvalue * p ≤ 0.05, ** p ≤ 0.01, n.s. = not significant (Independent t-tests comparing gene expression across cytokine stimulation conditions (Fig. 4))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper presents a sequential bioinformatic filtering pipeline to prioritize SNPs capable of creating STAT-binding GAS motifs, reducing ~3.3 million candidates to six for experimental follow-up. The core methodology is not inferential but rather rule-based attrition: genome-wide motif scanning (FIMO), histone-modification-defined open chromatin, gene-proximity annotation, dbSNP frequency filtering, spacing constraints, and cross-species conservation. As a secondary validation step, publicly available RNA-seq data (Lee et al.) were analyzed using DESeq2-normalized counts, and independent t-tests were applied to compare gene expression across cytokine stimulation conditions, with results reported as significance bands.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Independent t-test | Fig. 4 — pairwise comparisons of DESeq2-normalized gene counts for IRF3, IL7R, JAK2, JAK3, SOCS1, PTPN2 across IFNα, IFNβ, IFNγ, IL-6, IL-7 vs. control | — | not stated |
| FIMO motif scanning (position-weight matrix score with internal p-value threshold) | Genome-wide identification of nearly-complete GAS motifs (ncGAS) in hg38 | — | not stated |
-
Independent t-tests were applied to DESeq2-normalized RNA-seq counts to compare stimulation conditions↳ Could also: DESeq2's built-in Wald test (or likelihood ratio test) on raw count data, or a non-parametric rank-based test such as Mann-Whitney U on the normalized values — RNA-seq count data follow a negative-binomial distribution; DESeq2's native differential-expression test accounts for overdispersion and heteroscedasticity across conditions, which are assumptions the t-test does not model; a non-parametric alternative makes no distributional assumption and is robust to small n
-
Multiple pairwise t-tests were performed across six genes and five stimulation conditions with no stated correction↳ Could also: Benjamini-Hochberg false discovery rate (FDR) adjustment across the full family of gene × condition comparisons — Controlling the FDR across a joint family of tests reduces the expected proportion of false positives among declared significant results, which can matter when several comparisons are made simultaneously; this is standard practice in multi-condition transcriptomic analyses
-
Open-chromatin filtering used a binary presence/absence criterion based on H3K27ac ChIP-seq peaks↳ Could also: Quantitative ranking by ChIP-seq signal intensity (e.g., normalized read counts or fold-enrichment over input) to score candidates on a continuous scale — A continuous score would allow ranked prioritization rather than a hard binary cut, potentially retaining biologically relevant sites near the peak threshold and enabling sensitivity analysis of the cutoff choice
-
Cross-species conservation was applied as a binary filter (ncGAS motif present/absent in mm10 within 10 kb of the orthologous gene)↳ Could also: Quantitative phylogenetic conservation scores such as phyloP or phastCons from UCSC multi-alignment tracks — Continuous conservation scores capture the degree of constraint at each nucleotide position, allowing ranked filtering rather than an all-or-nothing comparison to a single species, and are routinely used in regulatory variant prioritization frameworks such as CADD
-
Proximity to gene regulatory regions was defined by a fixed 10 kb upstream window from the TSS↳ Could also: Integration of experimentally mapped enhancer–gene links from eQTL catalogs (e.g., GTEx), chromatin conformation data (Hi-C, HiChIP), or activity-by-contact scores — A fixed-distance window assumes all regulatory activity lies within 10 kb upstream, whereas long-range enhancers can act from tens to hundreds of kilobases away; activity-by-contact or eQTL evidence would anchor each candidate to a specific gene–SNP regulatory relationship rather than proximity alone
-
Significance was reported as categorical bands (* ≤ 0.05, ** ≤ 0.01) without effect sizes or dispersion measures↳ Could also: Reporting fold-change (or log2 fold-change) alongside 95% confidence intervals and the underlying n per group — Effect sizes communicate biological magnitude independently of sample size, and confidence intervals convey estimation uncertainty; these are complementary to p-values and are recommended by reporting guidelines (e.g., ARRIVE, Nature reporting checklist) to help readers assess practical significance
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40011812 (GAS_motifs)
Paper: Hoffmann et al. 2025, BMC Genomics. "Data-driven projections of candidate enhancer-activating SNPs in immune regulation." DOI 10.1186/s12864-025-11374-7. Repo: https://github.com/Firestar93/GAS_motifs (Python/R scripts, no env file, hardcoded Windows/Mac paths, no single entrypoint). Data: hg38 (UCSC), dbSNP155, many GEO ChIP/RNA sets.
Pipeline funnel reported (Results & Discussion)
ncGAS motifs (hg38, FIMO almost-GAS) 17,039,967 <-- HEADLINE, step 1 SNPs in ncGAS (dbSNP155) 3,301,512 <-- HEADLINE, step 2 SNPs in open chromatin (H3K27ac>20) 50,265 SNPs near immune genes (GO) 16,017 SNPs within 10kb upstream TSS 4,391 SNPs in >=2 subjects 416 SNPs spaced >=200bp from existing GAS 273 SNPs conserved in mm10 214 SNPs in JAK-STAT pathway 30 Final candidates 6 (rs560898780, rs1257658099, rs370669851, rs571421696, rs910130021, rs138606888)
In scope (deterministic, public data, reproducible 1:1)
- C1 — genome-wide ncGAS motif count on hg38. The repo detects GAS motifs by a pure regex (main.py/motif.py: "TTC...GAA|TTC....GAA"). The "almost/nearly-complete GAS" = exactly one mismatch at a fixed position, enumerated as 6 variant types (T1/T2/C/G/A1/A2_GAS, per get_create_GAS_motif_SNPs.py). Fully deterministic given hg38; no PWM/threshold subtleties for the exact-one-mismatch definition. Compare to 17,039,967.
- C2 — canonical complete GAS motif count on hg38 (TTCnnnGAA 3-gapper, TTCnnnnGAA 4-gapper) via the repo's own regex. Context/clean repo reproduction (deterministic).
Out of scope / not attempted (the hard 20%, stated honestly)
- C2b SNP intersection (3,301,512): needs full dbSNP155 (~tens of GB; paper cites hs1/T2T build for dbSNP while motifs are hg38 — a build inconsistency). Skipped per 80/20.
- Downstream funnel (50,265 -> ... -> 6): depends on many GEO ChIP/RNA-seq datasets, GO term lists, bedtools windows, manual curation. Multi-dataset + manual steps; out of 80/20 scope.
- Exact FIMO PWM/threshold replication (--thresh 0.05 ...): the paper's 17M is a FIMO count; we reproduce the definition (exact one-mismatch enumeration) independently on the same hg38 data — an equally-valid independent-tool reproduction (P16), not the FIMO binary.
Method for C1/C2
Download hg38 chromosomes from the exact URL the paper cites (https://hgdownload2.soe.ucsc.edu/goldenPath/hg38/chromosomes/), scan each chromosome with the repo's regex (canonical) and the 6 one-mismatch variant regexes (ncGAS). Count both non-overlapping (repo re.findall semantics) and all-overlapping (FIMO semantics) on the forward strand; report both. Run on «our HPC» SLURM, data on «infra».
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The paper's foundational claim — 17,039,967 nearly-complete GAS motifs on hg38 — reproduces essentially 1:1 (we got 17,042,730, +0.0162%) from a transparent regex on public hg38, strong evidence the value is genuine and pipeline-derived (no fabrication). The deviation is rounding-level and sits on the input/tooling side (FIMO PWM vs our regex, contig set), i.e. technical/expected. However, the paper's actual central conclusion — the 3,301,512-SNP dbSNP155 intersection and the downstream funnel to 6 candidate enhancer-activating SNPs — was not attempted (data volume, manual curation, missing/404 FIMO scripts, plus a flagged hs1-vs-hg38 build inconsistency), so the headline conclusion is only partially verified. Net: a clean, reproducible foundation but partial coverage with explainable deviations → overall yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.