Application of alternative de novo motif recognition models for analysis of structural heterogeneity of transcription factor binding sites: a case study of FOXA
The main results reproduced, with only marginal, non-material deviations.
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to attempt 1:1. Code = authors' own MultiDeNA (commit f3c1072), data = ReMap GSE90454.FOXA2.KerCT (hg38; the brief's geo:GSE92491 is a loose pointer, GEO ships no peak BEDs). Built the conda env on «our HPC» (python3.10, MEME 5.4.1, bedtools, openjdk; pip setuptools<81 to fix pkg_resources) and installed MultiDeNA. 3 of the 4 motif models (ChIPMunk PWM, diPWM, InMoDe) are bundled JARs and runnable; the bundled mouse example ran END-TO-END («job», exit 0): 500/500 peaks profiled, combined-scan + classification + AUC produced, PWM pAUC ~7.2e-4 matching the paper's PWM pAUC order (~8.0e-4) -> method-level reproduction CONFIRMED (C2). BLOCKER: the 4th model BaMMmotif2 would not compile - cmake needs Boost, and every conda Boost variant (libboost-devel/libboost/boost-cpp) pulls a python<3.10-pinned subpackage that conflicts with the env's python=3.10 pin (env_unresolvable for the BaMM component within the 80/20 budget). Therefore the headline 4-model figure 88.35% (3534/4000) was NOT reproduced; the full GSE90454.FOXA2.KerCT run (run_kerct.sbatch, prepared and ready) was not completed before finalization. NO value was fabricated. NOT attempted: the 16-dataset aggregate medians (the hard ~20%). Reproduce-from-here recipe in AUDIT.md: build BaMM in a separate python=3.9 Boost env, then submit run_kerct.sbatch and compare combined_scan_test recognized/4000 to 88.35% (treat as within-tol given ReMap-version drift).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 62assessed: 2026-06-15 ⛓ 4b1eeb83a8e8
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator headless) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe incomplete recognition of transcription factor binding sites (TFBS) in ChIP-seq peaks by the standard PWM model is partly due to structural heterogeneity of binding sites, and combining alternative de novo models (PWM, diPWM, BaMM, InMoDe) can significantly increase the fraction of recognized peaks and reveal this heterogeneity, tested here on FOXA2.
- ★ Combining four de novo models (PWM, diPWM, BaMM, InMoDe) significantly increases the fraction of recognized peaks versus PWM alone (by 26.3%). finding
- ★ The BaMM model provides the main contribution to TFBS recognition and best describes FOXA2 binding site structure in the ChIP-seq datasets studied. finding
- ★ FOXA2 binding sites are not fully described by any single model, indicating structural heterogeneity of the sites. finding
- ★ MultiDeNA pipeline integrates model training, accuracy assessment, threshold selection, peak scanning, and peak classification to combine multiple de novo TFBS search models. method
- Peaks can be classified by presence/absence and positional intersection of TFBS predicted by different models to assess structural diversity. method
- TomTom-based filtering against the HOCOMOCO FOXA2 matrix retained 16 of 22 ChIP-seq datasets as FOXA2-enriched for downstream analysis. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| ChIP-seq (computational reanalysis of preprocessed peaks) | human (hg38) and mouse (mm10) cell lines/tissues, TF FOXA2 | none | TFBS predictions/recognized peaks per de novo model and peak classifications | ReMap database (peaks in BED), MACS peak calling |
| PWM model de novo motif training and TFBS scanning | FOXA2 ChIP-seq peaks (top 4000 per dataset) | none | recognition score, pAUC, recognized peaks | ChIPMunk |
| diPWM (dinucleotide PWM) model training and scanning | FOXA2 ChIP-seq peaks | none | recognition score / recognized peaks | diChIPMunk |
| BaMM (Bayesian Markov model) training and scanning | FOXA2 ChIP-seq peaks | none | recognition score / recognized peaks | BaMM |
| InMoDe (Markov model) training and scanning | FOXA2 ChIP-seq peaks | none | recognition score / recognized peaks | InMoDe |
| Motif similarity comparison | PWM frequency matrices from found sites vs known FOXA2 matrix | none | matrix similarity p-value | TomTom; HOCOMOCO FOXA2_HUMAN.H11MO.0.A |
| Threshold selection via false positive rate estimation | whole-genome promoter dataset (2000 bp upstream of TSS; 19795 human / 19991 mouse genes) | none | FPR-based score thresholds (FPR 1.9e-4) | GRCh38.p13 / GRCm38.p6 reference genomes |
- ▲ Combination of four models increased the fraction of recognized peaks compared to PWM alone 26.3%
- – Median fractions of peaks containing TFBS of a sole model for PWM, diPWM, BaMM, InMoDe 1.08, 0.49, 4.15, 1.73%
- ▲ All four models jointly recognized 88.35% of peaks (3534 of 4000) in GSE90454.FOXA2.KerCT 88.35% (3534/4000)
- – Overlap fraction recognized by all models in GSE90454.FOXA2.KerCT 34.25% (1370/4000)
- ▲ BaMM makes the largest independent contribution to recognition in GSE90454.FOXA2.KerCT 17.4% (696 peaks) vs PWM 0.525% (21), InMoDe 0.975% (39), diPWM 0.2% (8)
- ▲ BaMM and InMoDe together add a fraction nearly equal to the all-model overlap 34.55% (696+647+39=1382/4000)
- – In paired classification, 'with intersection' peaks dominate, highest for methodologically close pairs BaMM–InMoDe 53.6%, PWM–diPWM 44.4%, diPWM–BaMM 41.0%, PWM–BaMM 37.3%, diPWM–InMoDe 35.4%, PWM–InMoDe 31.6%
- ▲ BaMM contributes greatly as the single model in pairs with PWM, diPWM, InMoDe 39.2, 36.4, 26.8%
- fold_change 26.3% increase (increase in recognized peaks combining four models vs PWM alone)
- count 1.08, 0.49, 4.15, 1.73% (medians of fraction of peaks containing sole-model TFBS for PWM, diPWM, BaMM, InMoDe)
- count 88.35% (3534/4000) (peaks jointly recognized by all four models in GSE90454.FOXA2.KerCT)
- count 34.25% (1370/4000) (all-model overlap fraction in GSE90454.FOXA2.KerCT)
- pvalue p-value<0.001 (TomTom threshold for matrix similarity to known FOXA2 motif)
- other FPR 1.9e-4 (fixed false positive rate for model threshold selection)
- count 16 of 22 (ChIP-seq datasets retained after TomTom filtering)
- other pAUC for FPR < 0.001 (partial area under ROC curve used to compare model accuracy)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper develops and applies MultiDeNA, a pipeline comparing four de novo transcription factor binding site (TFBS) models (PWM, diPWM, BaMM, InMoDe) across 22 FOXA2 ChIP-seq datasets (filtered to 16 after TomTom similarity screening). Model recognition accuracy was evaluated via partial AUC (pAUC at FPR < 0.001) from repeated random 90/10 cross-validation splits, with thresholds set uniformly at a fixed FPR of 1.9 × 10⁻⁴ against a whole-genome promoter negative set. Distributions of per-dataset peak fractions across models were compared using the Mann–Whitney U-test with Bonferroni correction, and results are reported primarily as medians and percentages.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Mann–Whitney U-test | Comparison of distributions of peak fractions assigned to different models across ChIP-seq datasets | 16 ChIP-seq datasets (retained after TomTom filtering) | not stated |
| Partial area under the ROC curve (pAUC at FPR < 0.001) | Assessment of recognition accuracy for each model (PWM, diPWM, BaMM, InMoDe) via cross-validation | 90 % training / 10 % test split of 4000 peaks per dataset; repeated multiple times (exact repeat count not stated) | not stated |
| TomTom p-value threshold (p < 0.001) | Filtering of the 22 ChIP-seq datasets to retain those enriched for canonical FOXA2 binding sites | 22 datasets input; 16 retained | not stated |
-
The 16 datasets are matched (all four models are applied to each dataset), and pairwise Mann–Whitney U tests with Bonferroni correction were used to compare peak-fraction distributions↳ Could also: A Friedman test followed by Dunn's post-hoc test with Bonferroni or Benjamini–Hochberg correction could also be used for repeated-measures non-parametric comparisons across four related conditions — Because the same 16 datasets serve as the unit of observation for all four models, a repeated-measures framework explicitly accounts for within-dataset correlation; the Friedman test is the non-parametric analogue of a one-way repeated-measures ANOVA and is designed for this matched structure
-
Multiple comparisons were corrected with the Bonferroni method↳ Could also: Benjamini–Hochberg false discovery rate (FDR) correction could also be applied — Bonferroni controls the family-wise error rate and is conservative when many pairwise tests are conducted; BH-FDR controls the expected proportion of false discoveries and tends to have greater power when the number of comparisons is moderate-to-large, which may be relevant when comparing six pairwise model combinations
-
Model recognition accuracy was summarized by pAUC at a single FPR cutoff (< 0.001)↳ Could also: The full area under the ROC curve (AUROC) or the area under the precision–recall curve (AUPRC) could also be reported alongside or instead of pAUC — Full AUROC is the most widely reported single-number accuracy metric and facilitates comparison with other published models; AUPRC is informative when positive examples (true TFBS) are sparse relative to the background, which is common in TFBS recognition tasks; pAUC at a strict FPR focuses on high-specificity operating points, which is appropriate for this task but may be less familiar to general readers
-
Cross-validation used a single random 90 %/10 % split repeated an unspecified number of times↳ Could also: Stratified k-fold cross-validation (e.g., 10-fold) with a fixed random seed could also be used — A fixed k-fold scheme with a stated seed makes the accuracy estimates fully reproducible and ensures each peak appears in exactly one test fold; reporting the number of repetitions and the variance of pAUC across folds would also quantify estimation uncertainty
-
Central tendency for per-dataset peak fractions is reported as medians without accompanying dispersion measures↳ Could also: Interquartile ranges (IQR) or bootstrap 95 % confidence intervals around the medians could also be reported alongside the medians — With n = 16 datasets, reporting spread (IQR or CI) alongside the median conveys how consistently each model performs across experimental conditions, which is directly relevant to the paper's claim about BaMM's superior contribution
-
Dataset retention was determined by a single TomTom p-value threshold (p < 0.001) applied to PWM-derived frequency matrices↳ Could also: A ranked or continuous similarity score (e.g., normalized TomTom E-value or Pearson correlation of position frequency matrices) could also be used to characterize dataset quality, with sensitivity analyses at alternative thresholds reported — A binary inclusion/exclusion threshold can be sensitive to the choice of cutoff; reporting results across a range of thresholds or using a continuous quality metric would make the filtering step more transparent and allow readers to assess whether conclusions depend on the specific cutoff chosen
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 34547062 (MultiDeNA / FOXA de novo motifs)
Paper: Application of alternative de novo motif recognition models for analysis of structural heterogeneity of TFBS: a case study of FOXA. Vavilov J Genet Breed 2021. DOI 10.18699/vj21.002. PMCID PMC8408018.
Pipeline / code: https://github.com/ubercomrade/MultiDeNA — the authors' own pipeline. Trains four de-novo motif models, assesses recognition accuracy (pAUC), scans ChIP-seq peaks at a fixed FPR, and classifies peaks by which model(s) recognise a site. Models:
- PWM via ChIPMunk (
chipmunk.jar, bundled) - diPWM via diChIPMunk (bundled in chipmunk.jar)
- InMoDe (
InMoDeCLI-1.1.jar, bundled) - BaMM via BaMMmotif2 (external, must compile)
- (also supports pwm-streme via MEME, sitega via external sitega — not needed here)
Data: Paper analysed 22 FOXA2 ChIP-seq datasets from the ReMap database
(brief's geo:GSE92491 is a loose pointer; the actual inputs are ReMap peak sets).
16 passed a TomTom filter. Per dataset: top 4000 peaks by MACS score
(−10·log10 p-value).
In scope (pipeline-derived, attempted)
- C1 (primary 1:1 anchor): dataset GSE90454.FOXA2.KerCT — "all models jointly recognized 88.35 % of the peaks (3534 out of 4000)" (Results text). Reproduce by running MultiDeNA with the 4 models at FPR 1.9e-4 (= tool default 0.00019), T=4000, on the top-4000 ReMap peaks of that dataset (hg38).
- C2 (method smoke test): bundled example (
foxa2_mm_chr19, mm10) runs end-to-end and produces a combined-scan classification — demonstrates the pipeline is runnable/reproducible (no paper number attached; method-level evidence).
Out of scope / not attempted (the hard ~20%)
- Aggregate medians over all 16 datasets (Table: median recognized % per model 47.3/46.4/65.8/54.0, combined 73.6; median pAUC; sole-model %). These require the full 16-dataset sweep + TomTom curation — too heavy for the 80/20 budget. Reported but not reproduced.
- SiteGA / pwm-streme model variants (not used for the headline numbers).
- Wet-lab / external (none — this is a pure-pipeline paper).
Threshold / params (from paper)
- FPR for TFBS recognition = 1.9·10⁻⁴ (tool default
-f 0.00019). - Top 4000 peaks per dataset (tool
-T 4000; training peaks-t 500default/example). - Peak ranking: MACS −10·log10(p-value) score (ReMap score column).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.