Comprehensive enhancer-target gene assignments improve gene set level interpretation of genome-wide regulatory data.
Part of the results reproduced; minor but material deviations remained.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
▸Reproduction agent’s raw note
DROP (no_code). The only resolvable code artifact for PMID 35473573 is TrimGalore, a generic adapter/quality-trimming tool, which does not implement the paper's reported computational claims (comprehensive enhancer-target gene assignments improving gene-set-level interpretation of genome-wide regulatory data). The headline results are not reproducible from this link: TrimGalore only trims reads and the paper provides no pinnable trimmed-read output to grade against. Public data (GSE180260) appears to exist, but without the actual analysis pipeline there is no in-scope, comparable pipeline-derived result. No scope.md/claims.tsv were produced and no SLURM/«our HPC» jobs were run in this room; nothing was fabricated. NOT ATTEMPTED: locating the paper's true analysis code (possibly a separate Babraham/author repo or supplementary), downloading/processing GSE180260, and any enhancer-target or enrichment comparison. Recommendation: re-screen the code-link extraction (likely a text-mining false positive that captured a preprocessing dependency instead of the analysis repo); if the real analysis pipeline is found, this RU can be upgraded from drop to a genuine reproduction attempt.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-14 ⛓ 9239c8b39f50
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether systematically combining multiple methods for defining enhancer locations and linking enhancers to target genes can produce genome-wide enhancer-to-target gene definitions (EnTDefs) that more accurately capture true regulatory targets—and thus improve gene set enrichment interpretation of genomic regulatory data—than the common naive approach of assigning distal regions to the nearest gene.
- ★ Combining multiple enhancer-definition and enhancer-gene link data sources yields 1860 genome-wide EnTDefs covering >500 cell types resource
- ★ The top 741 (40%) of EnTDefs significantly outperform the naive nearest-gene (>5kb LocDef) approach finding
- ★ The top 10 EnTDefs generalize well when applied to ChIP-seq data from other cell types not used in ranking finding
- ★ GSE-based ranking of EnTDefs is highly concordant with ranking based on overlap with curated enhancer-gene interaction benchmarks finding
- ★ Top general and cell-type-specific EnTDefs significantly outperform seven independent computational and experimental enhancer-gene pair datasets finding
- ★ General (non-cell-type-specific) EnTDefs outperform cell-type-specific EnTDefs (CT-EnTDefs) finding
- ★ Adding promoter regions to top EnTDefs (EnTDef_plus5kb) significantly outperforms the standard nearest-TSS method for GSE testing finding
- ★ Top EnTDefs used for GSE testing with DNA methylation or ATAC-seq data better recapitulate biological processes seen in parallel gene expression data than lower-ranked EnTDefs finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| ChIP-seq | human cell types (ENCODE, 34 transcription factors) | none | GO biological process enrichment concordance with curated TF GO annotations (F1 score) | ENCODE ChIP-seq datasets |
| ChromHMM chromatin state segmentation | multiple ENCODE cell types | none | enhancer region locations (UCSC tracks) | ENCODE ChromHMM |
| DNase-seq | multiple ENCODE cell types | none | DNase hypersensitive sites (DHSs) defining enhancer regions and signal correlation-based enhancer-gene links | ENCODE DNase-seq |
| CAGE (Cap Analysis of Gene Expression) | multiple cell/tissue types (FANTOM5 project) | none | enhancer regions and expression correlation-based enhancer-target gene links | FANTOM5 |
| ChIA-PET | ENCODE cell types | none | enhancer-promoter chromatin interactions | ENCODE ChIA-PET |
| Chromatin loop/Hi-C-derived boundary analysis with CTCF motif convergence | human cell types | none | loop boundary-based enhancer-target gene links (L1/L2/L3 gene limits) | — |
| Genome-wide DNA methylation profiling (bisulfite sequencing) | cells/tissue from parallel experiments with matched gene expression data | not specified | GSE-recapitulated biological processes compared to matched gene expression GSE results | — |
| ATAC-seq | cells/tissue from parallel experiments with matched gene expression data | not specified | GSE-recapitulated biological processes compared to matched gene expression GSE results | — |
- ▲ Top 741 of 1860 EnTDefs significantly outperform the >5 kb nearest-gene LocDef 40% of 1860 EnTDefs; FDR<0.05
- – For top 741 EnTDefs, median genes assigned per enhancer was 2 and median enhancers assigned per gene was 20 median 2 genes/enhancer (range 1-2); median 20 enhancers/gene (range 2-98)
- – Ranks 2-19 of top EnTDefs not significantly worse than the best-performing EnTDef p>0.01
- – Adding FANTOM5 or ChIA methods improved most EnTDefs; ChromHMM and L methods improved few FANTOM5/ChIA ~70%; DNase-seq ~24%; Thurman ~9-16%; ChromHMM ~5%; L ~1.7%
- ▲ EnTDef_plus5kb significantly outperforms the nearest TSS method ~0.05 increase in average F1 score, p<0.0001
- – Top 10 distal-only EnTDefs slightly underperform nearest TSS method overall but top half not significantly different ~0.03 lower F1; p>0.05 for top half
- ▲ Top 10 EnTDefs outperform GREAT, Fisher's exact test, and Poly-Enrich (5kb LocDef) methods F1=0.47 vs 0.45, p<0.007
- ▲ Top 10 EnTDefs and 5kb LocDef both significantly outperform the naive >5kb LocDef F1=0.47, 0.45 vs 0.27; p=2.37×10-14 and 1.32×10-8
- count 1860 EnTDefs generated (genome-wide enhancer-to-target gene definitions created from combinations of 4 enhancer-definition and 4 enhancer-gene link methods)
- count 1,768,201 individual enhancer-target links from 685,921 enhancers and 21,094 linked target genes (total possible enhancer-target links across >500 cell types)
- count 87 ENCODE ChIP-seq datasets for 34 transcription factors used for evaluation (GSE-based ranking of EnTDefs)
- fold_change 40% of EnTDefs (741/1860) significantly outperform >5kb LocDef (FDR<0.05, Wilcoxon signed-rank test)
- pvalue p=2.37×10-14 and p=1.32×10-8 (top 10 EnTDefs and 5kb LocDef vs >5kb LocDef, Wilcoxon signed-rank test)
- pvalue p<0.0001 (EnTDef_plus5kb vs nearest TSS method comparison)
- pvalue p=0.91 (Friedman test showing Poly-Enrich, GREAT, and FET performed equally well)
- mean average F1 = 0.47 vs 0.45 vs 0.27 (top 10 EnTDefs vs 5kb LocDef vs >5kb LocDef comparison)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This computational benchmarking study generated 1,860 genome-wide enhancer-to-target gene definitions (EnTDefs) by combining four enhancer-location sources with four linking methods across >500 cell types. Performance was evaluated using gene set enrichment (GSE) testing on 87 ENCODE ChIP-seq datasets for 34 transcription factors, with F1 scores (comparing significantly enriched GO biological process terms to curated TF GO annotations) as the primary metric. EnTDefs were ranked by average F1 score across TFs, and pairwise Wilcoxon tests — with FDR or p-value thresholds — were used to identify the set of significantly top-performing definitions. Results were reported as average F1 scores and p-values comparing key approaches.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Wilcoxon signed-rank test | Identifying the 741 EnTDefs that significantly outperform the >5 kb nearest-gene locus definition (LocDef) baseline | 87 ENCODE ChIP-seq datasets for 34 TFs | not stated |
| TF-paired Wilcoxon rank-sum test (labeled 'sum-rank' in Fig. 1 caption) | Sequential pairwise comparison of the top-ranked EnTDef against each lower-ranked EnTDef to identify the set not significantly worse (p > 0.01 threshold) | 34 TFs | not stated |
| Paired Wilcoxon test | Assessing the relative contribution of each individual enhancer-definition, extension, or linking method by comparing EnTDefs containing vs. excluding that method (Fig. 2B) | 741 top-ranked EnTDefs | not stated |
| Wilcoxon signed-rank test | Comparison of EnTDef_plus5kb (top 10) vs. nearest TSS method; top 10 EnTDefs and 5 kb LocDef vs. >5 kb LocDef; top 10 EnTDefs vs. nearest TSS (non-significant for top half) | 87 ChIP-seq datasets / 34 TFs | not stated |
| Wilcoxon rank-sum test | Comparison of top 10 EnTDefs vs. GREAT/FET/5 kb LocDef methods (average F1 = 0.47 vs 0.45, p < 0.007) | 34 TFs | not stated |
| Friedman test | Omnibus comparison of three GSE testing methods (Poly-Enrich, GREAT, Fisher's exact test with 5 kb LocDef) for overall equivalence (p = 0.91) | 34 TFs | not stated |
-
Performance was summarized as average F1 score at a single significance threshold across TFs, reported as a point estimate with no dispersion.↳ Could also: Area under the precision-recall curve (AUPRC) or AUROC, plus a standard deviation or 95% CI around mean F1 across TFs, could also be reported. — F1 depends on the significance threshold chosen; AUPRC/AUROC integrate across thresholds providing a threshold-independent summary, and dispersion measures would convey how consistently an EnTDef performs across diverse TFs rather than just on average.
-
Sequential pairwise Wilcoxon tests were performed between the top-ranked EnTDef and each lower-ranked one to identify the set not significantly worse (p > 0.01), without explicit multiplicity correction for the 1,859 comparisons.↳ Could also: A single multiplicity-corrected ANOVA or mixed-effects model with TF as a random effect, followed by Dunnett-style contrasts against the reference EnTDef, could also control the family-wise error rate across all pairwise comparisons. — Sequential pairwise tests against a reference accumulate type-I error across comparisons; a model-based approach with formal FWER control would provide tighter guarantees on which EnTDefs are genuinely equivalent to the top performer.
-
FDR correction was applied to the comparison of 1,860 EnTDefs vs. the nearest-gene baseline, but the specific procedure (e.g., Benjamini-Hochberg, Storey q-value) and the exact family of tests were not stated.↳ Could also: Explicitly naming the FDR procedure and the complete set of tests it covered could also be included. — Different FDR procedures (BH vs. Storey q-value vs. Bonferroni) have different assumptions and power; stating the method allows readers to reproduce the threshold and evaluate its appropriateness for the correlation structure among the 1,860 EnTDefs.
-
The three GSE methods (Poly-Enrich, GREAT, FET) were declared equivalent based on a non-significant Friedman test (p = 0.91).↳ Could also: Equivalence testing (e.g., two one-sided tests, TOST) or reporting confidence intervals around pairwise F1 differences could also be used to formally establish equivalence. — A non-significant omnibus test does not demonstrate equivalence; TOST or a CI-based approach would quantify the magnitude of any difference and whether it falls within a pre-specified equivalence margin, supporting a positive claim of similarity.
-
The evaluation used GO biological process annotations from the GO database as the ground truth for TF function, both to rank and to describe EnTDef performance.↳ Could also: Cross-validation with held-out TFs, or ranking on one independent benchmark (e.g., curated enhancer-gene pairs from ENCODE functional validation) while reporting performance on GO annotations separately, could also be used. — Using the same annotation source for both ranking EnTDefs and evaluating them can favor definitions that align with GO database coverage rather than with biological ground truth; an orthogonal benchmark provides a less circular performance estimate.
-
Method contribution (Fig. 2B) was assessed by comparing EnTDefs containing vs. excluding each method using paired Wilcoxon tests, reporting the percent of EnTDefs showing significant improvement.↳ Could also: A permutation-based or bootstrap importance analysis, or a factorial design decomposing variance in F1 attributable to each method factor and their interactions, could also quantify method contributions. — Reporting the percent of pairwise comparisons that are significant conflates effect size with statistical power; a variance-decomposition or effect-size approach would more directly quantify how much each method contributes to overall performance.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
FANTOM5 and ChIA-PET enhancer-gene assignment methods improve TF GO enrichment F1 score in ~70% of EnTDefs, versus Thurman correlation improving only ~9%.ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
-
Top 10 EnTDefs and 5 kb LocDef both significantly outperform >5 kb LocDef in TF GO enrichment F1 score (F1 0.47 and 0.45 vs 0.27).ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
-
EnTDefs without enhancer extension improve TF GO enrichment F1 score in ~60% of configurations versus only ~7% improvement with 1 kb extension, favoring no extension.ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
-
All ten EnTDef_plus5kb definitions significantly outperform the nearest-TSS method in TF GO enrichment F1 score (~0.05 F1 increase, p<0.0001).ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
-
EnTDefs ranked 2-19 are not significantly worse than the top-ranked EnTDef in TF GO enrichment F1 score, indicating robustness across top-ranked definitions (p>0.01).ChIP-seq human multiple cell types none 2022×1papers★ This paper is the founder (earliest)
-
Top 10 distal-only EnTDefs outperform GREAT, FET, and Poly-Enrich with 5 kb LocDef in TF GO enrichment F1 score (average F1 0.47 vs 0.45, p<0.007).ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
-
Top 40% of enhancer-target gene definitions (EnTDefs) significantly outperform the >5 kb nearest-gene location definition in TF GO enrichment F1 score (Wilcoxon FDR<0.05).ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
-
Adding FANTOM5 enhancer definition improves TF GO enrichment F1 score in >50% of EnTDefs, versus ChromHMM improving only ~5% and DNase ~24%.ChIP-seq human multiple cell types up 2022×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
- Genome-wide prediction of DNase I hypersensiti... L1 79/100
- Integrated analysis of post-transcriptional re...⚑ L1 32/100 ⚑
- Single-Cell Hi-C Technologies and Computationa... L1 80/100
- Sequencing of human genomes with nanopore tech... L1 75/100
- CRISPR/Cas9 Screens Reveal Multiple Layers of... L1 No data access
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
No individual results have been recorded for this entry yet.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a controlled DROP (no_code): the only resolvable code artifact, TrimGalore, is a generic adapter/quality-trimmer and not the enhancer→target-gene assignment / gene-set pipeline behind the paper's headline claims (Genome Biology 2022). No claims were pinned, GSE180260 was never processed, and no «our HPC» compute was run, so derivability and the core conclusion are undetermined, not refuted. The blocker is on our side — a likely text-mining false positive in code-link extraction — rather than an authors' defect or fabrication, so q4 is our-methodology and q5/q7/q8 stay yellow (cautionary) rather than red. Recommendation: re-screen the code link; if the real analysis repo is found this RU can be upgraded to a genuine reproduction attempt.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.