RegulonDB 11.0: Comprehensive high-throughput datasets on transcriptional regulation in Escherichia coli K-12.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
- ✓Reported values were directly comparable
- 🔴Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
RegulonDB 11.0 is a database/resource paper; in-scope reproducible results are the Table-1 HT dataset/object counts. KEY FINDINGS: (1) the cited archival data DOI zenodo:6376425 is EMPTY (526-byte docker-app pointer), so the only data source is the LIVE portal, now at v14.5.0 (4 years past v11.0) via its GraphQL API. (2) Despite that evolution, reproduction is strong: 7/20 claims EXACT - 6 of 8 dataset counts (RNA-seq 1864, ChIP-exo 94, gSELEX 164, TU 5, TSS 16, TTS 5) plus ChIP-seq TFBS sites 5108 (from exactly 28 in-house datasets, matching the paper). (3) TU/TSS/TTS/gene object counts reproduce within ~2-10% as lower bounds (API pagination drops rows at high page sizes - validated by re-counting at limit=100). (4) ChIP-seq grew (29->168 datasets, peaks->20990) and DAP shrank (215->107) - documented DB evolution, not fabrication. (5) Author peak collections (ChIP-exo/gSELEX/DAP) are not exposed per-dataset through the API -> those 3 counts uncheckable. NOT attempted: CRF growth-condition F1 (no training data/code), manual/LT curation, web-app infra. No fabrication detected; the central reproducibility gap is archival (no frozen v11.0 deposit), recorded honestly. Verdicts provisional pending human audit.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ dcb8f45b4007
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper addresses whether a single resource can comprehensively integrate the large and rapidly growing body of high-throughput datasets on transcriptional regulation in Escherichia coli K-12 (TF binding, transcription start/termination sites, transcription units, and gene expression) in a way that is comparable, traceable, and reproducible, since no such resource previously existed.
- ★ RegulonDB 11.0 is a radical upgrade offering access to more than 2000 high-throughput datasets on transcriptional regulation in E. coli K-12. resource
- ★ The resource integrates TF binding data from ChIP-seq, ChIP-exo, gSELEX and DAP-seq together with transcription start site, transcription termination site, transcription unit, and RNA-seq expression collections in a single place. resource
- ★ For ChIP-seq, both author-reported data and uniformly in-house reprocessed data are provided to enhance comparability, traceability, and reproducibility. method
- ★ An NLP and machine-learning assisted curation strategy was implemented to automatically extract detailed experimental growth conditions from metadata. method
- ★ No comprehensive resource previously existed that facilitated access to the diverse wealth of high- and low-throughput data on transcriptional regulation in E. coli K-12 in one place. finding
- New browsing and visualization tools were added, including a nucleotide-resolution genome viewer and a metadata-query interface. resource
- gSELEX binding datasets were curated using defined or inferred intensity thresholds, yielding 164 TFBS datasets for 121 TFs. finding
- A biotin DAP-seq collection covering 215 TFs was incorporated from the supplementary material of a prior study. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| ChIP-seq | E. coli K-12 MG1655 | various TFs (in vivo binding) | TF binding peaks/TFBS genomic coordinates | SRA raw reads processed with cutadapt, Bowtie2, SnakeChunks/snakemake |
| ChIP-exo | E. coli K-12 | TFs including OxyR, SoxR, SoxS, UvrY, and others | TF binding sites | proChIPdb |
| gSELEX | E. coli K-12 (in vitro) | individual TFs | binding intensity (%) relative to peak, target gene, peak location | TEC database |
| DAP-seq (biotin-DNA affinity purification sequencing) | E. coli K-12 (in vitro) | individual TFs | TF binding sites | — |
| RNA-seq (bulk expression profiling) | E. coli K-12 | various growth conditions and/or genetic backgrounds | gene expression profiles | — |
| 5′-end-protected RNA-seq variants | E. coli K-12 | none/various | transcription start sites at nucleotide resolution | — |
| RNA-seq (full-length transcript mapping) | E. coli K-12 | none/various | transcription termination sites and transcription units | — |
- ▲ Total high-throughput dataset collection expanded from 99 datasets (2019) to more than 2000 datasets. >20-fold
- ▲ More than 500 high-throughput datasets for TF DNA-binding were incorporated.
- ▲ 1864 RNA-seq datasets generated under different growth conditions and/or genetic backgrounds were incorporated.
- – 164 TFBS datasets corresponding to 121 different TFs were generated from gSELEX data.
- – A collection of experiments and metadata for 215 TFs was obtained using biotin DAP-seq.
- – 185 raw data files from 28 ChIP-seq datasets associated with 11 TFs were uniformly reprocessed.
- – RegulonDB was accessed an average of ~16300 times per year over the last 4 years.
- – 63 gSELEX datasets (41 TFs) were built using defined thresholds; 94 additional datasets (74 TFs) used the top 40 binding-intensity targets as cutoff.
- count >2000 high-throughput datasets (total HT datasets available in RegulonDB 11.0)
- count >500 (HT datasets for TF DNA-binding)
- count 1864 (RNA-seq datasets across growth conditions/genetic backgrounds)
- count 164 TFBS datasets (121 TFs) (gSELEX-derived TFBS collection)
- count 215 TFs (DAP-seq TF collection)
- count 28 ChIP-seq datasets, 185 raw files, 11 TFs (uniformly in-house processed ChIP-seq data)
- mean ~16300 accesses/year (average RegulonDB portal access rate over the last 4 years)
- count 63 datasets (41 TFs) with defined threshold; 94 datasets (74 TFs) with top-40 cutoff (construction method for gSELEX datasets)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a resource/database paper describing the curation, uniform processing, and integration of high-throughput datasets (ChIP-seq, ChIP-exo, gSELEX, DAP-seq, RNA-seq) for E. coli K-12 transcriptional regulation into RegulonDB, rather than a primary hypothesis-testing study. Statistical content is limited to aggregating results (including p-values for differential expression, peak-calling/motif-prediction statistics) as reported in the original source publications, alongside description of uniform bioinformatics pipelines (e.g., read trimming and alignment) used to reprocess a subset of ChIP-seq data. No dedicated inferential statistical framework for hypothesis testing across the aggregated collection is described in the available text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Unspecified statistical test for differential expression (test not named) | Flagging target genes as showing 'changes in expression and a significant p-value for differential expression' when ChIP-seq experiments were linked to gene expression data in the same source publication, to annotate TF function as activator or repressor | — | not stated |
| Unspecified peak-calling/motif-prediction statistics (test not named) | Curation of TF binding peak and TFBS features from ChIP-seq/ChIP-exo/gSELEX/DAP-seq source publications | — | not stated |
-
Significance calls for differential expression used to flag TF-target regulatory interactions were taken as reported by the original source publications, each presumably using its own test and threshold, without recomputation or a unified correction across the aggregated collection.↳ Could also: A unified reanalysis applying one differential expression method (e.g., DESeq2 or edgeR) with a single multiple-testing correction (e.g., Benjamini-Hochberg FDR) across all curated RNA-seq datasets — This would let significance thresholds be compared consistently across datasets originating from different labs and pipelines, rather than relying on heterogeneous criteria from each source study.
-
For 94 gSELEX datasets not analysed by their original authors, target selection used a fixed rank-based cutoff (the top forty binding intensities) rather than a statistically derived threshold.↳ Could also: Deriving a cutoff from the empirical intensity distribution (e.g., a percentile or FDR-based threshold estimated from a null/background model) — A distribution-based threshold can adapt to the number of true binding events per TF, whereas a fixed top-N rank may include weaker signals for TFs with few real targets or exclude true targets for TFs with many.
-
ChIP-seq peak/TFBS statistical values (from peak calling or motif prediction) were curated as provided by each publication's own pipeline and settings.↳ Could also: Uniform reprocessing of all raw ChIP-seq data with one peak-calling tool and consistent significance threshold (e.g., MACS2 with a fixed q-value cutoff), as was done for the subset of 28 datasets reprocessed with SnakeChunks — Applying one tool and threshold across all datasets, rather than only a subset, would make peak significance and counts more directly comparable across the full collection.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35584008 (RegulonDB 11.0)
Paper: Tierrafría et al. 2022, Microb Genom 8(5):000833. RegulonDB 11.0: Comprehensive high-throughput datasets on transcriptional regulation in E. coli K-12. PMCID PMC9465075 · DOI 10.1099/mgen.0.000833.
Nature of paper: A database/resource paper. Most reported numbers are
database content counts (Table 1) produced by uniform-processing pipelines
applied to publicly-deposited raw HT data. This is exactly the "third-party tool
on the paper's data" case the brief blesses (P16): the genuine analysis code is
the SnakeChunks ChIP-seq pipeline + standard tools, applied to GEO/SRA reads.
Code / data artifacts located
- Doc repo: github.com/PGC-CCG/RegulonDB-HT — 99.7% HTML; it is the manual (Manual/, Mapping/README, Uniformized-Data/), not runnable analysis code.
- Pipeline repo (real code): github.com/PGC-CCG/SnakeChunks — Snakemake NGS framework (R 69%, Python 25%) for ChIP-seq/RNA-seq. Maintainers Rioualen & van Helden. This is the ChIP-seq uniform-processing pipeline.
- App repos: github.com/regulondbunam/{RegulonDBHT-Web,GraphQL-api, RegulonDB-wdpservice} — the web app, not analysis.
- Zenodo 6376425 ("Data"): ⚠️ contains ONLY
description.json(526 B) — metadata for a Docker app image (regulondb-app), pointing at the web-app repos. It does NOT contain the HT object data. (See dataset_profile.) - Actual data portal: https://regulondb.ccg.unam.mx → "Integrated Views & Tools" → "RegulonDB-HT datasets" (downloadable object files). Raw reads from GEO/SRA/ArrayExpress/DEE2/proChIPdb/TEC.
Pipelines described (Methods)
- ChIP-seq uniform processing: cutadapt (q & len threshold 20) → Bowtie2
(local mode, genome NC_000913.3 MG1655) → MACS3 (
--nomodel --shift 0 --extsize 200, q-val 1e-3) → RSAT matrix-scan (motifs from RegulonDB 10.5, thresholds from RSAT matrix-quality) for TFBS. → peaks + sites. - RNA-seq: HISAT2 → DESeq normalization (geometric-mean); QC PASS tagging.
- Growth-condition extraction: Conditional Random Field (CRF) trained on 228 SOFT files → MCO term mapping.
Reported pipeline-derived results (Table 1) — reproduction targets
| Object | Strategy | Datasets | Objects (reported) |
|---|---|---|---|
| Gene expression | RNA-seq | 1864 | 4618 avg genes/dataset |
| ChIP-seq (TF binding) | ChIP-seq | 29 | 6585 peaks; 5108 sites |
| ChIP-exo | ChIP-exo | 94 | 23170 peaks |
| gSELEX | gSELEX | 164 | 35022 peaks |
| DAP-seq | DAP-seq | 215 | 19540 peaks |
| Transcription units (TU) | RNA-seq | 5 | 12347 |
| TSS | RNA-seq/dRNA-seq | 16 | 68049 |
| TTS | RNA-seq | 5 | 5326 |
| CRF growth-cond. | NLP/CRF | (228 SOFT) | F1 0.81 CV / 0.83 test; ~83% MCO |
IN SCOPE (attempt)
- Deposited-object counts vs Table 1 (PRIMARY, cleanest 1:1). Download the uniformized HT object files (TSS/TTS/TU/ChIP peaks/sites) from the RegulonDB-HT portal onto «infra» and count objects per type, comparing to Table 1. This directly tests whether the released data delivers the reported counts.
- ChIP-seq pipeline reproduction (HARDER, stretch). For ONE ChIP-seq dataset with a known reported peak count + SRA accession, run SnakeChunks-equivalent (cutadapt→bowtie2→MACS3 with the exact params) on «our HPC» and compare the reproduced peak count to the deposited/reported value.
OUT OF SCOPE (not attempted, why)
- Manual/LT curation, regulons, the curated knowledge graph — wet-lab/manual.
- The web app / MongoDB / GraphQL infrastructure — not a computational result.
- CRF growth-condition F1 — needs the 228 annotated SOFT training files + CRF code, neither shipped in the located repos (revisit if found).
- Full re-processing of all 1864 RNA-seq / 502 TF-binding datasets — infeasible & unnecessary; the per-type counts (item 1) capture the pipeline output.
Hard blocker as of start
«our HPC»/VPN tunnel DOWN (ssh timeout to «host».«infra».uni-hamburg.de:22). All data download + compute i
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.