Prediction of condition-specific regulatory genes using machine learning.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1, described well enough). ConSReg (logistic-LASSO classifier of condition-specific up/down regulatory genes) reproduced faithfully on «our HPC» using the authors' pinned conda environment. The repo is fully self-contained (ships 387 DAP-seq peak files, ATAC-seq peaks, TAIR10 GFF, 40 bulk DESeq2 contrasts + 1 single-cell contrast, AND reference outputs). Bulk mean AUROC reproduced to UR=0.8241/DR=0.8217 vs repo reference 0.8242/0.8246 and the paper's reported ~0.84 (mean abs diff 0.0001-0.0029); AUPRC within 0.007. Per-contrast structure agrees (Pearson r 0.72-0.92). Single-cell AUROC/AUPRC reproduced within single-contrast stochastic variance (UR>DR, ~0.6-0.7). Regulator importance ranking reproduced strongly (Spearman ~0.85, top-30 overlap ~70%). KEY CAVEAT (honesty): ConSReg/train_test.py uses sklearn train_test_split WITHOUT random_state, so per-contrast values are inherently non-deterministic; only means + rank structure are reproducible, and they match. NOT attempted: upstream DESeq2/scRNA DE generation (derived tables shipped), Maize cross-species + nitrogen-TARGET supplementary validations. Env gotcha recorded in kartei: bioconductor data packages (GenomeInfoDbData/GO.db/DO.db/TxDb) have stale post-link download URLs -> repaired via BiocManager.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-22no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetIntegrating TF-DNA binding (DAP-seq), open chromatin (ATAC-seq), and gene expression data through a supervised machine learning framework can more accurately identify condition-specific regulatory transcription factors in plants than existing enrichment-based approaches.
- ★ ConSReg integrates expression, DAP-seq TF-DNA binding, and ATAC-seq open chromatin data into machine learning models to predict condition-specific regulatory genes method
- ★ ConSReg predicts TFs regulating differentially expressed genes with average auROC of 0.84, 23.5-25% better than enrichment-based approaches finding
- ★ Undetected genes (UDGs) are better negative training data than non-differentially expressed genes (NDEGs) finding
- ★ Including ATAC-seq data significantly improves model performance regardless of experimental condition finding
- ★ Using 3KB upstream plus 0.5KB downstream of TSS as the promoter definition gives better performance than shorter regions across data sets finding
- ★ On an independent nitrogen-response validation data set, ConSReg ranks correct regulatory TFs better than other plant tools in 61.7% of cases, three times better than existing tools finding
- ★ Applying ConSReg to Arabidopsis single-cell RNA-seq data identifies candidate regulatory genes controlling cell wall formation in the root endodermis finding
- ConSReg is implemented as an open-source Python package generalizable to other eukaryotic species, demonstrated using maize DAP-seq and ATAC-seq data resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq differential expression | Arabidopsis thaliana | abiotic stress treatments (from 26 publications) | differentially expressed genes (fold change, mean expression, FDR-adjusted P-values) | — |
| DAP-seq | Arabidopsis thaliana | none | TF binding peak regions in gene promoters | — |
| ATAC-seq | Arabidopsis thaliana | none | open chromatin peak regions and signal scores in gene promoters | — |
| single-cell RNA-seq | Arabidopsis root (endodermis, cortex, quiescent center cells) | none | differentially expressed genes between cell types (ICI-assigned clusters) | Seurat v3.0 / DEsingle |
| TF ranking validation using independent expression data set | Arabidopsis thaliana | nitrogen treatment | ranking accuracy of known regulatory TFs | — |
| DAP-seq and ATAC-seq | Zea mays (maize) | none | generalizability of ConSReg predictions to another species | — |
- ▲ ConSReg achieved average auROC of 0.84 predicting TFs regulating DEGs, 23.5-25% better than enrichment-based approaches 23.5-25% improvement; auROC=0.84
- ▲ ConSReg gave better rankings of correct regulatory TFs in 61.7% of cases on independent nitrogen-response data, three times better than other plant tools 61.7% of cases; 3-fold better
- – Undetected genes (UDGs) outperform non-differentially expressed genes (NDEGs) as negative training sets
- ▲ Including ATAC-seq data improved model performance across all tested conditions
- ▲ 3KB upstream + 0.5KB downstream promoter definition outperformed shorter (e.g. 500bp) promoter definitions
- – ConSReg identified candidate regulatory genes for cell wall formation in root endodermis from single-cell data
- mean average AUC-ROC = 0.84 (cross-validation across expression data sets from 26 publications)
- fold_change 23.5-25% better than enrichment-based approaches (comparison of ConSReg auROC vs enrichment methods)
- other 61.7% of cases with better TF ranking (independent nitrogen-response validation data set comparison)
- fold_change three times better than other plant tools (ranking performance vs TF2Network, Cistome, PlantPAN 3.0)
- count >500 positive and >500 negative genes (threshold for selecting differential contrasts in evaluation data sets A and B)
- other 3KB upstream + 0.5KB downstream of TSS (best-performing promoter region definition)
- other normalization factor of 10,000 (Seurat log-normalization parameter for single-cell RNA-seq data)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes ConSReg, a machine learning method that integrates gene expression, DAP-seq binding, and ATAC-seq open-chromatin data to predict condition-specific regulatory transcription factors, framed as a supervised binary classification problem (DEGs vs. negative gene sets). Model performance is assessed primarily via cross-validation using average area under the ROC curve (AUC-ROC) across a compendium of expression data sets, and compared to enrichment-based tools and other published methods using percentage improvements and ranking-based comparisons on an independent validation set. Differential expression calls that feed the classifier's training labels were made using the DEsingle package for single-cell data (pairwise cluster comparisons) and FDR-adjusted p-values from bulk RNA-seq differential contrasts. The provided methods text is truncated mid-section, so some downstream statistical procedures are not fully described.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| DEsingle (differential expression test for single-cell/UMI count data) | pairwise comparisons between endodermis, cortex, and quiescent center cell clusters from single-cell RNA-seq | — | not stated |
| Differential expression testing with FDR-adjusted P-values (specific test/tool not specified in excerpt) | differential contrasts between control and treatment replicate groups in bulk RNA-seq data sets | — | not stated |
| Cross-validation with AUC-ROC as the evaluation metric | evaluating ConSReg's classification of TF-regulated vs. non-regulated genes across a compendium of expression data sets | — | na |
-
Model performance across data sets is summarized as a single average AUC-ROC (0.84) and as percentage improvements over enrichment-based approaches and other tools.↳ Could also: Reporting the distribution of per-dataset AUC-ROC values (e.g., mean with SD or a bootstrap/95% CI) alongside the average — This would convey how consistent the improvement is across the compiled data sets rather than only the central tendency.
-
Comparisons between ConSReg and other tools/approaches are expressed as percentage or fold improvements (e.g., '23.5–25% better', 'three times better') and ranking success rates.↳ Could also: A paired non-parametric test (e.g., Wilcoxon signed-rank test) across matched data sets or contrasts — This would provide a formal statistical framework (p-value or effect size with CI) for the magnitude of improvement claimed.
-
Differential expression for single-cell clusters was called using DEsingle, and bulk RNA-seq contrasts used FDR-adjusted p-values from an unspecified method.↳ Could also: Cross-checking DEG calls with alternative tools such as MAST or edgeR/DESeq2/limma-voom — Comparing DEG sets across methods can show how sensitive downstream training-label definitions are to the specific DE tool chosen.
-
The comparison of negative training set types (UDGs vs. NDEGs vs. LEGs) is evaluated empirically through resulting model performance.↳ Could also: A paired statistical comparison (e.g., paired t-test or Wilcoxon signed-rank test) of AUC-ROC differences between negative-set strategies across the same contrasts — This would quantify whether the observed advantage of UDGs as negative training data is systematic versus specific to particular data sets.
-
Model evaluation relies on cross-validation with AUC-ROC as the primary metric.↳ Could also: Nested cross-validation or an independent held-out test split, with per-fold variance reported — This can separate variance introduced by hyperparameter/feature selection from the estimate of generalization performance.
-
ATAC-seq and promoter-length choices are justified by improved model performance across data sets rather than by a formal enrichment or association test in this excerpt.↳ Could also: A statistical enrichment test (e.g., Fisher's exact test or a permutation test) comparing motif/binding site density in ATAC-seq peaks versus background regions — This would offer a complementary, hypothesis-testing-based line of evidence alongside the classifier performance comparison.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-32329779 (ConSReg)
Paper: Song et al. 2020, Prediction of condition-specific regulatory genes using
machine learning, Nucleic Acids Res 48(11):e62. PMID 32329779 / PMC7293043 /
doi:10.1093/nar/gkaa264.
Code: https://github.com/LiLabAtVT/ConSReg (Python package ConSReg, v1.1.7).
Data accession: GEO GSE122687 (+ GSE123013) — Arabidopsis root scRNA-seq.
What ConSReg does
Integrates TF binding (DAP-seq, 387 TFs), open chromatin (ATAC-seq) and a
differential-expression contrast (DESeq2 output) into a per-gene feature matrix,
then trains a regularized classifier (default logistic LASSO, lrlasso, via R
gglasso through rpy2) to predict whether a gene is up-/down-regulated in a given
condition. Performance is reported as AUROC/AUPRC under 5×-repeated train/val/test
cross-validation; regulator importance scores rank candidate condition-specific TFs.
In scope (pipeline-derived, attempted) — the repo ships ALL inputs + reference outputs
| id | result | pipeline | input (shipped in repo) | reference to compare |
|---|---|---|---|---|
| C1 | Bulk RNA-seq classification AUROC (mean ≈ 0.84, "23.5–25% better than enrichment"; Fig 2 / Results) | ConSReg lrlasso, rep=5, neg=udg, up_tss=3000/down_tss=500, DAP+ATAC |
data/diff_evalB/ (40 DESeq2 contrasts), 387 dap_seq_all_peaks/*.narrowPeak, atac_seq_all_peaks/all_merged.bed, TAIR10 GFF |
repo results/bulk_analysis/auroc_result.csv (per-contrast) + paper "average AUC–ROC 0.84" |
| C2 | Bulk AUPRC | same | same | repo results/bulk_analysis/auprc_result.csv |
| C3 | Single-cell (cortex vs endodermis) AUROC/AUPRC | ConSReg lrlasso, rep=5 |
data/diff_single_cell/cortext-endodermis.csv + DAP+ATAC |
repo results/single_cell_analysis/auroc_result.csv (UR 0.655 / DR 0.604) |
| C4 | Regulator importance scores (ranked TFs) | compute_imp_score(n_resampling=200) |
as C1 | repo results/bulk_analysis/imp_score_{UR,DR}.csv (secondary, qualitative) |
Grading note (honesty): ConSReg/train_test.py calls sklearn.train_test_split
without random_state → the CV splits are stochastic and not seeded. Therefore
per-contrast AUROC values cannot be reproduced bit-for-bit even with identical code +
data. The faithful, reproducible target is the mean AUROC/AUPRC across contrasts
(rep=5 + ~40 contrasts averages out the noise) compared to (a) the paper's reported
≈0.84 and (b) the repo's shipped reference means, graded within-tol.
Out of scope (not attempted; not pipeline-reproducible from shipped artifacts)
- Upstream generation of the DESeq2 contrasts from the 26 source studies' raw reads,
and the scRNA-seq DEG calling that produced
cortext-endodermis.csv(the repo ships the derived DEG tables; raw read processing is upstream wet-lab/external). - DAP-seq / ATAC-seq peak calling (peaks shipped pre-called).
- Maize cross-species application + nitrogen-response TARGET validation (Supplementary Tables S7/S8) — attempt only if primary results land and time permits.
- All wet-lab / manual interpretation (MYB63/MYB107 biology).
Environment
Authors ship a fully-pinned environment.yml (conda, python 3.6.15, r-base 4.1.3,
rpy2 2.8.6, gglasso, RRF, ChIPseeker; scikit-learn 0.19.1, numpy 1.19, scipy 1.1.0,
pandas 0.21.1). CoReg (devtools::install_github) is only needed for lglasso group
lasso / get_feature_group; the lrlasso notebooks do not call it → omitted.
Built inside a SLURM job on «our HPC» compute nodes («infra»).
</content>
</invoke>
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Clean reproduction. ConSReg's repo is fully self-contained (ships all inputs plus the authors' own reference outputs), and our independent rebuild on the authors' pinned conda environment reproduced the headline bulk mean AUROC ~0.84 (UR=0.8241/DR=0.8217), AUPRC within 0.007, single-cell ~0.6 with UR>DR, and the ranked-regulator signal (Spearman 0.85/0.86, ~70% top-30 overlap). The only deviations are sub-tolerance and stochastic, caused by train_test_split being called without random_state, so per-contrast values vary but the reproducible targets (means + rank correlations) match. No fabrication concern — every reported value is derivable from the shared code and data. All eight axes are green.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.