D2H2: diabetes data and hypothesis hub.
Part of the results reproduced; minor but material deviations remained.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
▸Reproduction agent’s raw note
D2H2 (PMID 38107655, doi:10.1093/bioadv/vbad178) is a resource/database web-tool paper from the Ma'ayan Lab. The GitHub repo MaayanLab/D2H2-site is a Flask web application (app + frontend + ETL/Docker), not analysis-pipeline code that emits numbers matching the paper. The publication reports no specific, pinnable quantitative pipeline-derived result (no fixed DGE tables with p-values/effect sizes for a named dataset, no overlap thresholds, no hypothesis counts, no benchmark metrics) - its computational outputs (differential expression, Enrichr enrichment, gene-set-overlap hypotheses) are generated on demand by the live platform at d2h2.maayanlab.cloud rather than published as static values. The P16 third-party-tool clause does not rescue it: running a standard DGE tool on the linked GSE160802 has no reported paper value to grade against (only the live site's current output, which is not a published claim and not a 1:1 reproduction). Therefore no claim is pinnable and this is a scope-level drop. NOT attempted: no «our HPC» compute was run, because there is no quantitative claim to reproduce; the repo and data (GSE160802) are public, so the obstacle is solely the absence of a pinnable reported result.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-15 ⛓ 4e468f237839
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper does not test a scientific hypothesis but presents a platform (D2H2) built on the premise that curating diabetes-relevant omics datasets and coupling them with LLM-powered tools (chatbot and automated hypothesis generation) can make published data and bioinformatics tools more accessible for knowledge discovery in diabetes research.
- ★ D2H2 is a web-based portal integrating hundreds of curated diabetes-relevant transcriptomics datasets with bioinformatics tools for gene/gene set queries resource
- ★ The D2H2 chatbot uses GPT-4 to parse free-text user queries and invoke the most relevant tool/API based on a predefined JSON tool catalog (processes.json) method
- ★ D2H2's hypothesis generation module selects random precomputed bulk RNA-seq gene set signatures, finds highly overlapping but abstract-dissimilar gene sets via Rummagene, and uses GPT-4 to speculate a mechanistic explanation method
- ★ D2H2 hosts 162 human and 176 mouse curated transcriptomics datasets with dedicated interactive report pages (as of December 2023) resource
- Case study: co-administration of GLP-1 and PYY3-36 peptides in obese/diabetic mice is associated with elevated Stat3 expression and upregulation of cholesterol metabolism genes (including SREBP-2) in the brainstem/hypothalamus finding
- ★ D2H2 queries curated gene set signatures against more than 600,000 gene sets extracted from PubMed Central supplementary materials via Rummagene resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| differential gene expression analysis (DESeq2/PyDESeq2) | mouse brain (brainstem, paraventricular nucleus of hypothalamus), GSE160802 | GLP-1 and PYY3-36 peptide co-administration vs vehicle control in obese/diabetic mice | differentially expressed genes, adjusted P-value, fold change | DESeq2 via PyDESeq2 Python package |
| gene set enrichment analysis | top 250 upregulated genes from GSE160802 mouse brain study | none (downstream analysis of above DE result) | enriched pathway terms (e.g., Cholesterol Metabolism WP4718) | Enrichr |
| bulk RNA-seq and microarray reanalysis (curated GEO studies) | human and mouse tissues, various diabetes-related studies | study-dependent (drug, diet, genotype, disease state) | gene counts, precomputed differential expression (limma), boxplot expression across conditions | ARCHS4, limma, Characteristic Direction, DESeq2 |
| single-cell RNA-seq analysis | curated scRNA-seq GEO studies (human and mouse) | none/other (study-dependent) | cluster identification, dimensionality reduction coordinates | scanpy (Leiden clustering, PCA, t-SNE, UMAP) |
| gene set overlap / hypothesis generation query | D2H2 precomputed bulk RNA-seq signatures vs PMC supplementary gene sets | none | overlapping gene sets ranked by abstract dissimilarity; GPT-4-generated hypothesis text | Rummagene, scikit-learn, GPT-4 |
| single-gene expression query (volcano plot) | type 2 diabetes transcriptomics signatures | none | gene-specific P-value and fold change across T2D signatures | Appyter (Gene Expression T2D Signatures) |
- ▲ Stat3 expression was elevated in the brainstem and paraventricular nucleus of the hypothalamus in peptide-treated mice compared to vehicle control
- ▲ Top 250 upregulated genes (adjusted P<0.05) from DESeq2 analysis were enriched for cholesterol metabolism terms, including Cholesterol Metabolism WP4718
- ▲ SREBP-2, a regulator of cholesterol metabolism, was among the significantly upregulated genes and a member of multiple cholesterol-related gene sets
- – D2H2 currently hosts 162 human and 176 mouse transcriptomics datasets with dedicated study report pages 162 human, 176 mouse datasets
- – 14 tools are currently available to the D2H2 chatbot spanning gene, gene set, term, and study metadata queries n=14 tools
- count 162 human datasets (curated human transcriptomics datasets hosted on D2H2 as of December 2023)
- count 176 mouse datasets (curated mouse transcriptomics datasets hosted on D2H2 as of December 2023)
- count n = 14 tools (bioinformatics tools currently available to the D2H2 chatbot)
- count more than 600,000 gene sets (gene sets extracted from PMC supplementary materials queried via Rummagene for hypothesis generation)
- pvalue adjusted P<0.05 (threshold for selecting top upregulated genes submitted to Enrichr in the GSE160802 case study)
- other top 100 terms ranked by abstract dissimilarity (daily automated hypothesis generation process using Rummagene results)
- other max_tokens=20, temperature=0 (OpenAI GPT-4 ChatCompletion API parameters used for tool selection in the chatbot)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This application note describes D2H2, a bioinformatics web portal aggregating curated diabetes transcriptomics datasets and integrated analysis tools; the authors conduct no primary experimental statistical analyses. Differential gene expression for curated datasets is precomputed using limma, with on-demand reanalysis available via DESeq2 (through PyDESeq2) or the Characteristic Direction method. An illustrative example applies DESeq2 to a single GEO study (GSE160802), filtering results by adjusted P-value < 0.05 before gene set enrichment via Enrichr. Abstract-level dissimilarity between publications is quantified with TF-IDF cosine distance using scikit-learn to support automated hypothesis generation.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma linear model with empirical Bayes moderation | Precomputed differential gene expression for all curated bulk RNA-seq and microarray studies | — | not stated |
| DESeq2 Wald test (via PyDESeq2) | On-demand differential expression between two user-selected conditions on study pages; illustrated with GSE160802 | — | not stated |
| Characteristic Direction method | Alternative on-demand differential expression available on study pages | — | not stated |
| Fisher's exact test / hypergeometric test (via Enrichr) | Gene set enrichment analysis applied to top differentially expressed genes in the example (GSE160802) | — | not stated |
| Leiden community detection algorithm | Cluster identification for scRNA-seq studies | — | na |
| TF-IDF cosine dissimilarity (scikit-learn) | Abstract dissimilarity scoring to select maximally divergent gene sets for GPT-4 hypothesis generation | — | na |
-
Differential expression for bulk RNA-seq studies is precomputed using limma (with voom-style count normalization implied)↳ Could also: edgeR with a quasi-likelihood F-test could also be applied to count-based RNA-seq data — edgeR models count overdispersion directly via a negative binomial GLM; cross-validating DE calls between limma-voom and edgeR is a commonly used strategy to increase confidence in reported gene lists
-
Top upregulated genes in the DESeq2 example are filtered by adjusted P-value < 0.05 alone (top 250 submitted to Enrichr)↳ Could also: A combined criterion of adjusted P-value < 0.05 and |log2 fold change| > 1 (or another effect-size threshold) is also widely used — Adding a fold-change filter reduces the influence of statistically significant but biologically small effects, which can dominate gene lists in large-sample studies
-
Abstract dissimilarity is quantified with TF-IDF cosine distance via scikit-learn to identify publications with divergent topics but overlapping gene sets↳ Could also: Dense biomedical language model embeddings (e.g., BioBERT or PubMedBERT sentence embeddings) could also be used to compute semantic similarity — Neural embeddings capture semantic relationships beyond lexical co-occurrence and may better distinguish abstracts that share vocabulary but differ conceptually, potentially surfacing more surprising cross-domain hypotheses
-
The Leiden algorithm is used for cluster identification in scRNA-seq studies↳ Could also: The Louvain algorithm or graph-based clustering with different resolution parameters could also be applied — Louvain is the direct predecessor to Leiden and is widely benchmarked; exploring a range of resolution parameters and reporting cluster stability (e.g., via bootstrapping) is a common approach to communicate sensitivity of cluster assignments
-
Gene set enrichment is performed using Enrichr's over-representation analysis (Fisher's exact test)↳ Could also: Rank-based gene set enrichment analysis (GSEA) applied to the full ranked gene list could also be used — GSEA uses the full ranked list rather than a binary DE/non-DE threshold, avoiding sensitivity to the chosen significance cutoff and detecting coordinated but modest expression changes across a pathway
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
No individual results have been recorded for this entry yet.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
D2H2 is a resource/web-tool paper (Ma'ayan Lab) whose computational outputs — differential expression, Enrichr enrichment, gene-set-overlap hypotheses — are generated on demand by the live platform and never published as static numbers, so there is no pinnable quantitative claim to put against our output. The data (GSE160802) and code (MaayanLab/D2H2-site) are public, so this is not a data-availability failure or an authors' defect — it is a legitimate scope drop (no_expected_result). Accordingly q2 is red (nothing comparable) but q5/q7/q8 are yellow rather than red: there is no deviation, no fabrication signal, and no critical discrepancy — simply nothing to grade 1:1.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.