D2H2: diabetes data and hypothesis hub.
Part of the results reproduced; minor but material deviations remained.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
▸Reproduction agent’s raw note
D2H2 (PMID 38107655, doi:10.1093/bioadv/vbad178) is a resource/database web-tool paper from the Ma'ayan Lab. The GitHub repo MaayanLab/D2H2-site is a Flask web application (app + frontend + ETL/Docker), not analysis-pipeline code that emits numbers matching the paper. The publication reports no specific, pinnable quantitative pipeline-derived result (no fixed DGE tables with p-values/effect sizes for a named dataset, no overlap thresholds, no hypothesis counts, no benchmark metrics) - its computational outputs (differential expression, Enrichr enrichment, gene-set-overlap hypotheses) are generated on demand by the live platform at d2h2.maayanlab.cloud rather than published as static values. The P16 third-party-tool clause does not rescue it: running a standard DGE tool on the linked GSE160802 has no reported paper value to grade against (only the live site's current output, which is not a published claim and not a 1:1 reproduction). Therefore no claim is pinnable and this is a scope-level drop. NOT attempted: no «our HPC» compute was run, because there is no quantitative claim to reproduce; the repo and data (GSE160802) are public, so the obstacle is solely the absence of a pinnable reported result.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-15 ⛓ 4e468f237839
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan published diabetes omics datasets and bioinformatics tools be made more accessible and useful for knowledge discovery and data-driven hypothesis generation by integrating curated transcriptomics data, gene/gene-set query tools, and an LLM-powered chatbot into a single web portal (D2H2)?
- ★ D2H2 is a web portal hosting hundreds of curated, uniformly reprocessed diabetes-relevant transcriptomics datasets from GEO with per-study visualization, differential expression, and single-gene queries. resource
- ★ A GPT-4-powered chatbot parses free-text user queries and invokes the most relevant bioinformatics tool via its API by matching against tool descriptions defined in a JSON file. method
- ★ D2H2 generates automated hypotheses by querying randomly selected precomputed RNA-seq signatures against Rummagene gene sets and using GPT-4 to explain high gene-set overlap between publications with dissimilar abstracts. method
- Setting GPT max_tokens to 20 and temperature to 0, and validating responses against the processes.json tool definitions, mitigates LLM hallucination risk. method
- Tool definitions stored in a single JSON file allow new tools to be added and the framework to be reused with other LLM services. method
- A case study suggests two-peptide (GLP-1 and PYY3-36) treatment may increase SREBP-2 activity and enhance cholesterol metabolism in the brainstem of diabetic mice. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq / microarray (curated reanalysis) | human and mouse tissues (GEO studies) | various (curated study conditions) | differential gene expression, gene expression levels | ARCHS4 gene counts; limma, DESeq2 (PyDESeq2), Characteristic Direction |
| single-cell RNA-seq (curated reanalysis) | human and mouse tissues (GEO studies) | various (curated study conditions) | cluster identification, dimensionality reduction coordinates, expression | scanpy (Leiden, PCA, t-SNE, UMAP) |
| bulk RNA-seq (case study) | obese and diabetic mice, brainstem and paraventricular nucleus of hypothalamus | co-administration of GLP-1 and peptide YY3-36 vs vehicle control | Stat3 expression and differential gene expression / enrichment | DESeq2; Enrichr (WikiPathway) |
| single-gene expression query (volcano plot) | type 2 diabetes transcriptomics signatures | none | gene-specific P-value and fold change per T2D signature | Appyter (Gene_Expression_T2D_Signatures) |
| gene set overlap / enrichment query | PubMed Central publication-derived gene sets | none | overlapping gene sets ranked by abstract dissimilarity | Rummagene; scikit-learn |
- – D2H2 hosts curated human and mouse transcriptomics datasets accessible via dedicated study pages. 162 human and 176 mouse datasets
- – D2H2 integrates a catalog of bioinformatics tools accessible to the chatbot. n=14 tools
- ▲ Stat3 shows elevated expression in brainstem and paraventricular nucleus of hypothalamus in peptide-treated vs vehicle mice (GSE160802).
- ▲ Top 250 upregulated genes (adjusted P<0.05) from peptide treatment enrich for cholesterol metabolism terms including Cholesterol Metabolism WP4718, with SREBP-2 among upregulated genes. top 250 genes
- – Hypothesis generation queries D2H2 gene sets against PMC-derived gene sets to find high-overlap, low-abstract-similarity matches. more than 600,000 gene sets; top 100 terms ranked
- count 162 human and 176 mouse transcriptomics datasets (datasets hosted on D2H2 (December 2023))
- count 14 (tools currently available to the D2H2 chatbot (Table 1))
- count more than 600 000 gene sets (gene sets extracted from PMC supporting materials via Rummagene)
- count top 100 returned terms (Rummagene terms ranked by abstract dissimilarity for hypothesis generation)
- count top 250 upregulated genes (genes with adjusted P-value <0.05 submitted to Enrichr in case study)
- pvalue adjusted P-value <0.05 (threshold for upregulated genes in peptide-treatment DESeq2 analysis)
- other max_tokens=20, temperature=0 (OpenAI gpt-4 ChatCompletion parameters for chatbot)
- count nine example queries (example queries on D2H2 landing page chatbot interface)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This application note describes D2H2, a bioinformatics web portal aggregating curated diabetes transcriptomics datasets and integrated analysis tools; the authors conduct no primary experimental statistical analyses. Differential gene expression for curated datasets is precomputed using limma, with on-demand reanalysis available via DESeq2 (through PyDESeq2) or the Characteristic Direction method. An illustrative example applies DESeq2 to a single GEO study (GSE160802), filtering results by adjusted P-value < 0.05 before gene set enrichment via Enrichr. Abstract-level dissimilarity between publications is quantified with TF-IDF cosine distance using scikit-learn to support automated hypothesis generation.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma linear model with empirical Bayes moderation | Precomputed differential gene expression for all curated bulk RNA-seq and microarray studies | — | not stated |
| DESeq2 Wald test (via PyDESeq2) | On-demand differential expression between two user-selected conditions on study pages; illustrated with GSE160802 | — | not stated |
| Characteristic Direction method | Alternative on-demand differential expression available on study pages | — | not stated |
| Fisher's exact test / hypergeometric test (via Enrichr) | Gene set enrichment analysis applied to top differentially expressed genes in the example (GSE160802) | — | not stated |
| Leiden community detection algorithm | Cluster identification for scRNA-seq studies | — | na |
| TF-IDF cosine dissimilarity (scikit-learn) | Abstract dissimilarity scoring to select maximally divergent gene sets for GPT-4 hypothesis generation | — | na |
-
Differential expression for bulk RNA-seq studies is precomputed using limma (with voom-style count normalization implied)↳ Could also: edgeR with a quasi-likelihood F-test could also be applied to count-based RNA-seq data — edgeR models count overdispersion directly via a negative binomial GLM; cross-validating DE calls between limma-voom and edgeR is a commonly used strategy to increase confidence in reported gene lists
-
Top upregulated genes in the DESeq2 example are filtered by adjusted P-value < 0.05 alone (top 250 submitted to Enrichr)↳ Could also: A combined criterion of adjusted P-value < 0.05 and |log2 fold change| > 1 (or another effect-size threshold) is also widely used — Adding a fold-change filter reduces the influence of statistically significant but biologically small effects, which can dominate gene lists in large-sample studies
-
Abstract dissimilarity is quantified with TF-IDF cosine distance via scikit-learn to identify publications with divergent topics but overlapping gene sets↳ Could also: Dense biomedical language model embeddings (e.g., BioBERT or PubMedBERT sentence embeddings) could also be used to compute semantic similarity — Neural embeddings capture semantic relationships beyond lexical co-occurrence and may better distinguish abstracts that share vocabulary but differ conceptually, potentially surfacing more surprising cross-domain hypotheses
-
The Leiden algorithm is used for cluster identification in scRNA-seq studies↳ Could also: The Louvain algorithm or graph-based clustering with different resolution parameters could also be applied — Louvain is the direct predecessor to Leiden and is widely benchmarked; exploring a range of resolution parameters and reporting cluster stability (e.g., via bootstrapping) is a common approach to communicate sensitivity of cluster assignments
-
Gene set enrichment is performed using Enrichr's over-representation analysis (Fisher's exact test)↳ Could also: Rank-based gene set enrichment analysis (GSEA) applied to the full ranked gene list could also be used — GSEA uses the full ranked list rather than a binary DE/non-DE threshold, avoiding sensitivity to the chosen significance cutoff and detecting coordinated but modest expression changes across a pathway
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
No individual results have been recorded for this entry yet.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
D2H2 is a resource/web-tool paper (Ma'ayan Lab) whose computational outputs — differential expression, Enrichr enrichment, gene-set-overlap hypotheses — are generated on demand by the live platform and never published as static numbers, so there is no pinnable quantitative claim to put against our output. The data (GSE160802) and code (MaayanLab/D2H2-site) are public, so this is not a data-availability failure or an authors' defect — it is a legitimate scope drop (no_expected_result). Accordingly q2 is red (nothing comparable) but q5/q7/q8 are yellow rather than red: there is no deviation, no fabrication signal, and no critical discrepancy — simply nothing to grade 1:1.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.