GeMI: interactive interface for transformer-based Genomic Metadata Integration.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH TO REPRODUCE THE CORE, 1:1 ON THE PIPELINE. GeMI's published artifact (github.com/armando2603/GeMI + a live ~510MB Google-Drive GPT2-small checkpoint) was run headless on «our HPC» (CPU; torch1.6/transformers3.3.1, deterministic greedy decoding) over (a) the repo's own shipped example inputs and (b) GEO GSE11670 -- the K562 dataset the paper itself uses for its 'Basic use case', fetched via GEOparse (the harvested code link, which is GeMI's GEO-fetch dependency). RESULT: the model reproduces the paper's central described output (a structured 16-attribute key-value table per sample) end-to-end, and on GSE11670/K562 produces biologically correct, high-confidence extractions (Organism=homo sapiens, Cell Line=k562, Health status=CML, Sex=female) -- a clean reproduction of the Basic use case. NOT ATTEMPTED / NOT 1:1: the headline QUANTITATIVE claims -- Fig 6 per-attribute accuracy and Table 3 timings -- because (i) the labeled held-out test set and gold attribute labels are NOT shipped in the repo (assets are inputs only), so accuracy cannot be recomputed, and (ii) timings are hardware-specific (TPU v2 training, authors' GPU). Table 4-5 (vs external OnASSiS over 6,627 samples) skipped per 80/20. NO fabrication signal: the published checkpoint behaves exactly as described; the gap is reproducibility of the eval set, not invented numbers. Honest organism-accuracy imperfection on raw ChIP-seq inputs (C2) is consistent with the paper's own perfect-string-match metric. Not the authors' code? It IS the authors' code+model; GEOparse is its dependency, so the P16 third-party-tool clause is moot/satisfied either way. Outputs + inputs + checksums under reproduction/outputs/.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 64assessed: 2026-06-14 ⛓ 34aeee59ba7c
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan a fine-tuned, continuously-learning transformer (GPT2) model, combined with active learning and gradient-based interpretability, automatically extract structured, complete metadata (including attributes not explicitly mentioned) from the free-text descriptions of GEO genomic experiments?
- ★ GeMI is a web tool that uses a fine-tuned GPT2 model to extract 15 structured key-value attributes from free-text GEO sample metadata. resource
- ★ GeMI reformulates attribute extraction as a multitask problem (task conditioning), predicting each attribute independently rather than sequentially, unlike the prior Cannizzaro et al. model. method
- ★ GeMI incorporates a real-time active learning framework that lets the model learn continuously from user feedback, minimizing the number of samples requiring manual annotation. method
- ★ GeMI integrates a gradient-based saliency map (InputXGrad) interpretability mechanism so users can verify whether predictions are based on correct patterns. method
- ★ The model can infer attributes not explicitly stated in the text (e.g., sex, tissue type, cell type, ethnicity, disease) by exploiting learned patterns. finding
- ★ A single model trained on the union of heterogeneous Cistrome and ENCODE attributes can integrate data from different genomic sources despite minimal sample overlap. method
- ★ GeMI shows considerable improvement over the state-of-the-art tool OnASSiS, especially in downstream analysis pipelines such as CombSAFE. finding
- Among confidence-scoring strategies for text generation, using the probability of the first generated token is the most calibrated proxy for prediction correctness. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Manual metadata curation / dataset aggregation | Cistrome Data Browser experiments (ChIP-seq/DNA-binding related) | none | Cell Line, Cell Type, Tissue Type, Factor Name attributes | — |
| Batch metadata download / dataset aggregation | ENCODE DNA sequencing experiments | none | 15 experiment attributes (e.g. Classification, Biosample term name, Assay, Organism, Sex, Age, Ethnicity, Health status) | — |
| Transformer-based text extraction (fine-tuned GPT2, multitask/task-conditioned) | GEO sample free-text metadata (merged Cistrome + ENCODE corpus, retrieved via GEOparse) | model fine-tuning on combined heterogeneous attribute set | extracted attribute-value predictions and prediction confidence | GPT2 (GEOparse for retrieval) |
| Model interpretability comparison (saliency mapping) | fine-tuned GPT2 attribute-extraction model applied to GEO sample text (e.g. GSM1348947) | comparison of LIME, Attention, and Gradient (InputXGrad) interpretation methods | quality/usefulness of saliency maps highlighting words driving a given attribute prediction | Ecco library with PyTorch |
| Active learning confidence estimation | fine-tuned GPT2 model outputs on GEO metadata samples | comparison of three token-probability aggregation strategies (multiply all, minimum, first token) | calibration of confidence score against prediction correctness | — |
- – Cistrome Data Browser contains 44,843 experiments manually annotated with Cell Line, Cell Type, Tissue Type and Factor Name.
- – ENCODE contains 20,734 experiments with 15 attributes each.
- – Only a minor fraction of samples overlap between the two source datasets. 3103 out of 58 235 (about 0.05%)
- – The model correctly inferred 'Sex = Male' for a sample solely from mention of benign prostate tissue, without an explicit sex field.
- – LIME was impractical for real-time use; the Attention-based saliency approach gave disappointing interpretability despite high computational efficiency; the gradient-based InputXGrad method produced similar or better results and was selected for GeMI.
- – Multiplying token probabilities or taking the minimum token probability produced too many low confidence values even for correct predictions; using the first token's probability was better calibrated and was adopted.
- ▲ GeMI demonstrated considerable improvement over the OnASSiS tool, particularly in the context of downstream analysis pipelines (e.g. CombSAFE).
- count >4 million (total digital samples in GEO repository)
- count 44 843 (experiments in Cistrome Data Browser)
- count 20 734 (experiments in ENCODE dataset)
- other 3103 out of 58 235 (~0.05%) (overlap between Cistrome and ENCODE samples)
- count 15 (number of attributes extracted by GeMI per experiment)
- count 4 (number of attributes per Cistrome sample used to split into multiple training samples)
- count 400 tokens (30-character word limit) (preprocessing constraint imposed by GPT2 technological limits)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
GeMI is a software/database paper describing a GPT2-based NLP web tool for extracting structured metadata from GEO genomic experiments. The evaluation approach in the provided text is empirical and demonstrative: saliency-map methods (LIME, Attention, Gradient/InputXGrad) were compared informally via empirical observation of output quality, and the active learning confidence metric was selected by qualitative calibration inspection. No formal hypothesis tests, p-values, or inferential statistics appear in the provided text; the paper is a system demonstration rather than a controlled experimental study.
-
Comparison among three saliency-map approaches (LIME, Attention, InputXGrad) was conducted by informal empirical inspection rather than a structured quantitative evaluation↳ Could also: A held-out human-annotation faithfulness study or automated faithfulness metric (e.g., AOPC, sufficiency/comprehensiveness scores from DeYoung et al. 2020) could also quantify explanation quality — Quantitative faithfulness metrics would allow numeric comparison across methods and reproducible reporting of which approach best identifies the tokens actually responsible for the model's predictions
-
The three confidence-aggregation strategies (product, minimum, first-token probability) were evaluated by observing whether low confidence correlated with incorrect predictions, without a formal calibration metric↳ Could also: Expected Calibration Error (ECE) or reliability diagrams could also be computed to compare how well each strategy's confidence scores track empirical accuracy — Calibration metrics provide a scalar, reproducible summary of alignment between predicted confidence and observed correctness, making the choice of strategy easier to justify and compare across settings
-
The merged Cistrome+ENCODE dataset was used without an explicit train/validation/test split description in the provided text↳ Could also: A pre-registered or clearly documented k-fold cross-validation or hold-out split strategy could also be applied, with results reported per fold — Explicit split documentation and cross-validation allow readers to assess variance in model performance across data partitions, which is particularly relevant for active learning systems where training data composition changes over time
-
GeMI's performance is described as a 'considerable improvement' over OnASSiS without numeric effect-size or statistical comparison in the provided text↳ Could also: McNemar's test or bootstrap confidence intervals on the difference in attribute-extraction accuracy between GeMI and OnASSiS could also be reported on the same held-out set — A paired significance test on matched samples would let readers assess whether the observed accuracy difference is likely to generalize beyond the evaluated sample, complementing the descriptive comparison
-
The 0.05% overlap figure between Cistrome and ENCODE (3,103 / 58,235) is reported as a raw proportion with no uncertainty estimate↳ Could also: A 95% Wilson or Clopper-Pearson interval could also accompany the proportion to convey estimation uncertainty — Confidence intervals on overlap proportions are informative when the overlap fraction informs decisions about dataset independence and potential leakage between training and evaluation sets
-
Method selection (InputXGrad over LIME and Attention) was partly supported by citing an external benchmark paper (Atanasova et al.) rather than an in-domain evaluation on GeMI's own outputs↳ Could also: An in-domain human-evaluation study with inter-annotator agreement (e.g., Cohen's kappa) could also be conducted, asking domain experts to rate whether highlighted tokens are the correct reason for each prediction — In-domain human evaluation directly measures whether the explanation is meaningful to the target users (bioinformaticians), which may differ from faithfulness results on general NLP benchmarks
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35657113 (GeMI: transformer-based Genomic Metadata Integration)
- Paper: Serna Garcia G, Leone M, Bernasconi A, Carman MJ. GeMI: interactive interface for transformer-based Genomic Metadata Integration. Database (Oxford) 2022. PMID 35657113 · PMCID PMC9216561 · DOI 10.1093/database/baac036
- Code (authors' own): https://github.com/armando2603/GeMI (branch
incremental_model, no license, last push 2022-08-21). Not the harvestedguma44/GEOparselink — GEOparse is GeMI's GEO-fetch dependency (used inbackend/app.py::searchGEOviaget_GEO), which is why the link harvester surfaced it. Reproducing GeMI necessarily runs GEOparse on the paper's data, so the P16 "third-party tool on the paper's data" clause is satisfied either way. - Pretrained model: Google-Drive checkpoint
checkpoint-4-8+-epoch=12-val_loss=0.287.ckpt(PyTorch-Lightning ckpt of a fine-tuned GPT2-small), linked from the repo README; verified live & downloadable 2026-06-14 (Drive large-file confirm page resolves). - Demo data: GEO GSE11670 — "Transcriptional profiling of ICL670-treated K562 cells" (human myeloid leukemia cell line; 6 GSMs). This is the paper's "Basic use case" dataset.
What GeMI is (the pipeline)
A single multitask GPT2-small model (12 layers/heads, 768 dim) fine-tuned to extract
16 structured key–value attributes (Cell Line, Cell Type, Tissue Type, Assay name/type,
Target of assay, Organism, Life stage, Biosample term name, Age, Age units, Sex, Ethnicity,
Health status, Classification, Investigated as) from the free-text title/description/
characteristics of a GEO sample. Extraction = conditional text generation P(value | input, "Field:")
with greedy argmax decoding (deterministic). Implemented in backend/model.py::Predictor.generateTable.
In scope (pipeline-derived → attempted)
| # | Reported result | Pipeline | Reproducible? |
|---|---|---|---|
| R1 | Core claim: the published model extracts structured key–value metadata from GEO free text | Predictor.generateTable (GPT2-small ckpt) |
Yes — run published ckpt on inputs |
| R2 | "Basic use case" on GSE11670 (K562): GeMI fills Cell Line / Organism / etc. (Fig.) | generateTable on GSE11670 (fetched via GEOparse) | Partial — qualitative; paper shows a screenshot, no printed values |
We pin a concrete, checkable subset of R1: on the repo's own shipped example inputs
(backend/data/input_1.json), the Organism attribute is explicitly present in each input's
"Characteristics" string, so a correctly-functioning extractor must reproduce it. We report
Organism-extraction accuracy on those N examples as a deterministic correctness check of the
published pipeline (this is our checkable point, not a number printed in the paper).
Out of scope (not pipeline-reproducible 1:1 → not attempted, with reason)
| Reported result | Why not attempted |
|---|---|
| Fig 6 per-attribute accuracy (GeMI vs baselines) | The labeled held-out test set (ground-truth attribute values for Cistrome/ENCODE samples) is not shipped in the repo; the repo's frontend/src/assets/* contain inputs only, no gold labels. Without ground truth, accuracy cannot be computed → no_expected_result/docs_insufficient. |
| Table 3 training time (~10 h) & inference sec/sample | Hardware-specific: training on TPU v2 (8 cores), inference on the authors' GPU. Not reproducible 1:1 on «our HPC» CPU/GPU; a re-measured number would not be comparable. |
| Table 4–5 GeMI vs OnASSiS semantic-annotation counts (6,627 ChIP-seq samples) | Requires running the full model over 6,627 samples and the external OnASSiS tool + ontology resources; large, multi-tool, last-20% — skipped per 80/20. |
| User study (intuitiveness/usefulness, 30 users) | Human survey — not a computational pipeline (non_pipeline). |
Plan
Build a py3.7 / torch 1.6.0 / transformers 3.3.1 env inside a «our HPC» SLURM job (compute node has internet)
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The authors' own published checkpoint reproduces the paper's core contribution 1:1 — a deterministic 16-attribute key-value table per GEO sample — and yields biologically correct, high-confidence K562 extractions (Organism=homo sapiens, Cell Line=k562, Health status=CML, Sex=female), with no fabrication signal. The unverified parts are on the availability/scoping side, not the authors' integrity: Fig 6 accuracy is not recomputable because the repo ships inputs only (no gold test set), Table 3 timings are hardware-specific, and Table 4-5 was skipped per 80/20. Minor C2 organism errors (e.g. high-conf mouse→human, 'musculus' vs 'Mus musculus' string mismatch) are honest real-model behavior consistent with the paper's own perfect-string-match metric. Overall: solid reproduction of the central claim with explainable, data-availability-driven gaps in the quantitative headlines.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.