Public Omics Explorer (POE): Enabling integrative semantic search across GEO omics datasets based on PubMed publications.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Any deviation was negligible
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the ONE shipped, re-runnable artifact 1:1. POE is a software/platform paper; its only reproducible computational kernel is reproduce_sbiobert_distance.py (the SBioBERT embedding + L2-distance core of the semantic search). Ran the authors' own script verbatim (pinned commit e397c03d) on a «our HPC» compute node against the paper's GSE298584 worked example, and independently re-implemented the same kernel as a cross-check. Confirmed EXACT: embedding dimension = 768 (C1), L2-normalized embeddings -> norm 1.0 and identical-text distance 0.0 (C3). The distance computation reproduces deterministically (Δ=0.0) and the shipped script and reimplementation agree exactly (squared-L2 = 1.084), but the paper reports no specific distance number, so C2 is graded partial (kernel reproduced, no paper number to match). Minor documentation discrepancy: paper Methods cites HF model dmis-lab/sentence-biobert while the shipped script loads pritamdeka/S-BioBert-snli-multinli-stsb; both are 768-d SBioBERT sentence-transformers (dimension agrees), not a fabrication. NOT attempted (out of scope, not shippable pipelines): production-DB coverage counts (GSE 262,173 / GSM ~9.68M / 130,548 PubMed links, scraped daily from live GEO/ENA/PubMed), the FAISS index benchmark table (Recall/Precision/latency/memory, needs the authors' ~250k-vector private index + their hardware), and the human relevance study (30 queries, Cohen's kappa 0.79, Wilcoxon p<0.05, expert 0-10 scoring). Env: sentence-transformers 3.4.1, torch 2.12.0+cpu, transformers 4.57.6, numpy 1.26.4, python 3.11.15. All heavy compute on «our HPC»; «host» holds results only.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 83assessed: 2026-06-14 ⛓ 70b438f86564
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a literature-informed semantic search platform that links GEO omics datasets to their associated PubMed publications (via biomedical embeddings) improve dataset discovery relative to standard keyword-based GEO search?
- ★ POE is a web platform that semantically links GEO datasets and ENA records through their associated PubMed publications for literature-informed dataset retrieval resource
- ★ POE employs the biomedical SBioBERT model to generate 768-dimensional dense vector embeddings of publication text, indexed with FAISS for context-aware retrieval method
- ★ POE achieved statistically higher relevance scores than standard GEO keyword search across benchmark biological queries finding
- ★ POE supports natural-language semantic queries, structured filters (organism, experiment type, library strategy, sample type, extracted molecule, publication year), accession-based retrieval (GSE, PMID, DOI), and a RESTful API method
- The FAISS IndexFlatL2 configuration provides exact nearest-neighbor search with perfect recall/precision serving as the ground-truth baseline, while approximate indexes trade accuracy for speed/memory method
- POE indexes data daily and does not store omics datasets locally, instead redirecting users to source repositories (GEO, ENA/SRA) resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Semantic embedding and FAISS indexing of publication text | PubMed titles/abstracts linked to GEO datasets | none | 768-dimensional dense vector embeddings / nearest-neighbor retrieval | SBioBERT (dmis-lab/sentence-biobert) via SentenceTransformers; FAISS IndexFlatL2 |
| Comparative quantitative retrieval evaluation (overlap analysis) | 30 natural-language biological queries across 5 thematic categories; PubMed-linked GEO datasets | none | GEO-unique, POE-unique, and intersection set sizes on aligned equal-sized result sets | — |
| Qualitative manual relevance scoring | 9 of 30 queries with >4 unique datasets; unique datasets from GEO and POE | none | Relevance score 0-10 from 5-criterion rubric, rated by two experts | — |
| FAISS index benchmarking | 100 representative semantic search queries from biomedical abstracts | none | Recall@10, Precision@10, average query time, memory usage (top 10 neighbors) | FAISS IndexFlatL2, IndexIVFFlat, IndexPQ, IndexHNSW |
- ▲ POE achieved statistically higher relevance than GEO keyword search across benchmark queries p<0.05
- – Two expert evaluators showed substantial inter-rater agreement in relevance scoring Cohen's κ=0.79
- – Wilcoxon signed-rank test showed expert ratings did not differ significantly, indicating reliable manual assessment p=0.097
- ▲ POE typically yields more results than GEO for a given query, requiring truncation to equal-sized sets for comparison
- – Approximate FAISS index methods offered substantial efficiency gains over IndexFlatL2 baseline
- other κ = 0.79 (Cohen's kappa, inter-rater agreement) (Agreement between two expert relevance evaluators)
- pvalue p = 0.097 (Two-tailed Wilcoxon signed-rank test comparing two experts' total scores (no significant difference))
- pvalue p < 0.05 (Wilcoxon signed-rank test: POE relevance scores higher than GEO)
- count 30 (Natural-language biological queries used in the benchmark)
- count 9 out of 30 (Queries with >4 unique datasets included in qualitative relevance evaluation)
- count 768 (Dimensionality of SBioBERT embeddings)
- count 256 (Maximum token length for text/query encoding)
- count more than 9.6 million samples (GEO sample count as of September 2025)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper presents POE, a web-based semantic search platform for GEO omics datasets. The statistical evaluation compared POE semantic search against GEO keyword search across 30 natural-language queries; a subset of 9 queries (those with >4 unique results per system) was subjected to manual relevance scoring (0–10 rubric, 5 criteria) by two independent bioinformatics experts. Inter-rater agreement was quantified with Cohen's κ, and non-parametric paired tests were used to evaluate both rater concordance and the difference in relevance between the two search systems.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Cohen's kappa (κ) | Inter-rater agreement between two expert evaluators on 0–10 relevance scores | — | not stated |
| Two-tailed Wilcoxon signed-rank test | Comparison of total relevance scores assigned by the two expert evaluators (agreement check) | — | not stated |
| Two-tailed Wilcoxon signed-rank test | Comparison of POE vs GEO mean relevance scores across the 9 benchmark queries | 9 queries | not stated |
-
The qualitative relevance analysis was restricted to 9 of 30 queries based on a threshold of >4 unique datasets per system↳ Could also: All 30 queries could have been included using a mixed-effects model or permutation test that accommodates varying group sizes, or sensitivity analyses could examine whether results hold under different inclusion thresholds — Including all queries — or reporting results under multiple thresholds — would reduce selection effects and allow readers to see whether conclusions are stable across the full benchmark
-
Inter-rater reliability was quantified with unweighted Cohen's κ applied to scores on a 0–10 ordinal scale↳ Could also: Weighted Cohen's κ (linear or quadratic weighting) or the intraclass correlation coefficient (ICC, two-way mixed model) could also be used for ordinal or continuous-like scales — Weighted κ and ICC penalize disagreements proportionally to their magnitude, which is often more appropriate when the scale has ordered levels, and ICC additionally yields a confidence interval for the reliability estimate
-
Final relevance scores were computed as the unweighted mean of two rater scores↳ Could also: Scores could also be averaged within an ICC framework, or a consensus adjudication protocol could be used when raters differ by more than a set threshold — Formalizing aggregation within a reliability model provides a principled estimate of measurement error and makes the weighting scheme explicit
-
Two Wilcoxon signed-rank tests were performed (inter-rater check and system comparison) without adjustment for multiplicity↳ Could also: A Bonferroni or Holm correction across the two tests could also be applied; alternatively, framing them as distinct inferential questions with pre-specified roles (confirmatory vs exploratory) would clarify the intended type-I error control — Making the multiplicity strategy explicit helps readers interpret the reported p-values in the context of the total number of tests performed
-
The primary comparison of POE vs GEO was based on n=9 query-level paired observations in a Wilcoxon test↳ Could also: A permutation test or bootstrap resampling procedure could also be used; alternatively, a multilevel model treating both datasets and queries as random effects would explicitly account for the nested structure of ratings within queries — With n=9 pairs the Wilcoxon test has limited power; permutation and bootstrap approaches do not assume a particular null distribution and can be more informative at small sample sizes
-
Relevance was operationalized as the sum of five rubric criteria (C1–C5, each 0–2), yielding a 0–10 composite score treated as a single outcome↳ Could also: Internal consistency of the rubric (e.g., Cronbach's α or McDonald's ω) could also be reported, and individual criterion scores could be analyzed separately or reduced via principal component analysis — Reporting scale reliability would inform readers whether the five criteria measure a single underlying construct or capture distinct dimensions, which affects interpretation of the composite score
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-41282419
Title: Public Omics Explorer (POE): Enabling integrative semantic search across GEO omics datasets based on PubMed publications. PMID: 41282419 · PMCID: PMC12636342 · DOI: 10.1016/j.csbj.2025.11.004 Code: https://github.com/DimitrisGrig/POE (MIT) · Data accession: GSE298584 (case-study example)
Paper type
Software/platform paper. POE is a web application (React + FastAPI + PostgreSQL/pgvector) that builds a semantic search index over GEO/PubMed metadata using SBioBERT sentence embeddings and FAISS (IndexFlatL2) nearest-neighbour search. Most reported numbers are properties of the authors' live, daily-updated production database or of human relevance studies — not of a shippable, re-runnable pipeline.
The one shipped, re-runnable artifact
The repo ships exactly one reproducibility entrypoint:
reproduce_sbiobert_distance.py — loads the SBioBERT sentence-transformer,
encodes title+abstract and a query with normalize_embeddings=True, and computes
the L2 (squared-Euclidean) distance sum((doc-query)**2), printed to 3 decimals.
This is the computational kernel underlying the entire semantic-search claim.
IN SCOPE (pipeline-derived, re-runnable on the paper's described method)
| id | result | where reported | pipeline | how reproduced |
|---|---|---|---|---|
| C1 | SBioBERT embedding dimension = 768 | Methods (embedding) | reproduce_sbiobert_distance.py / SBioBERT model |
load model on «our HPC», encode text, assert output dim |
| C2 | Distance metric is L2 (Euclidean) over normalized embeddings | Methods / UI ("geometric distance (L2)") | shipped script | run shipped script, confirm it computes L2; check determinism (re-run → identical to 3 dp) |
| C3 | Embeddings are L2-normalized → identical text gives distance ≈ 0; bounded in [0,4] | Methods (normalize) | shipped script | self-text distance ≈ 0.000; cross-text distance in (0,4) |
GSE298584 ("Fusobacterium nucleatum promotes metastasis of breast cancer via the miR-21-3p/FOXO3 axis") is used in the paper only as a worked search example, not as a dataset that is computationally processed. We use its linked title/abstract as concrete input text to the shipped kernel, which is the faithful use of "the paper's own data".
OUT OF SCOPE (not a re-runnable pipeline / not feasible / honest skip)
- Database coverage counts (GSE 262,173; GSM ~9.68M; PubMed links 130,548; index 250k+):
scraped daily from the live GEO/ENA/PubMed via the authors' production crawlers; values
drift daily and require re-crawling all of GEO. Not pinned, not shipped →
non_pipeline/infeasible. - FAISS index benchmark table (Recall@10/Precision@10/query-time/memory for IndexFlatL2/IVFFlat/PQ/HNSW): requires the authors' full ~250k-vector production index (not shipped) and is hardware-specific (their i3-10100/64 GB). Out of scope.
- Human relevance study (30 queries, Cohen's κ=0.79, Wilcoxon p<0.05, 0–10 expert scores): manual/wet-lab-equivalent expert annotation, not a pipeline → out of scope.
80/20 statement
We reproduce the SBioBERT embedding kernel (C1–C3) — the deterministic, fully-specified core the authors themselves chose to ship for reproducibility. We do not attempt the production-DB counts, the FAISS benchmark, or the human study (the hard, non-shipped ~20%+), and say why above.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The single shippable kernel of this platform paper reproduced exactly — embedding dimension 768 (C1), L2-normalization (norm=1.0, self-distance=0.0, C3), and a deterministic L2 metric where the authors' shipped script matched an independent reimplementation to 3 dp (C2, squared-L2=1.084). The only defect is a minor authors-side documentation inconsistency (Methods names dmis-lab/sentence-biobert, the script loads pritamdeka/S-BioBert-snli-multinli-stsb — both 768-d, no numeric impact). The paper's substantive empirical claims (coverage counts, FAISS benchmark, human relevance study) are not derivable from shared artifacts because they need the live DB, a private index, and manual annotation — a data-availability limit, not fabrication. Net: negligible factual deviation on what could be checked, but only partial confirmation of the central 'good search' conclusion.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.