MCPmed: a call for Model Context Protocol-enabled bioinformatics web services for LLM-driven discovery.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH TO REPRODUCE: yes, for the one pipeline-derived result in this perspective paper. MCPmed is a 'call for' paper; almost everything is narrative/architecture. The single deterministic computational result is the GEOmcp+STRING case study (Results, Phase 3). We reproduced it 1:1 by applying the THIRD-PARTY tool STRING DB v12.0 (the exact version pinned in the authors' own MCPmed/STRINGmcp repo) to the paper's 14-gene mouse list (species 10090) as a SLURM job on «our HPC». RESULT = HONEST PARTIAL, leaning strongly genuine: 49 PPI edges reproduce EXACTLY; BOTH reported GO-enrichment p-values reproduce to the EXACT 3 significant figures printed (6.73e-13 and 1.63e-06); 100% mapping and the top-3 hub identity (FOS/CREB1/BDNF) reproduce exactly; CREB1=11 and BDNF=10 exact. Two-p-value-to-3-sig-fig agreement is essentially impossible by chance -> the analysis is real, not fabricated. Three minor deltas: expected-edges 6 vs 9 (mismatch), FOS degree 12 vs 11 (off by one), and '278 GO terms' which matches neither 410 (all categories) nor 106 (true GO) - the metric is undefined in the paper. These are consistent with STRING DB content drift since the authors' run and with the paper's OWN internal inconsistency on gene count. NOT ATTEMPTED (out of scope): GEOmcp wall-clock latency benchmarks (environment-dependent), the autonomous GEO discovery act (non-deterministic LLM decision; GSE278158 existence/description verified only), Phase-4 biological interpretation (subjective), breadcrumbs JS unit tests, and the exact 15th gene (Supplementary Material 1 / LLM transcript not retrievable from the compute node - left as the 'last 20%'). FLAGS for human review: (1) registry accession GSE12345 is a textbook PLACEHOLDER, not the real dataset (real = GSE278158); (2) the paper claims 15 genes / 12 additions but enumerates only 14 / 11 in the main text - a wording/count slip, not a results fabrication.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 80assessed: 2026-06-14 ⛓ fdb02b901253
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusBioinformatics web servers are designed for humans and lack machine-actionable interfaces for LLM-driven autonomous agents; adapting the Model Context Protocol (MCP) to bioinformatics web server backends can provide a standardized, semantic, machine-actionable layer that enhances automation, reproducibility, and interoperability for next-generation research agents.
- ★ Adapting MCP to bioinformatics web server backends provides a standardized, machine-actionable semantic layer linking API endpoints to scientific concepts and metadata. method
- ★ MCP-enabled LLM implementations across GEO, STRING, and UCSC Cell Browser demonstrate enhanced data exploration capabilities. finding
- ★ MCP complements rather than replaces FAIR principles and GA4GH TRS by adding an AI-specific integration layer with semantic concept mapping, explicit versioning, and standardized error handling. mechanism
- ★ MCPmed is proposed as a community effort with lightweight HTML metadata 'breadcrumbs' and templates to bridge existing services toward MCP readiness. resource
- ★ GEOmcp is a proof-of-concept MCP implementation layering over NCBI E-utilities, enabling LLMs to autonomously search and download GEO data. resource
- MCP can lower the operational cost of exploring additional hypotheses against existing data corpora, helping extract latent findings from underused datasets. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| MCP layer / web service integration (proof-of-concept software) | Gene Expression Omnibus (GEO) database via NCBI E-utilities | none | autonomous LLM-driven dataset search, metadata retrieval, and data download | NCBI E-utilities API (ESearch, ESummary); GEOmcp |
| MCP layer / web service integration | STRING database | none | LLM-driven queries and cross-service integration | STRING REST API |
| MCP layer / web service integration | UCSC Cell Browser | none | LLM-driven queries and exploration | UCSC Cell Browser MCP |
- – Eight distinct MCP tools were implemented for GEOmcp, each with JSON Schema input/output specifications and semantic descriptions. 8 tools
- – Rate limiting implements 0.1 s delays between requests, allowing up to 10 requests/second with an API key. 0.1 s delay; 10 req/s
- – Download subsystem enforces configurable size limits and concurrent transfer caps for safety. 5 GB/file, 10 GB total, 3 parallel transfers
- ▲ MCP-enabled LLM implementations across GEO, STRING, and UCSC Cell Browser show enhanced exploration capabilities.
- count 1.3 M human omics samples (Most human omics samples deposited on GEO remain acutely underused (Piwowar and Vision, 2013))
- count 8 (Number of distinct MCP tools implemented in GEOmcp)
- other 0.1 s (Delay between requests for NCBI rate-limit compliance)
- count 10 requests per second (Maximum request rate with API key)
- other 5 GB per file, 10 GB total (Configurable download size limits)
- count 3 parallel transfers (Default semaphore-controlled concurrent downloads)
- other 300 s default (Configurable HTTP request timeout for downloads)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
-
The comparative summary of FAIR, GA4GH TRS, and MCP (Table 1) is presented as a qualitative narrative matrix without quantitative benchmarks.↳ Could also: A structured empirical evaluation — for example, measuring task-completion time, error rate, or number of API calls required for representative bioinformatics queries under each approach — could also have been conducted. — Quantitative benchmarking would allow readers to assess the magnitude of efficiency gains attributed to MCP adoption, complementing the conceptual argument.
-
The GEOmcp proof-of-concept is demonstrated through illustrative use-case descriptions rather than a systematic evaluation across a defined query set.↳ Could also: A benchmark suite of standardized bioinformatics queries with recorded success rates, retrieval precision/recall, and latency could also have been used to characterize system performance. — Standardized benchmark results would make it easier to compare future MCP implementations against the baseline established here.
-
The claim that bioinformatics datasets are 'acutely underused' is supported by citation of a 2013 study (Piwowar and Vision) without new quantitative analysis of current GEO reuse rates.↳ Could also: An updated bibliometric or download-count analysis of GEO dataset reuse — e.g., tracking citations per deposited study or download frequency by year — could also have been reported. — Current empirical data on reuse rates would ground the motivating claim in a more recent and directly comparable evidence base.
-
The usability of MCP-enabled services for LLM-driven discovery is argued conceptually and through a single breast-cancer workflow example.↳ Could also: A user study or task-based evaluation with domain researchers — comparing time-on-task, error frequency, and satisfaction scores between manual, API-only, and MCP-assisted workflows — could also have been conducted. — Human-factors evidence would complement the system-design argument by quantifying the practical benefit to actual end-users.
-
Rate-limiting parameters (e.g., 0.1 s delay between NCBI requests, 5 GB per-file limit) are stated as design choices without empirical justification.↳ Could also: Sensitivity analyses varying these thresholds against measured throughput and error rates could also have been reported. — Empirical parameter tuning results would help implementers choose appropriate values for their own deployments.
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-41729821 (MCPmed)
Paper: Flotho et al., MCPmed: a call for Model Context Protocol-enabled bioinformatics web services for LLM-driven discovery. Brief Bioinform 2026; 27(1):bbag076. PMID 41729821 · PMC12927880.
Paper type: Perspective / methods ("a call for…"). It proposes the MCPmed community effort and demonstrates MCP servers over GEO, STRING, and the UCSC Cell Browser. Most of the paper is narrative/architecture, not pipeline output.
In scope (pipeline-derived, reproducible)
The single concrete, deterministic computational result is the cross-service case study (Results → "Cross-service integration: GEOmcp + STRING DB"), whose network statistics come from the STRING database REST API applied to a fixed mouse gene list. STRING is a third-party tool; per BRIEF rule 2 (P16), running it on the paper's own gene list is a fully valid reproduction.
| Result (paper) | Pipeline | Reproducible? |
|---|---|---|
| 100% identifier mapping to mouse proteins | STRING get_string_ids |
yes — deterministic |
| 49 PPI edges observed | STRING ppi_enrichment/network |
yes |
| 6 expected edges | STRING ppi_enrichment |
yes (version-sensitive) |
| PPI-enrichment p < 1e-16 | STRING ppi_enrichment |
yes |
| Hubs FOS=12, CREB1=11, BDNF=10 | STRING network degree |
yes |
| GO:0048167 p=6.73e-13; GO:0007611 p=1.63e-06 | STRING enrichment |
yes |
| 278 GO terms total | STRING enrichment |
partly (metric ambiguous) |
Tool/version pinned from the authors' own repo MCPmed/STRINGmcp (stringmcp/ main.py): STRING v12.0 stable endpoint https://version-12-0.string-db.org/api,
network_type=functional, no required_score override (STRING default = medium
0.400), caller_identity=string_mcp_bridge, species 10090 (Mus musculus).
Out of scope (not attempted, with reason)
- GEOmcp performance benchmarks ("2–5 s per query", "<100 ms MCP overhead",
"~45 s workflow"): wall-clock latency, environment- and network-dependent, not
deterministically reproducible. →
non_pipelinefor grading purposes. - Autonomous GEO dataset discovery (query → GSE278158): the selection is an LLM/agent decision, non-deterministic. We only verify GSE278158 exists and matches the description; we do not reproduce the discovery act.
- Phase-4 biological interpretation ("NCOA1 as master regulator", etc.): subjective LLM synthesis, not a pipeline number.
- Breadcrumbs validation / integration tests: JS/Node mock-server unit tests, not a scientific result.
- Wet-lab / external: none in this paper.
Data note
The paper's registry-listed accession GSE12345 is a placeholder (textbook
example), not the dataset used. The case study actually uses GSE278158 (6-sample
aging mouse hippocampus RNA-seq) — but only its gene list feeds the reproducible
STRING step; the count matrix itself is not needed for the reported network stats.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a perspective/'call for' paper whose single deterministic result (the GEOmcp+STRING case study) reproduces essentially 1:1: 49 edges, both GO p-values to the exact 3 sig figs (6.73e-13, 1.63e-06), and the top-3 hubs (FOS/CREB1/BDNF) all match exactly — strong evidence the analysis is genuine, not fabricated. The few deviations are minor and explainable: FOS 12→11 and expected-edges 6→9 (STRING DB drift and/or the unlisted 15th gene), and the '278 GO terms' metric is undefined in the paper. Responsibility is split between expected technical drift and the authors' own internal inconsistency (15 vs 14 genes; placeholder accession GSE12345), none of which alter the central conclusion. Overall yellow: solid and clearly reproducible with small, well-understood discrepancies rather than a pristine match.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.