Wide-Open: Accelerating public data release by automating detection of overdue datasets.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH: yes. 1:1 reproduction, mostly. Wide-Open is a text-mining + repository-querying pipeline (regex-scan PMC OA for GSE/SRX accessions; check GEOmetadb; live-query NCBI to classify the rest as private/overdue). The paper ships both inputs and outputs as Supporting Information: S4 = odw.sqlite (the WideOpen database of extracted references, GEOmetadb snapshot 2017-04-16), S2 = overdue-over-time CSV (Fig 2), S3 = lag CSV (Fig 3). On «our HPC»/«infra» («job») we recomputed the figure data from the shipped S4 database using the repo's OWN documented logic (WideOpen/datawatch @90d33a50, build_page.get_hidden_df + load_dataframes SQL, faithful Py3 port), with NO live NCBI calls, and compared to S2/S3 and the reported numbers. CLEANLY REPRODUCED (exact/within-tol): submission->release lag trend (2006 87d -> 89.2d; 2016 >8mo -> 258.6d), GSE2436 11-year wait (2005-03-23 -> 2016-11-27 = 11.7y, exact), GSE17200 as the sole soil-name false positive (exact, documentary), and the headline ~400-datasets-released-in-one-week event (shipped S2 drop 477->76 = 401; independent recompute 1123->742 = 381, same early-Feb-2017 dates). The reported '473 identified' matches the shipped S2 figure-data plateau (~475) EXACTLY. THE ONE GAP (honest): a from-scratch recompute of the absolute overdue COUNT (C1=473) and oldest-overdue-year (C5=2010) from S4 alone over-counts ~2.4x, because the pipeline's live-NCBI validation step (GEOCacher: discard invalid/deleted/already-public candidates) is NOT encoded in the shipped database; the reproduced curve tracks S2's shape+timing+drop but sits ~+640 above it. NO FABRICATION INDICATORS: all reported headline numbers are internally consistent and largely recomputable from the shipped artifacts. NOT ATTEMPTED (hard 20% / non-reproducible by design): 84 overdue SRA datasets (needs SRAmetadb + live SRA queries; S4 has SRX mentions but no SRA release table); the live-state '455/473 released since' (time-dependent; today's GEO no longer reflects the 2017 state and S4 is already a post-drop April-2017 snapshot); re-running the full pipeline from raw PMC OA + current GEOmetadb (would produce a CURRENT snapshot, not the 2017 one); Fig 1 sample counts (scraped from NCBI history page, not a pipeline output).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 74assessed: 2026-06-14 ⛓ a41fad1ecf95
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan text mining be used to automatically identify datasets that are overdue for public release—i.e., referenced in published articles but still private in public repositories—to help enforce open-data policies that currently rely on lagging manual efforts?
- ★ Wide-Open is a general text-mining approach that automatically detects overdue datasets by scanning PubMed articles for dataset accession identifiers and querying repositories to determine if the datasets remain private. method
- ★ Applying Wide-Open to GEO and SRA identified a large number of overdue datasets that authors had referenced in publications but not released. finding
- ★ Notifying GEO of the Wide-Open-identified list spurred administrators to release approximately 400 datasets within one week. finding
- ★ Wide-Open attained a precision of 97% in identifying overdue datasets, even counting author citation errors. finding
- ★ The time lag between dataset submission and public release in GEO has steadily increased over time. finding
- Wide-Open is open-source and extensible to new repositories via three steps: creating accession regular expressions, identifying the dataset-access web API, and adapting the query-result parser. resource
- Wide-Open first checks indexing databases (GEOmetadb, SRAdb) before calling repository web APIs to minimize unnecessary API calls. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Text mining / regular-expression accession extraction from full-text articles | Open-access subset of PubMed Central (~1.5 million papers) | none | candidate dataset unique identifiers (GSE for GEO, SRX for SRA) | Wide-Open (regex GSE[0-9]+ and SRX[0-9]+) |
| Database lookup for release status | GEO repository | none | whether dataset accession has been released | GEOmetadb |
| Database lookup for release status | SRA repository | none | whether dataset accession has been released | SRAdb |
| Web API query and result parsing for accession validity/privacy | GEO repository | none | valid vs invalid accession; public vs private status (e.g., string 'is currently private') | GEO web API (acc.cgi) |
| Web API query and result parsing for accession validity/privacy | SRA repository | none | valid vs invalid accession; public vs private status | SRA web API (sra/?term=) |
- ▼ Approximately 400 GEO datasets were released within the first week after notifying GEO of the Wide-Open list ~400 datasets
- – Of 473 datasets identified by Wide-Open in February 2017, 455 were subsequently released by GEO 455 of 473
- – Wide-Open precision in identifying overdue datasets 97%
- ▲ Average GEO submission-to-release delay rose from 87 days (2006) to over 8 months (2016) 87 days to >8 months
- ▲ GSE2436 submitted March 2005 was not made public until November 2016, an 11-year wait 11 years
- – Wide-Open identified 84 overdue SRA datasets as of March 2017 84 datasets
- – Of 18 remaining unreleased candidates, only one was a true precision error (GSE17200 referring to a soil name); 14 were author typos, 3 legitimately unreleasable 1 of 18 false positive
- count 80,985 public datasets and 2,097,543 samples (GEO contents as of February 2017)
- count 473 (datasets identified by Wide-Open in February 2017 (GEO))
- count 455 (of the 473 GEO datasets released since identification)
- other 97% (Wide-Open precision including author errors)
- count ~400 (GEO datasets released within first week of notification)
- mean 87 days (average GEO submission-to-release delay in 2006)
- mean over 8 months (average GEO submission-to-release delay in 2016)
- count over 10,000 (GEO datasets currently remaining private)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a systems and informatics paper presenting Wide-Open, a text-mining pipeline for detecting overdue public datasets in GEO and SRA. All quantitative reporting is purely descriptive: counts of overdue datasets over calendar time, average submission-to-release delays by year, and a precision point estimate for the detection system based on manual verification. No inferential statistical tests are performed; the paper does not test hypotheses between groups but rather characterizes system output and longitudinal trends.
-
System precision was reported as a single point estimate (97%) based on manual verification of 473 detected candidates↳ Could also: A Wilson score or Clopper-Pearson 95% confidence interval for the precision proportion could also be reported — Confidence intervals quantify uncertainty around a proportion estimate and are especially informative when the denominator is moderate in size, making the reliability of the estimate more transparent to readers
-
Average submission-to-release delay per year was summarized using the mean (e.g., 87 days in 2006, >8 months in 2016)↳ Could also: Median and interquartile range (IQR) could also be reported alongside or instead of the mean — Release-delay distributions are likely right-skewed (the paper itself cites an 11-year outlier), a setting where the median is more robust to extreme values than the mean and IQR conveys spread without being pulled by outliers
-
The trend of increasing average delay over time (Fig 3) was described verbally without a statistical model↳ Could also: A linear or log-linear regression of average delay on year could also be fitted — A regression model would provide a quantified rate of change per year with confidence bounds, making the trend magnitude formally estimable and comparable across sub-periods
-
The drop in overdue dataset counts following GEO notification (Fig 2, magenta portion) was described narratively↳ Could also: An interrupted time-series (ITS) model or changepoint analysis could also be applied to the count series — These methods formally estimate the timing and magnitude of a step-change or slope-change due to an intervention, providing a statistically principled characterization of the GEO notification effect
-
System evaluation focused solely on precision; recall was not estimated↳ Could also: Recall (the fraction of all truly overdue datasets successfully detected) could also be estimated using a manually curated gold-standard sample of known overdue accessions — Precision alone characterizes the false-positive rate but not coverage; a recall estimate and F1 score would give a more complete picture of system performance for readers weighing deployment decisions
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-28594819 (Wide-Open / datawatch)
Paper: Grechkin M, Poon H, Howe B. Wide-Open: Accelerating public data release by automating detection of overdue datasets. PLoS Biol 2017. PMC5464523. Code: https://github.com/WideOpen/datawatch (MIT, last push 2017-07-27).
What the tool is (pipeline summary)
A text-mining + repository-querying pipeline, not a sequencing pipeline:
parse_papers.py— scan PMC Open-Access full-text XML; extract dataset accessions with regexesGSE[0-9]+(GEO) andSRX[0-9]+(SRA); build a sqlite DB (odw.sqlite) ofpapers,mentions(paperid,acc),datasets.parse_geometadb.py— join in GEOmetadb to populatedatasets(first_submitted_on,first_public_on) = the released (public) sets.build_page.py— a GSE mentioned in a published paper but not public is overdue.get_hidden_df()computes the overdue-count-over-time curve; a live NCBI GEO double-check (GEOCacher) confirms each candidate's status.build_sra.py— analogous for SRA (uses SRAmetadb + live SRA queries).
Key enabler for a faithful, time-independent reproduction
The paper ships its inputs and outputs as Supporting Information:
- S4 Data (
.s004, XZ) = the actualodw.sqliteWideOpen database of extracted references — the pipeline's intermediate. - S2 Data (
.s002, CSV) = Number of overdue GEO datasets over time (Fig 2). - S3 Data (
.s003, CSV) = Average submission→release lag (Fig 3). - S1 Data (
.s001, CSV) = number of GEO samples over time (Fig 1; wet/manual count from NCBI history page — out of scope).
This lets us recompute the published figure data from the shipped database using the repo's own documented logic, and compare to the shipped CSVs and the reported numbers — a deterministic 1:1 reproduction that does not depend on the (now-changed) live state of GEO. This is the canonical "third-party-tool-or- own-code on the paper's own data" reproduction the brief endorses.
IN SCOPE (pipeline-derived, deterministic from S4 odw.sqlite + repo code)
- C1 Overdue GEO datasets identified = 473 (Feb 2017). Recompute as GSEs
mentioned in a published paper but absent from the released
datasetstable (minus the repowhitelist.txt), mirroringbuild_page.load_dataframes. - C2 S2 overdue-over-time curve — regenerate via the exact
get_hidden_df()algorithm fromodw.sqlite; compare row-by-row to shipped S2. - C3 S3 lag-over-time — avg(
first_public_on−first_submitted_on) by release year fromdatasets; check 2006 ≈ 87 days, 2016 > 8 months; compare series to shipped S3. - C4 GSE2436: submitted Mar 2005, public Nov 2016 ≈ 11-year wait
(exact query on
datasets). - C5 Oldest overdue paper published in 2010 = min(first_mention year) over the overdue candidate set.
- C6 GSE17200 is the sole true precision error (a soil name) — confirm it is an overdue candidate and the documented false positive (whitelist line 1).
OUT OF SCOPE / NOT ATTEMPTED (the hard ~20%, or non-reproducible-by-design)
- 84 overdue SRA datasets (Mar 2017): needs SRAmetadb + live SRA web queries;
odw.sqlitecarries SRX mentions but no SRA release table → not cleanly recomputable from shipped data. Noted, not attempted. - "455/473 released since" and "≈400 released in first week": time-dependent live-state outcomes (post-notification GEO releases). The 2017 snapshot cannot be reproduced from today's GEO; the magenta drop is only partially encoded in S4 depending on when it was dumped. Treated as provisional / context, not a clean target.
- Re-running the full pipeline from raw PMC OA + current GEOmetadb: would produce a current snapshot, not the paper's 2017 one (corpus + GEO state have moved on; GEOmetadb is deprecated). Explicitly not the goal — that would not be a 1:1 reproduction of the reported numbers.
- **Fig 1 sample cou
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
A faithful Py3 port of the authors' own datawatch logic recomputes the figure data from the shipped S4 odw.sqlite: the lag trend (87->89.2d 2006; >8mo->258.6d 2016), the GSE2436 11.7-year wait (exact), the GSE17200 soil false-positive (exact) and the headline ~400-released-in-a-week event (381-401) all reproduce, and the reported 473 matches the shipped S2 plateau (~475) exactly. The only deviations — a ~2.4x inflated from-scratch overdue count and oldest-year 2010 vs 2004 — share one root cause: the pipeline's live-NCBI validation step is not encoded in the static shipped DB and cannot be replayed on a 2017 snapshot. This is an inherent reproducibility limit, not an authors' discrepancy; numbers are internally consistent with no fabrication signal.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.