An informatics research platform to make public gene expression time-course datasets reusable for more scientific discoveries.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL (strong). Faithful reproduction of the GETc / Dynamics4GenomicBigData time-course pipeline (authors' verbatim MATLAB step-functions run under GNU Octave 10.3.0 + Ramsay fdaM commit 4927ef7; MATLAB unavailable on «our HPC») on GEO GSE14103. Table A1 targets: DRG=6295, GRM=33. RESULT: C1 DRG=6295 reproduced EXACTLY (1:1) under the DMSO 8-timepoint synchronization series (lambda=0.37673, df=5, F_crit=9.11718); the paper labels the row 'Condition=Nocodazole, Time point=8' but Nocodazole alone has only 4 GEO samples, and of 4 candidate 8-sample selections tested only DMSO-8tp reproduces 6295 (others: all-12-avg=13887, DMSO0-6h+Noco7-10h=35136, Noco-4tp=4883) — so the analyzed condition is unambiguously the DMSO series. C2 GRM=37 vs reported 33: a faithful near-miss (+4, ~12%), NOT tuned to the target. The IHC clustering input was verified 1:1 against repo step_4 (double z-score of top-3000); the residual gap comes from MATLAB->Octave numerical differences in pdist(Spearman)/linkage(average) and the floating-point convergence stop of the iterative jumping-gene pruning, which cannot be eliminated without MATLAB. Two behaviour-preserving Octave adaptations were needed in the authors' IHC code (a numeric tabulate, and a manual distance-criterion flat cut replacing the MATLAB-incompatible cluster() option parsing) plus rewriting the shipped non-conformant convergence-metric line to its unambiguous scalar form. Out of scope (not attempted): step_5/6 ODE gene-regulatory network (needs SBEToolbox; no pinned GSE14103 number), step_7 DAVID GO enrichment (external web service), and the platform's IR/recommendation/human-judge evaluations. Note: this room's «infra» workdir had been reclaimed by the janitor between sessions, so the entire environment + data were rebuilt this session from the verbatim pipeline scripts recovered from the audit transcript; DRG=6295 was thus independently reproduced twice.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 75assessed: 2026-06-20 ⛓ 2988eb24a70c
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-23
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-20no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetA web-based platform combining an ODE-based analytic pipeline with dataset, literature, and collaborator recommendation functions can make public time-course gene expression datasets from GEO more findable, accessible, interoperable, and reusable, while facilitating collaboration between domain experts and data scientists.
- ★ GETc enables discovery and visualization of time-course gene expression data and analytical results from GEO resource
- ★ An ODE-based analytic pipeline identifies dynamic response genes, clusters them by expression pattern, and constructs gene regulatory networks method
- ★ A regular expression-based NLP method can identify time-course datasets from unstructured GEO metadata method
- A Dirichlet process mixture model clusters researcher publications into research topics for dataset recommendation via cosine similarity method
- A BM25-based literature recommendation system outperforms TF-IDF, word2vec and doc2vec for retrieving relevant literature per dataset finding
- Collaborators are recommended per dataset by scoring and ranking authors of the top similar retrieved literature method
- ★ 555 time-course gene expression datasets with more than 7 time points were identified from GEO, and the analytic pipeline was applied to 37 of them finding
- ★ GETc is a freely accessible, generalizable web-based platform applicable across biomedical research areas resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| regular expression-based text mining/NLP | GEO dataset metadata (titles, summaries, descriptions) | none | identification of time-course datasets and extraction of time point information | — |
| MetaMap entity recognition | GEO dataset metadata text | none | Human Disease Ontology (DOID) terms annotated per dataset | MetaMap |
| ODE-based time-course gene expression analytic pipeline | 37 time-course gene expression datasets from GEO (>7 time points) | none/observational time-course | dynamic response genes, expression-pattern clusters, gene regulatory network | — |
| Dirichlet process mixture model (DPMM) clustering | researcher publication titles/abstracts from PubMed | none | research interest clusters used for dataset recommendation via cosine similarity | — |
| BM25 text retrieval/ranking (compared to TF-IDF, word2vec, doc2vec) | 770,537 filtered MEDLINE articles | none | precision at 10 for literature recommended per dataset | BM25 |
| author scoring/ranking algorithm (weighted by author position) | authors of top 1000 similar literature per dataset; affiliations parsed | none | ranked collaborator recommendations with geographic distance | affiliation_parser; geopy |
- – Regular expression-based system identified time-course datasets from GEO metadata with 83.5% accuracy (167/200 test datasets) 83.5%
- – 862 candidate time-course datasets were identified by the automated system, of which 555 were manually filtered for processing 555/862
- – 555 time-course gene expression datasets with more than 7 time points were identified from GEO 555 datasets
- – The ODE-based analytic pipeline was applied to 37 of the identified time-course datasets 37 datasets
- ▲ BM25 achieved better precision at 10 than TF-IDF, word2vec and doc2vec for the literature recommendation system
- – 770,537 MEDLINE articles were utilized for developing the literature and collaborator recommendation systems 770,537 articles
- – GETc platform is freely accessible at genestudy.org
- other 83.5% accuracy (167/200) (accuracy of regex-based time-course dataset identification method)
- count 862 (candidate time-course datasets identified by automated system)
- count 555 (manually filtered time-course datasets with >7 time points hosted in GETc)
- count 37 (datasets analyzed with the ODE analytic pipeline)
- count 770537 (MEDLINE articles used for literature/collaborator recommendation development)
- count 122222 (GEO dataset series available as of 18 December 2019)
- count 34 (datasets added to GEO repository daily in 2019)
- count 16187835 (works indexed by DataCite data discovery index)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a web-based informatics platform (GETc) for hosting, visualizing, and disseminating time-course gene expression datasets and analysis results from GEO, along with associated recommendation systems; it is a tool/platform description rather than a hypothesis-testing study. Reported methodological components include a regular-expression/NLP classifier for identifying time-course datasets (evaluated on 200 random datasets), an ODE-based analytic pipeline that flags genes with significant expression changes over time (detailed in cited prior work), a Dirichlet process mixture model (DPMM) used to cluster researcher publications into research topics, and a comparison of BM25 against TF-IDF, word2vec, and doc2vec for literature recommendation using precision at 10. The provided text is truncated before the Results section, so reporting practices for that section could not be assessed.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Regular-expression/keyword-based NLP classification, evaluated as accuracy | identification of time-course datasets from GEO metadata | 200 randomly selected datasets (167 correctly identified) | not stated |
| ODE-model based identification of genes with significant expression change over time (method detailed in a separate cited publication, not fully described in this text) | identification of 'dynamic response genes' within the analytic pipeline | — | not stated |
| Retrieval-method comparison using precision at 10 (BM25 vs TF-IDF, word2vec, doc2vec) | evaluation of the literature recommendation system (detailed in a separate cited publication) | — | not stated |
-
Time-course datasets were identified using a regular-expression/keyword-based NLP classifier evaluated on 200 random datasets (83.5% accuracy).↳ Could also: A supervised machine-learning text classifier (e.g., logistic regression, SVM, or a fine-tuned transformer) trained on labeled examples — this could complement the rule-based approach and be evaluated with additional metrics such as precision, recall, and F1, which give a fuller picture of classifier performance than accuracy alone, particularly if the positive/negative classes are imbalanced.
-
Genes with significant expression changes over time were identified via an ODE-based modeling pipeline, with details deferred to prior cited work.↳ Could also: Alternative time-course differential expression methods such as maSigPro, EDGE, or spline-based mixed-effects models — these are commonly used alternatives for time-course gene expression analysis and could be used alongside ODE-based calls for comparison or cross-validation of which genes are flagged as dynamically responsive.
-
Literature recommendation performance was compared across BM25, TF-IDF, word2vec, and doc2vec using precision at 10.↳ Could also: Additional ranking metrics such as recall@k, NDCG, or mean average precision (MAP), evaluated via cross-validation or a held-out test set — these would provide a broader view of retrieval quality beyond a single precision cutoff and could help characterize the robustness of the method comparison.
-
Researchers' research interests were identified by clustering their publications with a non-parametric Dirichlet process mixture model (DPMM).↳ Could also: Alternative topic-modeling or clustering approaches such as Latent Dirichlet Allocation (LDA) or embedding-based clustering (e.g., k-means or HDBSCAN on document embeddings) — these are standard alternatives for grouping textual research profiles and could be compared for cluster stability or interpretability against the DPMM results.
-
The time-course identification accuracy (83.5%) was reported as a single point estimate based on 200 datasets.↳ Could also: Reporting a confidence interval around the accuracy estimate (e.g., a binomial or Wilson CI) — a confidence interval would convey the precision of the accuracy estimate given the sample size, which is common practice when reporting classifier performance.
-
Collaborator recommendation scores were computed with a fixed, manually specified weighting scheme (1 for first/last author, 0.1 otherwise) rather than a fitted statistical model.↳ Could also: A learned ranking model (e.g., logistic regression or a gradient-boosted ranking model) trained on relevance-labeled examples — a learned model could estimate weights empirically from data, and its performance could then be assessed with standard information-retrieval evaluation metrics.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 33247935 (GETc / Dynamics4GenomicBigData platform)
Paper: "An informatics research platform to make public gene expression time-course
datasets reusable for more scientific discoveries." DOI 10.1093/database/baaa074.
Code: https://github.com/AutumnTail/Pipeline (MATLAB; j142857z/Pipeline redirects here —
j142857z = Juan C. Ramírez, former maintainer). Pipeline designed by M. Carey & H. Wu.
Data: GEO GSE14103 ("Synchronized HCT116 cells: time course", GPL570 Affy HG-U133
Plus 2, 54675 probes, 12 samples = DMSO 0,2,4,6,7,8,9,10h + Nocodazole 7,8,9,10h).
What the pipeline computes (8-step time-course analysis)
- step_2 preprocess: pick a condition's samples, sort/dedup time points, log2-center
per gene →
gene_expression(M×n centered matrix).Est_Sub_Selpools ~well-behaved genes (per zero-crossing class, by IQR, per-gene GCV) to choose a shared smoothing λ. - step_3 FDA: B-spline basis (order 3, knots = time points, nbasis = n+2, penalty
order 2); GCV-optimal λ over the pool;
smooth_basisall genes; per-gene F-test (smooth model vs centered-mean null). DRG = #genes with F > F_{0.05, df-1, n-df}. - step_4 IHC clustering (Iterative Hierarchical Clustering, Carey): z-scored top-3000 DRGs, Spearman distance, average linkage, cutoff 1-α with α=0.75, prune "jumping genes", iterate to convergence. GRM = # final modules.
- step_5 ODE gene-regulatory network (needs SBEToolbox), step_6 graph stats, step_7 DAVID GO enrichment (external web service).
In scope (pipeline-derived, attempted)
The Table A1 (appendix) row for GSE14103 (SL=7): Time point=8, Condition=Nocodazole, # of DRG = 6295, # of GRM = 33. These two integers are the concrete, pipeline-derived numerical claims for this dataset and are the reproduction targets:
- C1 — DRG count = 6295 (step_2+step_3).
- C2 — GRM count = 33 (step_4).
Note: "Condition=Nocodazole" with "Time point=8" is ambiguous because Nocodazole has only 4 samples in GEO. The exact 8-sample selection the authors used is not specified, so we test the plausible 8-timepoint selections (DMSO-only; all-12 averaged; DMSO 0-6h + Noco 7-10h) and report which reproduces 6295/33; this disambiguation is part of the reproduction.
Out of scope (not attempted)
- step_5/6 ODE network + graph stats (require SBEToolbox; no concrete per-dataset number in Table A1 to compare against for GSE14103).
- step_7 DAVID GO enrichment (external authenticated web service; not a pinned numeric claim).
- The platform's recommendation systems (dataset/literature/collaborator), MetaMap disease annotation, regex time-point mining, MEDLINE filtering — these are platform-engineering / IR-evaluation results (Precision@10 by human judges, 770k articles, 555 datasets, etc.), not a re-runnable bioinformatic pipeline on GSE14103. Manual/human-judge evaluations are explicitly out of scope.
Method / environment
MATLAB is unavailable on «our HPC». We run the authors' verbatim step functions under GNU
Octave 10.3 with Ramsay's fdaM toolbox (mzerter/fdaM — the same FDA library the
pipeline uses; its R twin fda would build a different basis, so Octave+fdaM is the faithful
path). Only fminbnd (Octave needs an anonymous-function wrapper) and clusterdata
(replaced by the equivalent pdist/linkage/cluster) are adapted; all numeric logic is the
authors'. Compute runs as a SLURM job on «our HPC»; data + envs live on «infra».
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is an in-progress, partial reproduction of the GETc/Dynamics4GenomicBigData pipeline on public GEO data (GSE14103). The first reported value, DRG=6295 (Table A1), was reproduced exactly via the authors' FDA F-test, which is reassuring — no fabrication concern. The main caveats are on our side: MATLAB was substituted with Octave+fdaM, and the exact 8-sample selection was ambiguous (4 candidates tested, n_observed=12 vs reported 8). The second value, GRM=33, is still PENDING compute, so the core claim is confirmed only partially.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.