Evaluation of classification and forecasting methods on time series gene expression data.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the DETERMINISTIC half cleanly. Reproduced Table 4 forecasting RMSE (ARIMA + Holt-Winters) by running the authors' OWN shipped code/data on «our HPC» (statsmodels 0.11, the legacy ARIMA API). 1:1 RESULT: ARIMA on GSE6186 and GSE3406 reproduces essentially EXACTLY — all 6 reported cells (10/20/30% test) within +/-0.005 RMSE (e.g. GSE6186@10% 0.2437 vs 0.244; GSE3406@30% 0.6335 vs 0.63). ARIMA is the paper's best forecaster and the basis of its headline claim, so this corroborates those numbers as genuine. DIVERGENCES (honest, flagged): Holt-Winters matches at 10% but runs ~0.1-0.2 RMSE higher at 20-40% (real trend-extrapolation blow-up; paper likely used a different HW averaging/horizon); GSE1723 fails for both methods (reproduced 45-101 vs reported 0.35-1.07) because the shipped FORECASTING loader does not normalise that larger-scale 12-point dataset, while the paper's small values imply an undocumented normalisation. No fabrication indicated for the ARIMA result; HW/GSE1723 gaps point to undocumented preprocessing, not fabrication. NOT ATTEMPTED (hard ~20%): classification Tables 1-3 (CNN/LSTM/DNN/SVM/OC-SVM/DeepTrust/STEM) — non-deterministic train_test_split(random_state=None) + stochastic DL, unpinned TF/Keras, no seeds; and forecasting ANN/LSTM/GluonTS rows (stochastic NNs).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 74assessed: 2026-06-15 ⛓ b02d28867b8c
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusAlthough time series gene expression data shares characteristics with time series from other domains, most gene expression analyses ignore the time-ordered nature of the data and focus on clustering; this study tests whether classification and forecasting approaches (traditional statistical, machine learning, and deep learning) are adequate and effective for time series gene expression classification and forecasting.
- ★ Deep learning based methods generally outperform traditional approaches for time series gene expression classification finding
- ★ Supervised classification on gene expression is more effective than clustering when labels are available finding
- ★ For forecasting, an autoregressive statistical approach (ARIMA) has the best performance for short term forecasting, while deep learning based methods are better suited for long term forecasting finding
- ★ Novel CNN and LSTM based architectures are proposed for time series gene expression classification method
- ★ LSTM and ANN deep learning methods are implemented for gene expression forecasting method
- A comprehensive evaluation of unsupervised, supervised machine learning, and deep learning techniques for time series gene expression classification and forecasting across five real datasets resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| microarray time series gene expression classification | five real gene expression datasets (e.g. GSE6186, GSE3406, GSE1723; Saccharomyces cerevisiae/cell cycle data) | none | predicted gene class/label accuracy | — |
| time series gene expression clustering (short time series) | gene expression datasets | none | gene cluster assignments | STEM software |
| time series gene expression forecasting | five real gene expression datasets | none | predicted future expression values vs actual values (MSE) | GluonTS toolkit (Feed Forward Neural Network) |
- ▲ Deep learning based methods generally outperform traditional approaches for time series classification
- ▲ Supervised classification outperforms clustering when labels are available
- – ARIMA (autoregressive statistical approach) performs best for short term forecasting
- – Deep learning based methods are better suited for long term forecasting
- count 5 real gene expression datasets (number of datasets used for evaluation)
- count 3 classes for GSE6186 dataset (embedded space dimension equal to number of classes in DeepTrust)
- other DNN with ten hidden layers of 100 nodes, learning rate 0.0001, 5% dropout (DNN architecture hyperparameters)
- other LSTM classification layer with 200 cells (proposed LSTM classification architecture)
- other LSTM forecasting two layers with 100 and 50 neurons, batch size 100, 50 iterations (LSTM forecasting model configuration)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a machine-learning benchmarking study that evaluates multiple classification and forecasting methods on five real time-series gene expression datasets. For classification, proposed CNN and LSTM architectures are compared against SVM, One-Class SVM, DNN, and DeepTrust using predictive performance metrics. For forecasting, ARIMA, Holt-Winters, a GluonTS feed-forward neural network, ANN, and LSTM are compared across short and long horizons using MSE-based evaluation. The available text (methods section) describes no formal inferential statistical hypothesis tests; comparisons appear to be based on point-estimate performance metrics.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| MSE (Mean Squared Error) — used as training loss and as a forecasting evaluation metric | LSTM and ANN forecasting model training and evaluation | — | not stated |
| Classification performance metric(s) — specific metric (e.g. accuracy, F1) not named in the available text excerpt | Comparison of CNN, LSTM, SVM, One-Class SVM, DNN, DeepTrust across five datasets | — | not stated |
-
Forecasting methods are compared using MSE as the primary evaluation metric↳ Could also: RMSE, MAE, or MAPE could also be used as forecasting evaluation metrics — RMSE is on the same scale as the original data and is more interpretable than MSE; MAE is less sensitive to outlier errors; reporting multiple metrics together gives a more complete picture of forecasting accuracy across different error regimes
-
Method comparisons are based on point-estimate performance metrics across datasets without formal statistical significance testing↳ Could also: A Friedman test with post-hoc Nemenyi test, or pairwise Wilcoxon signed-rank tests across datasets, could also be applied — Formal non-parametric tests across multiple benchmark datasets are a standard practice in ML comparison studies (e.g., Demšar 2006) and allow assessment of whether observed performance differences are statistically reliable rather than dataset-specific
-
Models appear to be trained and evaluated on a single train/test partition per dataset↳ Could also: k-fold cross-validation or leave-one-out cross-validation could also be applied, particularly for smaller gene expression datasets — Cross-validation reduces variance in performance estimates and provides more stable generalization estimates, which is especially relevant when individual datasets may have limited numbers of gene profiles
-
Forecasting performance is reported without uncertainty quantification around predictions↳ Could also: Prediction intervals or probabilistic metrics such as CRPS (Continuous Ranked Probability Score) could also be reported — Prediction intervals convey forecast uncertainty, which is relevant for biological interpretation; the GluonTS toolkit already used in the paper natively supports probabilistic forecasting models
-
Clustering (via STEM) is compared to supervised classification informally, based on qualitative inspection of outputs↳ Could also: External cluster validity indices such as Adjusted Rand Index (ARI) or Normalized Mutual Information (NMI) computed against available ground-truth labels could also be used — Quantitative external validity indices would make the supervised-vs-unsupervised comparison more precise and reproducible, since both approaches could then be evaluated on the same numerical scale
-
Deep learning hyperparameters (layer sizes, dropout rate, batch size, optimizer) are fixed and described but not systematically varied↳ Could also: Systematic hyperparameter search (e.g., grid search or random search with cross-validation) could also be applied to each method — Hyperparameter sensitivity analysis helps distinguish whether observed performance differences reflect fundamental method properties or the particular configuration chosen, which strengthens the validity of benchmark conclusions
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
ARIMA achieves best performance for short-term forecasting of time series gene expression.microarray multiple-gene-expression-datasets 2020×1papers★ This paper is the founder (earliest)
-
Deep learning classification methods outperform traditional approaches for time series gene expression classification.microarray multiple-gene-expression-datasets up 2020×1papers★ This paper is the founder (earliest)
-
Deep learning methods outperform statistical approaches including ARIMA for long-term forecasting of time series gene expression.microarray multiple-gene-expression-datasets 2020×1papers★ This paper is the founder (earliest)
-
Supervised classification outperforms unsupervised clustering for time series gene expression when class labels are available.microarray multiple-gene-expression-datasets up 2020×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33156855
Paper: Tripto NI, Kabir M, Bayzid MS, Rahman A. Evaluation of classification and forecasting methods on time series gene expression data. PLoS One 2020; 15(11):e0241686. PMID 33156855 / PMC7647064.
Code: https://github.com/mahi045/time-series-gene-expression (commit
27d4047610eb456c12d4d3b032407ce4452e03f7, pushed 2023-11-21, public, no LICENSE).
The repo is the authors' own code. Datasets are shipped inside the repo
(Code/Datasets/GSE{3406,6186,1723,20305}_series_matrix.txt) as pre-processed
matrices: column 1 = gene/profile id, column 2 = integer class label, columns 3..N =
expression values across time points. So GEO re-download is not needed; the
exact inputs the authors used are in the repo.
Reported results and their pipelines
The paper has two computational halves:
- Classification (Tables 1–3): CNN, LSTM, SVM, One-Class SVM, DNN, DeepTrust, and a STEM clustering baseline, scored by accuracy / F1 on 5 datasets.
- Forecasting (Table 4): Holt-Winters, ARIMA, ANN, LSTM, GluonTS FFNN, scored by RMSE at test fractions 10/20/30/40 %.
In scope (attempted — deterministic, clean 1:1)
| Result | Pipeline | Why in scope |
|---|---|---|
| Table 4, ARIMA rows | Code/Forecasting/Arima model.py — statsmodels ARIMA(order=(3,1,0)), walk-forward one-step forecast, RMSE per gene averaged over the first 100 genes |
Fully deterministic given the shipped data + fixed order + fixed train/test split (LAST_IDX = ceil(N_TIME*(1-test%))). No RNG. |
| Table 4, Holt-Winters rows | Code/Forecasting/Statistical Methods.py — triple exponential smoothing (slen=2, alpha=0.4, beta=0.05, gamma=0.4), RMSE per gene averaged over all genes |
Fully deterministic, closed-form. No RNG. |
Targets: GSE3406 (this room's named dataset), GSE6186, GSE1723 at test% 10/20/30/40.
Out of scope (not attempted, with reason)
- Classification Tables 1–3 (CNN/LSTM/DNN/SVM/One-Class-SVM/DeepTrust/STEM).
The shipped classification scripts use
train_test_split(..., random_state=None)(seeCode/New Data Anomaly/supervised ml.py) — i.e. a fresh random split every run, so a single printed accuracy is not reproducible to a fixed value; and the CNN/LSTM/DNN headline methods are stochastic deep nets with unpinned TensorFlow/Keras versions and no fixed seeds. This is the hard ~20%; per the brief we do not chase it. (A single deterministic SVM number could be produced only as a mean±sd over many splits, which would not be a 1:1 match to the paper's single cell.) - Table 4 ANN / LSTM / GluonTS rows. Stochastic neural forecasters (Keras / GluonTS), unpinned versions, no seeds → not 1:1. Skipped.
- Wet-lab / dataset-curation steps: none (this is a pure methods-benchmark paper).
Reproducibility hazards noted (for the human auditor)
- The authors'
create_GSE_dataset()callsf.readline()before the loop, which silently skips the first data row (the files have no header line). We faithfully replicate this quirk so our averages match theirs. Arima model.pyaverages over the first 100 genes only (sample_size=100); Holt-Winters averages over all genes. We replicate each method's own averaging.- Table 4 ARIMA @ 40 % is reported as "—" for every dataset; we let ARIMA attempt it and record whether it likewise fails (too few training points for order (3,1,0)).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The paper's primary, headline result — ARIMA as the best short-term forecaster — reproduces essentially 1:1 by running the authors' own shipped code and data (all 6 reported GSE6186/GSE3406 cells within ±0.005 RMSE), so the central conclusion holds and there is no fabrication signal. The deviations are confined to secondary cells: Holt-Winters drifts ~0.1–0.2 RMSE higher as the horizon grows (unspecified HW averaging/horizon), and GSE1723 is ~100× off (45–101 vs 0.35–1.07) because the shipped loader does not normalise that dataset, implying an undocumented normalisation on the authors' side. Severity is therefore mixed — negligible for the core result, severe but explainable (scaling/preprocessing artifact) for GSE1723 — and blame splits between our self-chosen HW steps and the paper's incomplete preprocessing documentation. Overall: a solid reproduction with explainable, non-critical deviations.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.