Dynamic Gene Attention Focus (DyGAF): Enhancing Biomarker Identification Through Dual-Model Attention Networks.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- 🔴Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to RUN, but not to REPRODUCE the headline number. DyGAF is the authors' own package (PyPI 0.0.3 == github @852d25b); it installs (after adding two deps the README omits: tensorflow-addons, matplotlib) and runs 1:1 as documented on the only input shipped, input/covid_sampled.csv. That file is a 30-sample x 19939-gene DEMO subset (Target 15/15; repo calls it 'Normalized Sample input'), NOT the ~260-sample normalized GSE188678 matrix used for Table 1. So the pipeline yields a real but different number: testing accuracy 83.33% (5/6) vs reported 94.23% (=49/52, which needs a 52-sample test set => ~260 samples). Training accuracy reproduces (100%, though the RF overfits this tiny set so it is weak evidence). Result is deterministic: the final RandomForest + 80/20 split are hard-coded random_state=4, and n_splits in {2,5} both gave 83.33%. NOT a fabrication flag: 94.23% is plausibly derivable from the full dataset, which is simply not shipped (data-completeness gap). NOT attempted (hard 20%): reconstructing the full normalized input from raw GSE188678 (no normalization recipe given), the Table-1 control models (DEA-RF/RF/SVM/KNN, not in the package), edgeR DEGs, and the GO/KEGG/PPI downstream analyses (external/manual).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 44assessed: 2026-06-14 ⛓ 43d3f66f51ce
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe authors hypothesize that biomarker identification for COVID-19 can be improved by a method that simultaneously captures both the independent molecular changes of individual genes in response to infection and the dependent, interconnected roles of genes within regulatory networks, rather than addressing only one aspect as DEA or conventional machine learning do.
- ★ DyGAF, a dual-model attention-based neural network combined with random forest feature selection, identifies and ranks genes by significance for COVID-19 biomarker detection. method
- ★ DyGAF classifies COVID-19 gene-expression profiles to diagnose patients with 94.23% accuracy. finding
- ★ DyGAF identifies novel gene ontologies, KEGG/Wiki pathways, and key genes compared with traditional methods such as DEA and random forest. finding
- ★ Model A uses element-wise multiplication attention to weight independent gene functions while Model B uses dot-product/sparsemax attention to capture gene interdependencies. mechanism
- ★ A custom combined scoring function integrates attention weights from Model A and Model B to produce a unified gene importance ranking. method
- ★ DyGAF outperforms benchmarked conventional models in identifying and categorizing COVID-19 cases. finding
- DyGAF is released as an open-source software package. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq gene expression profiling | human nasopharyngeal swab samples (COVID-19 and non-viral/healthy individuals) | SARS-CoV-2 infection (disease vs control) | gene-level read counts / expression (~19,939 genes) | GEO accession GSE188678; Kallisto quantification, tximport, edgeR TMM normalization |
| Differential expression analysis (DEA) | human nasopharyngeal swab RNA-seq (169 control, 90 COVID-19) | COVID-19 vs control | differentially expressed genes (adj-P ⩽ .05, Benjamini-Hochberg FDR) | edgeR |
| Dual-model attention-based feature selection (DyGAF deep learning) | COVID-19 RNA-seq dataset | none (computational feature selection) | attention-weighted gene importance scores / ranked genes | TensorFlow/Keras, Adam optimizer, binary cross-entropy |
| Random forest feature selection | COVID-19 RNA-seq dataset | none | ranked gene features for comparison/validation | — |
| Classification / diagnostic model (multi-head attention) | COVID-19 gene-expression profiles | none | COVID-19 vs control classification accuracy | TensorFlow/Keras; 4 attention heads, 5-fold cross-validation, 20% validation split |
| Functional enrichment analysis | DyGAF-ranked genes | none | enriched KEGG pathways, Wiki Pathways, and Gene Ontology terms | — |
- – DyGAF diagnosed COVID-19 patients by classifying gene-expression profiles with high accuracy. 94.23% accuracy
- – Batch size of 16 over 100 epochs yielded the best (state-of-the-art) results during hyperparameter tuning.
- – Multi-head attention model with 4 heads best balanced performance and interpretability. 239,269 trainable parameters per head
- – DyGAF identified novel GOs, pathways, and key genes versus DEA and random forest.
- other 94.23% accuracy (DyGAF COVID-19 classification accuracy)
- count 169 control samples and 90 COVID-19 samples (dataset composition from nasopharyngeal swabs)
- count approximately 19,939 genes (number of genes covered in dataset)
- count 239,269 trainable parameters per head (4-head multi-head attention model)
- pvalue adjusted P-value threshold ⩽ .05 (Benjamini-Hochberg FDR for DEG significance)
- other 87.22% accuracy (prior study SVM classifier with 6 COVID-19 genes (BRSA) — cited reference)
- other AUC of 85% (prior 18-gene severity signature by Peterson et al — cited reference)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The study preprocessed COVID-19 RNA-seq count data (169 controls, 90 COVID-19 patients, ~19,939 genes) using Kallisto, tximport, and TMM normalization via edgeR, then identified differentially expressed genes using edgeR with Benjamini-Hochberg FDR correction at adj-p ≤ 0.05. The primary contribution is the DyGAF dual-attention neural network, which combines element-wise (independent) and dot-product (dependent) attention scores into a composite gene-ranking metric, trained with Adam/binary cross-entropy and evaluated by 5-fold cross-validation. Classification performance was reported as a single accuracy value (94.23%), and DyGAF gene rankings were compared against DEA and random forest outputs.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| edgeR differential expression test (exact test or quasi-likelihood F-test; specific test variant not stated) | COVID-19 vs. non-viral control comparison across ~19,939 genes | 259 samples (169 control, 90 COVID-19) | not stated |
| Random forest feature selection (importance metric not specified) | Gene ranking comparison benchmark against DyGAF | 259 samples | not stated |
| 5-fold cross-validation with binary cross-entropy loss | DyGAF model training and performance estimation | 259 samples partitioned into 5 folds | not stated |
| Classification accuracy | DyGAF COVID-19 vs. control classification benchmark against other models | exact held-out n not stated; derived from 259-sample 5-fold CV | not stated |
-
Classification performance was reported as a single accuracy value (94.23%)↳ Could also: AUC-ROC, F1-score, Matthews Correlation Coefficient, or sensitivity/specificity could also be reported alongside accuracy — With a class-imbalanced dataset (~65% controls, ~35% COVID-19), accuracy alone is influenced by the majority class; AUC and F1 characterize the sensitivity/specificity tradeoff and are widely used complementary metrics for imbalanced clinical classification tasks
-
Cross-validation performance was reported as a point estimate without dispersion across folds↳ Could also: Reporting mean ± SD (or 95% CI) of the performance metric across the 5 folds would also be standard practice — Fold-to-fold variance communicates the stability of the performance estimate; reporting it alongside the mean is common in k-fold validation workflows and helps readers gauge result reliability
-
Hyperparameters (number of attention heads, batch size) were selected by comparing configurations on the validation split and the same data were used to report final accuracy↳ Could also: A separate held-out test set or a nested (outer) cross-validation loop could also be used to evaluate the final hyperparameter-selected model — When hyperparameters are chosen based on validation-set performance, a truly independent test set or outer CV fold provides a less optimistic estimate of generalization; this is a standard design in model-selection workflows
-
Differential expression was identified using edgeR with TMM normalization↳ Could also: DESeq2 with median-of-ratios normalization and its shrinkage-based dispersion estimation could also be applied to the same two-group RNA-seq comparison — DESeq2 and edgeR use different negative-binomial dispersion estimation strategies; running both and comparing overlapping DEGs is a recognized sensitivity analysis in RNA-seq studies
-
Gene importance ranking was assessed via the composite DyGAF attention score without uncertainty quantification↳ Could also: Bootstrap resampling or permutation testing on the attention-weight scores could also be used to attach confidence intervals to gene rankings — Uncertainty estimates around importance scores would allow readers to distinguish genes whose rankings are stable across data subsets from those sensitive to sampling variation, which is particularly relevant for biomarker prioritization
-
A fixed 80/20 training/validation split was used alongside 5-fold cross-validation↳ Could also: Stratified k-fold cross-validation (preserving the ~65%/35% class ratio within each fold) could also be applied — Stratification ensures each fold reflects the overall class distribution; with a 1.9:1 control-to-case ratio, unstratified splits may yield folds with unequal class proportions, which can affect both training stability and performance estimates
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
6 downstream papers · 1 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Identifying novel host-based diagnostic biomarker pa... 2022 · 13 cites
- A 2-Gene Host Signature for Improved Accuracy of COV... 2023 · 9 cites
- A machine learning classifier using 33 host immune r... 2023 · 7 cites
- The two-stage molecular scenery of SARS-CoV-2 infect... 2023 · 3 cites
- Exploring COVID-19 Pandemic Disparities with Transcr... 2024 · 2 cites
- Identification of gene classifier and modules associ... 2023 · 0 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40160891 (DyGAF)
Paper: Islam, Wagh, Wei. Dynamic Gene Attention Focus (DyGAF): Enhancing Biomarker Identification Through Dual-Model Attention Networks. Bioinform Biol Insights 2025. PMID 40160891 · DOI 10.1177/11779322251325390.
Code: https://github.com/hiddenntreasure/DyGAF (MIT, public, last push
2025-04-09) — also a PyPI package DyGAF==0.0.3. This is the authors' own
tool (P16 not relevant here).
Data: GEO GSE188678 (COVID-19 nasopharyngeal-swab RNA-seq). The repo
ships a normalized, sampled input matrix input/covid_sampled.csv (tab-separated
despite the .csv name; column Target) — the exact input the package consumes.
How the package works (read from DyGAF 0.0.3 source)
DyGAF(df_path, target_column, seed, n_splits) →
- dependent + independent multi-head (4-head) attention nets, trained in
StratifiedKFold (TF 2.10 + tensorflow-addons sparsemax), seeded by
seed(np/random/tf). Each writes per-gene mean attention weights. analyze_feature_importance: normalize both weight vectors → custom score2*((min+prod)/2)^2→ multiply each feature by its score → RandomForest (n=100, max_depth=10, min_samples_split=5, random_state=4) on a fixedtrain_test_split(test_size=0.2, stratify=y, random_state=4).- Returns
(features_df ranked by RF importance, test accuracy).
The final classifier + split are hard-coded random_state=4 → deterministic
given the attention weights; only the TF attention step carries seed-dependent
(and possible CPU float) variation.
IN SCOPE (pipeline-derived, attempted)
- C1 — DyGAF testing accuracy 94.23% (Abstract; Table 1). Directly returned by the package. PRIMARY target.
- C2 — DyGAF training accuracy 100% (Table 1). Re-derived from the same final RF.
- C3 — DyGAF F1 96, specificity 91.9, sensitivity 100 (Table 1). Re-derived from the same final RF confusion matrix (label-orientation noted).
- C4 — DyGAF feature/biomarker ranking (top genes; Table 1 / Fig context):
produced as
features_importanceCSV. Compared qualitatively (the paper's exact top-100 list lives in Supplemental Tables; full overlap = the hard 20%).
OUT OF SCOPE (not attempted, why)
- edgeR DEG count (1702 DEGs), DEA-RF, DyGAF-SVM/KNN control rows of Table 1 — these control models are not implemented in the shipped package (only DyGAF).
- KEGG / WikiPathways / GO enrichment, STRING PPI, top-10 hub genes (Figs 5-6) — downstream external-tool analyses, not in the package; manual/web steps.
- Re-deriving
covid_sampled.csvfrom raw GSE188678 counts — the normalization recipe to the shipped matrix is not specified; we use the shipped matrix (the most faithful 1:1 input).
Drop check
NOT a drop: code public+installable, data shipped in repo, expected value (94.23%) pinnable. Eligible.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
DyGAF is the authors' own package and it installs/runs 1:1 as documented, reproducing the end-to-end attention->custom-score->RandomForest pipeline and the 100% training accuracy. The headline testing accuracy (94.23%), F1 (96) and specificity (91.9) do not reproduce because the repo ships only a 30-sample demo (covid_sampled.csv), not the ~260-sample normalized GSE188678 matrix (94.23% = 49/52 needs a 52-sample test set), and no normalization recipe is provided. This is an authors'-side data-completeness gap, not fabrication — the numbers are plausibly derivable from the full dataset that simply was not deposited. Severity is moderate: the classifier still works (83.33% acc, 100% sensitivity) and the core conclusion holds in a limited sense, but the exact reported values are not derivable from the shared artifacts.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.