SEMdag: Fast learning of Directed Acyclic Graphs via node or layer ordering.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough -> reproduced, close to 1:1. Ran the authors' own SEMdag pipeline (SEMgraph 1.2.4 / SEMdata 1.1.1, mirroring SEMgraph/SEMdag/R/main.R) on the paper's ALS data (GSE124439, bundled as SEMgraph::alsData) on «our HPC» (SLURM «job», node n095, 3m10s). EXACT reproduction of every deterministic claim: KEGG ALS pathway 364 nodes/333 edges -> LCC 190/261; ALS N=160 (139 ALS + 21 ctrl). Table-4 DAG structures reproduce closely: knowledge-based KB configs tightly (KB_TL edges 522 EXACT, KB source/target/degree/betweenness EXACT, vertices 169 vs 168), data-driven BU configs within a few percent (98 vs 100 vertices, edges 286 vs 284 / 264 vs 278). The only deviations trace to a benign DATA-VERSION DRIFT: the bundled alsData now has 17695 genes whereas the authors' main.R comments document 15936, so the top-100 limma DEGs differ slightly and caret::findCorrelation(0.99) drops 2 genes vs the authors' documented 0 -> BU graphs are 2 vertices smaller and KB ~1 vertex larger. Not a method/code error. NOT ATTEMPTED (out of scope, 80/20): Fig-5 disease-prediction MCC on test set GSE153960 (separate large download + stochastic run_fda/run_rf); the 6 comparison methods (PC/GES/ARGES/LiNGAM/CAM/NOTEARS, extra heavy packages); BRCA/COVID-19/STEMI cases (different accessions). Infra note: front1 has no usable local disk (/tmp 50M) and the home conda quota is exceeded + «infra» gave repeated conda ESTALE; resolved by building the conda env INSIDE the SLURM job with the package cache on the compute node's local /tmp (gotcha #4). Grades provisional; human signs off in AUDIT.md.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 81assessed: 2026-06-20 ⛓ 4ae567a5ace7
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-20
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-20no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a two-step, order-based structure learning algorithm (SEMdag), using either knowledge-based or data-driven node/layer ordering followed by penalized regression, can efficiently and accurately recover high-dimensional linear SEM/DAG structures with disease prediction performance comparable or superior to six established causal discovery methods.
- ★ SEMdag() is a two-step order-based algorithm for fast learning of high-dimensional linear SEMs, using knowledge-based (KB) or data-driven bottom-up (BU) node/layer ordering followed by penalized (L1) DAG estimation method
- ★ When the initial graph is sparse, the bottom-up (BU) approach yields better disease prediction performance than the knowledge-based (KB) approach finding
- ★ When the initial graph is denser, both BU and KB approaches achieve high performance, with KB based on topological layers achieving the highest score finding
- ★ SEMdag() achieves superior disease predictive performance compared to six established causal discovery methods (ARGES, GES, PC, LiNGAM, CAM, NOTEARS) finding
- ★ SEMdag() can handle high-dimensional data with less computing load than alternative structure learning methods finding
- SEMdag() is implemented in the R package SEMgraph, freely available on CRAN resource
- Exact DAG recovery (beyond Markov equivalence class) is achievable via correct topological ordering under identifiability conditions such as equal error variances, non-Gaussian errors, or additive non-linear errors mechanism
- The directLiNGAM procedure estimates causal ordering by iteratively regressing variables and testing independence of residuals from regressors, exploiting non-Gaussianity mechanism
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| RNA-seq expression profiling | Amyotrophic Lateral Sclerosis (ALS) patient/control samples (train/test datasets) | none (observational case-control) | DAG structure recovery and disease classification performance (Random Forest) | — |
| RNA-seq expression profiling | Breast cancer (BRCA) patient/control samples (train/test datasets) | none (observational case-control) | DAG structure recovery and disease classification performance (Random Forest) | — |
| RNA-seq expression profiling | Coronavirus disease (COVID-19) patient/control samples (train/test datasets) | none (observational case-control) | DAG structure recovery and disease classification performance (Random Forest) | — |
| RNA-seq expression profiling | ST-elevation myocardial infarction (STEMI) patient/control samples (train/test datasets) | none (observational case-control) | DAG structure recovery and disease classification performance (Random Forest) | — |
| Comparative benchmarking of causal discovery algorithms (SEMdag vs PC, GES, ARGES, directLiNGAM, CAM, NOTEARS) | Human disease expression datasets (ALS, BRCA, COVID-19, STEMI) | none | graph structure plausibility and downstream disease prediction accuracy (RF) | R package SEMgraph |
- – BU approach outperforms KB approach for disease prediction when the underlying graph is sparse
- ▲ KB approach (topological layer ordering) achieves the highest performance when the graph is denser
- ▲ SEMdag() shows superior disease predictive performance versus previous research/comparison methods
- – SEMdag() handles high-dimensional problems with reduced computational load
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper is a methods paper introducing SEMdag(), a two-step order-based algorithm (in the R package SEMgraph) for learning high-dimensional linear structural equation model DAGs under an equal-error-variance assumption. Evaluation is presented as a computational benchmarking exercise: DAG structure recovery and downstream disease-prediction performance (via a supervised learning algorithm, RF) are compared against six established causal discovery methods (PC, GES, ARGES, LiNGAM, CAM, NOTEARS) using paired training/testing expression datasets for four diseases (ALS, BRCA, COVID-19, STEMI). The excerpted text (Introduction and Materials and Methods) does not include the Results/statistical-comparison section, so specific significance tests, effect sizes, or dispersion measures used to compare method performance are not present in the provided text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| conditional independence (partial correlation) test at a chosen significance level α | skeleton/edge-removal step of the PC algorithm (one of the six comparator methods described) | — | stated |
| regression-coefficient significance pruning (P > 0.05 excluded) | edge-strength estimation step following causal ordering in the LiNGAM-style order-based procedure | — | stated |
-
Method performance is compared via a single training/testing split per disease dataset and downstream random forest prediction accuracy.↳ Could also: k-fold cross-validation or repeated random train/test splits with performance summarized as mean ± SD (or CI) across folds — Repeated resampling would characterize the variability of the predictive-performance estimates and support a formal comparison (e.g., a paired test across folds) rather than relying on a single split.
-
Seven structure-learning approaches (SEMdag KB/BU plus six comparator methods) are benchmarked across four disease datasets.↳ Could also: an omnibus rank-based test across algorithms and datasets (e.g., Friedman test with post-hoc Nemenyi correction), a common approach in multi-algorithm/multi-dataset benchmarking — This would jointly account for the many pairwise algorithm comparisons being made across datasets and control the family-wise error rate of the overall comparison.
-
Individual regression coefficients (edges) in the order-based strength-estimation step are retained or pruned using an uncorrected P > 0.05 threshold.↳ Could also: a false discovery rate procedure (e.g., Benjamini-Hochberg) applied across all tested edges — When many coefficients are tested simultaneously, an FDR-based threshold controls the expected proportion of false positive edges rather than relying on a fixed per-edge p-value cutoff.
-
The PC algorithm's skeleton estimation performs many conditional independence tests at a single significance level α without a stated correction.↳ Could also: a Bonferroni-adjusted α or a stability-selection approach (e.g., StARS) across the sequence of conditional independence tests — This would help account for the cumulative testing burden incurred while iteratively removing edges during skeleton construction.
-
Disease-prediction performance is assessed using a single supervised learning algorithm (random forest).↳ Could also: benchmarking with additional classifiers (e.g., logistic regression, SVM, gradient boosting) and reporting performance with confidence intervals — Comparing across multiple classifiers and reporting interval estimates would provide additional evidence on how classifier-agnostic and precise the predictive-performance differences between structure-learning methods are.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-39775401 (SEMdag, Grassi & Tarantino, PLoS One 2025)
Paper
SEMdag: Fast learning of Directed Acyclic Graphs via node or layer ordering.
Method paper presenting the SEMdag() function in the SEMgraph R package.
Evaluated on 4 real disease datasets (ALS, BRCA, COVID-19, STEMI) + comparison
against 6 causal-discovery methods (PC, GES, ARGES, LiNGAM, CAM, NOTEARS).
In scope (pipeline-derived, reproduced)
The brief's data accession is GSE124439 = the ALS training dataset, which
ships inside the package as SEMgraph::alsData (expression + group) and the KEGG
"Amyotrophic lateral sclerosis" pathway lives in SEMdata::kegg.pathways. The
authors' own reproduction code is SEMgraph/SEMdag/R/{main.R,help.R}.
Reproducible, clearly-specified, low-compute outputs for the ALS case:
-
KEGG ALS reference pathway reduction (Methods/Results text): original 364 nodes / 333 edges → largest connected component 190 nodes / 261 edges. Pipeline:
properties(kegg.pathways[["Amyotrophic lateral sclerosis"]]). -
Table 4 — descriptive analysis of the recovered DAG structures for the 4 SEMdag configurations on ALS:
- SEMdag_BU_TO / BU_TL: empty graph + top-100 limma DEGs (data-driven ordering)
- SEMdag_KB_TO / KB_TL: KEGG ALS pathway + all genes (knowledge-based ordering)
Metrics: vertices, edges, source nodes (in-deg 0), target nodes (out-deg 0),
mean degree, betweenness.
Pipeline:
huge::huge.npn→limmatop-100 DEGs →caret::findCorrelation(cutoff 0.99) →SEMgraph::SEMdag(graph, data, LO=..., beta=0.1, penalty=FALSE).
Out of scope (not attempted, why)
- Disease prediction MCC (Fig 5) on the test set GSE153960/GSE153960_SE:
requires the held-out test data file (
GSE153960_SE.RData) not shipped in the repo's small data; downstreamrun_fda/run_rfrandom-forest evaluation is stochastic and depends on a separate large download. Lower priority (80/20). - Comparison methods (PC/GES/ARGES/LiNGAM/CAM/NOTEARS): need many extra packages (pcalg, CAM, gnlearn, bnlearn) and are not the SEMdag method itself.
- BRCA / COVID-19 / STEMI cases: different accessions, not this RU's dataset.
Determinism note
SEMdag uses glasso (step 1, fixed λ) + glmnet/OLS (step 2, penalty=FALSE);
ordering is from the fixed graph (KB) or residual variance (BU) — all
deterministic given identical input data + graph, which ship in the package.
Vertices/edges counts are therefore expected to be exactly reproducible.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Near-1:1 reproduction: the version-independent claims (KEGG ALS pathway 364/333 → LCC 190/261, N=160 = 139+21) match exactly, and knowledge-based DAGs reproduce tightly (KB_TL edges 522, betweenness 80/8, degree 6 EXACT; vertices off by 1). The only deviations are in the data-driven BU configs (98 vs 100 vertices, ~5% edges, betweenness 45→54) and trace cleanly to a benign data-version drift in the bundled SEMgraph::alsData (17695 vs the authors' documented 15936 genes), which alters the top-100 DEGs and the 0.99 correlation filter. This is a technical/version issue on neither side's logic — no code error, no fabrication signal, and the central conclusion holds; overall yellow (solid with explainable deviations).
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.