Design considerations for workflow management systems use in production genomics research and the clinic.
The main results reproduced, with only marginal, non-material deviations.
- ✓No authors-side cause for any deviation
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Reproduced the engine-scalability experiment (the paper's in-scope, pipeline-derived result) on «our HPC» using the paper's own third-party benchmark repo azzaea/scalability-tst@6448f64: Nextflow 21.04.3 (DSL1) vs Cromwell 63 + WDL, wall-clock vs task-count sweep [1,2,4,8,16,32,64,128,256] x3 repeats, 1-step + 2-step hostname-scatter workflows, local executors on one 128-core std node (isolates pure engine overhead, the paper's own framing). SLURM «job» completed cleanly (1.5 h, 111 timed runs). RESULTS: (C1) both engines near-flat to 256 tasks (max/min <=2.49x) -> within-tol; (C2) Nextflow much faster, median Cromwell/NF ratio 6.87x -> partial (same direction/order as the paper's ~4x AWS figure, but platform-dependent factor ~7x not exactly 4x); (C5) NF faster at every single point -> partial (50x is a multi-node Biocluster peak, not reachable single-node). C3/C4 (512/1024-task failure thresholds) are tied to the paper's AWS 100-node x 96-core cluster and are NOT reproducible on a single node nor pipeline-derived -> recorded out-of-scope. The variant-calling sub-result was dropped (synthetic WES inputs on-request only). NOT attempted: absolute-second matching to AWS/Biocluster (cluster-specific, not the reproducible essence) and the platform failure thresholds. Net: the qualitative essence of the scalability claim reproduces well (flat scaling + Nextflow decisively faster than Cromwell); the exact 4x factor is platform-dependent. Verdicts provisional, must be independently checked.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 57assessed: 2026-06-21 ⛓ aab25bfd3733
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-21
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper investigates which workflow management system (WfMS) should be chosen for a given bioinformatics application regardless of analysis type, systematically evaluating Nextflow, CWL, WDL, and Swift/T using two use cases (a variant-calling pipeline and a scalability-testing framework) run locally, on an HPC cluster, and in the cloud.
- ★ The choice of WfMS depends both on intrinsic language/engine features and on adoption, collaboration, and technical support within bioinformatics consortia. finding
- ★ CWL and WDL are declarative language specifications prioritizing reproducibility and readability at the cost of expressiveness, while Nextflow and Swift/T are complete systems (language + execution engine) offering greater expressiveness. finding
- ★ Swift/T offers the most expressive and extensible language, inherited from C and Tcl, but has a steep learning curve. finding
- ★ WDL provides the most intuitive modular workflow scripting via explicit tasks, workflows, and subworkflows. finding
- ★ CWL workflow code is extremely verbose compared to the other three WfMSs, despite being modular and extensible by design. finding
- Conditionals were not directly supported in CWL until version 1.2.0, released in August 2020. finding
- Nextflow lacked the ability to import and reuse processes until the introduction of Nextflow DSL-2. finding
- ★ CWL and WDL are supported by multiple independent execution engines (e.g., cwltool, arvados, toil-cwl-runner, cwl-airflow, REANA, Cromwell, cwl-tes, rabix; Cromwell, toil-wdl-runner, miniWDL), causing feature-support inconsistencies, whereas Nextflow and Swift/T package language and engine together to avoid such issues. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Variant-calling genomics pipeline (WfMS comparative benchmark) | Local workstation, HPC cluster, and cloud compute environments | other (same pipeline implemented independently in four WfMSs: Nextflow, CWL, WDL, Swift/T) | Language expressiveness, modularity, scalability, robustness, reproducibility, portability, interoperability, and ease of development | — |
| Scalability-testing framework | Local workstation, HPC cluster, and cloud compute environments | other (workflow scaled across dozens to hundreds of nodes under each WfMS) | Scalability and robustness of task execution across WfMSs | — |
- – CWL added direct support for conditionals only in version 1.2.0 (August 2020)
- – toil-wdl-runner did not support nesting conditionals within loops until March 2020
- – Nested loops in WDL draft-2 code are not executable by Cromwell
- – Nextflow DSL-2 added the ability to import and reuse processes, a capability previously lacking
- – CWL code is markedly more verbose than Nextflow, WDL, and Swift/T code, increasing development time
- – WDL offers the most intuitive/readable modular workflow scripting among the four WfMSs, aiding maintainability
- other CWL v1.2.0 (August 2020) (Version/date CWL added direct support for conditionals)
- other March 2020 (Date toil-wdl-runner gained support for nesting conditionals within loops)
- count 4 (Number of WfMSs systematically compared (Swift/T, Nextflow, CWL, WDL))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper is a systematic qualitative and quantitative benchmarking evaluation of four bioinformatics Workflow Management Systems (Nextflow, CWL, WDL, Swift/T) across two use cases (a variant-calling pipeline and a scalability-testing framework) run in local, HPC, and cloud environments. The evaluation is primarily descriptive and comparative, assessing feature dimensions (modularity, scalability, robustness, reproducibility, portability, interoperability, ease of development) rather than testing biological hypotheses. No inferential statistical tests are reported in the text provided; results appear to be communicated through qualitative narrative, feature matrices (tables), and performance benchmarks.
-
The comparison of WfMS feature dimensions is presented primarily through qualitative narrative and binary feature-presence tables↳ Could also: A structured scoring rubric with weighted criteria (e.g., analytic hierarchy process or a Likert-scale expert survey across multiple independent raters) could also be used to summarize feature assessments — A formal scoring or multi-rater approach would make implicit weighting decisions explicit and allow inter-rater reliability to be reported, supporting reproducibility of the evaluation itself
-
Scalability and runtime performance comparisons are described qualitatively; the number of replicate benchmark runs and variability of runtime measurements are not stated in the provided text↳ Could also: Reporting median wall-clock time with interquartile range or mean ± SD across repeated runs, combined with a non-parametric test (e.g., Kruskal-Wallis with post-hoc Dunn correction) across WfMSs at each scale point, would also characterize performance — Repeated-run statistics would distinguish true performance differences from run-to-run noise (scheduler jitter, I/O variability) inherent in HPC and cloud environments
-
Evaluation of 'ease of development' and related qualitative dimensions relies on expert judgment by the authors↳ Could also: A user study with participants of varying bioinformatics experience completing standardized tasks in each WfMS, scored with established usability instruments (e.g., System Usability Scale, NASA-TLX), could also quantify this dimension — Structured usability metrics would generalize beyond the authors' own experience and allow statistical comparison of perceived difficulty across systems and user skill levels
-
Adoption and community usage of each WfMS are described narratively without a defined measurement strategy in the provided text↳ Could also: Quantitative proxies such as GitHub stars, forks, commit frequency, Stack Overflow tag volume, or PubMed citation counts over time could also be reported with a time-series or cross-sectional comparison — Objective adoption metrics complement expert opinion and allow readers to assess community momentum independently of the authors' experience
-
The two use cases (variant-calling pipeline and scalability framework) serve as the evaluation substrate; generalizability to other pipeline types is discussed narratively↳ Could also: A broader benchmark suite covering additional pipeline archetypes (e.g., RNA-seq, metagenomics, image analysis) could also be used to assess whether conclusions hold across workflow structures — Multiple pipeline types would support or qualify the scope of the recommendations, since WfMS performance characteristics may depend on task granularity, I/O patterns, or dependency graph topology
-
Resource consumption (CPU, memory, I/O overhead) attributable to the WfMS layer itself versus the analysis tools is not separated in the provided text↳ Could also: Profiling runs with a minimal no-op task DAG of the same topology could also be used to isolate WfMS orchestration overhead from tool runtime — Isolating overhead would let readers distinguish WfMS engine efficiency from confounding differences in how each system interacts with the job scheduler or container runtime
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.