Sustainable data analysis with Snakemake.
The main results reproduced, with only marginal, non-material deviations.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Snakemake software/methods paper. In-scope pipeline-derived quantitative results reproduced on «our HPC» (snakemake 9.23.1, single core, AMD EPYC 9654). FIG 9 (DAG-build runtime+memory vs #jobs) REPRODUCED via a synthetic N-job Snakefile dry-run (snakemake -n, DAG build only, «path» -v) spanning the paper's full 11..90000-job axis: both quantities scale LINEARLY (memory R2=0.99997 @ 13.5 KB/job; runtime R2=0.99964 for N>=5000). Memory anchors match within ~15% (11 jobs 75 vs 90 MB; 90000 jobs 1.23 vs 1.1 GB). Absolute wall-clock is higher at small N (5.8 vs 0.5 s) due to a ~5 s snakemake-9 import/startup floor and a slower-clock EPYC core vs the paper's i7-1370P @ up to 5.2 GHz -- honestly noted; the linear shape + memory are the hardware-independent reproducible claims and they hold. FIG 2a: cumulative commit curve rises monotonically (5920 commits, public snakemake/snakemake history). An authors'-OWN 1:1 hello-world benchmark (Zenodo workflow.tar.gz) is being re-run as a cross-check. NOT attempted (out of scope): conceptual figs 1,3-8; bibliometric figs 2b/2c (proprietary dimensions.ai). All grades provisional; a human reviewer signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 71assessed: 2026-06-20 ⛓ fb07b3a33606
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper postulates that in silico reproducibility alone is not sufficient for data analysis to have lasting, sustainable value to a research field, and that transparency and adaptability are equally necessary; it examines whether the Snakemake workflow management system can provide all three properties.
- ★ Reproducibility alone is insufficient for sustainable data analysis; transparency and adaptability are equally important additional properties. finding
- ★ Snakemake supports reproducibility, transparency, and adaptability by enabling automated, scalable, portable, readable, documented, traceable, and modular workflow definitions. finding
- ★ Snakemake workflows are specified via rules that describe how to obtain output files from input files, using wildcards for generality. method
- ★ Dependencies between jobs are inferred implicitly and automatically by matching input filenames to rules that can generate them, yielding a directed acyclic graph (DAG) of jobs. mechanism
- Snakemake can automatically generate a suite of unit tests (executable via Pytest) from a source workflow with pre-verified results. method
- ★ Snakemake is one of the most widely used workflow management systems in science, evidenced by citation counts and download counts. finding
- Workflow definition languages can be categorized by design niche (GUI-based, generic-language libraries, DSL-based, purely declarative, system-independent languages like CWL/WDL). finding
- Snakemake workflows can be automatically exported to CWL and can use CWL tool definitions, promoting interoperability. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| citation analysis | Snakemake publications (original 2012 article and this article) | none | citations by year and by scientific discipline of citing articles | Dimensions.ai (badge.dimensions.ai) |
| software repository/version-control analysis | Snakemake codebase | none | cumulative number of git commits over time and release markers | — |
| readability/knowledge-requirement categorization | example Snakemake workflow (Figure 3) | none | per-line classification into domain, technical, Snakemake, or trivial knowledge categories | — |
| automated unit test generation and comparison | Snakemake workflow rules run against source workflow input data | none | byte-by-byte comparison of generated rule outputs to source workflow results | Pytest |
- ▲ Snakemake averages more than 12 new citations per week and has accumulated almost 3000 citations in total. ~12 citations/week; ~3000 total
- ▲ Snakemake has a download count of over 1 million. >1,000,000 downloads
- – In the example workflow (Figure 3), only six lines required Snakemake-specific knowledge to understand, with most lines understandable via general, domain, or technical knowledge. 6 lines
- – DAG construction time scales linearly with the number of jobs and is negligible relative to typical workflow step runtimes.
- count more than 12 new citations per week (citation growth rate of Snakemake publications)
- count almost 3000 citations in total (cumulative citations of original and this Snakemake publication)
- count over 1 million (Snakemake download count)
- count six lines (lines requiring Snakemake-specific knowledge out of example workflow in Figure 3)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software methods paper describing the Snakemake workflow management system; it contains no inferential statistical analyses. Quantitative content is limited to descriptive counts and trends: cumulative git commits over time, citation counts by year and scientific discipline sourced from Dimensions.ai, and a categorization of example workflow code lines by required knowledge level, motivated by reference to the Dale-Chall readability formula. All results are reported as raw counts or qualitative groupings without formal hypothesis testing.
-
Citation counts were drawn from a single external source (Dimensions.ai) accessed on one specific date (2024-09-20) and reported as a point estimate↳ Could also: Multiple bibliometric databases (e.g., Web of Science, Scopus, Google Scholar) could also be queried and cross-referenced to triangulate citation totals — Different databases index different journals and conference proceedings; reporting a range across sources would give readers a sense of the uncertainty inherent in any single-database count
-
Readability of the example workflow was assessed by authorial categorization of each code line into one of four knowledge tiers (domain, technical, Snakemake-specific, trivial)↳ Could also: A formal inter-rater reliability study using independent annotators with agreement quantified by Cohen's kappa or Fleiss' kappa could also be applied — Kappa statistics would make the categorization scheme empirically verifiable and allow readers to judge how consistently the tiers can be assigned across raters and workflows
-
Cumulative git commit counts over time were presented as the primary measure of development activity (Figure 2a)↳ Could also: Additional metrics such as number of unique contributors per period, issue resolution rate, or time-between-releases could also be reported alongside commit counts — Commit counts can reflect commit granularity conventions rather than effort volume; complementary metrics would provide a more multi-dimensional view of project health and community engagement
-
Total download count (>1 million) was cited as a single cumulative figure with no temporal breakdown or confidence bound↳ Could also: A time-series of download rates (e.g., monthly or quarterly) could also be presented to show the adoption trajectory — A cumulative total cannot distinguish accelerating growth from a historical peak followed by plateau; a time series would allow readers to assess current momentum
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34035898
Paper: Mölder F, Jablonski KP, … Köster J. "Sustainable data analysis with Snakemake." F1000Research 2021 (rolling, v3). PMID 34035898 · PMCID PMC8114187 · DOI 10.12688/f1000research.29032.3.
This is a software/methods paper describing the Snakemake workflow management system. Most of the paper is conceptual (design patterns, scheduling theory, language design). The paper is itself "self-reproducing": its quantitative figures are produced by a Snakemake workflow archived on Zenodo.
Code / data artifacts as stated in the paper
- Software availability: Snakemake — MIT, https://github.com/snakemake/snakemake (the tool itself).
- Data availability: "The Snakemake workflow generating the results presented in this work, along with the corresponding Snakemake report … is available at https://doi.org/10.5281/zenodo.4244143" (CC-BY 4.0).
- Zenodo 4244143 (concept DOI) → latest version 10.5281/zenodo.13819541 (2024-09-27).
Files:
workflow.tar.gz(501.2 MB),report.html(3.1 MB). Linked GitHub:github.com/koesterlab/snakemake-paper-2020→ HTTP 404 (gone/private). → The archived Zenodoworkflow.tar.gzis the surviving runnable artifact.
NOTE: the registry code_url (github.com/pditommaso/awesome-pipeline) is a
harvesting artifact — that is an unrelated curated list of pipeline tools, not
this paper's code. The real artifact is the Zenodo workflow archive above.
Reproducible pipeline-derived results (IN SCOPE)
| Result | What it is | Reproducibility |
|---|---|---|
| Figure 9 (PRIMARY) | Snakemake DAG-build runtime & memory vs number of jobs (single core, i7-1370P). Reported: linear scaling; 11 jobs → 0.5 s / 90 MB; 90,000 jobs → 37 s / 1.1 GB. | Pure-CPU synthetic benchmark, no external data. Reproducible on «our HPC». Hardware-dependent absolute runtime (i7-1370P @ up to 5.2 GHz ≠ «our HPC» node) → expect linearity + memory to match, wall-clock to differ. |
| Figure 2a | Cumulative git commits of Snakemake over time. | Reproducible from the public snakemake/snakemake git history (git log). Snapshot-date dependent. |
| Figure 3d | Knowledge-requirement counts per statement category for the example workflow. | Small hand-codeable count; low value, secondary. |
OUT OF SCOPE (not pipeline-derived / not reproducible 1:1)
- Figures 1, 3a–c, 4, 5, 6, 7, 8 — conceptual diagrams / illustrative example workflow DAGs (design patterns, scheduling theory). No quantitative output.
- Figure 2b, 2c — citation counts by year / discipline. Sourced from dimensions.ai (badge.dimensions.ai) at a fixed snapshot date 2024/09/20; external proprietary bibliometric service, not derivable from shipped code/data → out of scope (external data dependency).
Pipeline named per in-scope result
- Fig 9 & Fig 2a: the paper's own Snakemake workflow (Zenodo
workflow.tar.gz), run with Snakemake itself on the paper's synthetic inputs (Fig 9) and on the public Snakemake git repo (Fig 2a).
Primary target
Figure 9 scaling benchmark — the single clearest pipeline-derived quantitative claim with an explicit code+data pointer.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.