Ten simple rules and a template for creating workflows-as-applications.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🔴The deviation was non-trivial in magnitude
- 🔴The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
▸Reproduction agent’s raw note
DROP (non_pipeline). The publication is a 'Ten Simple Rules' guidance + software-template article, not a research paper with pipeline-derived results. There is no reported computational value, no analyzed dataset, and no resolvable analysis repo tied to a result (the brief's repo URL is a 404 placeholder). Nothing is reproducible because nothing was computed/claimed as a result. Verified the paper type via PubMed (PMID 36520686 / PLOS Comput Biol 18(12):e1010705) and confirmed the placeholder repo 404. Not attempted: any «our HPC» job, since there is no in-scope pipeline result. Verdict is provisional and must be human-audited.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-19 ⛓ edee7e42cd81
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe paper argues that bioinformatics tools built on workflow managers should be shipped as user-friendly command-line applications rather than as raw workflow scripts, and presents ten simple rules plus templates to guide developers in doing so.
- ★ Writing bioinformatics software using a workflow manager (e.g., Snakemake, Nextflow) is beneficial, but shipping it as the native workflow script creates barriers for end users. finding
- ★ Wrapping a workflow behind a convenience launcher with a simple command-line interface lets users install and run a tool in as few as two steps (install with a package manager, run the tool). method
- ★ Ten simple rules are presented to help developers convert workflows into command-line applications, covering troubleshooting, configuration stacking, flexible execution, real-time feedback, packaging, environment isolation, environment/database location, test datasets, HPC profiles, and workflow-manager interaction. method
- ★ Cookiecutter-based templates (Snaketool for Snakemake, Nektool for Nextflow) are provided as both starter scaffolds and fully functioning examples of the ten rules. resource
- ★ A workflow-as-application consists of two key components: a convenience launcher (CLI that compiles configuration) and the workflow pipeline with associated files. mechanism
- Shipping tools via package managers (Conda/Bioconda, pip/PyPI) simplifies installation, enables non-root installs, version pinning, and isolated environments. method
- Immutable settings should be read directly by the workflow script while user-customisable settings are passed via a config file in the working directory, improving reproducibility and version control. method
- Existing command-line tools such as Hecatomb, ATLAS, VirSorter2, spacegraphcats, BlobToolKit, and PGAP exemplify hiding a workflow-manager backend behind a simple interface. resource
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36520686
Publication: Roach MJ, Pierce-Ward NT, Suchecki R, Mallawaarachchi V, Papudeshi B, Handley SA, Brown CT, Watson-Haigh NS, Edwards RA. Ten simple rules and a template for creating workflows-as-applications. PLoS Comput Biol 18(12):e1010705 (2022-12-15). DOI 10.1371/journal.pcbi.1010705 · PMCID PMC9754251.
Article type
"Ten Simple Rules" guidance / methods-commentary article in the PLOS Computational Biology education series. It teaches best practices and ships a software template (a cookiecutter-style scaffold for packaging Snakemake pipelines as installable command-line applications), referencing Snakemake/Nextflow.
In-scope pipeline-derived results
None. The article reports no computational result obtained by running a pipeline on data:
- no datasets are analyzed,
- no accession-backed inputs,
- no benchmark numbers / quantitative figures or tables produced by a bioinformatic run,
- no expected value that could be recomputed and compared.
Out-of-scope (everything in the paper)
The entire content is guidance text + a reusable template repository. The template is a starting scaffold, not an analysis whose output is a reported result.
Link checks
- Brief "Code":
https://github.com/gituser/my_tool→ HTTP 404 (unfilled template placeholder; not a real repo for this paper). - Brief "Data":
zenodo:10.5281/zenodo.4892136→ resolves (HTTP 200) but is the template's default example deposit, not a dataset this paper analyzes.
Decision
DROP — drop_reason = non_pipeline (text-mining false positive; guidance article,
no reproducible pipeline-derived result). No «our HPC» compute spent. Verdict provisional;
human audit required.
No individual results have been recorded for this entry yet.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
PMID 36520686 is a PLoS Comput Biol 'Ten Simple Rules' guidance article that ships a cookiecutter software template, not a research paper with pipeline-derived results. There is no reported computational value, no analyzed dataset, and no resolvable analysis repo (the brief's github.com/gituser/my_tool is a 404 placeholder; the Zenodo accession is the template's example deposit). The drop verdict (non_pipeline) is correct: nothing was computed or claimed, so nothing is reproducible — this is a text-mining false positive, not an authors' defect or a fabrication concern. All eight questions land red because no comparison endpoint exists at all.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.