Using prototyping to choose a bioinformatics workflow management system.
Part of the results reproduced; minor but material deviations remained.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓No relevant deviation in data/preprocessing
- ✓Any deviation was negligible
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
▸Reproduction agent’s raw note
DROP (non_pipeline). 'Using prototyping to choose a bioinformatics workflow management system' (Jackson, Kavoussanakis, Wallace; PLoS Comput Biol 2021) is a software-engineering experience report: it advocates prototyping a subset of a real workflow in each candidate tool to choose software, using the authors' selection of a WMS for the RiboViz ribosome-profiling pipeline as the worked example (shortlist Snakemake/cwltool/Toil/Nextflow; chose Nextflow; ~10 person-days). It deposits NO biological dataset and reports NO pipeline-derived numeric output to reproduce. The brief's 'Data' DOI (zenodo 840663) is a documentation citation (CWL User Guide), and the 'Code' link (snakemake) is one of four cited WMS repos, not author code -- both are text-mining false positives. The only quantitative content is descriptive project metadata (Table 3, dated 28 Feb 2020) and subjective effort/feature observations (Table 5); none is a bioinformatic pipeline result. The single cleanly-verifiable claim (Nextflow start = 2013) was confirmed via the GitHub API but is metadata, not a computation. NOT ATTEMPTED: any SLURM/«our HPC» compute (nothing to compute), any data download (none exists). No fabrication indicated -- authors frame the numbers honestly as a dated snapshot and their own prototyping experience. Verdicts are PROVISIONAL; a human signs off (see AUDIT.md).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-18 ⛓ fb8376526f8e
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan rapid prototyping of a subset of a real analysis workflow within candidate workflow management systems provide a low-cost, more informed basis for selecting a workflow management system (for the RiboViz ribosome profiling pipeline) than relying on reviews and recommendations alone?
- ★ Prototyping a subset of a project's actual workflow in candidate software offers a low-cost (time and effort) way to make a more informed software selection than relying solely on reviews and recommendations. method
- ★ After surveying and prototyping, Nextflow was chosen as the workflow management system for RiboViz. finding
- ★ The full selection process (survey plus prototyping) took 10 person-days, a small cost relative to the assurance gained. finding
- ★ A rapid (non-systematic) survey using popularity, open-source licence, stability/future, and text-based development environment criteria shortlisted four candidates: Snakemake, cwltool, Toil, and Nextflow. method
- ★ The original RiboViz Python analysis script was effectively becoming a custom workflow management system lacking re-entrancy, incremental build, and parallel execution, motivating adoption of an off-the-shelf system. mechanism
- Workflow management systems provide a common workflow description language, reproducibility, reusable components, incremental build, re-entrancy, and portability via containers/HPC/cloud. finding
- RiboViz is an open-source (Apache 2.0) high-throughput sequencing analysis pipeline specialised for ribosome profiling data. resource
- Each candidate was allotted 2–3 person-days of prototyping; if nothing productive could be implemented, it would be abandoned for the next. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Prototyping/implementation of a workflow subset (first 5 RiboViz steps: build rRNA and ORF indices, adapter trimming, rRNA contaminant removal, ORF alignment) | RiboViz ribosome profiling pipeline | none | ease of use, documentation quality, ease of implementation, required/useful functionality, container/HPC/cloud support | Snakemake |
| Prototyping/implementation of a workflow subset (first 5 RiboViz steps) | RiboViz ribosome profiling pipeline | none | ease of use, documentation quality, ease of implementation, required/useful functionality, container/HPC/cloud support | Nextflow |
| Prototyping/implementation of a workflow subset (first 5 RiboViz steps) | RiboViz ribosome profiling pipeline | none | ease of use, documentation quality, ease of implementation, required/useful functionality, container/HPC/cloud support | cwltool (CWL reference implementation) |
| Prototyping/implementation of a workflow subset (first 5 RiboViz steps) | RiboViz ribosome profiling pipeline | none | ease of use, documentation quality, ease of implementation, required/useful functionality, container/HPC/cloud support | Toil (CWL production implementation) |
| Web-search-based survey of available workflow management systems | bioinformatics workflow management software landscape | none | popularity, licence, stability/maintenance to shortlist candidates | Google web search |
| Repository/web-statistics review of shortlisted systems | open-source repositories of Snakemake, Nextflow, cwltool, Toil | none | software licence, project start date, last updated, number of contributors, search results | — |
- – Nextflow was selected as the workflow management system for RiboViz following prototyping evaluation
- – Total selection effort was 10 person-days (3 for survey + 2–3 per candidate prototype) 10 person-days
- ▲ Nextflow had the most contributors among shortlisted systems 230 contributors
- ▲ Nextflow had the highest number of web search results among candidates 23,800 results
- count 10 person-days (total cost of the workflow management system selection process)
- count 2-3 person-days per candidate (effort allotted to prototyping each shortlisted system)
- count 122 (Snakemake contributors (28 February 2020))
- count 230 (Nextflow contributors (28 February 2020))
- count 72 (cwltool contributors (28 February 2020))
- count 81 (Toil contributors (28 February 2020))
- count 23,000 / 23,800 / 2,440 / 20,700 (Google search results for Snakemake / Nextflow / cwltool / Toil bioinformatics (28 February 2020))
- count 3 person-days (effort to survey available software and shortlist candidates)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software engineering methods paper describing a qualitative prototyping-based selection process for a bioinformatics workflow management system. The study surveyed available systems, shortlisted four candidates (Snakemake, cwltool, Toil, Nextflow), and evaluated each by prototyping a subset of the RiboViz workflow. No inferential statistics were employed; quantitative data were limited to descriptive counts (GitHub contributors, web search result counts) and effort estimates in person-days. Selection was based on qualitative criteria assessed during hands-on prototyping.
-
Popularity of candidate systems was proxied by Google search result counts for '<system> bioinformatics'↳ Could also: Package download statistics (e.g., Bioconda/conda-forge download counts), citation counts in peer-reviewed literature, or GitHub star/fork counts could also quantify community adoption — Download or citation metrics are more directly linked to active use than search result counts, which can be sensitive to search-term order and SEO; multiple converging metrics would give a broader picture of community uptake
-
Evaluation of the four candidates used informal qualitative criteria assessed subjectively during prototyping↳ Could also: A formal multi-criteria decision analysis (MCDA) or weighted-scoring matrix could also structure the same qualitative criteria — A weighted-scoring approach makes the relative importance of each criterion explicit and reproducible, allowing readers to re-weight criteria according to their own project priorities
-
Each candidate received a fixed time budget (2–3 person-days) as the evaluation window↳ Could also: A standardised benchmark task set with defined acceptance tests could also be used to ensure each system is evaluated against identical, objectively measurable outcomes — Fixed tasks with pass/fail criteria reduce evaluator-to-evaluator variability and make comparisons more reproducible across teams with different prior experience with each system
-
The shortlisting phase relied on informal web searches rather than a systematic literature review↳ Could also: A PRISMA-style systematic review of workflow management systems, or use of established catalogues (e.g., the bio.tools registry), could also generate the candidate list — A systematic approach documents the search strategy and inclusion/exclusion criteria, improving transparency and reducing the risk of overlooking relevant tools
-
The evaluation produced a single final decision (Nextflow selected) with no quantitative uncertainty estimate around that choice↳ Could also: Inter-rater reliability measures (e.g., Cohen's kappa) across multiple independent evaluators scoring the same criteria could also be reported — When selection relies on qualitative judgement, documenting agreement between independent raters conveys how reproducible the outcome would be if a different team performed the same evaluation
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33630841
Title: Using prototyping to choose a bioinformatics workflow management system. Authors: Jackson M, Kavoussanakis K, Wallace EWJ. PLoS Comput Biol 2021. PMID: 33630841 · PMCID: PMC7906312 · DOI: 10.1371/journal.pcbi.1008622 Full text used: Europe PMC JATS XML for PMC7906312 (PMC HTML was CAPTCHA-blocked).
What kind of paper this is
This is a software-engineering experience / methodology report, not a data-analysis paper. Its thesis: when a project needs to pick a tool (here, a workflow management system, WMS), prototyping a subset of the real workflow in each candidate is a low-cost way to make an informed choice. The authors needed a WMS to re-implement the RiboViz ribosome-profiling analysis workflow (a Python script), shortlisted four candidates — Snakemake, cwltool, Toil, Nextflow — prototyped a subset of the RiboViz workflow in each, and chose Nextflow. The whole selection took ~10 person-days.
In scope (pipeline-derived computational results)
NONE. The paper reports no biological pipeline outputs — no counts, no quantified-expression values, no figures generated by running RiboViz on data. There is no result of the form "running pipeline P on dataset D yields value V" that could be regenerated and compared.
Out of scope (everything the paper actually contains)
All reported numbers are descriptive / subjective experience-report content, not pipeline outputs:
| Reported content | Location | Why out of scope for pipeline reproduction |
|---|---|---|
| Repository statistics (licence, start date, #contributors, #search results) | Table 3 | Project-metadata snapshot taken 28 Feb 2020; time-dependent; not a pipeline output. |
| Feature-support matrix (iteration, error recovery, containers, dry-run, …) | Table 5 | Subjective hands-on observations made during prototyping. |
| #RiboViz steps prototyped per WMS (14/3/3/5) | Table 5 | Records the authors' own prototyping effort, not a reproducible computation. |
| Person-days to implement (1/1/1/2) and selection cost (10 person-days) | Abstract, Table 5 | Human effort measurement, irreproducible by definition. |
| Shortlisting criteria, evaluation criteria | Tables 1, 2, 4 | Qualitative methodology description. |
| Choice of Nextflow | Results/Discussion | Editorial conclusion, not a computation. |
Provenance correction (text-mining false positives in the RU brief)
The reproduction brief carried two auto-harvested links that do not denote this paper's artifacts:
Code: https://github.com/snakemake/snakemake— this is merely one of the four cited WMS repos (the others: nextflow-io/nextflow, common-workflow-language/cwltool, DataBiosphere/toil). It is not the authors' code. The authors' own artifacts aregithub.com/riboviz/riboviz(the RiboViz package) andgithub.com/riboviz/workflows(the prototype workflows).Data: zenodo:10.5281/zenodo.840663— this is reference 29 (Hodges T, Crusoe MR. Common Workflow Language User Guide. zenodo; 2017), i.e. documentation, not a biological dataset. The paper deposits no data; its only availability statement is "The source code is available on GitHub (riboviz/riboviz)."
Conclusion
There is no pipeline-derived computational result to reproduce. The eligible signal was a harvester false positive (a generic GitHub link + a documentation DOI mistaken for a dataset). Controlled outcome: drop · non_pipeline (secondary: no biological data accession; no pinnable pipeline result). See AUDIT.md for the honest cross-checks actually performed.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a correctly-identified non-pipeline drop: an experience report (Jackson et al., PLoS Comput Biol 2021) advocating prototyping to select a WMS, depositing no biological data and reporting no pipeline-derived numeric result. The flagged data DOI is a documentation citation and the code link is one of four cited WMS repos, not author code — both text-mining false positives. The single comparable claim (Nextflow start = 2013) matched the GitHub created_at exactly; the remaining figures are time-stamped metadata or human-effort counts that are non-derivable by nature, not an authors' defect, and there is no fabrication indication. Non-reproducibility here is intrinsic to the paper type (data-availability / non-pipeline), so the case is yellow, not critical.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.