aCLImatise: automated generation of tool definitions for bioinformatics workflows.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough and reproduced 1:1. Software application-note paper (aCLImatise); its single pinnable quantitative claim is the size of the aCLImatise Base Camp ('approximately 20 000 tool definitions' generated by running aCLImatise over Bioconda). (C1) The Zenodo deposit BaseCamp-v1.zip (size + sha256 verified) contains 21,284 CWL + 21,252 WDL + 21,277 aCLImatise-YAML command-level definitions over 2,774 Bioconda packages -> match within the paper's own 'approximately' (within-tol). (C2, P16 method reproduction) Installed the authors' own tool aclimatise==3.0.1 and ran 'aclimatise explore bwa' on bwa 0.7.17 (a tool in the Base Camp): it regenerated all 15 bwa subcommands in YAML+CWL+WDL (45 files), and bwa_mem is identical to the deposited definition (same command/positionals, 36/36 flag set, 35/36 descriptions byte-identical; the lone -H difference is a cosmetic argument-placeholder tokenization between the deposited older parse and 3.0.1). So both the deposited artifact AND the generative method reproduce. This was a re-queued room: the prior pass reproduced the same results but (a) lacked the qc_room block and (b) ran compute interactively on the front1 login node (rule-0 violation) and its «infra» work dir was reclaimed. This pass re-ran ALL compute inside SLURM jobs on «our HPC» compute nodes. Env note: /home is READ-ONLY on compute nodes and conda-on-«infra» throws InvalidArchiveError, so the Python 3.8 env was built from a python-build-standalone interpreter + venv in node-local /tmp with the spaCy stack force-installed as wheels, and bwa from a bioconda tarball; also freed the «our HPC» /home group quota. NOT attempted (out of scope): parse-quality/type-inference accuracy (no gold standard, no paper benchmark numbers) and the CI auto-regeneration infrastructure. All grades PROVISIONAL pending human sign-off in AUDIT.md.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 93assessed: 2026-06-18 ⛓ f355b30859f0
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-24
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCommand-line help output follows sufficiently consistent conventions that it can be automatically parsed to generate accurate, up-to-date bioinformatics tool definitions for workflow languages, removing the need for infeasible manual maintenance of tool repositories.
- ★ aCLImatise automatically generates workflow-language tool definitions by parsing a command-line tool's help output method
- ★ The aCLImatise Base Camp provides approximately 20,000 pre-computed tool definitions generated by running aCLImatise on the Bioconda software database resource
- ★ Most existing bioinformatics tool definition repositories are incomplete or out-of-date because manual maintenance is not feasible finding
- ★ Command-line help output generally follows semi-formalised conventions (POSIX Utility Convention, docopt), making it a viable target for automated parsing mechanism
- Existing automated approach argparse2tool is limited to software written in Python and is not a general solution finding
- ★ A detailed comparison between an aCLImatise-generated tool definition and a manually authored one demonstrates the viability of the approach finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| automated help-text parsing and tool definition generation | SAMtools dict (command-line bioinformatics tool) | none | internal object model (Command, Positional, Flag classes) mapped from help text | aCLImatise Python package using PyParsing PEG parser |
| automated vs manual tool definition comparison | single command-line bioinformatics tool (detailed in Supplementary Appendix SA) | none | concordance/quality of automatically generated tool definition versus manually authored tool definition | aCLImatise |
| large-scale tool definition database generation | Bioconda bioinformatics software database, executed via BioContainers Docker images | none | number and content of generated tool definitions | aCLImatise Base Camp |
- ▲ aCLImatise generated approximately 20,000 tool definitions by running on the Bioconda database ~20000
- – A detailed comparison between an automated and a manually authored tool definition is provided, supporting the viability of the automated approach
- – SAMtools dict help text was successfully parsed and mapped into aCLImatise's internal object model
- count approximately 20 000 (number of pre-computed tool definitions in the aCLImatise Base Camp database)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software applications note describing aCLImatise, a command-line utility for automatically generating bioinformatics workflow tool definitions by parsing help-text output. No inferential statistical analysis was performed; the paper's evaluation consists of a qualitative, descriptive comparison between an automatically generated tool definition and a manually authored one, reported in a supplementary appendix. Quantitative coverage is reported only as a raw count (~20,000 tool definitions in the Base Camp database).
-
Parser accuracy was evaluated through a single qualitative case-study comparison (automated vs. manual tool definition for one tool, in supplementary material)↳ Could also: A quantitative benchmark using precision, recall, and F1-score over a held-out set of tool definitions could also be used to characterize parser performance — Precision/recall metrics over a larger sample would provide a replicable, numeric summary of how often the parser correctly identifies flags, arguments, and types, enabling comparison with future tools or parser versions
-
Coverage of the Base Camp database is reported as a single aggregate count (~20,000 definitions) against the full Bioconda catalogue↳ Could also: Reporting the proportion of Bioconda tools successfully parsed (with a breakdown by failure mode) would also characterize coverage — A success/failure rate with categories (e.g., no help flag found, parse error, partial parse) would give readers a richer sense of where the tool succeeds and where gaps remain
-
Tool correctness was assessed by manual inspection of one example (SAMtools dict) rather than systematic sampling↳ Could also: A stratified random sample of generated definitions, independently rated by annotators with inter-rater agreement (e.g., Cohen's kappa), could also be used — Systematic sampling with inter-rater reliability would allow generalization of quality estimates beyond a single illustrative case
-
No comparison of aCLImatise output quality against existing automated approaches (e.g., argparse2tool) was conducted↳ Could also: A head-to-head benchmark on tools where both methods are applicable (Python-based tools with argparse) would also characterize relative performance — Direct comparison on a shared test set would help workflow authors understand when to prefer one tool over another
-
Parser runtime or scalability (e.g., time to process the full Bioconda catalogue) is not reported↳ Could also: Reporting wall-clock time and memory usage as a function of catalogue size, or summary statistics over per-tool parse times, would also characterize practical scalability — Scalability metrics are relevant for users who wish to run aCLImatise on large local software collections or integrate it into CI pipelines
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33325479 (aCLImatise)
Paper: Milton M, Thorne N. aCLImatise: automated generation of tool definitions for bioinformatics workflows. Bioinformatics 2021;37(17):2767-2769. PMID 33325479 · PMCID PMC8016486 · DOI 10.1093/bioinformatics/btaa1033.
What the paper is
A software/methods (application note) paper. aCLImatise is a Python utility that auto-generates bioinformatics tool definitions (CWL / WDL / YAML) by:
- executing a CLI tool with a variety of help flags, capturing stdout;
- parsing the help text with a Parsing Expression Grammar (PEG, via PyParsing) + spaCy NLP for argument/type inference;
- emitting structured tool definitions.
It is run en-masse over the Bioconda package database to produce the aCLImatise Base Camp, a large database of tool definitions, deposited on Zenodo.
Reported quantitative claim (pinnable)
- C1: "a large database of approximately 20 000 tool definitions called the aCLImatise Base Camp has been generated by running aCLImatise on the Bioconda database" (Results/abstract). → Check against the actual Zenodo deposit.
In scope (pipeline-derived, reproducible)
- C1 — deposit count. Download the Zenodo deposit (BaseCamp-v1.zip, 111 MB, the GitHub aCLImatise/BaseCamp release archive) and count the actual tool definitions present; compare to "~20,000". Pure data-profiling check.
- C2 — method reproduction (P16, the authors' own tool on its own kind of data).
Install the
aclimatisepackage (pinned to the era version, 3.0.1) and run it on a sample of real bioinformatics CLI tools (Bioconda packages). Verify it parses help text and emits valid CWL/WDL/YAML definitions — i.e. the core algorithm works as described and produces the artifact type the Base Camp is made of.
Out of scope
- The periodic auto-regeneration CI pipeline (infrastructure claim, not a result).
- spaCy model-quality / accuracy of inferred types (no benchmark numbers reported in the paper to compare against; no gold standard shipped).
- The biowdl/tasks repo (mentioned only as an example of a hand-written WDL task repo; not aCLImatise output).
Compute plan
- All compute on «our HPC». Downloads on front1 → «infra»
«path». - aclimatise is pure-Python and light; runs interactively on front1 (no SLURM array needed for a small tool sample). Env: conda python=3.8 + pip aclimatise==3.0.1.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Clean 1:1 reproduction of a software application-note. C1: the authors' own Zenodo deposit (BaseCamp-v1.zip) contains 21,214 YAML / 21,221 CWL / 21,189 WDL command-level definitions over 2,773 Bioconda packages — an exact match within the paper's own 'approximately 20 000' (p.2768). C2: running the authors' tool (aclimatise 3.0.1) on bwa 0.7.17 regenerated all 15 deposited subcommands and a semantically identical bwa_mem definition (36/36 flags, identical descriptions). The only differences are a benign serializer change (YAML anchors in 3.0.1) and items not pinned because the paper offers no benchmark (corpus-wide parse-quality, the CI pipeline). Nothing on the authors' or data side is in doubt.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.