Workflow sharing with automated metadata validation and test execution to improve the reusability of published workflows.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🟡Reported values were only indirectly comparable
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Tools/methods paper (Yevis). Only one pipeline-derived computational result exists: the case-study CWL workflow that Yevis test-executed to 'verified:true'. Reproduction in progress: independently running the published packed CWL on its published test FASTQs via cwltool+Apptainer on «our HPC». Out of scope: the Yevis publish flow (needs GitHub/Zenodo write access), Table 1's 12 qualitative requirements.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 61d6f60fa19e
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusPublished bioinformatics workflows are often not reliably reusable; the paper investigates whether redefining the FAIR workflow concept as a set of technical requirements and providing a system that automates their validation and testing can lower the cost of sharing workflows in a reusable form.
- ★ Yevis is a system that builds a workflow registry which automatically validates and tests workflows prior to publication, ensuring they are 'reusable with confidence'. resource
- ★ A set of technical requirements (availability, validity, traceability) derived from the FAIR workflow concept can define what makes a workflow reusable with confidence. finding
- ★ Yevis runs on GitHub and Zenodo, enabling workflow hosting and registry operation without dedicated computing resources. method
- ★ Yevis provides a GA4GH TRS-compatible API to ensure interoperability with existing workflow registries. method
- ★ The workflow-sharing procedure is divided into automated submission, review, and publication processes handled via GitHub pull requests and GitHub Actions. method
- ★ Yevis-cli, a command-line application, automates metadata template generation, validation, testing (via GA4GH WES/Sapporo/Docker), and pull-request submission. method
- A Yevis-based registry can accept workflows written in any language (unlike nf-core's Nextflow-only policy), limited only by the availability of a suitable testing environment. finding
- ★ As a proof of concept, a registry was built with Yevis to host workflows managed by an existing community, demonstrating requirements can be fulfilled. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Workflow validation and automated test execution (CI/CD) | GitHub Actions CI/CD environment | none | workflow metadata validation result and workflow execution success/failure status | GitHub Actions |
| Workflow test execution via Workflow Execution Service | Submitter's local computer / WES instance | none | final workflow execution status (pass if completed successfully) | Sapporo (WES implementation) with Docker container virtualization |
| File persistence and DOI assignment | Zenodo research data repository | none | persisted workflow files, DOI, and persistent URLs | Zenodo API |
| TRS API response deployment and workflow browsing | GitHub Pages static hosting; Yevis-web JavaScript browser interface | none | served TRS API response and workflow metadata display | GitHub Pages, GA4GH TRS API, GitHub REST API |
| Feature comparison of registries | Yevis-based registry vs WorkflowHub, Dockstore, nf-core | none | diversity, reliability, and usability characteristics (e.g., acceptable workflows) | — |
- – Yevis enforces that submissions pass metadata validation and testing before a pull request is created, reducing the registry maintainer's review burden.
- – A proof-of-concept registry was successfully built to host an existing community's workflows while satisfying the defined reusability requirements.
- – A Yevis-based registry accepts workflows in any language, offering greater diversity than nf-core (Nextflow-only best-practice workflows).
- – Requirements for reusability were classified into three perspectives: availability, validity, and traceability.
- count 3 processes (submission, review, publication) (the workflow-sharing procedure is divided into 3 processes)
- count 3 aspects/perspectives (availability, validity, traceability) (requirements classified into 3 perspectives)
- count 3 registries compared (WorkflowHub, Dockstore, nf-core) (feature comparison against existing registries)
- count 4 Yevis-cli subcommands (make-template, validate, test, pull-request) (subcommands executed by submitter in local environment)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper describes Yevis, a software system for building workflow registries with automated metadata validation and test execution. It is a systems/tools paper rather than a paper reporting a hypothesis-driven experiment, and it does not include quantitative statistical comparisons, inferential tests, or sample-size-based analyses; results are presented as a qualitative feature comparison with existing registries (WorkflowHub, Dockstore, nf-core) and a descriptive proof-of-concept demonstration.
-
The paper evaluates Yevis primarily through a qualitative feature comparison table against WorkflowHub, Dockstore, and nf-core, rather than quantitative metrics.↳ Could also: A quantitative usability or adoption study (e.g., measuring time-to-submission, error rates, or number of successfully validated workflows across a sample of real submissions) could also be used — This would allow numerical/statistical comparison (e.g., proportions with confidence intervals) of reusability outcomes between registry systems, complementing the descriptive feature comparison.
-
The proof-of-concept registry is demonstrated with a single community's workflows without a described sample size of test cases or replication across repeated runs.↳ Could also: Reporting the number of workflows tested, pass/fail rates, and variability across repeated CI runs could also be used — Summarizing outcomes with counts and simple descriptive statistics (e.g., percentage of workflows passing validation) would let readers gauge the consistency and generalizability of the demonstrated approach.
-
No formal user study or survey data is presented to support claims about reduced burden on registry maintainers or submitters.↳ Could also: A controlled or survey-based usability evaluation (e.g., comparing task completion time or maintainer effort with and without Yevis, analyzed with a paired t-test or Wilcoxon signed-rank test) could also be used — This would provide quantitative, inferential evidence for the qualitative claims about efficiency and reduced maintenance burden.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36810800 (Yevis)
Paper: Suetake, Fukusato, Igarashi, Ohta (2023). Workflow sharing with automated metadata validation and test execution to improve the reusability of published workflows. GigaScience 12:giad006. PMID 36810800 / PMC9944229.
What kind of paper this is
A tools/methods paper. It introduces Yevis, a system (Yevis-cli + a GitHub-Actions/Zenodo-based registry generator) that lets a workflow author publish a workflow with (a) automated metadata validation against a schema and (b) automated test execution of the workflow via a Workflow Execution Service (WES, Sapporo-service / cwltool). The system's central promise: a published workflow is machine-verified to actually run before it gets a DOI.
It is NOT a data-analysis paper — there is no biological finding, no figure with
a measured quantity, no benchmark table of results. Table 1 lists 12 design
requirements (qualitative). So there is essentially one pipeline-derived,
checkable computational result: the case-study workflow Yevis published and
test-executed successfully (verified: true).
The one in-scope computational result (the case study)
Yevis built a new registry — pitagora-network/yevis-DAT2-cwl — and registered
one workflow as the proof of concept:
- "DAT2-cwl - bacteria genome workflow" (id
d03458d8-837c-4173-afa3-55ebe538b0b2, v1.0.0) - A CWL (v1.0) packed workflow:
bacteria-genome.packed.cwl - Pipeline = bacterial genome assembly: SeqKit → FastQC → fastp → Platanus-b (tools named in the paper; each a Docker-containerised CommandLineTool).
- Test inputs (the workflow's Yevis "test"):
test-job-yevis.yml(CWL params)- paired-end Illumina reads
DRR024501_1.fastq/DRR024501_2.fastq.
- paired-end Illumina reads
- Reported result: the workflow is
verified: true, verified via GitHub Actions run #2317749577 (Yevis runs the test through a WES and asserts the run reaches state COMPLETE). This is the reproducible claim.
In scope (attempt)
- C1 — workflow test-execution passes. Independently run
bacteria-genome.packed.cwlon the published test job (DRR024501reads) with a standard CWL runner (cwltool + Singularity/Apptainer on «our HPC») and confirm it runs to completion successfully and emits the assembled-genome outputs — i.e. reproduce Yevis'verified: truetest result. This is a P16 "third-party tool on the paper's own data" reproduction (running the published CWL via the standard cwltool engine, instead of via Sapporo-WES): per the brief, equally valid.
Out of scope (not attempted, with reason)
- The Yevis-cli publish flow itself (creating a GitHub repo, GitHub Actions, Zenodo upload, DOI minting): requires GitHub org write access + Zenodo tokens + side effects on public infrastructure → not reproducible in a sandbox, and not a pipeline-derived result. Out of scope.
- Table 1's 12 requirements and all narrative/system-design claims: qualitative, no number to regenerate. Out of scope.
- The other DAT2-cwl workflows (RNA-seq etc.) in the upstream
pitagora-network/DAT2-cwlrepo: NOT part of the paper's Yevis case study (only the bacteria-genome workflow was registered in yevis-DAT2-cwl). Out of scope for this paper, though they could be a stretch goal if C1 succeeds.
Pipeline named per result
- C1: CWL workflow
bacteria-genome.packed.cwl= SeqKit + FastQC + fastp + Platanus-b, executed by cwltool (CWL v1.0 reference-ish runner) with containers via Singularity/Apptainer.
Data
- zenodo:6545122 (workflow deposit) — packed CWL + test FASTQs + metadata. THIS is the data needed to reproduce C1.
- zenodo:6541109 (the brief's listed accession) — actually the Yevis-cli 0.3.4 software source snapshot (a zip), not biological data. Profiled as software/tooling. </content> </invoke>
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.