FA-nf: A Functional Annotation Pipeline for Proteins from Non-Model Organisms Implemented in Nextflow.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED. FA-nf is an authors'-own DSL1 Nextflow functional-annotation tool paper; the reproducible object is the pipeline emitting the reported annotation summaries. FLOOR (C4): I independently ran FA-nf end-to-end on «our HPC» (Nextflow 20.10.0 + Java11, apptainer/singularity 1.4.5, SQLite, all 5 containers) on the shipped aMicUni debug example -> RC=0, all documented outputs emitted; the external gogo.test.crg.eu GO-mapping service is still live and its step succeeded. Engineering hurdles solved: 27GB CRG DB bundle (md5 OK); apptainer build-temp on auto-cleaned «path» broke large image pulls (fixed via node-local tmp + --disable-cache); InterProScan process bind-mounts a container-only properties path that breaks autoMounts (fixed by extracting the real interproscan.properties to a host file). STRETCH (C1-C3): full proteomes are NOT in the repo, but the authors deposited per-organism inputs+outputs at biocore.crg.eu/papers/FA-nf-2021/. ALL nine reported Section 3.2 numbers match the deposited total_stats.txt EXACTLY, and I independently recomputed every GO count (17475/17540, 45428/47279, 15628) as distinct GO-annotated proteins in the deposited go_terms.tsv -> exact. The paper's 'two modes' map to blastAnnotMode common/all. NO fabrication signal: every value is exactly derivable from shipped data/code. NOT attempted: from-scratch full-proteome rerun (disproportionate compute; version-drift would only allow within-tol). Also noted: the brief's 'data' Zenodo DOI resolves to AGAT software, not data; the convenience bundle ships InterProScan DATA 5.48-83.0 vs the 5.54-87.0 stated in params.config.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ 2ba6f270994b
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-25
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper presents FA-nf, a Nextflow-based pipeline designed to provide scalable, reproducible, and accessible functional annotation of proteins from non-model organism genomes without requiring extensive bioinformatics expertise or manual software dependency installation.
- ★ FA-nf, implemented in Nextflow with Docker/Singularity containerization, integrates NCBI BLAST+, DIAMOND, InterProScan, and KEGG (KAAS/KofamKOALA) into a single functional annotation pipeline. resource
- ★ Containerization of every pipeline process ensures full reproducibility and eases deployment across Linux/HPC environments. method
- ★ The pipeline can be broken into smaller parallelized processes (chunked by sequence number) to improve performance on HPC infrastructure. method
- ★ Results from all annotation steps are stored in a central relational database (SQLite or MySQL/MariaDB) to simplify data retrieval and report generation. method
- ★ Using Nextflow avoids the need for users to manually install software dependencies, making the pipeline usable by researchers with less bioinformatic expertise than required by other existing pipelines. finding
- ★ NCBI BLAST+ yields slightly more annotation hits than DIAMOND under the same parameters, but at substantially greater computational cost. finding
- ★ The pipeline was successfully applied to annotate genomes of Apis dorsata, Phaseolus vulgaris, and Abscondita terminalis. finding
- An initial GFF cleaning/verification step using AGAT Toolkit ensures diverse GFF3 format inputs can proceed through downstream processes without loss of predictions. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Functional annotation pipeline run (DIAMOND homology search + InterProScan + KOFAMscan + BLAST annotator GO retrieval) | Apis dorsata (20,508 translated proteins, 12,172 genes) | none | % proteins/genes annotated, number of GO terms retrieved, runtime, CPU hours | DIAMOND vs NCBI NR (202010), InterProScan 5.48-83, KofamKOALA 202103, MySQL |
| Functional annotation pipeline run (NCBI BLAST+ homology search, same downstream tools) | Apis dorsata (20,508 proteins, 12,172 genes) | none (method comparison to DIAMOND) | % proteins/genes annotated, number of GO terms retrieved, runtime | NCBI BLAST+ |
| Functional annotation pipeline run (DIAMOND, same configuration/datasets) | Phaseolus vulgaris (57,327 proteins, 41,885 genes) | none | % proteins/genes annotated, GO terms retrieved, runtime, CPU hours | DIAMOND |
| Functional annotation pipeline run (DIAMOND, BLAST annotator 'all' mode) | Abscondita terminalis genome (20,439 proteins) | none | % proteins annotated, number of GO terms retrieved | DIAMOND |
- ▲ Apis dorsata (DIAMOND): 19,599 proteins (95.57%) and 11,265 genes (92.55%) functionally annotated; 17,475-17,540 GO terms retrieved depending on BLAST annotator mode; run took <8h (815 CPU hours) 95.57%
- ▲ Apis dorsata (NCBI BLAST+): 19,638 proteins (95.76%) and 11,304 genes (92.87%) annotated; up to 17,636 GO terms retrieved; took up to 2 days 95.76%
- ▲ Phaseolus vulgaris (DIAMOND): 54,878 proteins (95.73%) and 39,506 genes (94.32%) annotated; 45,428-47,279 GO terms retrieved; took ~1 day (1604 CPU hours) 95.73%
- ▲ Abscondita terminalis: 20,434 of 20,439 proteins (99.98%) annotated with at least one method; 15,628 GO terms retrieved 99.98%
- – NCBI BLAST+ produced marginally more annotated proteins/genes and GO terms than DIAMOND but required substantially longer runtime (up to 2 days vs <8h)
- count 815 CPU hours (Total compute for Apis dorsata annotation with DIAMOND)
- count 1604 CPU hours (Total compute for Phaseolus vulgaris annotation with DIAMOND)
- other E-value threshold 1x10^-5 (Homology search significance cutoff used in example runs)
- mean 95.57% (Proteins annotated in Apis dorsata with DIAMOND)
- mean 92.55% (Genes annotated in Apis dorsata with DIAMOND)
- count 17,475-17,540 GO terms (GO terms retrieved for Apis dorsata via different BLAST annotator modes)
- mean 99.98% (Proteins annotated in Abscondita terminalis)
- count 45,428 and 47,279 GO terms (GO terms retrieved for Phaseolus vulgaris via 'common' and 'all' BLAST annotator modes)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software methods paper introducing FA-nf, a functional annotation pipeline. The 'results' section presents three demonstration runs on real genomes (Apis dorsata, Phaseolus vulgaris, Abscondita terminalis), reporting annotation coverage as raw counts and percentages of proteins/genes annotated. No inferential statistical tests are applied; all reporting is purely descriptive. Runtime and coverage comparisons between NCBI BLAST+ and DIAMOND are made narratively without formal testing.
-
The comparison between NCBI BLAST+ and DIAMOND (runtime, coverage) is presented as a single narrative example on one genome with no replication↳ Could also: A formal benchmarking design with multiple independent genomes as replicates, followed by a paired comparison (e.g., Wilcoxon signed-rank test on annotation coverage across genomes) could also characterize the difference — Replicated benchmarks with a formal test would allow quantification of how consistently one approach outperforms the other, and whether the observed differences exceed expected run-to-run variability
-
Annotation coverage is reported as a single percentage per run (e.g., 95.57%)↳ Could also: A 95% confidence interval around the proportion of annotated proteins could also be reported (e.g., using a Wilson or Clovis interval) — A confidence interval conveys the precision of the coverage estimate and is especially informative when comparing across pipelines or datasets of different sizes
-
Runtime is reported as a single observed value per run (e.g., 'less than 8 h', '815 CPU hours') with no variability measure↳ Could also: Reporting runtime across multiple independent runs (mean ± SD, or median with IQR) could also characterize execution time — HPC runtimes can vary due to queue load, node availability, and I/O contention; replicated timing with a dispersion measure would make the reported figures more generalizable
-
GO term counts from different retrieval modes (common, most, all) are reported as raw numbers without a measure of overlap or agreement↳ Could also: A Jaccard similarity index or Venn-diagram overlap statistic could also quantify the degree to which the three retrieval modes agree — A formal overlap measure would help users understand how much additional (or potentially spurious) annotation each less-conservative mode adds relative to the strictest mode
-
No statistical framework is used to compare annotation quality (e.g., precision/recall) against a gold standard↳ Could also: Benchmarking against a curated reference proteome (e.g., a well-annotated SwissProt subset) with precision, recall, and F1 score could also characterize annotation accuracy — Coverage (% annotated) reflects completeness but not correctness; precision/recall metrics would allow comparison with other pipelines on a quality dimension
-
Three demonstration genomes are described but treated as isolated examples rather than a systematic benchmark set↳ Could also: A stratified benchmark across organisms varying in evolutionary distance from well-annotated model organisms could also be used, enabling regression of annotation coverage on phylogenetic distance — Non-model organism pipelines are particularly challenged by sparse database coverage for distant taxa; a systematic gradient would reveal where the pipeline's performance degrades
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34681040 (FA-nf)
Paper: Vlasova et al. 2021, Genes 12(10):1645. "FA-nf: A Functional Annotation Pipeline for Proteins from Non-Model Organisms Implemented in Nextflow." DOI 10.3390/genes12101645 · PMID 34681040 · PMCID PMC8535801 Code: https://github.com/guigolab/FA-nf (authors' own pipeline — P16 own-repo) "Data" DOI (zenodo 4732260): ⚠️ resolves to NBISweden/AGAT v0.6.2 software (a dependency the pipeline vendors), not experimental data. The brief's "Data: zenodo" pointer is therefore a software archive, not a dataset. The actual paper inputs/results live elsewhere (see below).
What kind of paper
This is a software/pipeline tool paper. Its quantitative "results" are an illustration of applying FA-nf to three non-model proteomes. There is no biological discovery to reproduce — the reproducible object is the pipeline itself producing the reported annotation summaries.
Pipeline (all in scope — this is one Nextflow pipeline)
Inputs: protein FASTA (+ optional GFF, cleaned by AGAT). Steps/tools:
- BLAST+ / DIAMOND vs SwissProt/UniProt (default DIAMOND, e-value 1e-5)
- InterProScan 5.54-87.0 (domains/signatures → GO)
- KofamKOALA / kofamscan (KEGG KO), koVersion 2021-05-02
- SignalP / TargetP (
sigtarp) and CD-search — OFF by default (skip_sigtarp=true,skip_cdSearch=true); SignalP/TargetP are proprietary. - GO term consolidation; SQLite/MySQL backend; outputs go_terms.tsv, interProScan.res.tsv, protein_definition.tsv, total_stats.txt, annotatedVsnot.png Execution: Singularity containers only (no conda profile). Images: guigolab/fa-nf:0.4.0, ncbi/blast:2.12.0, diamond, agat, biocorecrg/interproscan:5.54-87.0, kofamscan:1.3.0. → Singularity on «our HPC» is the key feasibility gate.
Reported results to reproduce (claims) — Section 3.2 (Results)
| id | organism | input | annotated | GO terms |
|---|---|---|---|---|
| C1 | Apis dorsata (giant honey bee) | 20,508 proteins / 12,172 genes | 19,599 proteins (95.57%), 11,265 genes (92.55%) | 17,475 (cons.) … up to 17,540 |
| C2 | Phaseolus vulgaris (common bean) | 57,327 proteins / 41,885 genes | 54,878 proteins (95.73%), 39,506 genes (94.32%) | 45,428 / 47,279 (two modes) |
| C3 | Abscondita terminalis (firefly) | 20,439 proteins | 20,434 (99.98%) | 15,628 |
Also reproducible (pipeline-execution claims, not organism numbers):
| id | claim | location |
|---|---|---|
| C4 | Pipeline runs end-to-end and emits the documented output files (go_terms.tsv, interProScan.res.tsv, protein_definition.tsv, total_stats.txt) on the shipped example | README / repo dataset/ |
In scope
- C4 (floor): run shipped
aMicUnidebug example (debug=true, debugSize=5, sigtarp+cdSearch skipped) end-to-end on «our HPC» → verify documented outputs. - C1–C3 (stretch): reproduce the annotation %/GO counts for ≥1 organism using
the 27 GB DB bundle (
fa-nf-databases-202108.tar.gz, on biocore.crg.eu) + full proteome inputs. Heaviest: InterProScan on ~20k proteins.
Out of scope / risk
- SignalP/TargetP outputs — proprietary, off by default; not attempted.
- Exact GO counts depend on UniProt-GOA / InterPro DB versions (Aug-2021 snapshot in the bundle); version drift means numbers may be within-tol not exact.
- Full proteome FASTAs: must confirm they are inside the CRG bundle or fetch from the source genome projects; availability TBD at download time.
Data inventory (to profile)
- zenodo 4732260 → AGAT software (dependency, mislabeled as data).
- repo
dataset/→ tiny example: aMicUni + Rhodotorula selected proteins (KB-scale)- gene_ontology_ext.obo (34 MB).
- biocore.crg.eu/papers/FA-nf-2021/datasets/ →
fa-nf-databases-202108.tar.gz(27.4 GB) + .md5 — the reference DB bundle.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.