Pharokka: a fast scalable bacteriophage annotation tool.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1, with minor version-driven deltas). Pharokka is the authors' own tool (P16); reproduction = running pharokka v1.1.0 on the authors' own Zenodo 7227091 benchmark data (md5-verified) on «our HPC» SLURM. All 10 quantitative table claims (Tables 3&4) independently reproduced to within 0.06-1.7%: 5 EXACT (C3 unknown=154, C6 CDS=212, C7 annotated=93, C8 unknown=119), 5 within-tol, 2 partial (C11/C15: the large 673-genome PHANOTATE/Prodigal total-CDS counts off by 1.3%/1.7%). ZERO mismatches. Residual sub-2% deltas are fully explained by gene-caller patch versions (PHANOTATE 1.6.7, Prodigal 2.6.3) and the PHROG db version (v1.0.0); annotated-function counts agree to <1% across 9000+ CDS. The paper's reported values are corroborated, with no fabrication signal. Runtimes (C5/C10/C14/C17) recorded for relative comparison only (hardware-dependent): ours faster on AMD EPYC 9654, same PHANOTATE>>Prodigal ordering. NOT ATTEMPTED: Prokka-with-PHROGs comparison rows (out of scope per scope.md). Env hurdles overcome and documented in kartei: mamba cache->«infra»; biopython 1.79 pin (Bio.SeqUtils.GC removed in 1.80); SimpleLocation=FeatureLocation shim for bcbio-gff 0.7.1; install_databases.py is silently broken (curl without -L on Zenodo redirect) -> fetched DB tarball directly.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 3a77c4d7bd09
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThere is a need for a rapid, scalable, and consistent bacteriophage-specific genome annotation tool, since existing tools are either web/email-based requiring manual curation, not designed specifically for phage gene features, or too resource-intensive for routine use.
- ★ Pharokka is a one-line, fast, scalable bacteriophage annotation tool producing standards-compliant outputs, installable via a two-line bioconda command resource
- ★ Pharokka uses PHANOTATE by default for CDS prediction because it is designed for phage-specific gene features (small genes, high coding density, alternative start codons) method
- ★ Pharokka functionally annotates predicted CDS by matching them to the PHROGs database using mmseqs2 (default e-value 1e-5) method
- Pharokka predicts tRNAs, tmRNAs and CRISPRs using tRNAscan-SE 2, Aragorn and MinCED respectively method
- ★ Pharokka screens CDS for antimicrobial resistance and virulence genes against CARD and VFDB using strict mmseqs2 thresholds (80% identity, 40% coverage) method
- ★ Pharokka runs in under 5 minutes for an average 50-kb bacteriophage genome finding
- ★ For large metavirome datasets, Pharokka (especially in Prodigal meta mode) is considerably faster than Prokka because it uses mmseqs2 rather than HMMER3 for database searching finding
- Low coding density identified by Pharokka in some crAss-like phage contigs is consistent with possible stop codon reassignment in these genomes finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| genome annotation benchmarking (runtime, CDS count, coding density, functional annotation counts) | Enterobacteria phage Lambda (48,052 bp, GenBank J02459) | gene predictor tool: Pharokka-PHANOTATE vs Pharokka-Prodigal vs Prokka+PHROGs | time (min), CDS count, coding density (%), annotated vs unknown function CDS | Intel Xeon CPU E5-4610 v2 @2.30GHz, 16 threads/cpus |
| genome annotation benchmarking (runtime, CDS count, coding density, functional annotation counts) | Staphylococcus phage SAOMS1 (140,315 bp, GenBank MW460250) | gene predictor tool: Pharokka-PHANOTATE vs Pharokka-Prodigal vs Prokka+PHROGs | time (min), CDS count, coding density (%), annotated vs unknown function CDS | Intel Xeon CPU E5-4610 v2 @2.30GHz, 16 threads/cpus |
| large-scale metavirome annotation benchmarking in meta mode | 673 crAss-like metagenome-assembled phage genomes from human gut (Yutin et al., 2021) | gene predictor/tool: Pharokka-PHANOTATE meta mode vs Pharokka-Prodigal meta mode vs Prokka+PHROGs | total time, gene prediction time, tRNA prediction time, database search time, CDS count, min/max contig coding density, annotated vs unknown function CDS | Intel Xeon CPU E5-4610 v2 @2.30GHz, 16 threads/cpus |
| protein homology search (mmseqs2) against PHROGs database | predicted CDS from annotated phage genomes | none | functional category / product assignment per CDS (e-value threshold 1e-5) | mmseqs2 |
| protein homology search (mmseqs2) against CARD and VFDB databases | predicted CDS from annotated phage genomes | none | antimicrobial resistance and virulence factor gene hits (80% identity, 40% coverage threshold) | mmseqs2 |
- – Pharokka finishes within 5 minutes for both Lambda and SAOMS1 regardless of gene predictor used, though slower than Prokka 4.19-4.26 min (PHANOTATE), 3.88-3.89 min (Prodigal) vs 0.27-0.93 min (Prokka)
- ▲ PHANOTATE predicts more CDS with higher coding density than Prodigal or Prokka Lambda: 88 (PHANOTATE) vs 61 (Prodigal) vs 62 (Prokka) CDS; 94.55% vs 83.69% vs 84.96% coding density
- ▼ For 673 crAss-like genomes, Pharokka is considerably faster than Prokka, especially in Prodigal meta mode 11.88 min (Pharokka-Prodigal) vs 106.55 min (Pharokka-PHANOTATE) vs 252.33 min (Prokka)
- ▼ Database searching time is much lower for Pharokka (mmseqs2) than Prokka (HMMER3) on the large crAss-like dataset 6.75 min (PHANOTATE)/5.58 min (Prodigal) vs 238.77 min (Prokka)
- – Some crAss-like contigs show low coding density, suggesting possible stop codon reassignment contig min coding density 66.01% (PHANOTATE) / 46.18% (Prodigal) / 46.13% (Prokka)
- count 38,880 PHROGs containing 868,340 proteins from 17,473 complete genomes (size/composition of the PHROGs reference database used for functional annotation)
- other e-value threshold of 10^-5 (default mmseqs2 threshold for CDS-to-PHROG matching)
- other 80% identity over 40% coverage (strict mmseqs2 threshold for CARD/VFDB antimicrobial resistance and virulence gene matches)
- count 88 vs 61 vs 62 CDS (predicted CDS count for phage Lambda: Pharokka-PHANOTATE vs Pharokka-Prodigal vs Prokka)
- count 246 vs 212 vs 212 CDS (predicted CDS count for phage SAOMS1: Pharokka-PHANOTATE vs Pharokka-Prodigal vs Prokka)
- count 138,628 vs 90,497 vs 89,802 CDS (total predicted CDS across 673 crAss-like genomes: Pharokka-PHANOTATE vs Pharokka-Prodigal vs Prokka)
- fold_change 106.55 min vs 11.88 min vs 252.33 min total runtime (runtime comparison for annotating 673 crAss-like metagenome-assembled phage genomes)
- mean under 5 min (stated typical Pharokka runtime for an average 50-kb bacteriophage genome)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper presents a software tool (Pharokka) and evaluates it through a set of computational benchmarking comparisons rather than a hypothesis-testing study. Performance (run-time) and annotation output (CDS counts, coding density, annotated vs. unknown function CDS) were compared across three annotation approaches (Pharokka with PHANOTATE, Pharokka with Prodigal, and Prokka with PHROGs) on two individual phage genomes and one collection of 673 crAss-like metagenome-assembled genomes. Results are reported as single-run summary tables (times and counts) with narrative interpretation, without inferential statistical testing.
-
Run-time and annotation metrics for each tool were reported from what appears to be a single run per genome/dataset combination.↳ Could also: Repeating each benchmarking run multiple times (or across multiple independent genomes per category) and reporting mean ± SD or a confidence interval — Repeated measurements would allow the variability of run-time and annotation counts to be quantified, which can be informative for users assessing tool reliability across different hardware or datasets.
-
Differences between Pharokka (PHANOTATE/Prodigal) and Prokka in CDS counts, coding density, and annotated/unknown function CDS were described narratively without formal statistical comparison.↳ Could also: Applying a paired statistical test (e.g. Wilcoxon signed-rank test or paired t-test) across the 673 crAss-like genomes, since each genome was annotated by all three approaches — With 673 paired observations already available, a paired test could formally quantify whether differences in CDS counts or coding density between methods are consistent across genomes, complementing the aggregate counts shown in Table 4.
-
The 673 crAss-like genome benchmarking reports aggregate totals (e.g. total CDS, total annotated/unknown CDS) rather than a per-genome distribution.↳ Could also: Reporting summary statistics (median, IQR, or range) of per-genome metrics such as coding density or annotation rate across the 673 genomes — A distributional summary would let readers see genome-to-genome variability (e.g. in coding density, which the authors use to flag possible stop-codon reassignment) rather than only the minimum/maximum and totals.
-
Speed comparisons between Pharokka and Prokka are presented as single wall-clock times on one benchmarking machine.↳ Could also: Benchmarking run-time across multiple hardware configurations or thread counts with repeated trials, summarized with a measure of central tendency and spread — This would help characterize how sensitive the reported speed advantages/disadvantages are to hardware variation, which is a common consideration in bioinformatics tool benchmarking papers.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Reproduction scope — pmid-36453861 (Pharokka)
Paper: Bouras et al. 2023, Pharokka: a fast scalable bacteriophage annotation tool, Bioinformatics. PMID 36453861 · DOI 10.1093/bioinformatics/btac776.
Tool / code: https://github.com/gbouras13/pharokka — benchmarked version v1.1.0
(pinned commit cc76f29c1c3cdbf844f9d1a82b36d727ffdb8159). This is the authors' own
tool (P16: own-repo). Reproduction = run the tool on the paper's own benchmark data
with the described parameters.
Data: Zenodo 10.5281/zenodo.7227091 →
pharokka_v_1_1_0_benchmarking.tar.gz (447.9 MB, MD5 2670f376090541c5b68e950e69a75b2b).
Contains BOTH benchmark inputs (SAOMS1 = GenBank MW460250.1; 673 crAss-like phage
genomes; Lambda test) AND the authors' own Pharokka/Prokka output files +
calc_gff_coding_density_prokka.py. → we can compare reproduction against BOTH the
paper's printed values AND the authors' shipped outputs.
IN SCOPE — pipeline-derived, reproducible (Tables 3 & 4)
Gene-prediction CDS counts are fully deterministic (depend only on the PHANOTATE / Prodigal version bundled with Pharokka v1.1.0), independent of any database → cleanest reproduction targets:
- C1 SAOMS1 PHANOTATE total CDS = 246
- C6 SAOMS1 Prodigal total CDS = 212
- C11 673 crAss-like PHANOTATE-meta total CDS = 138,628
- C15 673 crAss-like Prodigal-meta total CDS = 90,497
- C4/C9 SAOMS1 coding density (92.27% / 89.69%) — deterministic given CDS set.
Annotated-function CDS counts depend on the PHROGs MMseqs2 db version shipped
with the v1.1.0 install_databases.py; reproducible if the same db is pulled:
- C2/C7 SAOMS1 annotated CDS (92 / 93)
- C12/C16 673 crAss-like annotated CDS (9,341 / 9,228)
CONTEXT-DEPENDENT — reported but NOT a strict match target
Runtimes (C5, C10, C14, C17: 4.26 / 3.89 / 106.55 / 11.88 min) are hardware- dependent (paper: 16 threads, unspecified CPU). We record wall-time for order-of-magnitude / relative comparison only, not exact match.
OUT OF SCOPE
- Prokka-with-PHROGs comparison rows (Table 3/4): requires building Prokka + a custom PHROG HMM db; secondary to the Pharokka claims. Attempt only if time permits after the Pharokka rows.
- Any wet-lab / manual curation (none in this paper).
Pipeline per result
PHANOTATE/Prodigal (gene calling) → MMseqs2 vs PHROGs/CARD/VFDB (function) →
tRNAscan-SE/Aragorn/MinCED (tRNA/tmRNA/CRISPR). Single pharokka.py invocation.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.