Macrel: antimicrobial peptide screening in genomes and metagenomes.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Macrel is a well-documented, third-party-installable open-source pipeline (P16: applying it to its own benchmark data is a valid reproduction). Described well enough to reproduce 1:1. Ran the authors' own training/prediction code AND the shipped models on the original-source data, all on «our HPC». RESULT: (1) the tool's deterministic example output reproduces BYTE-FOR-BYTE (897/897 rows); (2) Table 2 (hemolytic) reproduces within tolerance, with the shipped Hemo.onnx giving numbers identical to a fresh retrain (0.936/0.927/0.945/0.944/0.873 vs paper 0.94/0.92/0.96/0.96/0.88); (3) Table 1 (AMP) specificity & precision reproduce EXACTLY (0.999 vs 0.998), while accuracy/sensitivity/MCC are a few points lower in the honest de-overlapped retrain (0.927/0.854/0.862 vs 0.95/0.90/0.90) — and the paper's sensitivity 0.90 sits between the overlap-inflated naive shipped-model run (0.985) and the de-overlapped retrain (0.854), confirming the methodology; the gap is explained by the redistributed AmPEP non-AMP file having drifted (+~1000 seqs) and scikit-learn version differences. KEY FINDING: the shipped v1.6.0 ONNX models are silently broken under onnxruntime 1.26.0 (return ~2p-1 instead of probability -> tool under-predicts, AMP sensitivity collapses to 0.10) but reproduce exactly under onnxruntime 1.20.1; setup.py pins onnxruntime unversioned, so a fresh 2026 install hits the broken behaviour. NOT ATTEMPTED: full PRJNA289586 metagenome assembly + screening (221 Illumina runs, out of compute scope; profiled from ENA metadata only); the Macrel_X benchmark-training Table-1 variant (Xiao Supp-S1 does not extract cleanly from PDF). No fabrication indicators.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 83assessed: 2026-06-18 ⛓ 3a7b1875a0f9
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan high-quality antimicrobial peptide (AMP) candidates be reliably predicted directly from genomes and metagenomes despite the computational challenges of small ORF gene prediction and homology-based functional classification, using a machine-learning pipeline trained under realistic class-imbalance conditions?
- ★ Macrel is an end-to-end pipeline that predicts high-quality AMP candidates from peptides, contigs, or reads of (meta)genomes resource
- ★ A novel set of 22 peptide features (6 local and 16 global) captures the amphipathic nature and transmembrane-helix propensity of AMPs method
- ★ Macrel's classifiers perform similarly to state-of-the-art methods for predicting antimicrobial and hemolytic activity but with enhanced precision/specificity finding
- ★ A novel Free Energy Transition (FET) local feature group encodes the propensity of peptides to fold when transferring from water to membrane method
- ★ Macrel was trained with a low AMP:non-AMP ratio (~1:50) to simulate real (meta)genomic conditions where high specificity is prioritized over sensitivity method
- Macrel filters spurious smORF gene predictions by outputting only smORFs (10–100 aa) classified as AMPs method
- ★ Macrel recovers high-quality AMP candidates from realistic simulated and real sequencing data without introducing many false positives finding
- Macrel can generate abundance profiles of AMPs in metagenomes/metatranscriptomes method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Random forest AMP classifier training | Curated peptide sequence dataset (adapted from Bhadra et al. 2018) | none | AMP vs non-AMP binary classification | scikit-learn (101-tree random forest) |
| Random forest hemolytic activity classifier training | HemoPI-1 peptide dataset (Chaudhary et al. 2016) | none | hemolytic vs non-hemolytic binary classification | scikit-learn |
| AMP classifier benchmarking | Xiao et al. 2013 benchmark peptide dataset | none | classification performance vs CAMPR3, iAMP-2L, AMAP, iAMPpred, AMP Scanner v2, AmPEP | — |
| Homology-aware benchmarking | Cd-hit clustered peptide sets (80% identity, 90% coverage) | varying AMP:non-AMP training ratios (1:1 to 1:50) | classification performance comparing homology search, Macrel, iAMP-2L, AMP Scanner v2 | Cd-hit v4.8.1; blastp |
| Simulated metagenome short-read pipeline test | Simulated human gut metagenomes and source isolate genomes | 3 sequencing depths (40, 60, 80 million 150bp reads) | recovery of AMP candidates and false positives from metagenomic processing | ART Illumina v2.5.8 (HiSeq 2500 error profile); MEGAHIT; modified Prodigal |
| Real metagenome/metatranscriptome processing and abundance profiling | 182 metagenomes and 36 metatranscriptomes (Heintz-Buschart et al. 2016, ENA PRJNA289586) | none | AMP abundance profiles (reads per million transcripts) | NGLess; Paladin |
| Spurious sequence detection | Macrel-predicted smORFs/AMPs | none | spurious prediction score (spurious if ≥0.8) | Spurio |
| Homology-based validation/annotation of predicted AMPs | Macrel-predicted AMP sequences | none | homology hits to identify gene fragments, patented or known AMPs | blastp against NCBI nr and DRAMP database |
- – Macrel AMP and hemolytic classifiers perform comparably to state-of-the-art methods but with enhanced precision
- – Reducing the gene-length threshold without further filtering could yield up to 61.2% false-positive predicted smORFs (motivating Macrel's AMP-classification filter) 61.2%
- – Even with large numbers of potentially artifactual input smORFs, Macrel outputs only a small number of high-quality candidates
- – Macrel recovered the same AMP candidates from simulated metagenomes as from source genomes without introducing many false positives
- count 3,268 AMPs and 165,138 non-AMPs (~1:50 ratio) (AMP classifier training set)
- count 442 hemolytic and 442 non-hemolytic peptides (HemoPI-1 hemolytic classifier training set)
- count 920 AMPs and 920 non-AMPs (Xiao et al. 2013 benchmark dataset)
- count 110 hemolytic and 110 non-hemolytic proteins (HemoPI-1 benchmark testing set)
- count 500 AMPs:500 non-AMPs (homology-aware benchmark testing set)
- count 1,197 AMPs (homology-aware benchmark training set)
- other 22 features (6 local, 16 global) (novel peptide feature set)
- count 61.2% (predicted smORFs that could be false positives without filtering (Sberro et al. 2019))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
Macrel is evaluated as a machine learning pipeline using classifier benchmarking rather than classical inferential statistics. Two random forest classifiers (AMP activity, hemolytic activity) were trained and assessed on held-out benchmark datasets and via out-of-bag (OOB) estimates, with performance reported as point estimates of classification metrics (precision, recall, accuracy, F-measure, AUC). A homology-aware evaluation further partitioned sequences by 80%-identity clustering before train/test splitting across seven AMP:non-AMP class-imbalance ratios. Pipeline performance was additionally tested on six simulated metagenomes at three sequencing depths and on 182 published metagenomes; no classical inferential hypothesis tests were employed.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Random forest classifier evaluation with out-of-bag (OOB) error estimate for overlapping sequences | AMP activity benchmark (Xiao et al. 2013 dataset); OOB used when training sequences overlapped test set | 920 AMPs + 920 non-AMPs (benchmark); 3,268 AMPs + 165,138 non-AMPs (training) | not stated |
| Held-out test set evaluation: accuracy, precision, recall, F-measure, AUC | AMP classifier compared against CAMPR3, iAMP-2L, AMAP, iAMPpred, AMP Scanner v2, AmPEP | 920 AMPs + 920 non-AMPs | not stated |
| Held-out test set evaluation: accuracy, precision, recall, F-measure, AUC | Hemolytic classifier compared against SVM, KNN (IBK), MLP, logistic regression, J48, RF (HemoPI-1 benchmark) | 110 hemolytic + 110 non-hemolytic (test); 442 hemolytic + 442 non-hemolytic (train) | not stated |
| Homology-aware train/test split evaluation at seven AMP:non-AMP ratios (1:1 to 1:50), with blastp-based homology search as baseline | Stricter benchmark comparing Macrel, iAMP-2L, AMP Scanner v2, and blastp homology search | 500 AMPs + 500 non-AMPs (test); 1,197 AMPs in training at varying non-AMP ratios | not stated |
| Simulated metagenome pipeline evaluation: AMP candidate recovery rate and false-positive counting relative to isolate genome processing | 6 simulated metagenomes at 3 sequencing depths (40, 60, 80 million 150 bp reads) | 6 simulated metagenomes | not stated |
-
Classifier performance metrics (precision, recall, AUC) are reported as single point estimates on a fixed test set, with no measure of uncertainty↳ Could also: Bootstrap resampling of the test set (e.g., 1,000 iterations) to derive 95% confidence intervals around each metric — CIs around AUC and precision quantify how much of the observed difference between tools is attributable to sampling variability in the benchmark, enabling more nuanced comparison when metrics are close
-
Competing classifiers were compared by inspecting tabulated metric values side by side, without a formal statistical comparison of their performance on the same test set↳ Could also: McNemar's test for comparing error rates of two classifiers on the same held-out set, or DeLong's method for comparing paired AUC values — Formal pairwise tests provide a principled way to assess whether observed differences in accuracy or AUC between methods exceed what would be expected by chance on a fixed benchmark
-
A single random partition was used for the homology-aware benchmark (500 AMPs:500 non-AMPs test set from one clustering run)↳ Could also: Repeated stratified k-fold cross-validation (e.g., 5×5-fold) partitioned at the homology-cluster level — Repeated cross-validation reduces variance arising from a single random split and yields a more stable performance estimate, particularly when cluster sizes are unequal
-
The paper uses ROC-AUC as a primary summary metric, evaluated at the same seven class-imbalance ratios explored in training↳ Could also: Precision-recall (PR) curves and area under the PR curve (AUPRC) plotted across all decision thresholds for each imbalance ratio — Under high class imbalance (up to 1:50), PR curves are more informative than ROC curves because they focus on the positive (AMP) class, directly reflecting the precision-recall trade-off that matters most for genomic prospection
-
Random forest hyperparameters were fixed at scikit-learn defaults except for the number of trees (set to 101)↳ Could also: Nested cross-validation with random or grid search over key hyperparameters (e.g., max_features, min_samples_leaf, max_depth) — Hyperparameter tuning within an inner cross-validation loop prevents optimistic bias from implicit selection and characterizes the sensitivity of performance to parameter choices
-
Pipeline fidelity on simulated metagenomes was assessed at three sequencing depths (40, 60, 80 million reads) using six fixed metagenome compositions↳ Could also: Factorial or Latin-hypercube design varying depth, community diversity, and read-error rate simultaneously — A multi-factor simulation design would identify which parameters most strongly influence AMP recovery and false-positive rate, and whether depth effects are consistent across community compositions
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Strong third-party-tool reproduction: the deterministic macrel peptides example is byte-identical (897/897), Table 2 (hemolytic) reproduces within tolerance, and Table 1 AMP specificity/precision reproduce exactly (0.999 vs 0.998). The only material deviations — AMP accuracy 0.95→0.927, sensitivity 0.90→0.854, MCC 0.90→0.862 — sit on the input/data side (a redistributed AmPEP non-AMP file that drifted +~1k seqs plus a scikit-learn version gap), and the paper's 0.90 sensitivity is correctly bracketed between the naive (0.985) and de-overlapped (0.854) runs, so this is technical/expected, not an authors' defect. No fabrication indicators. The notable finding is a packaging bug on our/installer side (shipped ONNX models break under onnxruntime 1.26.0 but reproduce exactly under 1.20.1), which is an environment issue independent of the paper. Overall yellow: solid with fully explainable, modest deviations.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.