nf-core/isoseq: simple gene and isoform annotation with PacBio Iso-Seq long-read sequencing.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> 1:1 functional reproduction. nf-core/isoseq is a Bioinformatics applications note whose contribution IS the pipeline; it reports NO quantitative benchmark values, so this is a functional reproduction (BRIEF rule P16: valid third-party-tool-on-its-own-data). Pinned to the paper-era release 1.1.1 and run with its documented -profile test (human GRCh38 chr19 Iso-Seq CI fixture) on a «our HPC» compute node, the pipeline completed cleanly (Nextflow exit 0, 14m31s, 225 output files) and produced EVERY documented output: per-chunk collapsed bed12 transcriptomes (14 transcripts total), the final TAMA-merged gene/isoform annotation bed12 (13 genes G1-G13, all chr19), and the gene report. The exact documented tool chain ran at the manifest versions (pbccs 6.2.0, lima 2.2.0, isoseq3 3.4.0, bamtools 2.5.1, gstama/TAMA, uLTRA 0.0.4.1, samtools 1.15.1). Newly characterized upstream counts: 531 ZMWs in -> 326 CCS -> 275 FLNC. NOT attempted: any numeric paper comparison (none exists); large-scale / real biological Iso-Seq runs (out of scope — paper only documents the test profile). Grades are provisional; a human reviewer signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ bfcc13158c8c
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper addresses whether the manual, stepwise processing of PacBio Iso-Seq long-read data (a laborious, error-prone workflow with no existing published pipeline) can be automated into a single reproducible pipeline for gene and isoform annotation.
- ★ nf-core/isoseq is a new automated Nextflow-based pipeline that processes raw Iso-Seq subreads through to genome annotation (BED format) without requiring transcriptome assembly. method
- ★ No Iso-Seq annotation pipeline had been published before this tool; manually running each program individually is long, laborious, and prone to errors. finding
- ★ Using FLNC reads instead of HiFi reads avoids information compression and allows the complete workflow to be run in parallel. finding
- ★ The pipeline follows nf-core guidelines/templates, is written in Nextflow DSL2, and each program is containerized (Singularity, Docker, or Conda), requiring only Java, Nextflow, and Docker/Singularity as dependencies. method
- The pipeline offers a choice of aligners: uLTRA (better small-exon detection using reference annotation) or minimap2 (when no reference annotation is available). method
- ★ TAMA collapse and TAMA merge are used to collapse alignments into transcript models and merge batch/sample annotations; the resulting annotation is left unfiltered so users can apply their own filtering criteria. method
- The pipeline is freely available on the nf-core website and GitHub under an MIT License and can be tested with an nf-core test dataset. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| PacBio Iso-Seq long-read RNA sequencing (subread generation via reverse transcription, template switching, SMRTbell adapter addition) | not organism-specific; general pipeline applicable to any genome/tissue (cited prior uses: animal, plant, fish genomes) | none | raw subread BAM files | PacBio sequencing instrument |
| Subread preprocessing (ccs consensus calling, lima primer selection, isoseq3 refine chimera removal, TAMA polyAcleanup, bamtools BAM-to-FASTA conversion) | Iso-Seq subread BAM input (any sample/organism) | none | cleaned Full-Length Non-Chimeric (FLNC) reads without polyA tails | PacBio isoseq3 tools; bamtools |
| Long-read mapping to reference genome | reference genome of the sample under study | none | spliced alignments (BAM) | uLTRA or minimap2 |
| Alignment post-processing / collapsing into transcript and gene models | mapped FLNC alignments, batched by sample | none | unfiltered genome annotation of transcripts/isoforms in BED format | TAMA collapse and TAMA merge |
- – The pipeline can be run end-to-end (subread preprocessing, mapping, alignment post-processing) to automatically generate genome annotation from raw Iso-Seq subreads.
- – uLTRA outperforms other splice-aware aligners at detecting small exons (<30 nucleotides).
- – The pipeline can be tested using the nf-core test dataset.
- other small exon size threshold <30 nucleotides (criterion for uLTRA's improved small exon detection versus other splice-aware aligners)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is an applications-note paper describing nf-core/isoseq, a Nextflow-based computational pipeline for processing PacBio Iso-Seq long-read sequencing data into gene/isoform annotations. The text describes pipeline architecture, component tools (ccs, lima, isoseq3, minimap2/uLTRA, TAMA), and implementation choices, but does not report an experiment with statistical hypothesis testing, group comparisons, or quantitative benchmarking results.
-
The paper introduces the pipeline without including a quantitative benchmarking section comparing its output (e.g., isoform/gene calls) against a reference annotation or other existing Iso-Seq pipelines.↳ Could also: A benchmarking analysis reporting metrics such as sensitivity, precision, or concordance against a gold-standard annotation, potentially across multiple samples or replicate runs — such a comparison, especially if summarized with a measure of spread (e.g., range or CI across samples), would let readers quantify how the pipeline's output compares to alternatives; this is a common addition in papers introducing new bioinformatics pipelines, though not required for an applications note.
-
The pipeline offers two alignment options (uLTRA and minimap2) and describes their relative strengths qualitatively (e.g., small-exon detection) without a reported quantitative comparison on shared data.↳ Could also: A paired comparison of alignment/annotation outcomes between the two aligners on the same dataset, using standard concordance metrics — reporting such a comparison would give users a quantified basis for choosing between aligner options, complementing the qualitative rationale already given.
-
The final annotation is left unfiltered, with filtering (e.g., by read count) left entirely to user discretion, without a suggested statistical or model-based default.↳ Could also: An optional model-based filtering step, such as a count-based threshold derived from a null/expected distribution (an approach used in some isoform-classification tools) — this could offer a reproducible, principled default for users while still preserving the flexibility to apply custom filters, as the pipeline currently allows.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36961337 (nf-core/isoseq)
Paper: Guizard et al. 2023, Bioinformatics 39(4):btad150. "nf-core/isoseq: simple gene and isoform annotation with PacBio Iso-Seq long-read sequencing." Type: nf-core Applications Note describing a Nextflow pipeline.
Nature of the paper's "results"
This is a software/pipeline note. The full text reports no quantitative benchmark values — no read counts, isoform counts, runtimes, mapping rates, or BUSCO scores. The paper's contribution is the pipeline itself and the claim that it produces an unfiltered gene/isoform BED annotation from raw PacBio Iso-Seq subreads via a documented sequence of tools.
Therefore the reproducible, pipeline-derived result is functional:
Running the published pipeline on its documented test data completes successfully and produces the documented gene/isoform annotation outputs.
This is a valid third-party-tool-on-its-data reproduction (BRIEF rule 2 / P16):
we run the authors' own published pipeline on the published test dataset using
the documented -profile test parameters and check it delivers what it promises.
Pipeline steps (in scope — all computational)
- PBCCS — subreads → CCS consensus
- LIMA — primer removal
- ISOSEQ3 REFINE — chimera removal → FLNC
- BAMTOOLS CONVERT — bam → fasta
- GSTAMA_POLYACLEANUP — polyA trimming
- uLTRA (or minimap2) — splice-aware genome mapping
- SAMTOOLS SORT
- GSTAMA COLLAPSE — collapse →
*_collapsed.bed(bed12 transcriptome) - GSTAMA MERGE —
*.bedfinal merged annotation +*_gene_report.txt
In-scope reproduction targets (functional claims)
- C1 Pipeline runs to completion on
-profile test(Nextflow, all processes succeed). - C2 Produces a bed12 collapsed transcriptome (
08_GSTAMA_COLLAPSE/*_collapsed.bed). - C3 Produces final merged annotation (
10_GSTAMA_MERGE/*.bed) + gene report. - C4 (stretch) Record observed counts (CCS reads, FLNC reads, collapsed transcripts, merged genes) as concrete derived values — the paper gives none, so these are newly characterized reference numbers, not paper comparisons.
Out of scope
- No wet-lab / sequencing steps (subread generation is upstream of the pipeline).
- No numeric paper claims exist to grade against (none reported).
Test dataset (what -profile test actually uses)
- input samplesheet: nf-core/test-datasets @ isoseq branch
- reference:
Homo_sapiens.GRCh38.dna.chromosome.19.fasta - gtf:
Homo_sapiens.GRCh38.104.chr.13_18_19.gtf - params: rq=0.9, chunk=5, five_prime=100, three_prime=100, splice_junction=10, capped=true, aligner=ultra, max_cpus=2, max_memory=6.GB, max_time=6.h
- organism: human (GRCh38, chr19 subset) — NOT chicken (despite Roslin authors).
Data-deposit note (profiling)
The brief's "Data: zenodo 10.5281/zenodo.7116979" resolves to the archived pipeline source code (nf-core/isoseq-1.1.1.zip, 2.9 MB) — i.e. software, not an experimental dataset. The actual analysis inputs are the small nf-core/test-datasets isoseq files. This is recorded in dataset_profile.json.
Version pin
Paper-era release = v1.1.1 (the Zenodo archive). Reproduction uses that tag.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.