Vespucci: a system for building annotated databases of nascent transcripts.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
The Vespucci GRO-seq-to-transcript-atlas pipeline (authors' own code, github.com/karmel/vespucci) was run end-to-end on «our HPC» inside apptainer containers (bowtie2+mm9 alignment, Postgres 9.2-backed tag loading, transcript stitching with the paper's own default MAX_EDGE=500/DENSITY_MULTIPLIER=10000, density setting, edge drawing and scoring) using the 4 of 5 GRO-seq replicates from GSE48759 that are actually public (GSM1183906-08, GSM1183914; the paper's 5th replicate is not in this GEO series and was unavailable). The pipeline completed successfully and produced a full transcript atlas: 119794 total transcripts (paper: 84076) and 40627 scored >=1 'confident' transcripts (paper: 34743, 41%), i.e. same order of magnitude but 20-40% higher than reported, with a lower confident-fraction (34% vs 41%) -- plausibly explained by one fewer input replicate and/or unknown/undocumented read pre-processing (adapter trimming) differences. Strand balance (49.9%/50.1%) matches the paper's reported near-symmetric sense/antisense transcription. NOT reproduced in this pass: the paper's detailed RefSeq/ncRNA.org annotation-overlap breakdown and its benchmark against the Hah et al. HMM (IRR metric) -- these require external annotation databases and a third-party HMM implementation that were not set up in this run and are flagged as not-attempted rather than failed. The dataset (GSE48759) is open, complete for the samples actually used, and delivered what the paper's methods promised for the GRO-seq subset; the series as a whole is far larger (139 samples, multi-assay) than the 4 samples this room validated.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-07-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-07-31no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan transcriptional units be identified de novo from GRO-seq short reads by an algorithm that merges overlapping reads and separates regions of disparate read density, thereby capturing RefSeq-like mRNA boundaries while resolving short non-coding RNAs such as eRNAs? The paper tests this by building Vespucci and applying it to primary mouse macrophage GRO-seq data.
- ★ Existing ChIP-seq and RNA-seq analysis platforms (e.g. Cufflinks, peak callers) are unsuited to GRO-seq because they assume spliced/exonic reads, uniform density and paired-end data, and cannot identify transcriptional units de novo across the whole genome. finding
- ★ Vespucci is a novel algorithm and system for de novo transcript identification from GRO-seq that determines transcript regions, stores them in a PostgreSQL relational database and associates them with known reference annotations by 2D genomic overlap. method
- ★ Transcript assembly can be framed as merging proto-transcripts in a 2D space where the x-axis is genomic position and the y-axis is read density, with merging governed by two parameters (DENSITY_MULTIPLIER and MAX_EDGE). mechanism
- ★ Vespucci expands existing annotations for mRNAs and lincRNAs by defining the primary transcript beyond the polyadenylation site; many nascent mRNA transcripts extend well beyond RefSeq annotated termination sites. finding
- ★ Vespucci generates assemblies for un-annotated non-coding RNAs, including those transcribed from enhancer-like elements, and predicts approximately twice as many non-coding transcripts as the Hah et al. HMM. finding
- The Hah et al. two-state HMM is optimized to recover RefSeq-defined boundaries and therefore loses or merges many shorter non-coding transcripts, and its flat-file storage impedes integration with other genomic data. finding
- A custom length-sensitive score (RPKM scaled by log base 100 of transcript length, with transcripts <200 bp set to 0) reduces noise from short stitched transcripts while retaining lowly transcribed long transcripts. method
- ★ The Python/PostgreSQL codebase and a pre-loaded Amazon EC2 AMI are publicly available at https://github.com/karmel/vespucci for use and extension. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| GRO-seq (global run-on sequencing) library preparation and sequencing | Thioglycollate-elicited primary peritoneal macrophages from 6–8-week-old C57Bl/6 mice | none (serum starvation overnight in 0.5% FBS following thioglycollate elicitation) | Position, length, orientation and abundance of nascent transcripts (strand-specific short reads) | — |
| GRO-seq (previously published, re-analyzed) | MCF-7 human breast cancer cells | none stated | Nascent transcript short reads used for transcript calling | — |
| ChIP-seq (previously published, re-analyzed) | Mouse macrophages (mm9) | none | H3K4me1 enrichment versus input; peaks called with HOMER findPeaks -nfr -style histone | — |
| ChIP-seq (previously published, re-analyzed) | MCF-7 cells | none | H3K4me2 enrichment versus input; peaks called with HOMER | — |
| Short-read alignment / computational mapping | mm9 mouse reference genome | none | Uniquely mapped read coordinates (chromosome, strand, start, end) | Bowtie2 with default alignment options (bowtie2 --no-unal -x) |
| De novo transcript assembly and database storage (Vespucci) | Mapped GRO-seq tag files (BAM/SAM) from mouse macrophages and MCF-7 | Parameter variation (DENSITY_MULTIPLIER, MAX_EDGE) | Merged transcript coordinates, tag counts, RPKM and custom length-sensitive scores, annotation associations | PostgreSQL 9.2, Python 2.7+, Django 1.2+ with psycopg2; Amazon EC2 small instance AMI |
| SQL-based quantification of transcript classes (case study counts) | Vespucci transcript database from primary murine macrophages | none | Counts of coding and non-coding transcript categories | — |
- ▲ Vespucci predicts approximately twice as many non-coding transcripts as other systems such as the Hah et al. HMM. ~2-fold
- ▲ Many nascent mRNA transcripts identified by Vespucci extend well beyond RefSeq-annotated termination sites (e.g. run-off past the 3' end of Mmp12).
- – Vespucci can be tuned by parameter choice either to respect RefSeq transcript boundaries or to call the entire nascent transcript at the same locus.
- – Two neighboring sense-strand transcription regions separated by only a short genomic distance were kept as separate transcripts because of differing reads-per-basepair densities. 363 bp apart
- – Promoter-associated RNA is resolved as a distinct unit overlapping but not part of the genic transcript, antisense at the 5' end (Tmbim6/Cstb locus).
- – eRNAs appear as bi-directional transcripts centered on transcription factor binding sites and marked by H3K4me1, and each strand is called as a separate contiguous unit.
- ▼ RPKM tends to decrease with transcript length, whereas the custom score scaled by log(length) levels out scores across the wide range of transcript lengths.
- – Transcript identification is incremental: new samples can be added to the full data set without re-processing all prior data.
- fold_change approximately twice as many (Non-coding transcripts predicted by Vespucci versus the Hah et al. HMM)
- other 35–100 bp (Length range of GRO-seq short reads assumed by the method)
- other 10 000 (Default DENSITY_MULTIPLIER scaling factor relating density to basepairs)
- other 500 (Default MAX_EDGE, maximum allowed 2D distance between proto-transcripts to be stitched)
- other 363 bp (Genomic distance between two adjacent sense-strand transcripts kept separate due to density difference)
- other <200 bp (Transcript length threshold below which the custom score is set to 0)
- count 4 replicates previously published under GSE48759 (GSM1183906–GSM1183908, GSM1183914) (Previously published macrophage GRO-seq replicates reused in this study)
- other 6 to 8-week-old C57Bl/6 mice; 2.5 ml thioglycollate; lavage 3–4 days later; 10% then 0.5% FBS (Primary macrophage isolation and culture conditions)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational-methods paper describing an algorithm and software system (Vespucci) for de novo transcript identification from GRO-seq short-read data, rather than a study built around classical hypothesis-testing statistics. Transcript boundaries are inferred using a 2D merging algorithm governed by two tunable parameters (DENSITY_MULTIPLIER, MAX_EDGE), and transcript abundance is quantified with either standard RPKM or a custom length-scaled score. Comparisons to other approaches (e.g., the Hah et al. HMM) and to RefSeq annotations are reported as descriptive counts and examples rather than through formal statistical tests, and ChIP-seq peaks were called with HOMER using specified style options.
-
The comparison of transcript counts between Vespucci and other methods (e.g., 'approximately twice as many non-coding transcripts' as the Hah et al. HMM) is presented as a descriptive count difference.↳ Could also: a formal statistical comparison such as a bootstrap-based confidence interval or paired proportion test on the count difference — this would quantify the uncertainty around the reported difference and support a statement about whether it exceeds what might be expected from sampling variability alone
-
Transcript abundance is quantified with RPKM or a custom length-scaled score, computed from pooled/merged replicate data.↳ Could also: variance-stabilized, replicate-aware count models such as DESeq2 or edgeR — these provide model-based dispersion estimates and formal significance tests for abundance differences across conditions or replicates, in addition to a point estimate like RPKM
-
Multiple biological replicates (e.g., the four previously published replicates) are merged into a single transcript-calling pass rather than compared to one another quantitatively.↳ Could also: reproducibility metrics such as the Irreproducible Discovery Rate (IDR) or pairwise Pearson/Spearman correlation of replicate tag densities — this would give a quantitative measure of concordance between biological replicates prior to or alongside pooling
-
ChIP-seq peaks for H3K4me1/H3K4me2 were called with HOMER using specified style parameters, without a stated significance threshold (e.g., FDR or p-value cutoff) for the resulting peak set.↳ Could also: explicitly reporting the FDR or p-value threshold used by the peak-calling software — this would let readers gauge the stringency of the peak set and its expected false-positive rate
-
The DENSITY_MULTIPLIER and MAX_EDGE parameters are selected based on qualitative use-case reasoning (e.g., 'if the user desires to focus on large transcripts...').↳ Could also: a systematic parameter sweep benchmarked against a gold-standard annotation set using precision/recall or F1 score — this would offer an objective, reproducible criterion for parameter choice in addition to qualitative guidance
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The authors' own Vespucci code ran end-to-end to completion (bowtie2/mm9 → tag load → stitch → density → edges → score, ALLDONE) on the open GSE48759 data, and the qualitative result reproduces: a populated transcript atlas of the right order of magnitude with a near-even strand split (49.9%/50.1% vs the paper's ~11% sense / 11% antisense symmetry). Quantitatively the counts drift upward — 119,794 vs 84,076 total transcripts (1.42x) and 40,627 vs 34,743 at score ≥ 1 (1.17x, but 34% vs 41% of total). The deviation sits on our/data-availability side, not the authors': the paper used 5 GRO-seq replicates and only 4 are deposited in GSE48759, and Methods do not document any read trimming, so both the input cohort and pre-processing had to be self-defined. Additionally, two substantive Results blocks — the RefSeq/ncRNA.org overlap breakdown (8,742/34,743 = 25% full-gene; 21,916 = 63% unannotated) and the Hah et al. HMM IRR benchmark (1.8% vs 12.7%) — were not attempted, so overall this is a solid but incomplete reproduction rather than a 1:1 verification, with no indication of anything suspect in the published values.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.