IsomiR_Window: a system for analyzing small-RNA-seq data in an integrative and user-friendly manner.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Software/tool paper (IsomiR Window). Described well enough: code on GitHub plus a fully-preserved tool site (VM 9GB, standalone Built tarball 261MB, per-species Knowledge Bases on Google Drive, doc PDFs). Reproduced by artifact verification: the site ships Built_Shared_Folder containing a COMPLETE executed run of the paper's Basal Cell Carcinoma demonstration, so each reported number was recomputed from the shipped pipeline output. 7 of the quantitative claims reproduce EXACTLY (mapping 94.15%; miRNA proportion >80% both conditions; isomiR 3'-trimming 54.6%/62.1%; novel miRNAs 155 with sub-counts 79/47 — the randfold+RNAcentral+seed filtering logic was fully recovered). 1 partial (top-15 DE: 15 total exact but isomiR/canonical split 13/2 vs reported 12/3, an editing/length boundary + DESeq2-version effect). The miR-183-5p '1756 isoform-exclusive targets' could not be derived because only one functional run is shipped (needs variant-vs-canonical difference). Two simulation-benchmark numbers (4-5% FP, 100% accuracy) were not attempted (simulated data not shipped). NOT attempted: full independent re-execution from raw FASTQ (needs multi-GB KB + 12 pinned tools + web-app orchestration; standalone wiki broken). Dataset PRJNA148721: all 30 runs present on ENA; note the shipped 2-condition analysis used 26 (15 Nodular + 11 Infiltrative). Verdict provisional; values are auditable from the cited shipped files.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 78assessed: 2026-06-18 ⛓ fc240c58af67
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-18
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusMost existing small-RNA-seq tools cannot identify all isomiR types and require bioinformatics expertise; the authors hypothesize that an integrated, user-friendly platform (IsomiR Window) can systematically detect, quantify, and functionally explore all types of isomiRs, revealing biologically relevant isomiR-level signals missed by canonical miRNA analysis.
- ★ IsomiR Window is an integrated, user-friendly platform that systematically identifies, quantifies, and functionally explores isomiR expression in small-RNA-seq datasets without requiring computational skills resource
- ★ The pipeline incorporates a novel algorithm that detects all types of isomiRs (3'end, 5'end, 3'tailings, internal SNPs/editings) and all possible fuzzy combinations method
- ★ On simulated small-RNA-seq data, isomiR detection and quantification identified different isomiR types with high confidence and 100% accuracy for correctly mapped reads finding
- ★ In Nodular BCCs, up-regulation of miR-183-5p is predominantly due to a novel 5'end variant with a different seed region finding
- ★ The novel miR-183-5p 5'end isomiR has 1756 isoform-exclusive mRNA targets significantly associated with disease pathways (including STAT5 and NOTCH2) finding
- IsomiR Window processes multiple datasets automatically with no input file size restriction and integrates over ten third-party tools plus required databases (miRBase, RNAcentral, Ensembl) method
- IsomiR Window supports analysis of 21 species of animals and plants and is distributed as a Linux virtual machine with all required software resource
- isomiR-focused analysis reveals biologically relevant signals (differential isomiR expression between BCC subtypes) missed by canonical miRNA-level analysis finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| small-RNA-seq simulation / isomiR detection benchmarking | simulated human 22-nt and 16-nt miRNA reads (miRBase v22.1 hairpin and mature sequences) | in-silico insertion of single nucleotide variants (SNVs) and 3'end tailing additions | true positives, false positives, false negatives; sensitivity and specificity of isomiR detection | ART read simulator; in-house Perl scripts |
| small-RNA-seq analysis (isomiR identification, quantification, differential expression, functional analysis) | human Basal Cell Carcinomas (BCCs), 30 datasets (SRA PRJNA148721, SRR364267–SRR364296) | none (comparison of distinct BCC types, e.g. Nodular vs other) | isomiR/miRNA expression levels, differential expression, mRNA target prediction, pathway enrichment | Flexbar (adapter trimming), Prinseq (quality filtering) |
- – IsomiR Window identified different types of isomiRs with high confidence and 100% accuracy for correctly mapped simulated reads 100% accuracy
- ▲ miR-183-5p is up-regulated in Nodular BCCs, predominantly driven by a novel 5'end isomiR variant
- – The novel miR-183-5p 5'end variant displays a different seed region motif and 1756 isoform-exclusive mRNA targets significantly associated with disease pathways 1756 targets
- – Differentially expressed isomiRs target disease-associated genes including STAT5 and NOTCH2
- other 100% accuracy (isomiR identification accuracy for correctly mapped simulated reads)
- count 1756 isoform-exclusive mRNA targets (targets of the novel miR-183-5p 5'end variant)
- count 30 datasets (SRR364267–SRR364296) (BCC small-RNA-seq datasets from SRA PRJNA148721)
- count 13 tools (existing isomiR analysis tools compared)
- count 21 different species (animal and plant genomes supported)
- count average read depth 1000 for hairpins and 10,000 for mature derived transcripts (simulated dataset read depth)
- other reads 18–28 nts, average quality score > 20 (read filtering criteria for BCC data)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is primarily a software tool paper introducing IsomiR Window, an integrated pipeline and browser interface for small-RNA-seq isomiR analysis. Algorithm performance was validated on simulated datasets using sensitivity and specificity metrics. A biological case study applied the tool to 30 publicly available small-RNA-seq datasets from Basal Cell Carcinomas (BCCs), using the pipeline's integrated differential expression and functional enrichment modules to identify differentially expressed isomiRs and their associated disease pathways. Statistical details for the BCC case study are reported qualitatively rather than numerically in the text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Sensitivity (TP/(TP+FN)) and specificity (TP/(TP+FP)) on simulated reads | Algorithm validation: isomiR detection on simulated 22-nt and 16-nt small-RNA-seq datasets | Simulated at 1000x average depth for hairpins and 10,000x for mature-derived transcripts; exact read counts not stated | not stated |
| Differential expression analysis (specific test/model not named in text; described as using an integrated third-party tool) | Comparison of isomiR expression between BCC subtypes using 30 SRA datasets | 30 datasets (SRR364267–SRR364296) | not stated |
| Pathway/functional enrichment analysis (specific method not named in text) | Functional analysis of 1756 isoform-exclusive mRNA targets for disease pathway association | 1756 isoform-exclusive mRNA targets | not stated |
-
Simulated-data algorithm performance was summarized at a single operating point as sensitivity and specificity↳ Could also: A receiver operating characteristic (ROC) curve with area under the curve (AUC) could also characterize performance — ROC/AUC describes the full sensitivity–specificity trade-off across all decision thresholds, rather than at one fixed point, which is particularly informative when downstream users may want to tune the detection stringency
-
The specific differential expression statistical model applied to the BCC datasets is not named in the text↳ Could also: Explicitly stating the DE method and key parameters (e.g., DESeq2 Wald test with negative binomial model, or edgeR exact test) would be an alternative reporting choice — Naming the model and its assumptions (e.g., negative binomial dispersion estimation strategy) allows readers to assess suitability for their own data and supports full computational reproducibility
-
Multiple isomiR features across multiple BCC comparisons were tested without a documented multiplicity correction strategy↳ Could also: Reporting an explicit family-wise error control procedure such as Benjamini-Hochberg FDR applied across all isomiR tests simultaneously would also be standard practice — When hundreds to thousands of isomiRs are tested simultaneously, documenting the correction scope and method helps readers interpret the expected false-discovery rate among reported hits
-
Functional enrichment significance is described only qualitatively ('significantly associated with disease pathways')↳ Could also: Reporting the enrichment method (e.g., Fisher's exact test, hypergeometric test, or GSEA), the reference gene set database and version, and corrected p-values or FDR would also convey this result — Quantitative enrichment statistics together with the database version enable independent replication and direct comparison with other isomiR or miRNA studies in BCCs
-
Simulated data were generated at two fixed read depths (1000x hairpins, 10,000x matures) for performance evaluation↳ Could also: Evaluating sensitivity and specificity across a range of simulated depths (e.g., 10x, 100x, 1000x, 10,000x) would also characterize robustness — Real small-RNA-seq experiments vary widely in sequencing depth; a depth-stratified benchmark would show how detection accuracy degrades at lower coverage, informing users about minimum sequencing requirements
-
The BCC case study used 30 publicly available datasets without describing the statistical power for detecting differential isomiR expression↳ Could also: A post-hoc power analysis or simulation-based sample-size justification for detecting the reported fold-changes could also be reported — For small-RNA-seq differential expression, power is sensitive to sequencing depth, dispersion, and effect size; a power estimate contextualizes the sensitivity of the analysis and guides future study design
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33522913 (IsomiR Window)
Software/tool paper (BMC Bioinformatics 2021). Code: github.com/andreiaamaral/ IsomiR-Window (Perl pipeline + R DESeq/topGO). Tool distributed as a VirtualBox VM (8.98 GB) and a standalone "Built" tarball (261 MB) at https://isomir.fc.ul.pt.
In scope (pipeline-derived results, demonstration dataset PRJNA148721, BCC)
- Read mapping rate to GRCh38 (bowtie 1.2.2) [pipeline: find_ncRNAs.pl/bowtie]
- ncRNA classification, miRNA proportion [find_ncRNAs.pl + RNAcentral gff3]
- isomiR type distribution (3'/5' trimming/addition %) [find_isomiRs.pl]
- novel miRNA prediction counts (155/79/47) [miRDeep2 2.0.1.2]
- differential expression top-15 isomiRs/miRNAs [deseq.pl + Rdeseq.R, DESeq2 1.24]
- functional targets of miR-183-5p 5' variant (1756) [targetscan_70.pl + miRanda + topGO]
Out of scope / not attempted
- Simulation benchmark (4-5% FP, 100% complex-isomiR accuracy): derived from an in-silico spike-in dataset not shipped in the example; would need their simulation generator. NOT ATTEMPTED.
- Wet-lab / clinical sample collection: external. OUT OF SCOPE.
Reproduction strategy used
Artifact verification. The site ships Built_Shared_Folder.zip containing a COMPLETE EXECUTED RUN of the BCC demonstration (project "1"). Each reported number was recomputed by us from those shipped pipeline output files and compared to the paper. This is a direct check that the paper's numbers are the actual pipeline output (fabrication detection). A full independent re-run from raw FASTQ was not performed (boundary documented in AUDIT.md).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.