iCOMIC: a graphical interface-driven bioinformatics pipeline for analyzing cancer omics data.
The main results reproduced, with only marginal, non-material deviations.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
iCOMIC (Sithara et al. 2022) is a GUI-driven Snakemake cancer-omics pipeline. The harvested triple (paper, code=Picard, data=GSE34515) is a text-mining artifact: Picard (MarkDuplicates) belongs to the DNA-seq germline benchmark on NA12878; GSE34515 is the Affymetrix microarray reference for the RNA-seq DE benchmark (raw RNA-seq=SRP082682). We reproduced the paper's own pipeline results, naming the tool per result (Brief P16). RESULTS: (C1a) GSE34515 microarray reference DE (RMA+limma) -> 21368-gene log2FC, markers textbook-correct (FCGR3B/CD16, CDKN1C up in non-classical; CCR2, S100A12, CD163 up in classical), 3563 genes adj.P<0.05 [partial: no scalar in paper]. (C1) RNA-seq<->microarray FC Pearson r = 0.80-0.82 over DE genes, matching reported 0.80-0.85 [within-tol]; over ALL genes r=0.39 but the paper correlates DE genes. (C2s) germline SNP F1 0.971 vs reported 0.988 [within-tol; recall 0.976~=0.978]. (C2i) germline INDEL F1 0.748 vs reported 0.971 [mismatch], explained by raw HaplotypeCaller indels without the pipeline's variant-filtering step + build/truth-version deviation (GRCh38+GIABv4.2.1 vs b37+v2.19) -- methodological, not fabrication. Scope's 'SRP082682 labels unknown' gap was resolved via SRA sample_alias. Overall: the paper is described well enough to reproduce; SNP-level and transcriptomic-concordance claims reproduce within tolerance, the INDEL F1 claim does not at raw-HC settings. All grades PROVISIONAL -> human reviewer signs off.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-19 ⛓ 755f039bb671
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-29
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetA user-friendly, GUI-driven, Snakemake-based bioinformatics toolkit (iCOMIC) can make comprehensive DNA-Seq and RNA-Seq cancer genomic data analysis, including driver mutation and TSG/OG prediction, accessible to users with minimal programming expertise while matching the performance of expert-built pipelines.
- ★ iCOMIC provides a GUI-driven, Snakemake-based pipeline integrating multiple tools for DNA-Seq and RNA-Seq analysis with minimal command-line interaction. resource
- ★ cTaG classifies genes as tumor suppressor genes (TSG) or oncogenes (OG) using ratio-metric/entropy features from somatic mutation data rather than background mutation rate. method
- ★ NBDriver uses raw nucleotide sequence context surrounding mutations, combined with genomic/structural/evolutionary features, to distinguish driver from passenger mutations. method
- ★ iCOMIC's BWA MEM-GATK HC DNA-Seq pipeline achieved top benchmarking performance (F1 0.971 indels, 0.988 SNPs) against the GIAB NA12878 dataset. finding
- ★ iCOMIC's RNA-Seq pipelines (HISAT2/STAR-StringTie-ballgown) achieved r=0.85 correlation on a human monocyte benchmark dataset. finding
- An ensemble predictor combining NBDriver, CONDEL and MutationTaster outperformed existing pan-cancer models in prioritizing curated driver/passenger mutations. finding
- Nonsense mutations are more commonly found in TSGs than in OGs. finding
- Methods relying on background mutation rate are biased toward high-mutation-rate genes, missing low-mutation-rate driver genes. mechanism
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Germline variant calling benchmarking (WGS/DNA-Seq, 18 tool-combination pipelines) | GIAB/NIST NA12878/HG001 benchmark dataset | none | Recall, precision, F1 score for SNPs and indels | hap.py (qfy.py) with RTG Tools Vcfeval |
| RNA-Seq differential expression pipeline benchmarking (4 tool combinations) | Human monocyte RNA-Seq dataset, NCBI-SRA SRP082682 | none (classical vs non-classical monocyte comparison) | Differentially expressed genes; correlation coefficient between pipelines | HISAT2/STAR-StringTie-ballgown |
| Machine learning model training/validation (NBDriver) | Missense mutations from 58 genes with experimentally validated functional impact; 5 independent validation datasets | none | Classification of driver vs passenger mutations | TF-IDF Vectorizer, Count Vectorizer, One-Hot Encoder; repeated cross-validation |
| Machine learning model training (cTaG) | Somatic mutation data from COSMIC v79, labeled via Cancer Gene Census (CGC) | none | Classification of genes as TSG or OG (pan-cancer and tissue-specific) | Random forest classifier |
- – Highest F1 score for BWA MEM-GATK HC-SnpEff/Annovar pipeline on NA12878 benchmark F1=0.971 (indel), F1=0.988 (SNP)
- – Correlation between RNA-Seq pipeline outputs on human monocyte dataset r=0.85
- ▼ Lowest-performing DNA-Seq combination was Gem3-Bcftools-SnpEff for indels F1=0.486
- – NBDriver true-positive predictions identified driver genes with known functional evidence 138 driver genes
- ▲ Ensemble of NBDriver, CONDEL and MutationTaster outperformed existing pan-cancer driver/passenger prediction models
- – Nonsense mutations occur more frequently in TSGs than OGs
- fold_change F1=0.971 (indel), F1=0.988 (SNP) (BWA MEM-GATK HC pipeline vs GIAB NA12878 truth set)
- correlation r=0.85 (HISAT2/STAR-StringTie-ballgown RNA-Seq pipelines, human monocyte dataset SRP082682)
- count 138 driver genes (True positive predictions from NBDriver with known functional evidence)
- count 58 genes (NBDriver training data of missense mutations with experimentally validated functional impact)
- other F1=0.486 (lowest indel score) (Gem3-Bcftools-SnpEff pipeline, NA12878 indel benchmarking)
- other 18 workflow combinations benchmarked (Germline variant calling benchmarking with GIAB dataset)
- other default percentile=5; threshold >2000 mutations excluded (cTaG algorithm parameters for defining highly-mutated samples)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a bioinformatics tool/pipeline paper (iCOMIC) rather than a classical hypothesis-testing study; it reports computational benchmarking of pipeline combinations against reference truth sets. Performance was quantified using recall, precision and F1 score against GIAB/GA4GH benchmark standards for variant calling, a correlation coefficient for RNA-Seq pipeline concordance, and repeated cross-validation with median performance metrics for the in-house machine-learning classifiers (cTaG, NBDriver). No conventional inferential statistical tests (e.g., t-tests, ANOVA) or p-values are reported in the provided text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Benchmarking metrics (recall, precision, F1 score) via GA4GH/hap.py framework with RTG Tools Vcfeval | Comparison of 18 aligner–variant caller–annotator pipeline combinations against GIAB/NIST HG001 v2.19 truth set (NA12878) | 18 pipeline combinations run on one benchmark dataset (NA12878/HG001) | not stated |
| Correlation coefficient (r) | Comparison of differentially expressed genes/expression concordance between HISAT2-StringTie-ballgown and STAR-StringTie-ballgown RNA-Seq pipelines on human monocyte dataset (SRP082682) | not stated | not stated |
| Repeated cross-validation with reported median performance metrics | Training/evaluation of NBDriver machine-learning models (feature–classifier pairs) for driver vs passenger mutation prediction | missense mutations from 58 genes with experimentally validated functional impact | not stated |
| Random forest classification (pan-cancer model) | cTaG classification of genes as tumor suppressor genes (TSG) vs oncogenes (OG) | somatic mutation data from COSMIC v79 across cancer types, labels from Cancer Gene Census | not stated |
| Kernel density estimation (descriptive comparison of distributions) | Comparing neighborhood sequence feature distributions surrounding driver vs passenger mutations (NBDriver) | not stated | not stated |
-
Pipeline performance is summarized with single point estimates of recall, precision and F1 score for each of the 18 (DNA-Seq) and 4 (RNA-Seq) tool combinations.↳ Could also: Reporting a measure of uncertainty around these estimates, such as bootstrapped confidence intervals over variant subsets or genomic regions. — This would convey how much the F1/precision/recall values might vary with resampling, complementing the single-dataset point estimates already reported.
-
NBDriver cross-validation results are summarized using median values of performance metrics across a repeated cross-validation strategy.↳ Could also: Reporting the full distribution (e.g., mean ± SD or interquartile range) across folds/repeats, or nested cross-validation for hyperparameter selection. — Showing spread across folds in addition to the median would give readers a sense of model stability, and nested CV can guard against optimistic bias from hyperparameter tuning on the same folds used for evaluation.
-
RNA-Seq pipeline concordance between two aligner/DE workflows is summarized using a single correlation coefficient (r = 0.85).↳ Could also: Accompanying the correlation coefficient with a scatter plot, a formal test/CI for the correlation, or a Bland-Altman-style agreement plot. — These would help characterize not just the strength of association but also the pattern and magnitude of any systematic differences between the two pipelines' expression estimates.
-
Differences between driver and passenger mutation sequence-feature distributions were assessed descriptively via kernel density estimation.↳ Could also: A formal distributional comparison test (e.g., Kolmogorov-Smirnov or Mann-Whitney U on derived features) could also be used. — This would provide a quantitative statistic and significance level alongside the visual/descriptive KDE comparison already presented.
-
Multiple tool combinations (18 DNA-Seq, 4 RNA-Seq workflows) are compared without an explicit multiplicity adjustment.↳ Could also: Where formal significance testing across many pipeline comparisons is desired, a correction such as Benjamini-Hochberg FDR could also be applied. — This would help control the chance of highlighting a top-performing pipeline combination by chance alone when many combinations are screened, though it is worth noting the paper's core comparisons are benchmarking metrics rather than hypothesis tests, so this may not be a core need for this type of analysis.
-
Differential expression analysis relies on DESeq2 and ballgown, which have built-in normalization and testing procedures not elaborated on in the text.↳ Could also: Explicitly reporting the internal statistical model and multiple-testing correction used by these tools (e.g., DESeq2's Wald test with Benjamini-Hochberg FDR) could also be stated for transparency. — Making explicit which internal test and FDR correction were used (if any settings were changed from defaults) helps readers relate the reported DE gene lists to a known statistical framework.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35899080 (iCOMIC)
Paper: Sithara et al. 2022, iCOMIC: a graphical interface-driven bioinformatics pipeline for analyzing cancer omics data. NAR Genom Bioinform 4(3):lqac053. PMID 35899080 · PMCID PMC9310080 · DOI 10.1093/nargab/lqac053.
Authors' own code: https://github.com/RamanLab/iCOMIC (Snakemake + PyQt GUI).
Assigned harvested code link: https://github.com/broadinstitute/picard — a
third-party dependency iCOMIC calls for MarkDuplicates in the DNA-Seq path.
Assigned harvested data: GEO GSE34515.
Harvested-triple inconsistency (important)
The (paper, code=Picard, data=GSE34515) triple is internally incoherent and is a text-mining artifact:
- Picard is used by iCOMIC only as
MarkDuplicates, a step in the DNA-Seq / germline variant-calling benchmark (on NGS reads, NA12878). It operates on BAM/SAM. - GSE34515 is an Affymetrix microarray (U133 Plus 2.0, 9 samples) used only as the reference to validate the RNA-Seq differential-expression benchmark (raw RNA-seq = NCBI-SRA SRP082682). Picard cannot process microarray CEL data.
So Picard and GSE34515 do not combine into one result. We treat the paper's own data as the reproduction target (per Brief P16) and reproduce the pipeline results those data underpin, naming the tool per result.
In scope (pipeline-derived results)
| id | result | pipeline / tools | data | reported | location |
|---|---|---|---|---|---|
| C1 | RNA-seq↔microarray fold-change correlation | HISAT2/STAR → StringTie → ballgown (and STAR→HTSeq→DESeq2) vs microarray DE | SRP082682 (RNA-seq) + GSE34515 (microarray ref) | Pearson r = 0.85 (StringTie-ballgown); 0.8 (HTSeq-DESeq2); Galaxy 0.66 | Figs 5–6 |
| C1a | microarray-side reference DE fold changes (sub-result of C1, fully specified) | RMA + limma, non-classical (CD16+) vs classical (CD16−) | GSE34515 only | (no explicit number; the per-gene FC vector feeding C1) | Methods / Fig 5 |
| C2 | germline SNP/indel benchmark F1 | BWA-MEM → Picard MarkDuplicates → GATK HaplotypeCaller → SnpEff; hap.py vs GIAB | NA12878/HG001 (GIAB) | SNP F1 0.988 (recall 0.978, prec 0.998); INDEL F1 0.971 (recall 0.967, prec 0.976) | Table 2 |
C2 is the result that actually exercises the assigned tool (Picard). C1/C1a is the result that actually uses the assigned dataset (GSE34515).
Out of scope (not pipeline-derived / not attempted)
- The PyQt GUI itself, usability, "graphical interface" claims — software-engineering, not a numeric pipeline output.
- Machine-learning modules (cTaG, NBDriver driver-gene prediction) — separate models, not the assigned data.
- Galaxy/other-tool comparison numbers except as context for C1.
- Any wet-lab step (none; fully computational paper).
Known reproducibility gaps (surfaced from the paper text)
- SRP082682 sample assignment unspecified. 34 RNA-seq runs exist; the paper does not list which runs are "classical" vs "non-classical" monocytes nor how many it used. Group labels must be recovered from BioSample/GEO metadata.
- Correlation methodology thin. Paper states "Pearson correlation of fold changes" between GSE34515 and RNA-seq DE, but not gene-ID mapping, filtering, or which contrast on the microarray. Exact r=0.85 is not 1:1-pinnable → best honest grade for C1 is partial/within-tol, not exact.
- C2 underspecified: NA12878 read set (WGS vs exome, which accession), reference build (hg19/hg38), and GIAB truth version (text says "HG001 v2.19") are only partially given. v2.19 is an old truth set; using a newer one shifts F1.
Compute plan
All heavy compute on «our HPC» (SLURM, «infra»). Microarray side (C1a) = light R/ Bioconductor. RNA-seq side (C1) = align ≤34 samples (heavy). C2 = BWA→Picard→GATK (heavy; prefer exome NA12878 if available to bound cost).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.