KARAJ: An Efficient Adaptive Multi-Processor Tool to Streamline Genomic and Transcriptomic Sequence Data Acquisition.
The main results reproduced, with only marginal, non-material deviations.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL, healthy. KARAJ is a Bash download-manager whose only quantitative claims are download-throughput ratios. The Aspera-vs-others speedups (C1-C3) hinge on IBM Aspera (ascp), and ENA's anonymous era-fasp service no longer authenticates (post-2024) -- a genuine EXTERNAL blocker, so the exact Aspera ratios are not reproducible via the live service; we measured wget/curl/fastq-dump baselines on two link conditions and bounded the ratios instead. Independently, KARAJ hard-codes ascp -l 300m (35.8 MiB/s); measured HTTPS speeds straddle that cap, so the Aspera-wins ordering is environment-specific, not universal. The reproducible CORE -- the multi-processor parallel framework (C4) -- DOES reproduce: clean interleaved medians on «our HPC» give a j8/j1 speedup of 3.31x, close to the reported 3.6x. The C3 premise (fastq-dump far slowest, 1.79 MiB/s) and the C5 disk-check halt logic (5/5) are confirmed. NOT attempted: text-mining/summary-report features (no numeric claim), Figure 1 (schematic), the 277 GB download for C5 (anecdote, not a computation). The paper's central mechanism reproduces; the Aspera-specific magnitudes are blocked + environment-specific.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-20 ⛓ 941b71b2fa8a
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-23
- Rubric version
- not recorded
- Assessed by
- —
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan an automated, end-to-end command-line tool streamline the querying, size-checking, metadata retrieval, and rapid parallel downloading of genomic/transcriptomic sequence data and article supplementary tables more efficiently than existing manual or partial tools?
- ★ KARAJ automates end-to-end querying and downloading of genomic/transcriptomic sequence data from a list of PMCIDs, URLs, or accession numbers method
- ★ KARAJ's parallel downloading framework, powered by Aspera, significantly reduces downloading time compared to using Aspera alone finding
- ★ Aspera is faster than wget, curl, and fastq-dump for downloading FASTQ files finding
- ★ KARAJ mines full-text research articles for accession numbers and supplementary materials using the Lynx package method
- ★ KARAJ calculates dataset file size and checks local disk space availability before downloading method
- KARAJ generates a metadata table containing sample information and experimental design for a study method
- ★ KARAJ enables retrieval of supplementary data tables attached to publications, a feature not offered by existing tools resource
- KARAJ supports PRJNA accession IDs by converting them to SRP IDs via Entrez Direct then fetching links via ffq method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| download speed benchmarking | 10 FASTQ files from GSE126379 | download tool (Aspera vs wget vs curl vs fastq-dump) | time to download | IBM Aspera connect / wget / curl / fastq-dump |
| parallel download benchmarking | GSE126379 sequence data | KARAJ parallel framework with 8 cores vs Aspera alone | downloading time reduction (fold-change) | Aspera connect |
| literature text mining for accession numbers | multiple published research articles (PMCIDs/URLs, e.g., PMC6492329, PMC7182534, etc.) | none | list of accession numbers, PubMed URL, title, abstract, PMID (info.txt summary table) | Lynx |
| local storage capacity check | sequence data from MacParland et al. study (277 GB) vs local disk (7.6 GB free) | none | halt of download process with memory size limitation error | — |
| metadata table retrieval | GSE126379 accession | none | metadata table with sample/experimental design info | ffq |
| supplementary table retrieval | multiple articles (e.g., PMC6492329, PMC7182534, PMC8000127, etc.) | none | downloaded supplementary data tables saved per PMCID directory | — |
- ▲ Aspera is faster than wget for downloading 10 FASTQ files from GSE126379 3.2-fold
- ▲ Aspera is faster than curl for downloading the same FASTQ files 3.1-fold
- ▲ Aspera is faster than fastq-dump for downloading the same FASTQ files 5.0-fold
- ▼ KARAJ parallel framework using 8 cores reduced downloading time of GSE126379 data compared to Aspera alone 3.6-fold
- ▲ Parallel framework speeds up downloading process by at least two times, scaling up to the number of local system cores ≥2-fold
- – KARAJ correctly detected insufficient local disk space (7.6 GB) for a 277 GB dataset and halted the download with an error
- fold_change 3.2 (Aspera vs wget download speed, GSE126379 FASTQ files)
- fold_change 3.1 (Aspera vs curl download speed, GSE126379 FASTQ files)
- fold_change 5.0 (Aspera vs fastq-dump download speed, GSE126379 FASTQ files)
- fold_change 3.6 (KARAJ parallel framework (8 cores) vs Aspera alone, GSE126379)
- other 277 GB dataset vs 7.6 GB free disk space (MacParland et al. dataset memory check demonstration (PMC6197289))
- count 7 required software packages (ffq, pysradb, Lynx, Aspera connect, axel, wget, Entrez Direct) (KARAJ Installer dependency check)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software tool paper describing KARAJ, a Linux command-line utility for automated querying and downloading of genomic/transcriptomic sequence data. The paper contains no inferential statistical tests; its quantitative claims are limited to descriptive performance benchmarks (fold-difference in download speed) derived from a single in-house experiment on one dataset (GSE126379). Results are reported as simple ratios without measures of variability or formal hypothesis testing.
-
Download speed benchmarks were derived from a single run on one dataset (GSE126379, 10 FASTQ files) and reported as single fold-difference values↳ Could also: Repeated-measures benchmarking across multiple runs and/or multiple datasets of varying sizes, reporting mean ± SD or 95% CI for elapsed time — Repeated trials would quantify run-to-run variability attributable to network conditions and I/O jitter; multiple datasets would indicate whether the speed advantage generalises across file sizes and repository endpoints
-
Tool comparisons (Aspera vs. wget, curl, fastq-dump) are summarised as ratio estimates without a formal test of the difference↳ Could also: A paired Wilcoxon signed-rank test or paired t-test on replicated download times for each tool, with effect size (e.g., Cohen's d or rank-biserial r) — Formal testing with an effect size would let readers judge whether observed speed differences are likely to persist beyond this specific experimental context and how large the practical advantage is
-
The parallel-framework benefit (3.6-fold with 8 cores vs. Aspera alone) was measured at a single core count on one dataset↳ Could also: A scaling experiment reporting wall-clock time across a range of core counts (e.g., 1, 2, 4, 8, 16) with replicated runs, and fitting a parallel speedup model (e.g., Amdahl's law) — A speedup curve would characterise how the parallel benefit scales with hardware resources, helping users predict performance on their own systems
-
Memory-check scenario (Scenario 5) reports a single illustrative case (dataset size 277 GB vs. 7.6 GB free) with no systematic evaluation of the module's accuracy↳ Could also: A table of tested accession numbers with stated vs. KARAJ-estimated file sizes and observed disk-space checks, allowing readers to assess estimation accuracy — Systematic validation across multiple datasets and file types would support the claim that the size-estimation module is reliable in practice
-
Accession-number mining accuracy (how many accession numbers KARAJ correctly identifies in article text) is demonstrated through scenarios but not quantified↳ Could also: Precision and recall metrics computed over a held-out set of articles with manually curated ground-truth accession numbers — Precision/recall would give users a calibrated expectation of how often the text-mining step succeeds or misses accession numbers across varied article formats
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36430895 (KARAJ)
Paper: KARAJ: An Efficient Adaptive Multi-Processor Tool to Streamline
Genomic and Transcriptomic Sequence Data Acquisition. Int J Mol Sci 2022.
PMID 36430895 · PMCID PMC9694301 · DOI 10.3390/ijms232214418.
Code: https://github.com/GTP-programmers/KARAJ (Shell, MIT, active — last
commit 2026-05-01). Cloned at «infra» …/reproductions/pmid-36430895/KARAJ.
Data: GEO GSE126379 → SRA study SRP185454 / BioProject PRJNA521732 →
10 single-end FASTQ runs SRR8556723–SRR8556732 (CD19+ B cells & EBV-LCLs),
total ≈ 8.23 GB (fastq_bytes summed from ENA). Public, resolvable on ENA;
ENA still publishes fastq_aspera (fasp) links and the fasp control port 33001
is reachable.
What KARAJ is
A Bash command-line wrapper that, given a PMCID/URL/accession, text-mines
accession numbers (Lynx), summarises datasets, checks free disk vs dataset size,
and downloads sequence + supplementary data. Its download engine uses
wget/axel for supplementary files and ascp (IBM Aspera Connect) driven by
GNU parallel -j K for sequence data (KARAJ.sh L2505–2536):
cat urls | parallel -j K ascp -QT -l 300m --retry-timeout=1800 -P33001 -i <key> -q {} out.
fasp URLs are built by string-rewriting ENA FTP paths
(ftp://ftp.sra.ebi.ac.uk/ → «email»:).
In scope (pipeline-derived, attempted)
All quantitative claims in this paper are download-throughput measurements — the only "pipeline" output the tool produces. They are network/IO-bound, not deterministic computations, so faithful numeric reproduction is not expected; the reproducible question is whether the relative ordering / rough magnitude holds in an independent environment.
- C1–C3 Aspera vs wget / curl / fastq-dump on the 10 GSE126379 FASTQs (reported 3.2× / 3.1× / 5.0×). Pipeline = the four download tools. Reproduce by downloading the same 10 files with each tool on one «our HPC» compute node and computing time ratios.
- C4 KARAJ parallel framework:
parallel -j8 ascpvs singleascp(reported 3.6×). Reproduce by running KARAJ's exact ascp+parallel command at j=1 and j=8.
Out of scope / not attempted (with reason)
- C5 memory-check (277 GB > 7.6 GB free → halt): a control-flow example, not a benchmark. "277 GB" = size of the MacParland et al. dataset (PMC6197289); "7.6 GB" = the authors' machine's free disk — an environment-specific anecdote, not a derivable number. We verify the logic (does KARAJ compare size vs free space and halt?) by code inspection only; we do not download 277 GB.
- Text-mining / summary-report / metadata-table generation: described
qualitatively with no reported numeric result to compare against
(
no_expected_resultfor those features) — not attempted. - Figure 1 is an architecture schematic (no data).
Key caveats (honesty)
- The headline numbers are a single "in-house experiment" (paper's words), n≈1, on undisclosed hardware/network → inherently environment-specific.
- KARAJ caps Aspera at
-l 300m(≈37.5 MB/s). The «our HPC» compute link measured ≈43 MB/s for plain wget in our probe — i.e. faster than the Aspera cap. On such a link a capped Aspera can be slower than wget, the opposite of the paper. This is the central generalizability finding, not a tool defect. - IBM's Aspera Connect installer URL hard-coded in KARAJ's
Installer.sh(ibm-aspera-connect_4.1.0.46…) is dead (download failed on «our HPC») — a reproducibility gap in the shipped installer. We substitute biocondaaspera-cli(sameascp+ ENA public key) to run the equivalent command. - fastq-dump (C3) also converts SRA→FASTQ from NCBI, so it is not a pure download — part of why it is the slowest; noted when grading.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.