Intrinsic suppression of type I interferon production underlies the therapeutic efficacy of IL-15-producing natural killer cells in B-cell acute lymphoblastic l
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Reported values are derivable from the shared data
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL 1:1 reproduction of the paper's PUBLIC-DATA computational claim. Sudan et al. (JITC 2023) is mainly a wet-lab + mass-cytometry (CyTOF) immunology paper; its primary pipeline (CyTOF immune profiling, Figs 1-2) is normalized with the brief's cited repo github.com/nolanlab/bead-normalization and gated in Cytobank, but the raw FCS data is 'available on reasonable request' (not deposited) -> NOT reproducible. The Fig 1A survival analysis needs COG P9906 outcome/RFS data which GEO states cannot be provided (restricted). EGAS00001003266 in Fig 4B is EGA controlled-access. The reproducible public-data result is Fig 4B/4C: IL-15 transcript inversely correlates with MYC in B-ALL. Reproduced on «our HPC»/«infra» from the cited PUBLIC GEO series matrices using canonical HG-U133 probes (MYC 202431_s_at[+244089_at], IL15 205992_s_at): (C1a) GSE11877 (COG P9906, n=207) Pearson r=-0.175 p=0.012 / Spearman -0.184 p=0.008 -> significant NEGATIVE, matches paper. (C1b) GSE13159 (MILE, n=576 B-ALL) r=-0.111 p=0.0075 / Spearman -0.129 p=0.0019 -> significant NEGATIVE. (C2) subtype contrast: MYC-driven+MLL group n=83 (EXACTLY the paper's stated 'MYC driven+MLL n=83') has lower IL-15 (median 0.195) than ETV6::RUNX1+Ph+ n=180 (0.336), MWU p=7.5e-15 -> aggregate claim reproduced; HONESTY NUANCE: at single-subtype resolution the t(8;14) MYC-translocation class alone (n=13) has the HIGHEST IL-15 (0.42), so the low signal is carried by KMT2A/MLL t(11q23) + hyperdiploid, and MILE lacks BCL2-tx/hypodiploid classes -> graded partial. (C3) GSE132929 Burkitt's IL-15 lower than non-Burkitt's, MWU p=7.5e-22. NOT attempted, and why: CyTOF normalization/gating (the repo's real use) raw FCS restricted; Fig 1A survival COG outcome data restricted; EGA arm controlled-access; all mouse/flow/qPCR/CRISPRa NK-92 wet-lab (non-pipeline). Fabrication check: no discrepancy -- every reproduced Fig 4B/4C value is derivable from the cited public GEO data and holds in the stated direction with significance; the non-reproduced parts are out of scope due to restricted data, not due to any detected inconsistency. All grades provisional; human audit sheet in AUDIT.md.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 76assessed: 2026-06-15 ⛓ 49d8651ad5ee
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator headless) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe authors hypothesized that the poor efficacy of type I interferons (IFN-Is) as single-agent leukemia therapy stems from an intrinsic block in IFN-I production and/or response during primary B-ALL leukemogenesis, which suppresses anti-leukemia immune subsets (especially NK cells) and could be countered by IL-15-producing NK-cell therapy.
- ★ High expression of IFN-I signaling/response genes (IFNAR1, IFNAR2, STAT1, MX1, OAS1) predicts favorable relapse-free survival in B-ALL patients finding
- ★ Human and mouse B-ALL microenvironments exhibit an intrinsic defect in autocrine (B-cell) and paracrine (pDC) IFN-I production and IFN-I-driven immune responses finding
- ★ Reduced IFN-I production is sufficient to suppress the immune system and promote leukemia development in MYC-driven B-ALL-prone mice finding
- ★ Suppression of IFN-I production most markedly lowers IL-15 transcription, reducing NK-cell number and effector maturation in B-ALL microenvironments mechanism
- ★ Adoptive transfer of healthy NK cells significantly prolongs survival of overt B-ALL-bearing transgenic mice finding
- In vivo IFN-I administration reduces leukemia progression and increases circulating NK-cell/effector frequencies in B-ALL-prone mice finding
- ★ IL-15 suppression is most severe in MYC-overexpressing B-ALL, and MYC overexpression sensitizes B-ALL cells to NK-mediated killing finding
- ★ CRISPRa-engineered IL-15-secreting human NK-92 cells kill high-grade B-ALL in vitro and block leukemia progression in vivo more effectively than IL-15-non-producing NK cells resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| gene-expression-based survival analysis | COG P9906 high-risk B-ALL patient cohort (n=207 total, n=15/n=14 subgroups) | none (stratified by IFN-I gene expression) | relapse-free survival probability, WBC count, IRF7/CD123 transcript levels | — |
| flow cytometry (intracellular IFNα2b staining) | human B-ALL patient BMMCs and PBMCs vs age-matched healthy donors | class C CpG ODN stimulation | frequency of IFNα2b+ cells | BD FACSymphony |
| mass cytometry (CyTOF) | human/mouse B-ALL immune cell subsets | PMA/Ionomycin stimulation | surface protein and intracellular cytokine expression across immune subsets | Standard BioTools CyTOF, analyzed in Cytobank |
| transgenic mouse model characterization | Eμ-Myc and Eμ-Myc/IFNAR1−/− mice | germline IFNAR1 knockout | leukemia development/progression, immune cell frequencies | — |
| adoptive cell transfer | Eμ-Myc B-ALL-bearing mice | intravenous injection of syngeneic UBC-GFP NK cells vs PBS | leukemia-free survival | — |
| in vivo cytokine administration | Eμ-Myc B-ALL-prone mice | intraperitoneal IFNβ (50,000 IU) vs PBS for 9 days | leukemia progression, circulating NK/NK-effector frequencies | flow cytometry |
| NK-cell cytotoxicity and proliferation assay | CRISPRa control-sgRNA vs IL-15-sgRNA NK-92 cells against B-ALL targets | CRISPRa-mediated IL-15 activation | specific cytotoxicity (%7-AAD+), NK proliferation (cell counts) | BD FACSymphony flow cytometer, FlowJo v10.7.1 |
| bioluminescence imaging xenograft | luciferase-labeled P493-6 B-ALL cells engrafted in NSG mice, treated with CRISPRa control or IL-15 NK-92 cells | IL-15-producing vs control NK-92 cell therapy | leukemia burden via bioluminescence signal | Lago-X imaging system |
- ▲ B-ALL patients with high IFN-I response gene expression (n=15) had significantly longer relapse-free survival than low IFN-I response patients (n=14)
- ▼ High IFN-I response patients were less likely to have WBC counts ≥100,000/µL (13%) versus low IFN-I response patients (43%) 13% vs 43%
- ▼ Low IFN-I response patients showed reduced IRF7 and CD123 expression compared with high IFN-I response patients
- ▼ IFNα2b+ cell frequency after CpG stimulation was reduced in B-ALL patient BMMCs and PBMCs versus healthy donors
- ▼ Peripheral blood pDC frequencies were reduced in B-ALL patients versus healthy donors
- ▲ Adoptive transfer of healthy NK cells significantly prolonged survival of B-ALL-bearing Eμ-Myc mice versus PBS control
- ▼ IL-15 suppression was most severe in B-ALL patients with high MYC expression
- – CRISPRa IL-15-secreting NK-92 cells killed B-ALL cells in vitro and blocked leukemia progression in vivo more effectively than IL-15-non-producing control NK cells
- count n=15 (High IFN-I Response) vs n=14 (Low IFN-I Response) (COG P9906 RFS survival comparison groups)
- count 13% vs 43% with WBC ≥100,000/µL (WBC counts in High vs Low IFN-I response patients)
- count n=7 BMMC, n=10 PBMC (B-ALL) vs n=7 BMMC, n=10 PBMC (healthy) (IFNα2b+ cell frequency comparison, CpG-stimulated BMMC/PBMC)
- count n=10 (B-ALL) vs n=10 (healthy) (peripheral blood pDC frequency comparison)
- count n=7 (B-ALL) vs n=10 (healthy) (CXCR4 MFI on peripheral blood pDCs)
- count n=10 (B-ALL) vs n=10 (healthy) (HLA-DR MFI on peripheral blood cDCs)
- count n=13 (high) vs n=9 (low) (patients with concomitant high/low expression of IFN-I response genes, IRF7, and CD123)
- other 7×10^5 NK cells administered intravenously; effector:target ratio 10:1 (dosing for adoptive NK-cell transfer and in vitro cytotoxicity assay)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper employs high-dimensional flow and mass cytometry, murine survival analyses, qPCR, and in vitro/in vivo NK-cell efficacy assays to characterize IFN-I suppression in B-ALL and evaluate IL-15-producing CRISPRa-engineered NK cells. Pairwise group comparisons between mouse cohorts used two-tailed Mann-Whitney U tests; Kaplan-Meier curves were compared with the log-rank test; and exact p values were reported for significant (p<0.05) and trending (0.05<p<0.1) results. Sample sizes were determined a priori with the 'cpower' function in R, and qPCR biological replicates were each run in three technical replicates.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| two-tailed Mann-Whitney U | pairwise comparisons between mouse cohorts (explicitly stated in Methods); statistical test used for human patient vs healthy donor flow cytometry comparisons is not named in the provided methods text | mouse cohort sizes not stated in methods text; patient and healthy donor flow cytometry groups n=7–10 per group as stated per figure | not stated |
| log-rank | Kaplan-Meier relapse-free survival comparison (COG P9906 IFN-I-high n=15 vs IFN-I-low n=14); also used for leukemia-free survival in NK-cell adoptive transfer mouse experiment | n=15 vs n=14 (COG P9906 RFS); mouse cohort sizes for survival not specified in provided text | not stated |
-
IFN-I response genes were dichotomized at their median for Kaplan-Meier survival analysis, producing two groups of n=15 and n=14 from 207 total patients↳ Could also: Cox proportional hazards regression using IFN-I gene expression as a continuous covariate, optionally adjusted for known prognostic variables such as WBC count, age, and cytogenetic subtype — Continuous modeling preserves all variation in gene expression rather than compressing it into two levels, and produces a hazard ratio with a confidence interval that facilitates comparison with other studies; covariate adjustment can clarify whether the survival association is independent of established risk factors
-
Multiple pairwise comparisons were made across many figures and endpoints without a stated multiplicity correction↳ Could also: A false discovery rate correction (e.g., Benjamini-Hochberg) or family-wise error rate correction (e.g., Bonferroni) applied across related comparisons within each figure or experimental unit — When many tests are performed simultaneously, controlling the expected proportion of false positives helps distinguish reproducible signals from chance findings; making the correction approach explicit adds interpretive transparency
-
Survival between two patient subgroups was compared with the log-rank test alone↳ Could also: A multivariable Cox proportional hazards model including IFN-I response status and established clinical prognostic covariates — Adjusted Cox regression quantifies the independent prognostic contribution of IFN-I response status and yields a hazard ratio with a confidence interval, enabling effect-size estimation and cross-study comparison beyond a binary significant/not-significant conclusion
-
Pairwise group comparisons used the Mann-Whitney U test (stated explicitly for mouse cohorts); where three or more groups are compared the analysis framework is not described↳ Could also: A Kruskal-Wallis omnibus test followed by Dunn's post-hoc pairwise test (or one-way ANOVA with Tukey HSD if normality is met) when more than two groups are compared simultaneously — Testing all groups jointly before pairwise comparisons controls the experiment-wise error rate within that family; Dunn's post-hoc procedure accounts for the number of pairwise contrasts naturally
-
Dispersion around central tendency values is not specified in the statistical methods section↳ Could also: Explicit reporting of SD (for approximately normal data), IQR (paired with nonparametric tests), or 95% confidence intervals alongside group medians or means — Dispersion measures are particularly informative at small n (n=7–15 per group throughout), allowing readers to assess variability and effect magnitude independently of p values; SD and IQR also signal whether parametric or nonparametric methods are more appropriate
-
Effect sizes are not reported alongside p values for any comparison↳ Could also: Rank-biserial correlation (for Mann-Whitney U), Cohen's d (for t-tests), or hazard ratios with 95% CIs (for survival analyses) — Effect size estimates convey biological or clinical magnitude of differences separately from sample-size-dependent statistical significance, and are increasingly expected by journals and meta-analyses for quantitative synthesis
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37217248
Paper. Sudan, Kandarpa, et al. Intrinsic suppression of type I interferon production underlies the therapeutic efficacy of IL-15-producing natural killer cells in B-cell acute lymphoblastic leukemia. J Immunother Cancer 2023;11:e006649. PMID 37217248 · PMCID PMC10231005 · DOI 10.1136/jitc-2022-006649. Open access.
Brief metadata sanity check (resolved). The room brief paired this paper with
github.com/nolanlab/bead-normalization (a MATLAB CyTOF bead-normalization tool)
and GEO:GSE11877 (a 2009 COG pediatric B-ALL microarray set whose GEO record
lists other PMIDs). Both are genuinely cited by the paper, not a mismining:
- The repo is used as a black-box: "Data were normalized using MATLAB
(https://github.com/nolanlab/bead-normalization/releases)" — i.e. mass-cytometry
(CyTOF) bead normalization. The raw FCS data are "available on reasonable
request" (not deposited) → CyTOF normalization is not reproducible from
public data →
data_restricted, out of scope. - GSE11877 (COG P9906) and GSE13159 (MILE) are used in Figure 4B for the gene-expression analysis (MYC vs IL-15 transcript correlation across B-ALL subtypes). Their expression matrices are public → in scope.
Pipeline-derived results and their data sources
| Result (paper) | Pipeline | Data | Public? | In scope |
|---|---|---|---|---|
| Fig 4B IL-15 transcript correlates negatively with MYC across B-ALL; IL-15 lower in MYC/BCL2-tx, KMT2A-r, hypodiploid vs Ph/Ph-like/ETV6::RUNX1 | microarray expr → per-gene probe extraction → linear regression / group comparison | GSE11877, GSE13159, EGAS00001003266 | GSE11877 ✅, GSE13159 ✅, EGA ❌ controlled | YES (on the 2 public GEO sets) |
| Fig 4C Burkitt's lymphoma (MYC-driven) expresses lower IL-15 than non-Burkitt's B-NHL | microarray expr → group comparison | GSE132929 | ✅ | YES (secondary) |
| Fig 1A COG P9906 RFS by IFN-I-signature (IFNAR1/2,STAT1,OAS1,MX1) high vs low (Kaplan-Meier, log-rank) | expr stratification → survival | GSE11877 expr + COG outcome/RFS data | expr ✅ but RFS/outcome COG-restricted ("cannot be provided… contact COG") | NO — data_restricted (outcome not public) |
| CyTOF immune profiling (Figs 1,2; pDC/cDC/NK frequencies) | CyTOF: bead-normalize (nolanlab repo) → Cytobank gating | raw FCS | ❌ "on reasonable request" | NO — data_restricted |
| Mouse Eµ-Myc experiments, qPCR, flow, CRISPRa NK-92, in-vivo survival (Figs 2,3,5,6) | wet-lab | — | — | NO — non_pipeline (wet-lab) |
What we attempt (80/20)
Primary (C1): Reproduce the Fig 4B headline — IL-15 (IL15) transcript
expression is negatively correlated with MYC transcript expression in B-ALL —
directly from the public series matrices of GSE11877 and GSE13159, using
the canonical HG-U133 probes (MYC 202431_s_at, IL15 205992_s_at). Pearson +
Spearman; expect r<0, p<0.05.
Secondary (C2): Fig 4B subtype contrast — IL-15 lower in MYC-driven/KMT2A-r subtypes vs ETV6::RUNX1/Ph — using GSE13159 (MILE) leukemia-class sample labels.
Tertiary (C3): Fig 4C — Burkitt's lymphoma lower IL-15 than non-Burkitt's B-NHL — GSE132929 (only if cheap).
What we do NOT attempt, and why
- CyTOF normalization/gating (the
bead-normalizationrepo's actual use): raw FCS not public →data_restricted. - Fig 1A survival on COG P9906: RFS/outcome data COG-restricted →
data_restricted. - EGAS00001003266 arm of Fig 4B: EGA controlled-access →
data_restricted. - All mouse / flow / qPCR / CRISPRa NK-cell wet-lab: not a pipeline.
The paper's core mechanistic computational claim that is checkable on public data is the MYC↔IL-15 inverse relationship (Fig 4B). That is our reproduction target.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.