A Bayesian approach to accurate and robust signature detection on LINCS L1000 data.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the ALGORITHM but NOT the numbers, due to a dead data host. PARTIAL. The authors' core Bayesian peak-deconvolution C++ code (src/dpeak.cpp) builds with g++ 12.2 on «our HPC» and runs deterministically -> zscore_hi = -1.57776 (identical on repeat; the example uses an unseeded RNG), reproducing the central method 1:1 at the code level (the repo gives no ground-truth number for this mock example, so graded partial not exact). The PRIMARY numeric target - the published Bayesian GSE70138 signature matrices (Level5 116218x978, Level4 335465x978) - could NOT be verified: their host callisto.astro.columbia.edu («ip») is offline (TCP 80 and 443 time out from a «our HPC» compute node and from «host», while NCBI/GitHub work from the same node, so it is the host not our network); the files are not mirrored on Zenodo/figshare and are too large for the Wayback Machine. Only the published sha512 checksum manifest was recoverable (Wayback 2024-04-26), confirming the file roster and filename-encoded dimensions but not the matrices themselves -> data_unavailable for the reference artifacts. NOT ATTEMPTED: the full Wolfram pipeline (needs Wolfram Engine, which requires a licensed account not present; and per the repo README the bundled partial-plate sample would not match the database anyway, so no 1:1 was possible) - intentionally not chased per 80/20. NOT ATTEMPTED: the paper's downstream Bayesian-vs-MODZ signature-detection benchmarks (that analysis code is not in the repo; out of scope). No fabrication signal: the code is real, open, and runs; the gap is a dead third-party hosting server, an availability failure not an integrity one.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 37assessed: 2026-06-15 ⛓ 189e7495bc1b
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a Bayesian peak deconvolution algorithm with probabilistic z-score inference recover gene expression levels from LINCS L1000 data more accurately and robustly than existing methods (k-means, GMM, AGMM), thereby producing less noisy perturbagen signatures?
- ★ A novel Bayesian-based peak deconvolution algorithm gives unbiased likelihood estimations for peak locations and characterizes peaks with probability-based z-scores. method
- ★ A full pipeline processes raw L1000 FI data into perturbagen signatures via LISS/QC, Bayesian peak deconvolution, quantile normalization, probabilistic z-score inference, and replicate combination. method
- ★ Signatures from the Bayesian pipeline show higher similarity between bio-replicates and between drugs with shared targets than existing methods, indicating more reliable and less noisy representations. finding
- ★ On simulated data the Bayesian method achieves lower MSE and higher correlation between true and recovered peak positions than k-means and AGMM. finding
- L1000 single-peak shapes are well modeled by a Student's t-distribution with three degrees of freedom across all expression levels. mechanism
- Code and precomputed data for LINCS L1000 Phase II (GSE70138) are publicly released. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| L1000 gene expression profiling (Luminex bead peak deconvolution) | LINCS L1000 Phase II small-molecule compound data (GSE70138), multiple human cell lines | small molecule compounds | deconvoluted peak positions / gene expression z-scores for 978 landmark genes | Luminex FlexMap 3D |
| Simulated L1000 dataset peak deconvolution benchmark | 10 000 simulated samples modeled on L1000 characteristics | none (simulation) | MSE and Pearson correlation between true and predicted peak positions across Bayesian MLE, k-means and AGMM | — |
| Student's t-distribution peak-shape fitting (reference shape construction) | LINCS L1000 level 2 data | none | best-fit degrees of freedom and scale parameter vs log2 expression | — |
| LISS calibration and quality-control fitting | L1000 wells/plates with 80 control invariant transcripts | none | goodness of fit (chi^2) and slope a of scaling relation | — |
- ▲ Bayesian MLE shows smaller MSE and higher Pearson correlation than k-means and AGMM in recovering simulated peak positions
- ▲ Bayesian pipeline signatures give higher similarity between bio-replicates and between drugs with shared targets than existing methods
- – Student's t-distribution with three degrees of freedom fits peak shapes appropriately across all expression levels DOF=3
- – Empirical scale-parameter relation sigma_x = 0.15 + 5.3*e^(-0.52x) fits with low error mean absolute error 0.009
- ▲ Bayesian method performed best across additional simulated datasets (Gaussian and other t-distribution parameters)
- other sigma_x = 0.15 + 5.3 e^(-0.52x), mean absolute error 0.009 (fitted scale parameter vs log2 expression)
- count DOF = 3 (degrees of freedom of Student's t-distribution for peak shape)
- count 978 landmark genes (genes measured per L1000 sample)
- count 10 000 simulated samples (simulated benchmark dataset size)
- other mixing ratio 2:1 (two gene barcodes coupled to same bead color)
- count 49.5 (50 x 99%) reads, noise mean 0.5 (Poisson means for simulated reads and noise)
- count remove additional 3.0% with a>3.0 or chi^2>4.0; 4.0% already missing (quality control thresholds and sample removal)
- other color misidentification rate alpha_c ~1% (background read rate in Bayesian model)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational methods paper presenting a Bayesian peak deconvolution algorithm for LINCS L1000 gene expression data, modeling fluorescent intensity reads with Student's t-distributions (DOF=3) and Bayesian posteriors to estimate gene expression peak locations. Performance was evaluated on 10,000 simulated samples using MSE and Pearson correlation against ground-truth peak positions, and on real LINCS L1000 Phase II (GSE70138) data using inter-replicate and inter-drug-target signature similarity. Results were reported primarily as Pearson correlation coefficients and MSE values, with Spearman rank correlation used internally for replicate weighting; no formal hypothesis tests with p-values were reported.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Mean Squared Error (MSE) | Comparison of Bayesian MLE, k-means, and AGMM peak deconvolution accuracy on simulated data (Figure 5) | 10,000 simulated samples | na |
| Pearson correlation coefficient | Comparison of true versus predicted peak positions across three deconvolution methods on simulated data (Figure 5) | 10,000 simulated samples | not stated |
| Chi-squared goodness-of-fit (χ²) | Quality control: samples with χ² > 4.0 from LISS linear scaling fit were excluded | null | not stated |
| Student's t-distribution fitting (MLE for shape and scale parameters, DOF selection by median across expression levels) | Characterizing peak shapes at 11 discrete log2 expression levels to derive empirical σ(x) relation (Figures 3–4, Equations 1–2) | null | stated |
| Spearman rank correlation | Computation of replicate-combination weights when aggregating bio-replicate z-score profiles (Equation 14) | null | na |
| Bayesian posterior / Maximum Likelihood Estimation (MLE) | Core peak deconvolution algorithm: posterior distribution over peak locations given observed fluorescence intensity reads (Equations 3–5) | ~50 reads per bead color per sample (Poisson mean of 49.5 stated for simulations) | stated |
-
Method comparison on simulated data used MSE and Pearson correlation between predicted and true peak positions↳ Could also: Lin's concordance correlation coefficient (CCC) or Spearman correlation could also quantify agreement between predicted and true values — CCC simultaneously captures both precision (correlation) and accuracy (mean bias), which can distinguish between a method that is well-correlated but systematically shifted versus one that is accurate but noisy; Spearman r would be robust to non-normal distributions of peak position residuals
-
Peak shape was modeled with a Student's t-distribution at a fixed degree of freedom (DOF=3), selected based on the median fitted DOF across all expression levels (Figure 3)↳ Could also: A model allowing DOF to vary as a function of expression level, or a non-parametric kernel density approach, could also represent peak shapes — Per-level DOF estimation would allow the model to capture any systematic variation in tail heaviness across expression levels rather than assuming DOF=3 is globally appropriate
-
Bio-replicates were combined via a Spearman-correlation-weighted average (Equations 13–14)↳ Could also: An unweighted average or a Bayesian hierarchical model aggregating replicate z-scores could also be used — An unweighted average avoids the potential circularity of down-weighting replicates that are genuinely discordant due to biology rather than noise; a hierarchical model would explicitly propagate replicate-level uncertainty into the combined estimate
-
A uniform (non-informative) prior over peak locations was adopted in the Bayesian deconvolution model (Equations 4–5)↳ Could also: An empirical or weakly informative prior derived from the distribution of observed peak locations across all samples could also be used — An empirical prior would regularize estimates for samples with few reads or heavily overlapping peaks, potentially improving deconvolution stability in edge cases without requiring strong prior assumptions
-
Quality control applied fixed thresholds (χ² > 4.0 or slope a > 3.0) to exclude low-quality samples↳ Could also: Data-adaptive thresholds based on the empirical percentile distribution of χ² and slope values, or a mixture-model-based outlier detection step, could also identify poor-quality samples — Data-adaptive thresholds do not require pre-specification of cutoffs and can accommodate shifts in the QC metric distributions across different experimental batches
-
Uncertainty in the fitted DOF was summarized as a 68% confidence interval (Figure 3), equivalent to ±1 SD for a normal distribution↳ Could also: 95% confidence or credible intervals could also be reported for key fitted parameters — 95% intervals are the more widely adopted convention in biomedical reporting and facilitate direct comparison with benchmarks and parameter estimates from other published methods
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Bayesian-derived L1000 signatures show higher similarity between biological replicates and between compounds sharing targets compared to existing methods.other human cell line up 2020×1papers★ This paper is the founder (earliest)
-
Student's t-distribution with three degrees of freedom appropriately fits L1000 peak shapes across all expression levels.other l1000 2020×1papers★ This paper is the founder (earliest)
-
Empirical L1000 scale parameter follows sigma_x = 0.15 + 5.3*exp(-0.52x) with mean absolute error of 0.009.other l1000 2020×1papers★ This paper is the founder (earliest)
-
Bayesian MLE achieves lower MSE and higher Pearson correlation than k-means and AGMM when recovering peak positions in simulated L1000 data.other simulated l1000 up 2020×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-32003771
Paper: Qiu Y, Lu T, Lim H, Xie L. A Bayesian approach to accurate and robust
signature detection on LINCS L1000 data. Bioinformatics 36(9):2787, 2020.
PMCID PMC7203754 · DOI 10.1093/bioinformatics/btaa064.
Code: https://github.com/njpipeorgan/L1000-bayesian (pinned 617504ed12a9bf5738e3e2707b2251b3315da644)
Data: GEO GSE70138 (L1000 Phase II); precomputed Bayesian outputs at
callisto.astro.columbia.edu/files/L1000/.
What the repo ships (pipeline-derived results)
The repo implements a Bayesian peak-deconvolution pipeline that turns raw L1000 fluorescent-intensity (LXB) data into z-score signatures. Components:
src/dpeak.{cpp,cu,h}— core C++/CUDA deconvolution engine (dpeak_single).example/C++/example_dpeak.cpp— standalone deterministic demo of the core algorithm on built-in mock reads (unseededdefault_random_engine⇒ fixed input ⇒ reproducible z-score). No Wolfram needed.pipeline/{pipeline.wl,functions.wl}+sample_lxb/(REP.A028_MCF7, 24 LXB) — full end-to-end pipeline (LISS/QC → deconvolution → quantile norm → z-score → combine replicates). Requires Wolfram Mathematica / Wolfram Engine.- Precomputed published outputs (HDF5): Level4 ZSPC
n335465x978, Level5 COMPZn116218x978for GSE70138 (+ GSE92742), with a sha512 checksum file.
In scope (attempted)
- C1 — core deconvolution algorithm (their C++ code). Build
example_dpeakwith g++ on «our HPC» and run it; the mock input is deterministic, so the printedzscore_hiis a reproducible output of the published algorithm. Pipeline:g+++src/dpeak.cpp. - C2 — published signature dataset (numeric 1:1). Download the authors' precomputed Bayesian GSE70138 Level4/Level5 HDF5 + sha512 checksums; verify integrity (sha512) and verify the reported matrix dimensions (Level5 = 116218×978 signatures, Level4 = 335465×978 z-scored profiles) and 978 landmark genes. Report a few concrete z-score values as auditable points.
Attempted-but-blocked / not chased (honest 80/20)
- C3 — full pipeline on sample_lxb via Wolfram. Requires Wolfram Engine,
which needs a (free) license activation tied to a Wolfram account; no such
credential is available in this environment →
env_unresolvablefor this leg. Also note the repo README explicitly states sample-data z-scores do not match the database (sample covers only part of a plate), so even if run, it would not yield a numeric 1:1 against C2 — it would only confirm the workflow executes. Therefore not chased (the "last 20%").
Out of scope (not a repo pipeline)
- The paper's downstream benchmarks (Bayesian-vs-MODZ signature-detection AUC / connectivity to known MoA): the benchmark/analysis code is not in the repo; only the z-score-generation pipeline is shipped. Wet-lab L1000 assay itself is out of scope.
Plan
One «our HPC» SLURM job (compute node = internet + «infra» scratch): clone repo →
build+run C++ core (C1) → download+sha512-verify+inspect reference HDF5 (C2) →
probe wolframscript availability (C3). Small results back to «host»; large HDF5
stay on «infra» with SHA recorded.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.