Combining evidence of preferential gene-tissue relationships from multiple sources.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
The deterministic Perl aggregation stage of the tissue-specificity pipeline (inner_score.pl + combine-data.pl) was reproduced bit-exactly (modulo benign tie-order artifacts) against the shipped combine-data/results.txt for all 4 constituent datasets, after diagnosing and fixing a uniq-file-format parsing bug thatinitially blocked 3 of 4 datasets. The ROKU-SPM and Decision-Function approaches were confirmed non-reproducible from the shipped repo content (undocumented required parameters, no driver script). The Bayesian Factor sampler stage (the paper's primary computational method for GSE7307) was found to have two independent, real problems: (1) as literally specified (nrruns=10000) it is computationally infeasible within any realistic SLURM ceiling, extrapolated from a real completed nrruns=50 benchmark (81m42s) to roughly 253 hours: not a guess, a measured extrapolation; and (2) even at reduced nrruns=50, the script crashes with a genuine NA-comparison error at the final classification step, a bug independently confirmed to NOT be caused by probe/rowname mismatch (0/42369 missing after a corrected diagnostic) and instead hypothesized, but not conclusively traced, to be a stochastic numerical edge case inside the BFN4 sampler's internal variance computation. GSE7307 was independently re-downloaded from GEO (677 samples confirmed, cross-validated against the repo's bundled 195-sample analysis-ready file), with the 677-to-195 reduction only partially explained by the paper's stated normal-tissue-only filter. The paper's headline aggregate numbers (3434 specific / 424 2-selective genes) were confirmed to span all 5 Bayesian datasets, not GSE7307 alone, so no paper-stated GSE7307-only figure exists to validate against even in principle. This RU does not claim completeness: the Bayesian stage's root Bayesian-Factor computation was not brought to a successful completed state for GSE7307, and this is recorded as a genuine partial/error outcome rather than a fabricated success.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-07-30
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-07-31no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan predictions of preferentially (tissue-specifically) expressed human genes be improved by combining, at the interpretative (high) level, the outputs of multiple methods applied to multiple independent expression datasets, thereby eliminating method- and study-specific biases? The authors test whether such a consensus/rule-based scoring approach yields more reliable tissue-specific gene predictions than any single data source or database.
- ★ A high-level integration approach combining three methods across four human microarray datasets, merged by consensus voting and a rule-based inner/total score, predicts preferentially expressed genes while reducing method- and study-specific bias. method
- ★ The approach identifies a resource of 3434 tissue-specific human genes. resource
- ★ Predictions are largely corroborated by existing public resources: 99% of predictions are supported by at least one of PaGenBase, TiGER or HPA. finding
- ★ The combined approach outperforms any individual public database (PaGenBase, TiGER, HPA) at identifying drug targets and biomarkers with known tissue specificity. finding
- ★ Consensus voting across methods yields better predictions than the individual methods alone. finding
- ROKU was modified into ROKU-SPM by incorporating the SPM specificity measure (two SPM values plus entropy as optimization parameters) to classify genes as specific, 2-selective or ubiquitous. method
- The Bayes factor method was extended to test 2-selectivity and modified so that the maximal tissue mean must exceed the others by a constant c times the between-tissue variability, because replicate variance was very small relative to between-tissue variance and otherwise supported specificity for nearly all genes. method
- Parameter settings are robust: parameters trained on any one of five tissue-specific gene sets gave similar accuracies on the other four, so the mixed-tissue parameter set was adopted as the most general. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Microarray expression profiling (GNF1H, Human U133A/GNF1H Gene Atlas; MAS5 + log2) | Normal human tissues (84 tissue terms, mapped to 34) | none | Probe-set expression intensities across tissues; classification as specific / 2-selective / ubiquitous | Affymetrix HG-U133a (22283 probe sets) |
| Microarray expression profiling (NCBI GEO GDS3113; Limma normalization) | Normal human tissues (32 tissue terms, mapped to 27) | none | Probe-set expression intensities across tissues; specificity classification | ABI Human Genome Survey Microarray (44928 probe sets) |
| Microarray expression profiling (GeAZr, licensed from GeneLogic; MAS5 + log2) | Normal human tissues (100 tissue terms, mapped to 55) | none | Probe-set expression intensities across tissues; specificity classification | Affymetrix HG-U133a,b (32878 probe sets) |
| Microarray expression profiling (NCBI GEO GSE7307; MAS5 + log2) | Normal human tissues (105 tissue terms, mapped to 42) | none | Probe-set expression intensities across tissues; specificity classification | Affymetrix HG-U133plus2 (53998 probe sets) |
| Computational specificity calling with ROKU-SPM (Shannon entropy + SPM, outlier detection) | All four human microarray datasets | none | Predicted tissue(s) per probe set; specific / 2-selective / ubiquitous call | — |
| Computational specificity calling with the decision function (gap g, specificity sp, decision value d) | All four human microarray datasets | none | Whether a gene is preferentially expressed and in which tissue(s) | — |
| Bayes factor hypothesis testing (modified for 2-selectivity and effect magnitude) | Human microarray datasets with multiple samples per tissue only | none | Bayes factor evidence for tissue specificity / 2-selectivity | — |
| Benchmarking/cross-database comparison against public resources | Human genes; PaGenBase (microarray), TiGER (EST), HPA (protein/antibody staining); training and test gene sets from HuGEindex.org (5 disjoint sets, 7 tissues, plus 10 house-keeping genes each) | none | Overlap of predicted tissue-specific genes; fraction of correctly predicted training/test genes; recovery of tissue-specific drug targets and biomarkers | PaGenBase SPM default cutoff; TiGER enrichment score >5 and P-value threshold; HPA 'Strong' staining or 'High' annotated protein expression |
- – In total 3434 tissue-specific genes were identified by the combined approach. 3434 genes
- – Genes with the highest scores overlap 85% with PaGenBase, 71% with TiGER and only 28% with HPA. 85% / 71% / 28%
- – 99% of the predictions are supported by at least one of PaGenBase, TiGER or HPA. 99%
- ▲ The approach performs better than any of the compared databases at identifying drug targets and biomarkers with known tissue specificity.
- – Training on the Liver gene set gave 95% correctly predicted genes on the liver set and 95% (Mixed), 95% (Kidney), 95% (Muscle), 94% (Lung or prostate) when applied to the other test sets. 94-95%
- – Parameters trained on the mixed-tissue set gave 91% correct predictions on the mixed-tissue set. 91%
- ▲ The consensus vote produced better predictions than those obtained from the individual methods.
- – Only minor differences were observed between the five parameter sets in the all-against-all training/testing procedure, indicating robust parameter settings.
- count 3434 (Total tissue-specific genes identified)
- other 85% (Overlap of top-scoring predictions with PaGenBase (microarray))
- other 71% (Overlap of top-scoring predictions with TiGER (EST))
- other 28% (Overlap of top-scoring predictions with HPA (protein expression))
- other 99% (Predictions supported by at least one of PaGenBase, TiGER or HPA)
- other 95%, 95%, 95%, 95%, 94% (Correct predictions using liver-trained parameters on Liver, Mixed, Kidney, Muscle, Lung-or-prostate test sets)
- other 91% (Correct predictions with mixed-tissue-trained parameters on the mixed-tissue set (Table 4))
- count 22283 / 44928 / 32878 / 53998 probe sets; 84 / 32 / 100 / 105 tissues before mapping and 34 / 27 / 55 / 42 after mapping (Dataset sizes for GNF1H, GDS3113, GeAZr and GSE7307 (Table 1))
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper uses a high-level integrative approach that applies three separate methods (ROKU-SPM, based on Shannon entropy and a normalized specificity measure; a decision function based on intensity gaps; and a Bayes factor hypothesis-testing approach) to four independently processed human microarray datasets to classify genes as tissue-specific, 2-selective, or ubiquitously expressed. Method parameters were tuned by minimizing an optimization score against five curated training/test gene sets from HuGEindex.org, using an all-against-all training/testing scheme across the five gene sets rather than a single held-out validation split. Predictions from the individual methods were merged via a rule-based consensus vote and then combined across probe sets and datasets using an inner-score/total-score averaging scheme, and the resulting gene list was compared to three public databases by reporting raw percentage overlap rather than a formal statistical test of concordance.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Bayes factor (modified hypothesis test comparing tissue-specific vs. ubiquitous/2-selective null hypotheses, with thresholds set by simulation) | per-gene classification of tissue specificity/2-selectivity in datasets with multiple samples per tissue | — | not stated |
| ROKU-SPM (Shannon entropy-based specificity/outlier detection combined with a normalized specificity measure, SPM) | per-gene classification of specific vs. ubiquitous expression across tissues in all four microarray datasets | — | not stated |
| Decision function (threshold rule using intensity gap, specificity value, and decision value parameters) | per-gene classification of specific vs. ubiquitous expression across tissues in all four microarray datasets | — | na |
-
Bayes factor thresholds and the constant c were determined via simulation, and gene-tissue calls were made across a large number of genes without an explicit multiple-testing correction step being described.↳ Could also: A false discovery rate procedure (e.g., Benjamini-Hochberg) applied across the full set of gene-by-tissue tests — This would give an explicit, corrected significance threshold when screening thousands of genes simultaneously, which is a standard control when performing many parallel comparisons.
-
Evidence from three heterogeneous methods (ROKU-SPM, Decision function, Bayes factor) was integrated using a rule-based consensus vote table.↳ Could also: A formal statistical meta-analysis (e.g., combining per-method/per-dataset p-values with Fisher's method, or a random-effects model) — This would let heterogeneity between methods and datasets be modeled explicitly and would yield a single combined significance estimate rather than a categorical voting rule.
-
Prediction accuracy was reported as a single percentage for each training/testing combination in the all-against-all cross-validation scheme (Table 4).↳ Could also: Reporting the mean and dispersion (e.g., SD or a confidence interval) of accuracy across the five training folds — This would convey how stable the parameter estimates are across folds, in addition to the point percentages already shown.
-
Overlap between the predicted gene list and three public databases (PaGenBase, TiGER, HPA) was reported as raw percentage agreement.↳ Could also: A concordance statistic such as Cohen's kappa, or a hypergeometric/enrichment test for the observed overlap — This would quantify agreement beyond what would be expected by chance and provide a formal measure of association alongside the raw overlap percentages.
-
The Bayes factor approach's underlying variability was captured by the sample standard deviation of replicate measurements within each tissue.↳ Could also: A mixed-effects or hierarchical model treating tissue and dataset as random effects — This could jointly model within-tissue replicate variability and between-dataset variability in a single framework, which may be useful given that variability differs between individual replicates and between tissues.
-
Training and test gene sets were selected from HuGEindex.org and manually curated/visually screened for a detectable expression pattern before being used for parameter optimization.↳ Could also: A fully blinded or pre-registered gene selection procedure not informed by visual inspection of expression patterns — This could reduce the chance that gene selection is implicitly tuned to patterns the methods are designed to detect, complementing the existing manual curation step.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
One of four arms reproduces bit-exactly: after fixing a genuine parsing bug in the shipped inner_score.pl (it reads the serial-number column of 5 of 6 *_uniq.txt files), the Perl aggregation pipeline reproduces combine-data/results.txt with only benign Perl-hash-order column swaps — though the reproducing invocation (inner_bg + inner_gse7307 + inner_gz + inner_log3113) had to be found by exhaustive search and matches neither README example. The other three arms are blocked on the authors' side: ROKU-SPM and Decision-Function require entropy/SPM/THRESHOLD parameters documented nowhere, and the Bayesian sampler is both computationally infeasible as specified (~253 h extrapolated from a real nrruns=50 benchmark) and crashes with missing value where TRUE/FALSE needed at nrruns=50 after all BF calls complete. Compounding this, the headline numbers (3434 specific / 424 2-selective genes; testis ~41% vs ~46% — the paper's own internal inconsistency) are 5-dataset aggregates with no GSE7307-only counterpart, and the shipped 195-sample analysis matrix cannot be rebuilt from the public 677-sample GSE7307 series. Severity is incompleteness, not contradiction: nothing reproduced disagrees numerically with the paper, so this is a solid-but-limited reproduction with real reproducibility defects, not a fabrication signal.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.