FEM: mining biological meaning from cell level in single-cell RNA sequencing data.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce the CORE algorithm 1:1. The FEM tool (qingyunpkdd/single_cell_fem @ f8dd0638) ships a sample input (PBMC 2700x32738) + Reactome gmt (1499 sets) + its precomputed FEM output. Re-running the UNEDITED shipped batch code on «our HPC» reproduced the shipped 1499x2700 FEM matrix EXACTLY: all 1,464,764 non-zero -log10(p) values agree to <1.5e-14 (Jaccard 1.0, Pearson 1.0). This is the paper's central 'functional expression matrix' computation and is fully deterministic. KEY DISCREPANCY (methods vs code, flagged for audit): the executed path uses the hypergeometric sf, min overlap >2, and -log10 of the RAW p-value with NO multiple-testing correction, despite the paper text claiming Fisher's exact test and 'adjusted p-value' (a Fisher implementation exists but is unused). NOT attempted (hard ~20%): the Table 2 (liver/GSE81252) and Table 3 (PBMC/GO) per-cell-type enrichment numbers, which need a GEO download plus a stochastic Seurat clustering + cell-type-assignment + aggregation pipeline; the shipped sample (PBMC x Reactome) matches neither table's dataset x gene-set combination, so neither is checkable from shipped data alone.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 75assessed: 2026-06-15 ⛓ c34103131692
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan functional gene set enrichment be performed at the single-cell level by using all expressed genes in each cell to convert a gene expression matrix into a functional expression matrix (FEM), thereby preserving information normally discarded by gene filtering and revealing the biological significance of individual cells?
- ★ The FEM algorithm converts each cell's gene expression matrix (GEM) into a functional expression matrix by applying Fisher's exact test enrichment per cell and per gene set, then encoding adjusted p-values as information content. method
- ★ FEM utilizes information from all expressed (non-zero) genes rather than only well-characterized filtered genes, capturing functional groups that conventional pipelines miss. method
- ★ FEM performs well for cell clustering and cell-type-specific function annotation across PBMC, human liver, and human pancreas datasets. finding
- ★ FEM can be integrated with GEM as a second omics modality in Seurat for combined downstream clustering and differential expression function analysis. method
- A gene-set–based functional representation is more robust to technical covariates (count depth, batch) than individual high-expression genes because the number of expressed genes per biological process follows a hypergeometric distribution. mechanism
- Algorithm runtime is reduced via 0/1 matrix multiplication (A×Bᵀ) and multi-core parallel processing to make per-cell, per-function Fisher tests tractable. method
- FEM cannot fully replace GEM for cell classification because it ignores gene expression magnitude, only presence/absence. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq functional enrichment (FEM via Fisher's exact test) + clustering | Human peripheral blood mononuclear cells (PBMCs), 2,700 cells | none | per-cell gene-set enrichment converted to information content; cell clusters and cell-type function annotation | 10X Genomics; SciPy (Fisher's exact test); Seurat |
| scRNA-seq functional enrichment (FEM) + clustering | Human pancreas, 2,126 cells, 10 cell types (Muraro et al., 2016) | none | FEM-based clustering and cell-type-specific function annotation | SciPy; Seurat |
| scRNA-seq functional enrichment (FEM) + clustering | Human liver, 777 cells, 7 cell types (Camp et al., 2017) | none | FEM-based clustering and cell-type-specific function annotation | SciPy; Seurat |
| Differential expression of genes/functions (DEG/DEF, GC-DEF, FC-DEF) | PBMC / liver / pancreas scRNA-seq clusters | none | differentially expressed genes and functions between clusters | Seurat (multi-modal integration) |
| Dimensionality reduction and UMAP visualization | scRNA-seq GEM/FEM matrices | none | PCA principal components, clusters, UMAP embedding | Seurat; PCA; UMAP (McInnes et al., 2018) |
- – FEM achieved good cell clustering and cell-type-specific function annotation in all three datasets (PBMC, liver, pancreas).
- – Per-cell, per-function Fisher's exact test enrichment yields adjusted p-values encoded as information content GS = -log(adj-p) to form the FEM.
- – GEM was used for clustering (retains expression magnitude) while FEM provided functional annotation, motivating fused GEM+FEM analysis.
- count 2,700 cells (PBMC dataset size)
- count 2,126 cells; 10 cell types (human pancreas dataset)
- count 777 cells; 7 cell types (human liver dataset)
- count 1,499 (C2 Reactome gene sets used)
- count 9,996 (C5 GO gene sets used)
- count 4,872 (C7 immunologic signature gene sets used)
- other FDR < 0.02; no more than 300 genes per gene set (ImmuneSigDB validation gene set criteria)
- other clustering resolution 0.4–1.2 (Seurat cluster function setting per data)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper introduces FEM, a computational algorithm that converts per-cell gene expression matrices into functional expression matrices by applying Fisher's exact test to every cell × gene-set pair; the resulting p-values are transformed to information content scores (−log(adjusted p)) to form the FEM. The FEM is then fed into Seurat's standard pipeline (PCA, graph-based clustering, UMAP visualization) for cell clustering and differential-expression-of-functions analysis. Performance is demonstrated on three public scRNA-seq datasets (PBMC n=2,700; human pancreas n=2,126; human liver n=777) primarily through visual inspection of UMAP embeddings and qualitative agreement with known cell-type annotations.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Fisher's exact test (hypergeometric) | Core FEM algorithm — applied to every cell × gene-set pair to score per-cell functional enrichment | N = total background genes in database; n = non-zero-expressed genes per cell; K = genes in focal gene set; k = overlap; exact numeric values are dataset- and cell-dependent and not reported as a single figure | not stated |
| Bonferroni correction | Multiple-comparison adjustment of Fisher's exact test p-values across gene sets per cell (explicitly described as optional) | Number of gene sets per functional database: 1,499 (Reactome), 9,996 (GO), 4,872 (immunologic signatures); actual application across datasets not confirmed | not stated |
| Seurat FindMarkers (underlying test not explicitly named; Seurat default is Wilcoxon rank-sum) | Differential expression of genes (DEG) and functions (DEF) between clusters for GC-DEG, GC-DEF, and FC-DEF analyses | — | not stated |
-
Clustering quality was evaluated through visual inspection of UMAP embeddings and qualitative agreement with known cell-type annotations↳ Could also: Quantitative clustering metrics such as Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), or average silhouette width could also be computed against ground-truth labels — Numeric metrics allow reproducible, bias-free comparison between FEM-based and GEM-based clustering across datasets and methods, and can be reported with uncertainty estimates (e.g., across random seeds), providing a complement to visual assessment
-
Bonferroni correction was used (optionally) to adjust Fisher's exact test p-values across gene sets per cell↳ Could also: Benjamini-Hochberg (BH) false discovery rate correction could also be applied at the same step — BH FDR controls the expected proportion of false discoveries rather than the family-wise error rate; because gene sets overlap substantially, tests are correlated and Bonferroni can be more conservative than necessary, while BH is the standard adjustment in genomic enrichment workflows under correlated tests
-
Fisher's exact test on binary expressed/not-expressed gene calls was chosen as the enrichment scoring method for each cell↳ Could also: Single-sample methods such as AUCell (area under the recovery curve on ranked expression) or ssGSEA (rank-based enrichment score) could also be applied — The paper itself discusses these as existing alternatives; rank-based methods use graded expression values rather than a binary threshold and may be more sensitive when non-zero expression levels carry biologically meaningful signal, offering a natural methodological comparison point
-
p-values were transformed to information content scores (−log(adjusted p)) to populate the FEM expression matrix↳ Could also: A z-score derived from the hypergeometric null distribution, or the directly signed log-odds ratio of the Fisher's test, could also serve as the per-cell activity score — Different score transformations affect the marginal distribution of the resulting matrix; z-scores place scores on a standardized scale that facilitates cross-gene-set and cross-cell comparisons, while log-odds ratios directly quantify the magnitude of enrichment independent of sample size
-
The Seurat differential-expression step did not specify the underlying statistical test (Seurat's default is the Wilcoxon rank-sum test)↳ Could also: The test could be explicitly set within FindMarkers to a negative-binomial model (DESeq2 or edgeR pseudobulk) or a hurdle model (MAST) — Explicit test selection and reporting aids reproducibility; negative-binomial and hurdle models are designed for count data with zero-inflation characteristic of scRNA-seq and have been shown to better control type-I error in some benchmarks, making the choice of test a meaningful parameter to document
-
Three specific public datasets were selected to demonstrate FEM performance without a systematic multi-dataset benchmarking framework↳ Could also: A broader panel of datasets with varying cell numbers, sequencing depths, and technologies could also be evaluated, with bootstrapped confidence intervals on ARI or NMI — Wider benchmarking with uncertainty quantification would support stronger generalizability claims and help characterize conditions under which FEM's clustering agreement advantages are largest or smallest relative to GEM-only or alternative enrichment-scoring approaches
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
89 downstream papers · 2 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Comprehensive Integration of Single-Cell Data. 2019 · 13,137 cites
- Integrating single-cell transcriptomic data across d... 2018 · 10,726 cites
- Batch effects in single-cell RNA-sequencing data are... 2018 · 2,019 cites
- A Single-Cell Transcriptome Atlas of the Human Pancr... 2016 · 1,073 cites
- Efficient integration of heterogeneous single-cell t... 2019 · 722 cites
- Fully-automated and ultra-fast cell-type identificat... 2022 · 650 cites
- Benchmarking algorithms for gene regulatory network... 2020 · 562 cites
- Predicting gene regulatory links from single-cell RN... 2023 · 49 cites
- Optimal Hypoxia Regulates Human iPSC-Derived Liver B... 2018 · 39 cites
- Effect of imputation on gene network reconstruction... 2022 · 22 cites
- scCAN: single-cell clustering using autoencoder and... 2022 · 19 cites
- scGREAT: Transformer-based deep-language model for g... 2024 · 13 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34909283 (FEM: mining biological meaning from cell level in scRNA-seq)
PeerJ 2021, DOI 10.7717/peerj.12570. Repo: github.com/qingyunpkdd/single_cell_fem @ f8dd063890581c99879d2210e7864fc063e734e9 (2021-11-02, authors' own Python+R).
What FEM is (the pipeline)
FEM = "functional expression matrix". Per single cell, it tests every functional gene set (Reactome / GO / immunologic, MSigDB) for over-representation among that cell's NON-ZERO expressed genes, then stores GS(i,j) = -log10(p) as a function x cell matrix (replacing the gene x cell matrix). Downstream: PCA + Seurat clustering + UMAP on the FEM instead of the GEM.
NOTE (code vs paper): the paper text says "Fisher's exact test" and "-log(adjusted
p-value)". The actual batch/GUI code path uses the hypergeometric survival
function scipy.stats.hypergeom.sf(k-1, M=all_genes, n=K=set_size, N=n_expr)
(Fisher code exists in repo but is NOT on the executed path), applies a hard
min overlap of >2 (k<=2 -> p=1), a per-cell write threshold p<0.5, and
-log10 of the RAW p-value with NO multiple-testing adjustment (the column is
mislabelled "Adjusted P-value"). Flagged for audit.
In scope (pipeline-derived, attempted)
- C1 (PRIMARY, deterministic): the shipped FEM batch code, run on the shipped
sample input (PBMC 2700 cells x 32738 genes) against the shipped Reactome gmt
(c2.cp.reactome.v7.0, 1499 sets), reproduces the shipped FEM output matrix
gui__versioin/result/hg191_c2(1499 functions x 2700 cells, 1,464,764 nnz, values = -log10 p). Fully self-contained, no stochastic step -> expect EXACT. Pipeline: single_cell_fem batch_processing_version (Fisher/hypergeom GSEA).
Out of scope / not attempted (the hard ~20%)
- Table 2 (liver GSE81252) & Table 3 (PBMC GO) per-cell-type top pathways (e.g. mature-hepatocyte reactome-gluconeogenesis adj-p 1.75E-11; B-cell go-mhc-class-ii 1.31E-299). These require: download GEO data, full Seurat pipeline (liver.r/pbmc.r), STOCHASTIC clustering (UMAP/Louvain), manual cell-type assignment, and an unspecified per-cluster p aggregation. Not cleanly specified -> skipped per 80/20. The shipped sample is PBMC+Reactome, which does not match either table's (dataset x gene-set) combination, so neither table is checkable from shipped data alone.
- GUI interaction, UMAP figures (visual/stochastic).
Data
- GSE81252 (liver) is referenced but NOT needed for C1; the repo ships a PBMC sample + its FEM result, which is the cleanest deterministic reproduction.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The paper's core FEM transform reproduces 1:1 deterministically: re-running the unedited shipped batch code on the shipped PBMC2700 input vs Reactome1499 regenerated the shipped 1499x2700 matrix (1,464,764 non-zeros) exactly (Jaccard 1.0, Pearson 1.0, max abs diff 1.4e-14) — no fabrication. The notable issue is authors'-side: the paper describes Fisher's exact test and 'adjusted p-value', but the actual code uses the hypergeometric SF on raw, uncorrected p-values with a mislabelled output column, which changes the meaning of the published Tables 2-3 p-values. Those biological enrichment claims (e.g. 1.75E-11, 1.31E-299) were out of scope to verify (GEO download + stochastic, underspecified pipeline). Overall a solid reproduction of the deterministic core with an explainable methods-vs-code discrepancy, hence yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.