Reducing language barriers, promoting information absorption, and communication using fanyi.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL, honest 1:1 on a third-party-tool pipeline applied to the paper's own data. 'fanyi' is a translation tool paper; its novel outputs are translations that need private API keys and are non-deterministic AI text -> out of scope. The pipeline-derived demo behind Figure 3 IS reproducible and was reproduced on «our HPC»: GSE52778(airway) -> DESeq2(~cell+dex) -> top 20 up/20 down -> clusterProfiler compareCluster GO-BP. Dataset is EXACT (same 8 SRR samples), DE-gene count/structure is EXACT (40 = 20+20). Gene identities are PARTIAL: 28/40 (70%) overlap with the Fig-3 heatmap, directionally consistent. A methodological finding: the paper omits the 'top 20' selection rule; ranking by adjusted p-value reproduces the figure (28/40) whereas fold-change ranking does not (5-6/40), so the authors ranked by significance. GO enrichment is PARTIAL: 2/7 terms exact + 4 thematically concordant (cardiac/Wnt/adhesion/hematopoietic themes match); metal-ion terms not recovered at cutoff, partly due to GO.db 2026 vs paper's 2022. Described well enough to reproduce the qualitative result strongly; NOT exact because selection rule, thresholds, and software/GO versions are unstated. NOT attempted (80/20): the PBMC3k Seurat marker demo (Fig 1/2), the translation outputs themselves, and gene_summary() NCBI retrieval. No fabrication concern: all values derivable from public data + standard tools.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 67assessed: 2026-06-14 ⛓ bed44e80196d
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetManual retrieval and interpretation of gene information plus language barriers impede biomedical researchers' (especially non-native English speakers') ability to interpret results and communicate findings; an integrated R package combining automated NCBI gene-summary retrieval with AI-driven multilingual translation can streamline both tasks.
- ★ The fanyi R package retrieves gene information from NCBI and translates it into multiple languages using AI-driven online translation services resource
- ★ fanyi integrates multiple translation engines including Baidu, Bing, Youdao, Volcengine, Caiyun, Tencent, and the ChatGLM large language model resource
- ★ fanyi can translate axis labels and other text in ggplot2 graphics via the translate_ggplot function method
- ★ fanyi's gene_summary function aids interpretation of single-cell marker genes by combining expression data with retrieved gene descriptions finding
- ★ fanyi's translate function aids interpretation of bulk RNA-seq differential expression results by translating gene names and enrichment terms finding
- Language barriers negatively impact science and science education, causing inequality for under-represented, non-English-speaking communities finding
- fanyi caches retrieved gene information via the yulab.utils package to avoid redundant NCBI queries method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| single-cell RNA-seq marker gene analysis | human PBMCs (2700 cells, Seurat example dataset) | none | cluster marker gene identification, expression levels, adjusted P-values, and NCBI gene summaries | Seurat, clusterProfiler, fanyi, ggsc |
| bulk RNA-seq differential expression analysis | human airway smooth muscle cells (GEO: GSE52778, airway Bioconductor dataset) | not stated in text (design formula ~donor + condition) | differentially expressed genes (up/down), hierarchical clustering, translated gene names | DESeq2, clusterProfiler, ggtree, fanyi |
| functional enrichment analysis (GO biological process) | gene clusters derived from airway RNA-seq differential expression data | none | enriched biological process terms, displayed in original and AI-translated form | clusterProfiler (compareCluster, enrichGO), fanyi translate_ggplot |
- ▲ CCR7 is highly expressed in naïve CD4 T cells; encodes a GPCR involved in memory T-cell migration to inflamed tissue and dendritic cell maturation
- ▲ LGALS2 is highly expressed in CD14+ monocytes; binds lymphotoxin-alpha and regulates its transcription, increasing myocardial infarction risk; anti-LGALS2 antibody increased perfusion restoration, arteriolar diameter, and M2 macrophage numbers in a murine hindlimb model
- ▲ S100A6 is highly expressed in CD14+ monocytes and is implicated in cell cycle progression and differentiation; associated with inflammatory monocyte subsets and angiogenesis in prior studies
- ▲ CD3E is highly expressed in CD4 T cells, CD8 T cells, and NK cells as a key TCR complex component
- – DESeq2 analysis of the airway dataset yielded 20 up-regulated and 20 down-regulated genes ranked by adjusted P-value for downstream demonstration
- – Hierarchical clustering of differentially expressed genes separated them into two clades, each characterized by distinct GO biological process enrichment via clusterProfiler
- – translate_ggplot translated biological process terms in the enrichment dotplot into Chinese with a single function call
- count 2700 PBMCs (Seurat example single-cell dataset used for marker gene demonstration)
- count 20 up-regulated and 20 down-regulated genes (top DESeq2 differentially expressed genes selected from airway dataset)
- other 98% of peer-reviewed scientific articles are published in English (motivation for language barrier problem, citing reference 16)
- count more than 200 languages supported (translation capability of fanyi)
- other top 2 marker genes per cluster with avg_log2FC > 1, ranked by adjusted P-value (marker gene selection criteria in Seurat/clusterProfiler pipeline)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software demonstration paper introducing the fanyi R package for gene information retrieval and AI-driven multilingual translation. Statistical analyses serve as illustrative use cases rather than primary study endpoints: Seurat's FindAllMarkers identifies marker genes across clusters in a 2700-cell PBMC dataset, DESeq2 identifies differentially expressed genes in a four-donor airway smooth muscle RNA-seq dataset (GSE52778) using a donor-blocked design, and clusterProfiler performs Gene Ontology enrichment on DEG clusters derived by hierarchical clustering. All results are presented as annotated visualizations demonstrating the package's translation and annotation capabilities.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| DESeq2 negative binomial GLM with Wald test | Differential expression analysis of the airway RNA-seq dataset (GSE52778), design ~donor + condition; top 20 up- and 20 down-regulated DEGs selected for heatmap demonstration | Four human airway smooth muscle cell donors with two conditions each (8 samples inferred from design formula; paper states 'four human airway smooth muscle cells') | not stated |
| FindAllMarkers (Seurat; Wilcoxon rank-sum test, the package default — not explicitly named in the paper) | Marker gene identification across all PBMC cell clusters; top 2 markers per cluster retained after filtering avg_log2FC > 1 | 2700 peripheral blood mononuclear cells | not stated |
| Hypergeometric test (via clusterProfiler enrichGO / compareCluster) | Gene Ontology biological process enrichment of two DEG clusters produced by cutree on the hierarchical clustering tree | 40 top DEGs (20 up- and 20 down-regulated) mapped to Entrez IDs | not stated |
| Hierarchical clustering (hclust with Euclidean distance; linkage method not stated) | Gene-level clustering of the 40 top DEGs on rlog-normalized expression profiles, visualized as a tree with gheatmap | 40 genes across 8 samples | na |
| k-means clustering (k = 2) | Sample-level column ordering in heatmap (kmeans applied to transposed expression matrix) | 8 samples (inferred from GSE52778 four-donor paired design) | na |
-
DESeq2 was used for differential expression in a small RNA-seq experiment with four donors and two conditions (approximately eight samples)↳ Could also: limma-voom or edgeR could also be applied; limma-voom in particular is frequently used when the number of samples is very small — limma-voom borrows variance information across genes using empirical Bayes shrinkage, which can improve power and control of false positives in small-n settings; benchmarks comparing DESeq2, edgeR, and limma-voom in small-sample designs exist and show method-dependent differences in sensitivity and FDR control
-
FindAllMarkers (Seurat) was applied with its default Wilcoxon rank-sum test to identify PBMC marker genes↳ Could also: MAST, a hurdle model for single-cell data, or pseudobulk methods (e.g., aggregating counts per donor then applying DESeq2/edgeR) could also be used for single-cell differential expression — MAST explicitly models the bimodal zero-inflated distribution of single-cell counts; pseudobulk approaches treat the biological replicate (donor) as the unit of analysis, which better controls false positives when donor-level variability is present
-
Genes were prioritized by ranking on adjusted p-value after a hard threshold of avg_log2FC > 1, then the top two per cluster were retained for display↳ Could also: Shrinkage-based fold change estimation (e.g., DESeq2's lfcShrink with apeglm) or a composite ranking statistic incorporating both significance and effect size could also be used — Shrinkage estimators reduce the influence of noisy fold change estimates at low expression levels; a single composite ranking avoids the threshold sensitivity inherent in two separate cutoffs applied sequentially
-
Euclidean distance with unspecified linkage was used for hierarchical clustering of 40 DEGs for heatmap visualization↳ Could also: Correlation-based distance (e.g., 1 minus Pearson correlation) with Ward's D2 or complete linkage is also widely used for expression heatmaps — Correlation-based distance captures the shape of expression profiles independent of absolute magnitude, which may better group genes with similar patterns when between-sample normalization leaves scale differences; the choice of linkage method can substantially affect cluster topology
-
GO biological process enrichment was performed separately on each of two DEG clusters without a stated multiplicity correction across the two enrichment analyses↳ Could also: Applying FDR correction jointly across all enrichment tests from both clusters, or using gene set enrichment analysis (GSEA) on the full ranked DEG list, would also be standard options — Joint FDR control across both enrichment families maintains the intended discovery rate across the full set of tests; GSEA avoids the need for an arbitrary fold-change or p-value cutoff by using the complete expression ranking as input
-
rlog-normalized expression values were used as input to Euclidean-distance hierarchical clustering and k-means↳ Could also: Variance-stabilizing transformation (vst, also from DESeq2) or log-CPM normalization could also serve as the normalized input; the choice of normalization can affect clustering results — vst is computationally faster than rlog for large datasets and behaves similarly; log-CPM with TMM normalization (edgeR) is another standard approach; documenting the normalization choice explicitly allows readers to assess the stability of the resulting clusters
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-39039634 (fanyi)
Paper: Wang D, ... Yu G. Reducing language barriers, promoting information absorption, and communication using fanyi. Chin Med J (Engl) 2024. PMID 39039634 · PMCID PMC11332769 · DOI 10.1097/cm9.0000000000003242 Repo: https://github.com/YuLab-SMU/fanyi Data: GEO GSE52778 (airway smooth-muscle RNA-seq; Himes et al. 2014)
Nature of the paper
fanyi is a software tool paper. The package itself is purely a *translation
- gene-information* utility (functions:
translate(), the per-service*_translate()wrappers,gene_summary(),ydict(),translate_ggplot()). It ships no bioinformatic analysis pipeline of its own. The biology in the paper is demonstration: standard third-party tools are run to generate inputs, whichfanyithen translates/annotates.
What is pipeline-derived (and reproducible)
The paper contains two demonstration analyses built from canonical tutorial datasets and standard Bioconductor/Seurat tools. Per BRIEF rule 2 (P16), applying these third-party tools to the paper's data is a fully valid reproduction.
| # | Result | Pipeline | Reproducible? |
|---|---|---|---|
| R1 | Fig 3 — GSE52778 differentially-expressed genes (heatmap of 20 up + 20 down) + GO-BP functional characterization of the up vs down clusters (dotplot, Fig 3B; Fig 3C = its Chinese translation) | airway (GSE52778) → DESeq2 (~cell + dex) → top 20 up / 20 down → clusterProfiler compareCluster/enrichGO(BP) |
YES — IN SCOPE (primary target). Deterministic; canonical dataset. |
| R2 | Figs 1–2 — PBMC 2700 scRNA-seq marker genes (CCR7, LGALS2, CCL19, LDHB; S100A6, CD3E) + UMAP/violin/density | Seurat pbmc3k standard tutorial → FindAllMarkers |
Partially — secondary. Seurat tutorial is largely deterministic but has cluster-id/labeling steps; NOT attempted in depth (80/20). |
Why R1 is the primary target — concrete comparison anchors in Fig 3
Fig 3 exposes checkable values even though the text reports no numbers:
- 8 samples, columns labelled
SRR1039508/09/12/13/16/17/20/21→ exactly theairway8-sample subset of GSE52778 (4 cell lines × untrt/Dex). - 40 gene symbols in the heatmap (20 up cluster, 20 down cluster).
- Enrichment clusters labelled Cluster 1 (19) and Cluster 2 (20)
→ 19 + 20 genes mapped into
compareCluster. - 7 GO-BP terms (Fig 3B): regulation of cell-cell adhesion; regulation of embryonic development; striated muscle cell differentiation; atrial cardiac muscle tissue development; hematopoietic stem cell homeostasis; iron import into cell; copper ion transport.
Out of scope (not attempted)
- The translations themselves (the package's actual novel output). They require private API credentials (Baidu/Bing/Youdao/Tencent/ChatGLM …) and are non-deterministic AI output → not pipeline-reproducible. (Fig 3C Chinese text, Fig 1/2 translated annotations.)
gene_summary()/ NCBI retrieval — depends on live NCBI gene records.- Wet-lab / manual interpretation — none claimed.
Caveats affecting grading
- The paper gives no DE thresholds, no fold-change/p cutoffs, no selection rule
for "top 20", and no software versions beyond "R (2022)" and httr2 1.0.1.
So R1 is reproduced with defensible standard choices (DESeq2 default
~cell+dex, significant by padj, top-20 by |log2FC| per direction) and the match to Fig 3 is judged qualitatively (gene-symbol overlap, GO-term overlap, cluster sizes). Exact 1:1 on the 20/20 list is not guaranteed because the selection rule is unspecified — this is recorded honestly, not chased.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a software/translation-tool paper; its novel outputs are out of scope, so reproduction targets the illustrative Fig 3 pipeline (GSE52778/airway → DESeq2 ~cell+dex → top 20 up/20 down → clusterProfiler GO-BP). Input data is identical and the DE structure (40 = 20+20) is exact; deviations sit at the input/methodology layer — the paper omits the selection rule, thresholds, and software/GO versions, and the genes/terms were read off a heatmap. Gene identity is partial (28/40, 70%, directionally consistent) and GO is partial (2/7 exact + 4 concordant), all explained by inferred padj-ranking + GO.db 2026-vs-2022 drift. No fabrication and no overturned finding — values are fully derivable from public data; overall a solid partial reproduction with explainable, authors-documentation-gap deviations.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.