Graph Random Forest: A Graph Embedded Algorithm for Identifying Highly Connected Important Features.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Reported values were only indirectly comparable
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce -> 1:1 on the simulation. The GRF repo (github.com/tianlq-prog/GRF @0c7856f) ships ONLY a self-contained simulation pipeline (sim_data.py + graph_nn.py + run_simulation.ipynb); no real-data script. I ran the authors' own code unchanged (one rpy2>=3.6 compatibility shim only: their code calls the removed numpy2ri.activate()) on 10 independently-seeded simulated datasets matching the notebook setting (two cores, N=500, 4000 nodes, 100 true predictors, logistic link) on «our HPC» SLURM «job». All four in-scope simulation claims reproduce: GRF feature-selection AUC (hop2/3) ~0.91-0.93 vs standard RF ~0.66 -- exactly the paper's '>0.9 vs 0.6-0.75' (Fig 2); and GRF selects a denser, far less fragmented top-100 sub-graph than RF (largest component ~85 vs ~49; density 2x; components 15 vs 44) -- matching Fig 4. The pipeline is stochastic (sklearn RF has no fixed random_state), so I seeded per replicate and report mean+-sd over 10 reps (paper's 'ten datasets'); no byte-match claimed, the comparison is to the reported regime/ordering, which holds in every replicate. NOT attempted (the hard ~20%): the real-data results -- GSE93593 hESC (Table 3), NSCLC (Table 1), GO enrichment (Tables 2/4) -- because the repo ships no real-data code; reproducing them would require reconstructing undocumented glue (1733-cell matrix download, HINT-PPI filter to 12,215 genes, DCX labelling, Enrichr GO). That is absence-of-code, not evidence of fabrication; no values were flagged as non-derivable.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 89assessed: 2026-06-15 ⛓ 99fa9d191e70
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan incorporating known biological network topology into the random forest tree-building process yield a feature-selection method that identifies highly connected sub-graphs of important genes while maintaining classification accuracy comparable to standard random forest?
- ★ Graph Random Forest (GRF) embeds graph/network information directly into the decision-tree building process by splitting on features in the k-hop neighborhood of a data-driven head-splitting node. method
- ★ GRF identifies effective features that form highly connected sub-graphs while achieving classification accuracy equivalent to standard random forest. finding
- ★ GRF adapts the Gini importance metric, aggregating feature importance across per-node forests weighted by head-node occurrence counts. method
- ★ Higher hopping steps (2 or 3) yield better feature-selection performance (AUC/PR-AUC) without sacrificing prediction accuracy. finding
- ★ Sub-graphs selected by GRF have higher density and larger connected components than those selected by RF, consistent with the biological assumption of functional consistency among related genes. finding
- GRF feature selection is reproducible/stable across repeated training runs. finding
- Open-source GRF implementation is provided as a public GitHub repository. resource
- ★ GRF can distinguish lung cancer subtypes (LUAD vs LUSC) using TCGA miRNA expression with high accuracy. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Simulation / synthetic gene expression classification | Scale-free network of 4000 features with 500 samples generated via Barabási–Albert model and multivariate Gaussian | none (synthetic true predictors via generalized linear model with logistic/absolute link) | Test accuracy, AUC, PR-AUC of feature importance, sub-graph density, number of connected components, size of largest connected component | Python; Linux workstation (5950X CPU, 128 Gb RAM, GTX 3060 GPU) |
| RNA-seq / miRNA expression classification | Non-small cell lung cancer (LUAD vs LUSC) patient tumors from TCGA | none (subtype comparison; LUAD=1, LUSC=0) | Classification accuracy distinguishing subtypes; connectivity of selected feature sub-graph | TCGA data; HINT miRNA network |
| RNA-seq classification | Human embryonic stem cells (GSE93593) from GEO | none | Classification and feature selection | GEO dataset GSE93593 |
- ▲ GRF with 2/3 hopping steps achieved AUC mostly higher than 0.9 for feature selection in one-core simulation versus RF AUC >0.9 (GRF) vs ~0.6-0.75 (RF)
- – GRF with hopping step 3 outperformed RF in test accuracy; hopping step 2 was similar to RF; hopping step 1 was lower than RF
- ▲ GRF sub-graphs had larger density in all experiments and largest connected component exceeded 80 nodes (of top 100 features) at hopping steps 2/3, whereas RF sub-graphs were more scattered >80 of 100 nodes
- – Feature selection was reproducible: across 20 repeated GRF runs (top 100 features each), 58 features appeared >14 times and 86 appeared >10 times 58/100 >14 times; 86/100 >10 times
- – Of 58 most stable features, 53 were true predictors and the other 5 were within one-step neighborhood of true predictors 53/58 true predictors
- – Across the union of 20 selected feature lists, 46.4% were true predictors and 89.3% were within one-step neighborhood of true predictors 46.4% true; 89.3% within one step
- ▲ Prediction accuracy showed an upward trend as the number of true predictors increased
- – GRF computation time per simulation experiment was around 7 seconds ~7 s
- other AUC >0.9 (GRF) vs ~0.6-0.75 (RF) (feature importance AUC, one-core logistic simulation, 10 datasets per setting)
- count 4000 features, 500 samples (simulation expression data X)
- count true predictors 30,60,90,120,150,180,210 (varying number of true predictors in simulation)
- count 53/58 (stable top features that were true predictors across 20 runs)
- other 46.4% true predictors; 89.3% within one-step neighborhood (union of selected feature lists over 20 training repeats)
- count 9819 miRNA features, 1020 samples (combined LUAD+LUSC TCGA dataset after processing (largest connected component of HINT network))
- count LUAD: 23,032 miRNA, 524 patients; LUSC: 23,652 miRNA, 496 observations (raw TCGA NSCLC subtype expression matrices)
- other ~7 s (GRF computation time per simulation experiment)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a methodological machine-learning paper that proposes Graph Random Forest (GRF) and evaluates it primarily through predictive-performance comparison rather than formal inferential hypothesis testing. Performance was assessed on simulated data (4000 features, 500 samples; 10 datasets per setting, 7:3 train/test split) and two real RNA-seq datasets, using test accuracy, AUC, and PR-AUC as metrics, with results reported as means with standard-deviation error bars across repeated runs (10 simulation replicates; 20 repeats on real data). No classical significance tests, p-values, or confidence intervals are reported.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Classification test accuracy (held-out test set) | GRF vs RF comparison in simulations and real datasets (Figure 2a,d; Table 1) | 500 samples (simulation) split 7:3; 1020 samples (NSCLC) split 7:3 | na |
| Area Under the ROC Curve (AUC) on estimated feature importance vs ground-truth predictors | Feature-selection power in simulations (Figure 2 second column; Figure 3) | 10 simulated datasets per setting | na |
| Precision-Recall AUC (PR-AUC) on estimated feature importance | Feature-selection power in simulations (Figure 2 third column) | 10 simulated datasets per setting | na |
| Graph density / connected-component descriptive metrics | Connectivity of selected top-100 sub-graphs (Figures 4 and 5) | — | na |
| Stability/reproducibility frequency counts of selected features | 20 repeated GRF trainings, top-100 features each (Section 3.1) | 20 repeats; 500 samples, 4000 features, 210 true predictors | na |
-
Method comparisons (test accuracy, AUC, PR-AUC) across GRF variants and RF were summarized with means and SD error bars over repeated runs.↳ Could also: A paired statistical comparison across the matched replicate datasets (e.g., paired t-test or Wilcoxon signed-rank test) or reporting 95% confidence intervals for the metric differences could also be presented. — A paired test or CI on the per-replicate differences would quantify how consistently one method exceeds another beyond the visual overlap of error bars, since the same simulated datasets are used for both methods.
-
Variability was conveyed with standard-deviation error bars computed from ten simulation replicates.↳ Could also: A 95% confidence interval or the interquartile range/individual replicate points could also be shown. — With a modest number of replicates (n=10), CIs or showing the individual points additionally communicates uncertainty in the mean and the shape of the distribution, which some readers prefer.
-
Feature-selection quality was assessed against the known ground-truth predictor list using AUC and PR-AUC.↳ Could also: Reporting precision/recall (or F1) at a fixed selection size such as the top-100, with a resampling-based interval, could also be included. — A fixed-threshold metric maps directly onto the practical task of choosing a finite gene list, complementing the threshold-free AUC/PR-AUC summaries.
-
Reproducibility of feature selection was characterized by counting how often features recurred across 20 repeated trainings.↳ Could also: A formal stability index (e.g., Jaccard-based stability or Kuncheva's consistency index) across the repeated selections could also be reported. — A single stability statistic gives a comparable, scale-aware number that can be contrasted directly between GRF and RF and across settings.
-
Models were trained with default parameters and a single 7:3 train/test split per dataset.↳ Could also: Repeated k-fold or nested cross-validation could also be used for performance estimation and any tuning. — Cross-validation uses the data more fully and yields lower-variance performance estimates, which is often valued when sample sizes are limited.
-
Real-data subtype classification (LUAD vs LUSC) performance was summarized as an averaged score over twenty experiments.↳ Could also: Additional class-imbalance-aware metrics (balanced accuracy, sensitivity/specificity, or ROC/PR curves with intervals) could also accompany accuracy. — Reporting per-class metrics alongside overall accuracy describes performance on each subtype, which is informative when class sizes differ (524 vs 496 here).
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
GRF prediction accuracy increases monotonically as the number of true predictors in the simulated network grows.other synthetic scale-free-network up 2023×1papers★ This paper is the founder (earliest)
-
Across the union of 20 GRF feature lists, 46.4% of selected features are true predictors and 89.3% fall within one hop of a true predictor.other synthetic scale-free-network 2023×1papers★ This paper is the founder (earliest)
-
Of 58 most reproducibly selected GRF features, 53 are true predictors and the remaining 5 are within one hop of a true predictor.other synthetic scale-free-network 2023×1papers★ This paper is the founder (earliest)
-
GRF feature selection is stable: 86 of 100 top features appear in more than 10 of 20 repeated runs; 58 appear in more than 14 runs.other synthetic scale-free-network 2023×1papers★ This paper is the founder (earliest)
-
GRF with 2–3 hopping steps achieves feature-selection AUC >0.9, substantially outperforming RF (AUC ~0.6–0.75) on one-core simulation.other synthetic scale-free-network up 2023×1papers★ This paper is the founder (earliest)
-
GRF-selected feature sub-graphs have higher density and largest connected components exceeding 80 of the top 100 features (hopping steps 2–3), versus more scattered RF sub-graphs.other synthetic scale-free-network up 2023×1papers★ This paper is the founder (earliest)
-
GRF test accuracy exceeds RF at hopping step 3, matches RF at step 2, and is lower than RF at step 1, indicating hopping depth is a critical hyperparameter.other synthetic scale-free-network mixed 2023×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37509188 (Graph Random Forest, GRF)
- Paper: Tian, Wu, Yu. "Graph Random Forest: A Graph Embedded Algorithm for Identifying Highly Connected Important Features." Biomolecules 2023;13(7):1153.
- Repo: https://github.com/tianlq-prog/GRF @ commit 0c7856fab371e92aba74cf3b7564329555bac428
- Data accession in brief: GEO GSE93593 (hESC single-cell RNA-seq, DCX status).
What the repo actually ships
sim_data.py—simulation_data(...): generates a Barabási–Albert (BA) graph, picks core nodes + "true predictors", draws X ~ MVN(0, 0.8^Dij) and a binary y via a logistic/abs link. SELF-CONTAINED synthetic data, no download.graph_nn.py—graph_rf(...): the GRF algorithm. Pre-trains an RrandomForest(ntree=500, maxnodes=2) via rpy2 to score each feature by how often it is a root split var, then fits per-feature sklearn RandomForests over graph neighborhoods (iter_near_sizehops), aggregating feature importances + predictions.rf(...): the standard-RandomForest baseline (sklearn), returns feature importances + test accuracy.sub_graph(...): connectivity metrics of the top-thresselected features (density, #components, largest-component size, mean distances).
run_simulation.ipynb— runs ONE simulation (num_core=2, N=500, num_nodes=4000, true_predictors=100, link=logistic) for GRF at hop-steps 3,2,1 and writes the acc/roc/prauc + connectivity metrics tosimulation_res.csv.
IN SCOPE (pipeline-derived, clearly specified, low-hanging — the 80%)
- Feature-selection AUC, simulation (paper's headline claim, Fig. 2):
GRF (hop-step 2/3) feature-selection ROC-AUC "mostly higher than 0.9" while
standard RF is "around 0.6–0.75". REPRODUCE by running
graph_rf(hops 1/2/3) andrfon simulated datasets and computingroc_auc_score(true_predictors, imp). - Sub-graph connectivity, simulation (Fig. 4): GRF selects a denser, more
connected sub-graph (larger density, larger largest-component, fewer components)
than standard RF. REPRODUCE via
sub_graphon both importance vectors.
These need NO external data and exercise the authors' own algorithm end-to-end. Note: the pipeline is stochastic (sklearn RandomForest has no fixed random_state; numpy/random unseeded in the algorithm). We seed per replicate for our own regenerability and report mean±sd over 10 simulated datasets (the paper's "ten datasets" per setting). Exact byte-match is not expected or claimed — the comparison is to the reported AUC regime (>0.9 vs 0.6–0.75) and the connectivity ordering.
OUT OF SCOPE (the hard ~20% — not attempted, by 80/20 rule)
- GSE93593 real-data run (Table 3), NSCLC (Table 1), GO enrichment (Tables 2/4). Reason: the repo ships no real-data script — only the simulation. Reproducing Table 3 would require reconstructing undocumented glue (download 1733-cell hESC matrix, build/filter the HINT PPI network to 12,215 genes, define the DCX label, GO/Enrichr calls) that is not in the shipped code. This is exactly the under-specified last 20%; we document it rather than fabricate it.
- Pipeline parameter search inside
rf()(10-fold CV grid over n_estimators/ max_depth): we fix n_estimators/max_depth to avoid a long search; documented as a minor, faithful deviation (it only tunes the baseline, not GRF's importances).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Running the authors' own simulation code unchanged (one rpy2 compat shim) on 10 seeded datasets reproduces all four in-scope claims in the reported direction and magnitude: GRF feature-selection AUC >0.9 vs standard RF 0.6–0.75 (Fig 2) and a denser, far less fragmented top-100 subgraph (Fig 4). The deviations are purely stochastic (per-replicate seeding) and the comparison is to figure-range regimes rather than exact values, so q2 is yellow. The unverified part — the real-data Tables 1–4 (GSE93593 hESC, NSCLC, GO enrichment) — is absence of shipped real-data code, not a derivability or fabrication problem, so q5/q7 stay green while q8 is yellow for partial coverage. No values flagged as non-derivable or 'too perfect'.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.