Tracing truth: dynamic temporal networks for multi-modal fake news detection.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- 🔴Could not use the authors’ exact input data
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🔴The deviation was non-trivial in magnitude
- 🔴The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Code described well enough to run, but the data shipment is incomplete/mismatched and the headline result does NOT reproduce 1:1. GitHub repo dongdong0012/DTN-demo is deleted (404); code + pre-trained checkpoints recovered from Zenodo 10.5281/zenodo.15545745 (v1.0.0, commit 76c5bb8); PHEME processed node features from the README's Dropbox link. The PHEME random-walk files required by pheme.py (rwr_results/pheme_n5_p5_u100/n_neighbors.txt + original_adj) and the graph_def edges that would regenerate them are absent from every author source; the rwr Dropbox link ships only GossipCop, and the processed-data Dropbox ships only PHEME, so neither dataset is runnable as-shipped. I made PHEME runnable by reconstructing the two missing files from the shipped node batch files: n_neighbors.txt faithfully (timestamped sequences present; node-own timestamp provably irrelevant via shift-invariant stable softmax), original_adj partially (news<->{news,post,user} edges only; post/user cross-edges unavailable). Then evaluated the authors' OWN shipped checkpoints on their OWN shipped test split with their OWN code (num_epoch=0). Result: PHEME accuracy 0.794 (5-repeat mean) vs reported 0.938; real P/R/F1 0.868/0.811/0.838 vs 0.967/0.964/0.936; fake 0.676/0.762/0.716 vs 0.895/0.913/0.924 -> MISMATCH (~0.14 below on accuracy). Decisive corroboration independent of my reconstruction: the 10 shipped checkpoints store the authors' own val_acc 0.779-0.819 (mean 0.804), i.e. the authors' own models scored ~0.80, not 0.938. Additional internal-consistency red flags in Table 2: DTN PHEME real F1 0.936 is below both its P(0.967) and R(0.964); fake F1 0.924 is above both its P(0.895) and R(0.913); GossipCop fake F1 0.989 > P=R=0.982 -- all impossible for a harmonic mean, and pheme.py itself computes F1 correctly, so the printed table is not what the code yields. NOT attempted (out of 80/20 scope / data unavailable): GossipCop Table 2 (C4-C6; node features not shipped) and the Table 3 ablation (C7; no ablation toggles in the shipped code). Caveat: original_adj reconstruction is partial and may modestly lower the absolute reproduced score, but does not affect the checkpoint-stored val_acc corroboration. All grades are provisional; a human reviewer decides (see AUDIT.md).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 33assessed: 2026-06-14 ⛓ ac85d5ec5486
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a dynamic temporal network model that captures temporal relationships among propagation nodes and dynamically fuses multi-modal information improve the accuracy of fake news detection in dynamic social networks?
- ★ The proposed dynamic temporal network (DTN) model improves multi-modal fake news detection accuracy by capturing temporal dynamics of propagation nodes and dynamically fusing multi-modal information. method
- ★ A time similarity strength metric measures temporal similarity among nodes in propagation sequences and dynamically weights neighboring nodes. method
- ★ A heterogeneous social propagation graph of users, news, and posts enhances node representation via temporal similarity and dynamic graph structure changes. method
- ★ A Transformer encoder extracts global semantic features of news propagation and integrates multi-modal information. method
- ★ Entropy analysis quantifies temporal dynamics and reveals abnormal short-term concentrated and long-term diffusion propagation patterns of fake news. mechanism
- ★ DTN outperforms existing methods in accuracy and robustness and generalizes across different datasets. finding
- The model enables early monitoring of news propagation and long-term diffusion warning. finding
- ★ A hybrid text encoding using fine-tuned RoBERTa and T5, ResNet18 for images, and one-hot social features provides multi-modal feature representations. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Text feature extraction / encoding | News titles, content, and social posts; user descriptions/profile metadata | none | Contextualized text embeddings (sentence-level [CLS] embedding, field embeddings e_t/e_c/e_p, user embedding e_u) | Fine-tuned RoBERTa and fine-tuned T5 encoders |
| Image feature extraction | News-related images | none | Dense visual embedding e_v from final average pooling layer | Pre-trained ResNet18 |
| Social media feature representation | Users who posted news and related posts (follower count, fan count, verification status, reposts, likes) | none | Sparse numerical feature matrix / user social embedding e_u | One-hot encoding |
| Heterogeneous graph construction and optimization | Social network of users, news, and posts | none | Optimized propagation graph capturing dynamic propagation relationships | Random walks with frequency sampling (parameter k) |
| Global semantic encoding / multi-modal fusion | News propagation sequences | none | Overall semantic features of news propagation; fake/real classification | Transformer encoder |
- ▲ DTN model outperforms existing methods in accuracy and robustness for fake news detection
- – DTN demonstrates generalization capability across different datasets
- – Fake news exhibits concentrated short-term propagation and long-term diffusion patterns revealed via temporal/spatial distribution analysis
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a deep learning paper proposing the DTN (Dynamic Temporal Network) model for multimodal fake news detection, integrating text (RoBERTa, T5), image (ResNet18), and social media features within a heterogeneous propagation graph enhanced by temporal similarity weighting, entropy analysis, and a Transformer encoder. Model performance is evaluated by comparing DTN against existing baseline methods on benchmark datasets, with accuracy and effectiveness cited as the primary outcomes. The provided text ends before the full experimental and results sections, so detailed reporting of metrics, dataset splits, and comparison protocols is not available in the supplied excerpt. No formal statistical hypothesis tests are described anywhere in the available text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Not stated — evaluation relies on classification performance metrics (accuracy, effectiveness) compared across models; no formal hypothesis tests are described in the available text | DTN model comparison against baseline fake news detection methods on benchmark datasets | — | not stated |
-
Model comparisons are reported as point-estimate accuracy figures without measures of variability or formal significance testing↳ Could also: Repeated k-fold cross-validation with confidence intervals, or classifier comparison tests such as McNemar's test or bootstrap-based paired tests, could also be applied — A single held-out test split may be sensitive to the particular partition chosen; cross-validation with significance tests would quantify uncertainty in observed performance differences and allow assessment of whether gaps between models exceed expected sampling variation
-
The paper cites accuracy as the primary evaluation metric↳ Could also: F1-score, precision, recall, AUC-ROC, and Matthews Correlation Coefficient could also be reported alongside accuracy — For binary classification tasks such as fake news detection, real-world datasets frequently exhibit class imbalance; metrics like macro-F1 and AUC-ROC are less sensitive to class skew and provide complementary perspectives on model behavior across both classes
-
The paper validates the DTN architecture as a whole against baselines, with no ablation study described in the available text↳ Could also: Ablation experiments that systematically remove or replace individual components — temporal similarity weighting, entropy analysis, each modality branch, the Transformer encoder — could also be conducted — Ablation studies isolate the marginal contribution of each architectural choice, allowing readers to attribute observed performance gains to specific design decisions rather than the ensemble
-
The heterogeneous graph is optimized via random walks with frequency sampling controlled by a hyperparameter k↳ Could also: Sensitivity analyses varying k and other hyperparameters, or alternative graph sampling strategies such as node2vec or metapath-based walks, could also be evaluated — Reporting performance across a range of k values demonstrates model robustness to hyperparameter choice and aids practitioners in selecting appropriate settings for different graph densities or dataset sizes
-
Social media numerical attributes (follower count, fan count, verification status) are encoded with one-hot encoding↳ Could also: Learned embedding layers or continuous min-max or z-score normalization could also be applied to these features — One-hot encoding represents each value as an unordered discrete category; for count-based or ordinal attributes, preserving the ordinal or continuous structure via normalization or learned embeddings captures the relative magnitude of user influence
-
The paper asserts 'robustness' as a model property without a formal protocol for measuring it in the available text↳ Could also: Robustness could also be formally quantified via cross-domain transfer experiments, performance under input perturbation, or evaluation on held-out time periods of propagation data — A concrete robustness protocol with defined perturbation types or domain shift conditions makes the claim reproducible and distinguishes generalization from within-distribution performance
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40989433 (DTN: Dynamic Temporal Networks for multi-modal fake news detection)
- Title: Tracing truth: dynamic temporal networks for multi-modal fake news detection
- PMID 40989433 · PMCID PMC12453775 · DOI 10.7717/peerj-cs.2998
- Code: https://github.com/dongdong0012/DTN-demo → HTTP 404 (repo deleted) as of 2026-06-14.
User
dongdong0012still exists (200); repo removed. - Code+data archive (preserved): Zenodo 10.5281/zenodo.15545745
DTN-demo-v1.0.0.zip(642 MB, CC-BY-4.0). Per BRIEF rule P16, the Zenodo snapshot of the release is equally valid for reproduction.
In scope (pipeline-derived results)
The reproducible computational outputs are the DTN model's classification metrics in Table 2 (a deep-learning training+eval pipeline):
- C1 PHEME accuracy 0.938
- C2 PHEME real P/R/F1, C3 PHEME fake P/R/F1
- C4 GossipCop accuracy 0.993
- C5 GossipCop real P/R/F1, C6 GossipCop fake P/R/F1
- C7 ablation accuracy gains (Table 3) — secondary
Pipeline: PyTorch deep model = RoBERTa+T5 text encoder + ResNet18 image encoder
- heterogeneous graph (random walks) + temporal fusion + GAT/Transformer head; Adam, batch 16, max 40 epochs patience 5. Requires GPU.
80/20 plan
- Primary target: PHEME (smaller: 6,425 news / 156,397 nodes) — reproduce Table 2 DTN accuracy + P/R/F1.
- GossipCop (1.64M nodes) is much heavier — attempt only if PHEME succeeds and time/compute allow.
- Ablation (Table 3) only if the training pipeline is cleanly toggleable.
Out of scope
- Baseline models (MIGCL, HCCIN, HetTransformer, …) — these are other papers' methods, re-tabulated for comparison; not the authors' pipeline.
- Any wet-lab / manual steps: none (pure computational paper).
Auditability flags (reported-table internal inconsistencies)
Several reported F1 values are mathematically inconsistent with their own precision/recall (F1 must be the harmonic mean, i.e. between P and R):
- C2 PHEME real: P=0.967, R=0.964 → F1 must be ≈0.965, but 0.936 printed.
- C3 PHEME fake: P=0.895, R=0.913 → F1 must be ≈0.904, but 0.924 printed (above both).
- C6 GossipCop fake: P=R=0.982 → F1 must be 0.982, but 0.989 printed. These are recorded as possible-fabrication / typesetting-error notes for the human reviewer.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Re-evaluating the authors' own shipped PHEME checkpoints on their own test split with their own pheme.py yields accuracy 0.794 (mean of 5) versus the paper's 0.938 — a ~0.14 gap — and the checkpoints even store the authors' own val_acc (~0.80), corroborating that the models never reached 0.938. This is squarely an authors-side problem, not a data-availability artifact: the deviation is reconstruction-independent. Compounding it, Table 2 contains mathematically impossible F1 values (real F1 0.936 below both P/R; fake F1 0.924 and GossipCop 0.989 above both P/R), and pheme.py computes F1 correctly — so the published table is not what the code produces. Verdict: severe, fabrication-suspect; the central performance claim does not hold.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.