Prediction, syntax and semantic grounding in the brain and large language models.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓No relevant deviation in data/preprocessing
- ✓Any deviation was negligible
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough for ONE result, blocked for the rest. EXACT 1:1 reproduction of Fig 1E word-class counts (NOUN 1376, VERB 936, ADJ 325, PROPN 413, total 3050) by counting the spaCy POS labels shipped in the repo's Excels/*_wordclass.xlsx across all 20 analyzed audiobook chapters (commit 7448078084ca, computed on «host» with Python stdlib, no GPU). These numbers are exactly recomputable from the released data -> fully grounded, no fabrication signal. The two Llama 3.2 analyses (Fig 8I semantic predictability; Fig 8K GDV + Fig 8M probe-classifier accuracy) could NOT be reproduced: both scripts require Transcript/{ch}_words.xlsx, the actual German audiobook words, which the README explicitly gates ('contact the authors'); the repo ships POS labels and numeric onsets only, so the LLM input text is data_restricted (the neural EEG/MEG .fif data IS open on Zenodo 15744486, but the text/audio is not). These figures also carry no numeric values in the text, so any comparison would be qualitative. The EEG/MEG neural contrasts (Figs 1-7) were NOT attempted: they are produced in the Brainstorm GUI (manual, non-scriptable), out of scope per the brief. Not a drop because a clearly-specified pipeline result reproduced exactly; not full because the headline LLM results depend on a restricted transcript. Brief's accession 10.5281/zenodo.1212303 does not resolve; used repo README's Zenodo 15744486 instead.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 86assessed: 2026-06-16 ⛓ d03aa92aa836
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-07-31
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusHow does the brain anticipate and process distinct word classes (nouns, verbs, adjectives, proper nouns) during naturalistic continuous speech comprehension, and do word-class-specific neural signatures of prediction and semantic grounding parallel the predictability and representational structure found in a transformer-based large language model (Llama 3.2)?
- ★ Nouns show significant pre-onset neural activity, suggesting enhanced anticipatory processing of this word class. finding
- ★ Event-related fields and potentials for different word classes show highly reproducible, characteristic spatio-temporal signatures replicable across independent exploration and validation cohorts. finding
- ★ Source-space activity for nouns extends beyond temporal regions into sensorimotor cortices, suggesting deeper semantic grounding of nouns in sensory experience than verbs. finding
- ★ A linear probe on Llama 3.2 hidden states shows nouns and adjectives are more easily predicted than verbs, aligning with the pre-onset neural signals for nouns. finding
- ★ Combined simultaneous MEG-EEG recording leverages complementary sensitivities and high temporal resolution to capture spatio-temporal dynamics of anticipatory language processing. method
- ★ Comparing word-class-specific neural responses with word-class-specific predictability/representational structure in an LLM provides a complementary computational reference frame (rather than direct encoding/decoding mapping). method
- Even neural networks trained only on next-word prediction can spontaneously internalize basic grammatical/word-type structure, supporting usage-based emergence of syntax. mechanism
- Use of an exploration/validation cohort split as an internal replication strategy to enhance reproducibility of naturalistic EEG/MEG findings. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Magnetoencephalography (MEG), event-related fields during continuous speech listening | 29 healthy right-handed German native speakers listening to German audio book 'Vakuum' | none (naturalistic listening) | word-class-specific ERFs (nouns, verbs, adjectives, proper nouns) around word onset | 248-channel Magnes 3600WH, 4D-Neuroimaging |
| Electroencephalography (EEG), event-related potentials during continuous speech listening | 29 healthy right-handed German native speakers listening to German audio book | none (naturalistic listening) | word-class-specific ERPs around word onset | 64-channel ANT Neuro system (with EOG and ECG) |
| Source reconstruction (minimum norm imaging / sLORETA) of ERFs/ERPs | human cortex template (ICBM 152 / MNI anatomy) | none | mean source activity per region across 600 Schaefer2018 parcellations, significant p-value source maps | Brainstorm software; Overlapping Spheres (MEG), BEM/OpenMEEG (EEG) |
| Part-of-speech tagging of stimulus text | German audio book transcript | none | POS tags assigning words to noun/verb/adjective/proper-noun classes | spaCy model 'de_core_news_sm' |
| Linear probe neural network on LLM hidden states | pre-trained multilingual Llama-3.2-1B (embedding layer + 16 transformer blocks) | none | word-class predictability / representational structure from hidden representations | Llama-3.2-1B (release Sept 25, 2024), PyTorch, RoPE positional encoding |
| Forced alignment for word onset extraction | audio files and corresponding transcripts | none | precise word onset times for epoching | WebMAUS software |
- ▲ Significant pre-onset activity observed for nouns, indicating heightened anticipatory readiness for this word class
- – Word-class-specific ERF/ERP spatio-temporal signatures were highly reproducible across exploration and validation cohorts
- – Noun source activity extended into sensorimotor cortices beyond temporal regions, consistent with deeper semantic grounding than verbs
- – Linear probe of Llama 3.2 hidden states predicts nouns and adjectives more easily than verbs
- – Cluster-based and permutation tests revealed significant differences between noun and verb ERFs/ERPs (p < 0.05) p<0.05
- count 29 participants (15 females) (study sample listening to audio book)
- mean mean age 22.8 years (range 18-28) (participant age)
- mean laterality quotient 85.4 ± 12.6 (right-handedness (Edinburgh-type))
- count Nouns: 1376, Verbs: 936, Adjectives: 325, Proper Nouns: 413 (word-type frequency in audio book)
- mean Noun word length µ:0.498 s σ:0.220 s (distribution of noun durations)
- mean Verb word length µ:0.420 s σ:0.198 s (distribution of verb durations)
- count exploration cohort subjects 1-19, validation cohort subjects 20-29 (internal replication split)
- other 5,000 randomizations, two-tailed cluster-based/Wilcoxon tests, FDR corrected (statistical testing of ERF/ERP differences)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
Twenty-nine participants listened to a naturalistic German audio book while simultaneous EEG and MEG were recorded. Neural responses were segmented by word class (nouns, verbs, adjectives, proper nouns) and analyzed via event-related fields and potentials (1–4 Hz filtered) in both sensor and source space. Primary inferential comparisons used cluster-based permutation tests (paired t-test and Wilcoxon signed-rank) against randomized baselines, with an FDR correction applied to the channel-level anticipatory analysis. A complementary analysis trained a linear probe on hidden states of Llama-3.2-1B to characterize word-class-level predictability in a large language model.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Cluster-based paired t-test (two-tailed, 5,000 randomizations) | Sensor-space comparison of noun vs. verb ERFs/ERPs across all 29 participants and all channels/time points | 29 participants | not stated |
| Cluster-based permutation test / sourcestatistics (FieldTrip within Brainstorm) | Source-space comparison of word-type-specific ERFs vs. randomly selected time points matched for trial number, across 600 Schaefer2018 parcellations | 29 participants | not stated |
| Two-tailed Wilcoxon signed-rank test (5,000 randomizations, FDR-corrected) | Anticipatory activity analysis in predefined left frontal MEG channels and bilateral temporal EEG channels, comparing noun/verb ERPs/ERFs to random time points | 29 participants | not stated |
| Linear probe (single-layer neural network classifier) | Word-class predictability from hidden states of Llama-3.2-1B across embedding and 16 transformer layers | — | na |
-
The primary multi-channel, multi-timepoint comparison used cluster-based permutation testing, which controls family-wise error by treating contiguous clusters of significant time-sensor points as the unit of inference↳ Could also: Mass-univariate testing with threshold-free cluster enhancement (TFCE) or FDR correction applied voxel-/sample-wise would also be standard in EEG/MEG research for this type of comparison — TFCE avoids the need to set an arbitrary cluster-forming threshold and can be more sensitive to spatially diffuse effects; FDR mass-univariate approaches similarly control for multiplicity while providing per-sample inference that can be easier to interpret
-
The cohort split (exploration n=19, validation n=10) was defined by chronological recording order, serving as an internal replication check↳ Could also: Random assignment to the two cohorts, or a leave-one-out / k-fold cross-validation approach across subjects, could also be used to assess generalizability of grand-average patterns — Random splitting removes any potential confound of session order (e.g., equipment drift, seasonal variation in recruitment); cross-validation provides a continuous measure of how well patterns replicate rather than a binary exploration/validation outcome
-
Word-class effects on neural time series were analyzed at the level of grand-average ERPs/ERFs after collapsing across all tokens of each class per participant↳ Could also: Linear mixed-effects models treating both participants and individual word tokens as random effects (the 'by-item' random slope, as recommended in Barr et al. 2013) would also be applicable — Mixed models account simultaneously for between-subject and between-item variability, and generalizability claims apply to both the sampled participants and the sampled vocabulary; averaging to the participant level before testing treats each token within a class as exchangeable
-
Source reconstruction used the standard ICBM 152 template MRI anatomy for all participants rather than individual structural MRIs↳ Could also: Individual-MRI-based source reconstruction (with participant-specific BEM and cortical surface) is also standard when individual scans are available — Individual anatomy can improve co-registration accuracy and reduce spatial smearing in the source estimate, particularly for EEG where the skull and scalp geometry strongly influence the forward model; template-based methods are a well-accepted alternative when individual MRIs are not available
-
Neural results were reported as binary significance (p < 0.05) without accompanying effect sizes or confidence intervals↳ Could also: Reporting Cohen's d (for paired differences), partial eta-squared, or 95% bootstrap confidence intervals around the cluster statistic or the mean difference waveform would also be standard — Effect size metrics and CIs communicate the magnitude and precision of differences independently of sample size, facilitating meta-analysis and sample-size planning for future studies; they are increasingly expected by journals and reporting guidelines (e.g., APA, OHBM)
-
The anticipatory analysis focused on a pre-selected set of left frontal MEG channels and bilateral temporal EEG channels, chosen a priori based on their relevance to language processing↳ Could also: A whole-scalp permutation test with subsequent post-hoc channel-level inspection, or a data-driven approach using multivariate pattern analysis (MVPA) decoding across all sensors in the pre-onset window, could also characterize anticipatory topography — Pre-selection of channels based on prior knowledge is a valid and common approach; whole-scalp or MVPA approaches are complementary in that they allow discovery of anticipatory signals outside the a priori ROI and provide a continuous decoding accuracy time-course as an effect-size analogue
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-41807493
Paper: Kölbl et al. (2026) Prediction, syntax and semantic grounding in the brain and large language models. Sci Rep. PMID 41807493 / PMC12979642 / DOI 10.1038/s41598-026-41532-0.
Code: https://github.com/nikolakoe/Prediction_Syntax_Semantic_Grounding
(commit 7448078084ca0aaaa99807f0cbba825af233e7b2, default branch main,
public, MIT-less / no license file, Python).
Data: EEG/MEG recordings on Zenodo record 15744486 ("EEG/MEG recordings
using german audio book", open access; ~33 subjects × ~750 MB preprocessed .fif
plus raw EEG + stimuli channels). NOTE: the brief's listed accession
10.5281/zenodo.1212303 does not resolve; the repo README points to record
15744486, which is the live open dataset.
The pipeline (from README + scripts)
The authors recorded combined EEG+MEG while participants listened to ~1 h of the German sci-fi audiobook "Vakuum". Words were force-aligned (BAS WebMAUS), POS-classified with spaCy, the neural data epoched per word, statistics run in the Brainstorm GUI, and two LLM analyses (Llama 3.2) computed on the transcript.
| # | Reported result | Pipeline | In scope? | Why |
|---|---|---|---|---|
| R1 | Fig 1E word-class counts: NOUN 1376, VERB 936, ADJ 325, PROPN 413 (Σ 3050) | spaCy POS tagging of transcript → counts | YES | Class labels are shipped in Excels/*_wordclass.xlsx (all 20 analyzed chapters). Reproducible from open data, no GPU, deterministic. |
| R2 | Fig 8I semantic predictability per word class (Llama 3.2 next-word prob.) | semantic_predictability_OpenAccess.py |
NO — blocked | Script reads Transcript/{ch}_words.xlsx["Word"] = the actual German words. README: "For audio signals and transcript of the german audio book contact the authors." Transcript not shipped → data_restricted. |
| R3 | Fig 8K GDV per Llama layer; Fig 8M probe-classifier accuracy per layer | syntactic_predictability_OpenAccess.py |
NO — blocked | Same: needs Llama hidden embeddings of each transcript word → requires the restricted transcript text. POS labels alone (shipped) are insufficient; the embeddings are the input. |
| R4 | EEG/MEG neural contrasts (Figs 1–7) | Brainstorm GUI statistical tests (README step 6) | NO — out of scope | Manual GUI analysis, not a scriptable pipeline. Preprocessing scripts (EEG/MEG) and epoching exist but produce intermediates, not a reported scalar; the reported neural stats are produced interactively in Brainstorm. |
Decision
Reproduce R1 (the one clearly-specified, fully-open, pipeline-derived numeric result). R2/R3 are genuinely blocked by the restricted transcript text (the audio
- transcript are "on request" even though the neural .fif data is open). R4 is a manual-GUI result, out of scope per the brief. Per the 80/20 rule we reproduce R1 exactly and document R2–R4 honestly rather than fabricate.
Verification that the transcript is genuinely absent
Excels/001_wordclass.xlsx (and all others) contains a single column Spacy
holding POS tags only (X, ADP, NUM, NOUN, PROPN, PRON, AUX, DET, ADJ, VERB, ADV,
CCONJ, PART, SCONJ, INTJ). No Word column / no German text. The companion
*_Vakuum_word_boundaries.npy holds onsets (numbers), not words. So the words
needed by the two Llama scripts are not derivable from the repo — consistent with
the README's "contact the authors".
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.