Online Learning for Wearable EEG-Based Emotion Classification.
Part of the results reproduced; minor but material deviations remained.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓No relevant deviation in data/preprocessing
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
This paper has a computational component, but its primary data is legally or ethically access-restricted — identifiable patient cohorts, rare-disease genomes, or controlled-access biobanks that cannot be openly shared. The reproduction therefore could not be attempted. That is a neutral verdict: it does not mean the result is wrong or that the authors fell short — only that, for legitimate privacy reasons, it cannot be independently checked from public data. We deliberately do NOT assign a 0–100 score here, because a low number would wrongly read as a failed reproduction.
▸Reproduction agent’s raw note
DROP (data_restricted). The paper is well described and the code exists, but EVERY dataset its computational pipeline consumes is access-gated, so none of the reported numeric results (Tables 3-6) can be regenerated autonomously. Dataset II/III (the authors' own EEGEMO deposit, Zenodo 7398263) is RESTRICTED: 'shared upon request for scientific use only', requires a research-institution email + written scientific justification + manual author approval, 156.2 GB. Dataset I (AMIGOS) is EULA-gated (sign license, email, receive credentials) and not openly downloadable. Code state: HPI-CH/EEGEMO main branch (ef875e9) ships ONLY data-collection + feature/filter utilities + dataset loaders and a 1-line README; the actual offline training/evaluation pipeline (River SRP/ARF + progressive validation that produces the tables) lives only on a secondary branch 'martemianov' (adb0c4a) as a bachelor-thesis development snapshot with a hard-coded DEBUG subject range (range(3,5)), commented-out alternative model configs, and an in-code model (SRPClassifier n_models=3 + HoeffdingAdaptiveTree + RandomOverSampler) that DIFFERS from the paper's described ARF/SRP with 5 trees -- i.e. docs_insufficient is a real co-factor. The pipeline IS in principle runnable (feature extraction, progressive validation, and model code are all present) so this is reproducible the moment a human with the EEGEMO-on-request grant or the AMIGOS EULA supplies the data; the blocker is purely data access, not method opacity. NOT attempted: any numeric regeneration (no input data). «our HPC»/VPN was also transiently down during this session but is NOT the cause of the drop -- the data restriction was verified entirely from public Zenodo + AMIGOS metadata.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-19 ⛓ 071f5661fff4
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusHow reliably an online (continuously updated) classifier can classify emotion (Valence and Arousal) from the state-of-the-art AMIGOS EEG dataset, and how accurately it can predict affective states in real-time from wearable consumer-grade EEG devices in live settings.
- ★ A lightweight online-learning pipeline trains subject-specific binary classifiers for Valence and Arousal from a streaming EEG source in real-time. method
- ★ On the AMIGOS dataset the pipeline achieves 23.9% (Arousal) and 25.8% (Valence) higher F1-Score than previous state-of-the-art work. finding
- ★ Applied to a curated wearable-EEG dataset (15 participants, two consumer devices, 16 emotional videos), the pipeline reaches mean F1-Scores of 87% (Arousal) and 82% (Valence) in an immediate-label setting. finding
- ★ The pipeline is fast enough to produce real-time predictions in a live scenario with delayed labels while being continuously updated. finding
- ★ An experimental data-collection framework (PsychoPy-based) was developed to curate neurophysiological data from a range of commercial EEG devices with live affect prediction, and released with the dataset and code. resource
- A significant discrepancy exists between performance on readily available labels versus delayed live labels, indicating more data are needed. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| EEG-based emotion classification (online learning) | AMIGOS dataset — 40 healthy human participants (13 female, ages 21–40, mean 28.3) watching 16 short movie-clip videos | emotional video stimuli | binary Valence and Arousal classification (F1-Score) from streamed EEG | Emotiv EPOC headset, 14 channels, 128 Hz, 14-bit |
| EEG-based emotion classification (immediate-label, curated) | Dataset II — 11 recruited human participants (6 female, 5 male, ages 25–42, mean 29.27); 2 discarded | 16 short emotional videos (51–150 s) from DECAF and MAHNOB-HCI datasets | binary Valence/Arousal classification F1-Score | Muse S Headband Gen 1 (4 channels, 256 Hz) and Neurosity Crown (8 channels, 256 Hz) |
| EEG-based emotion classification (live/delayed-label real-time prediction) | Dataset III — 4 human participants (1 female, 3 male, ages 25–34, mean 28.5) | 16 short emotional videos | real-time affective state prediction with delayed incoming labels | Muse S Headband and Neurosity Crown, streamed via python-osc/OSC |
| Behavioral self-assessment (PANAS) | Human participants (Datasets II/III) | pre/post experiment | Positive and Negative Affect scores on 5-point Likert scale (20 items) | PsychoPy v2021.2.3 |
| Behavioral affect self-assessment (Affective Slider) | Human participants (Datasets II/III) | after each stimulus video | continuous valence and arousal ratings (normalized 0–1, step 0.01) | PsychoPy Affective Slider |
- ▲ Higher F1-Score on AMIGOS than prior state-of-the-art for Arousal 23.9% higher
- ▲ Higher F1-Score on AMIGOS than prior state-of-the-art for Valence 25.8% higher
- – Mean Arousal F1-Score on curated wearable dataset (immediate labels) 87%
- – Mean Valence F1-Score on curated wearable dataset (immediate labels) 82%
- – Live real-time emotion prediction succeeded on incoming stream with delayed labels
- ▼ Notable drop in classification performance between immediate labels and delayed live labels
- other F1-Score 87% (Arousal), 82% (Valence) (Mean F1 on curated wearable-EEG dataset, immediate label setting)
- other 23.9% (Arousal), 25.8% (Valence) higher F1-Score (Improvement over prior work on AMIGOS dataset)
- count 16 videos (Short emotional movie clips used as stimuli (51–150 s, μ=86.7 s, σ=27.8 s))
- mean μ=29.27, σ=5.41 years (Age of Dataset II participants (6 female, 5 male))
- mean μ=28.5, σ=3.5 years (Age of Dataset III participants (1 female, 3 male))
- mean mean age 28.3, ages 21–40 (AMIGOS participants (13 female))
- count 128 Hz, 14 channels, 14-bit (Emotiv EPOC recording parameters for AMIGOS)
- count 256 Hz; 4 and 8 channels (Muse S and Neurosity Crown sampling rate and channel counts)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a real-time EEG emotion classification pipeline evaluated primarily through machine-learning performance metrics (F1-Score) on a public benchmark dataset (AMIGOS) and two newly curated wearable-EEG datasets, rather than through classical inferential hypothesis testing. Demographic and stimulus characteristics (e.g., participant age, video duration) are summarized with mean and standard deviation. Based on the provided text (which ends before the Results/Discussion sections), no formal significance tests, p-values, or multiplicity corrections are described for the classification comparisons.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| not stated (classification performance compared via F1-Score rather than a named inferential test) | comparison of the proposed online classifier vs. state-of-the-art results on the AMIGOS dataset (Arousal and Valence dimensions) | 40 participants (AMIGOS dataset) | na |
-
Classifier performance across the AMIGOS dataset and the two curated datasets is summarized using mean F1-Scores without an accompanying formal statistical comparison or p-value.↳ Could also: Paired comparisons (e.g., a paired t-test or Wilcoxon signed-rank test across subjects/folds) between the proposed pipeline and prior state-of-the-art F1-Scores — A paired significance test would quantify how likely the observed F1-Score difference is to reflect a systematic improvement versus sampling variability across participants or cross-validation folds.
-
Mean and standard deviation are used to summarize participant ages and stimulus durations.↳ Could also: Reporting the range alongside the mean/SD, or a 95% confidence interval for the mean — For small samples (e.g., 9-11 participants per dataset), a range or CI can convey the spread and precision of the estimate more transparently than SD alone.
-
Multiple binary classifiers (Valence, Arousal) and multiple online-learning algorithms are evaluated and compared on the same datasets.↳ Could also: A repeated-measures ANOVA or Friedman test across classifiers/dimensions, with a post-hoc correction (e.g., Holm-Bonferroni) for pairwise comparisons — When several classifiers or dimensions are compared on the same underlying data, a single omnibus test with corrected post-hoc comparisons can help control the overall false-positive rate across the multiple comparisons being made.
-
Model performance (F1-Score) is reported as a single mean value per condition.↳ Could also: Bootstrap resampling to obtain a confidence interval or distribution around the F1-Score — A bootstrap CI would communicate the uncertainty around the reported mean F1-Score, which can be informative given the relatively small number of participants (e.g., 9-15) contributing to Datasets II and III.
-
The discrepancy between immediate-label and delayed-label classification performance is described narratively as 'significant' without a corresponding statistical test in the provided text.↳ Could also: A formal comparison (e.g., paired t-test or non-parametric equivalent) between immediate- and delayed-label F1-Scores — A paired statistical test would let the size of the immediate-vs-delayed performance gap be assessed against expected variability, complementing the descriptive comparison already presented.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36904590
Paper: Moontaha S, Schumann FEF, Arnrich B. Online Learning for Wearable
EEG-Based Emotion Classification. Sensors 2023;23(5):2387. DOI 10.3390/s23052387.
Code: https://github.com/HPI-CH/EEGEMO (main ef875e9, branch martemianov adb0c4a)
Data: Zenodo 10.5281/zenodo.7398263 (concept 7397907) + AMIGOS (QMUL).
Reported results and whether they are pipeline-derived
The paper reports binary (high/low) valence and arousal classification with online/streaming learners (River library) under progressive validation (test-then-train). Three data settings:
| Result | Dataset | Pipeline | In scope? |
|---|---|---|---|
| Table 3 — AMIGOS LR/ARF/SRP F1+Acc (valence, arousal) | I (AMIGOS, EPOC 14ch, 40 subj) | PSD/Welch bandpower+ratios -> River ARF/SRP/LR, progressive validation | pipeline-derived, in scope but data EULA-gated |
| Table 4 — per-subject + mean F1, ARF/SRP/LR × Muse/Crown | II (EEGEMO, 9 analyzed) | same feature pipeline + per-subject progressive validation | pipeline-derived, in scope but data restricted |
| Fig 9/10 — subject-wise F1, window-length sweep (1–5 s) | I & II | same | pipeline-derived, in scope, data-gated |
| Table 5 — live delayed-label per-subject F1+Acc | III (4 subj, live) | delayed progressive validation on live stream | pipeline-derived but requires live recording + restricted data |
| Table 6 — Dataset II with artificial 86 s label delay | II | delayed progressive validation | pipeline-derived, in scope, data restricted |
| PANAS / Affective-Slider self-reports, study protocol | — | wet-lab / human-subject collection | out of scope (not a computational pipeline) |
| Comparisons to Siddharth/Topic/Miranda-Correa et al. | external | external prior work | out of scope (other papers) |
Pipelines named (per in-scope result)
- Feature extraction: Welch PSD per channel -> 5 band powers (Delta 0.5–4,
Theta 4–8, Alpha 8–16, Beta 16–32, Gamma 32–45 Hz) + total power + all 10
pairwise band ratios; relative power. (
utils/features.py,FeatureWindow.py) - Preprocessing (Dataset II/III): 50 Hz notch (IIR), 0.5–45 Hz 5th-order
Butterworth band-pass, average reference. (
utils/filter.py) - Windowing: tumbling windows of 1–5 s; one feature vector per window;
windows shuffled before offline training. (
Dataset.add_window,shuffle_data) - Models (River): Adaptive Random Forest (ARF), Streaming Random Patches (SRP, base = Hoeffding Adaptive Tree), Logistic Regression baseline; binary threshold 0.5 on the Affective Slider (≥5 for AMIGOS SAM 1–9).
- Validation: progressive (immediate) and delayed-progressive (live/Table 6);
10-fold CV for offline sanity check (
stream_csv.py:cross_validate).
Blocker (why nothing was executed)
All in-scope results require access-restricted data. See AUDIT.md and
data/dataset_profile.json. No openly downloadable data exists for any reported
table, so no numeric value was regenerated. Outcome: drop — data_restricted.
The method itself is reproducible (feature + model + validation code all present),
so a human holding the EEGEMO on-request grant or the AMIGOS EULA could run it.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a data_restricted drop: every dataset the pipeline consumes is access-gated (EEGEMO Zenodo 7398263 'on request' + 156.2 GB; AMIGOS EULA-gated), so no numeric claim (Table 4 Arousal F1 ~0.87 / Valence ~0.82, Table 3 AMIGOS 0.834/0.831) could be regenerated — q1/q2 are red on availability, not authors' defect. There is no fabrication signal; derivability and the core claim are simply untested (q5/q7 yellow), not refuted. A genuine authors-side co-factor is documented: the table-producing pipeline lives only on a thesis branch with a hard-coded DEBUG subject range and an in-code model (n_models=3 + HoeffdingAdaptiveTree) that diverges from the paper's described 5-tree ARF/SRP. Net: reproducible the moment a human supplies the gated data; blocker is access plus mild method/code under-specification, hence overall yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.