A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (1:1 on the headline result). PNAS 2022 Lake Geneva hybrid empirical(EDM)+parametric(Simstrat) DO model; authors' own R repo SugiharaLab/Geneva_Hybrid @91739e3 driven by legacy rEDM 0.7.x on shipped monthly limnological data (Zenodo 6587597 == repo snapshot, 545 monthly records). Environment rebuilt on «our HPC»: conda R 4.1.3 + dplyr 1.0.10 + rEDM 0.7.3 (compiled from CRAN archive); ran after a minor Linux path-case fix (DATA/LIB symlinks). The two headline numbers reproduce essentially exactly: hybrid DOB hindcast rho=0.898 (paper 0.89) and MAE=0.951 mg/L (paper 0.94); parametric baseline correctly far lower (0.651). CCM tables (C4/C5), Fig1D mEDM skill (C6), and greedy-vs-baseline models (C7/C8) all reproduce -- and the shipped precomputed mEDM caches recompute BIT-FOR-BIT from code+data (max diff ~1e-16), i.e. no fabrication in the deposited intermediates. The TP x warming hypoxia scenario grid (C10, 138 hybrid runs) reproduces the full monotonic structure with corners within ~1.5-8 percentage points of the figure-read values. NOT separately quantified: C9 (S-map coefficient series, ran but not isolated) and C11 (seasonal depletion-rate metric, scenario data saved). OUT OF SCOPE (not attempted): the Simstrat hydrodynamic model itself (parametric/Windows-binary; its outputs are shipped and consumed as fixed EDM inputs) and field/wet-lab monitoring (external data collection).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 69assessed: 2026-06-21 ⛓ 3bc02c749203
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-21
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper asks whether the major interdependencies among lake physics, biogeochemistry, and ecology driving deep-water dissolved oxygen (DOB) in Lake Geneva can be resolved without building a model too complex to reliably fit or interpret, particularly given confounded, nonstationary effects of reoligotrophication and climate change.
- ★ A hybrid model combining empirical dynamic modeling (EDM/S-map) for biogeochemical sink terms with the equation-based Simstrat hydrodynamic model for physical source terms produces substantially better historical DOB forecasts than conventional parametric models. method
- ★ The effects of total phosphorus (TP) on chlorophyll (CHL) and of CHL on DOB are state-dependent (nonlinear) rather than fixed, changing systematically as reoligotrophication progresses. finding
- ★ A moderate 3°C increase in air temperature would impact Lake Geneva water quality on the same order of magnitude as the eutrophication that occurred over the previous century. finding
- ★ Convergent cross-mapping (CCM) identifies air temperature, lake temperature, thermal structure, CHL, and phosphorus as causal drivers of DOB. finding
- ★ Nonlinear (state-dependent) S-map models with θ > 0 substantially outperform the linear multivariate autoregressive (MAR) equivalent (θ = 0) in predicting DOB. finding
- Multiple modes of management intervention (both nutrient control and climate mitigation) are likely necessary to achieve a healthy lake. finding
- S-map regression provides a general empirical dynamic modeling framework for quantifying time-varying (Jacobian-like) interaction strengths among ecosystem variables. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Convergent cross-mapping (CCM) causality analysis | Lake Geneva long-term limnological time series | none (observational) | causal coupling among DOB and candidate drivers (air temp, lake temp, thermal structure, CHL, phosphorus) | Empirical Dynamic Modeling (EDM) |
| Multivariate S-map (nonlinear state-space) regression | Lake Geneva time series (hmix, Tsurf, Tatm, Q, CHL, TPsurf, TPlake, DOB) | none (sequential variable addition to embedding) | forecast skill (Pearson correlation between observed and predicted DOB) as function of nonlinearity parameter θ | EDM / S-map |
| S-map partial derivative (Jacobian coefficient) estimation | Lake Geneva time series | none (state-dependent analysis across TP levels) | ∂CHL/∂TP and ∂DOB/∂CHL as functions of TPlake state | EDM / S-map |
| 1D hydrodynamic modeling | Lake Geneva water column | none / atmospheric forcing input | daily thermocline depth prediction accuracy | Simstrat model |
| Hybrid empirical–parametric scenario modeling | Lake Geneva (simulated 6-month stratified periods) | fixed-background TP concentrations (reoligotrophication states) and air-temperature increase scenarios | DOB depletion rate over stratified season | Simstrat + EDM hybrid model |
| Sediment core analysis | Lake Geneva sediments | none (historical reconstruction) | historical hypoxia status relative to TP levels | — |
- – Simstrat reproduces daily thermocline depth across four decades with low error nrmse = 6.7% (rmse = 21 m)
- ▲ Adding biogeochemical variables (CHL, TP surf, TP lake) to the physical embedding sequentially improves multivariate DOB forecast skill
- ▲ Nonlinear S-map models (θ > 0) predict DOB much better than the linear MAR model (θ = 0)
- – CHL showed essentially no response to TP when TP lake was high (1980s); phosphorus limitation of CHL only emerged once TP lake fell below a threshold threshold ~40 µg·L⁻¹ (near-zero effect at TP lake > 60 µg·L⁻¹)
- – DOB was insensitive to CHL during the eutrophic period but developed a consistent negative response to CHL as reoligotrophication proceeded threshold ~50 µg·L⁻¹ TP lake
- – Lake Geneva sediment cores show the lake was not hypoxic prior to 1945 TP < 10 µg·L⁻¹
- – Hybrid model scenarios show DOB depletion rate is high at high background TP and decreases with lower TP, with warming scenarios producing effects comparable to historical eutrophication
- other nrmse = 6.7%, rmse = 21 m (Simstrat thermocline depth prediction accuracy over four decades)
- other TP lake < 40 µg·L⁻¹ (threshold below which CHL begins responding to phosphorus limitation)
- other TP lake < 50 µg·L⁻¹ (threshold below which DOB shows consistent negative response to CHL)
- other TP < 10 µg·L⁻¹ (phosphorus level prior to 1945 when lake was not hypoxic (sediment core evidence))
- other TP lake > 60 µg·L⁻¹ (1980s phosphorus level at which mean annual effect of TP on CHL was essentially zero)
- other DOB_init = 7.5 and 6 mg·L⁻¹ (initial dissolved oxygen conditions used in hybrid model stratified-period scenario experiments)
- other 3°C air temperature increase (climate scenario whose predicted water-quality impact is comparable to 20th-century eutrophication)
- other P<36 µg·L⁻¹ (phosphorus level below which initial TP-reduction actions began to affect algal biomass and DOB)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper uses a two-stage empirical/parametric hybrid modeling approach rather than classical hypothesis testing: first, Empirical Dynamic Modeling (EDM), including convergent cross-mapping (CCM) for causal inference and multivariate S-map regression for state-dependent nonlinear prediction, is used to identify and quantify time-varying relationships among lake variables (TP, CHL, DO_B, physical drivers); second, this empirical model is combined with a deterministic, equation-based hydrodynamic model (Simstrat) to forecast dissolved oxygen decades into the future under different climate/nutrient scenarios. Model performance is reported primarily as forecast skill (Pearson correlation between observed and predicted values) and error metrics (nrmse, rmse) rather than through group-comparison significance tests.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Convergent cross-mapping (CCM), a nonlinear time-series causality detection method | Causal driver identification for DO_B (Fig. 1 B and C, SI Appendix Fig. S2, Table S1) | — | not stated |
| Multivariate S-map (sequential locally weighted global linear map) regression, tuned by nonlinearity parameter θ | Prediction of DO_B and CHL from embeddings of driver variables (Fig. 1D, Fig. 2, SI Appendix Figs. S3, S5, S6) | — | not stated |
| Comparison of linear MAR model (S-map with θ=0) versus nonlinear S-map (θ>0) forecast skill | Fig. 1D, evaluating predictability of DO_B across embeddings | — | not stated |
| Forecast skill quantified as Pearson correlation between observed and predicted values | Multivariate EDM/S-map predictions of DO_B (Fig. 1D) and hybrid model validation against historical data | — | na |
| Deterministic hydrodynamic model (Simstrat) validated with normalized RMSE and RMSE | Prediction of daily thermocline depth over four decades | four decades of daily thermocline depth data (exact n not given) | not stated |
-
Causal relationships among lake variables were assessed with convergent cross-mapping (CCM), a nonlinear, state-space-based causality detection method.↳ Could also: Granger causality testing (or its nonlinear extensions) is a standard alternative for inferring predictive causal relationships in time series. — Granger causality is widely used and well understood for linear or weakly nonlinear systems, and comparing its results with CCM could help characterize how much of the detected causality depends on nonlinear state-dependence versus simpler linear predictive relationships.
-
State-dependent relationships (e.g., effect of TP on CHL, CHL on DO_B) were quantified via S-map regression coefficients without reported confidence intervals or uncertainty bounds.↳ Could also: Bootstrap resampling (e.g., block bootstrap for time series) could also be used to generate confidence intervals around the time-varying S-map coefficients. — This would let readers gauge the sampling uncertainty of the estimated Jacobian-like coefficients shown in Fig. 2, complementing the point estimates already presented.
-
Model predictability was reported as forecast skill (Pearson correlation between observed and predicted DO_B) across different embeddings and θ values.↳ Could also: Reporting additional accuracy metrics such as RMSE, MAE, or a formal cross-validated log-likelihood alongside correlation could also be used. — Correlation-based skill can be insensitive to systematic bias or scale errors, so pairing it with an absolute error metric can give a fuller picture of predictive accuracy, similar to how nrmse/rmse were already used for the Simstrat model.
-
The nonlinearity tuning parameter θ was varied to compare linear (θ=0, equivalent to a MAR model) versus nonlinear S-map fits, with the best θ apparently chosen by inspection of the forecast-skill curve (Fig. 1D).↳ Could also: A formal leave-one-out or blocked time-series cross-validation procedure with a defined selection criterion (e.g., maximum out-of-sample skill with reported variability across folds) could also be used to select θ. — This would make the θ selection procedure fully reproducible and would allow quantification of how sensitive the chosen embedding is to the particular data split used.
-
The hybrid empirical-parametric model's improved performance is described qualitatively as leading to 'substantially better forecasts' relative to the parametric-only approach.↳ Could also: A formal model comparison framework, such as comparing information criteria (AIC/BIC where applicable) or a paired test on out-of-sample forecast errors between the hybrid and parametric-only models, could also be used. — This would provide a quantitative, statistically grounded comparison of the two modeling approaches rather than a descriptive comparison of forecast plots.
-
Scenario projections (different TP and air-temperature combinations) are presented as point trajectories of DO_B depletion (Fig. 3) without stated uncertainty ranges.↳ Could also: Ensemble or Monte Carlo propagation of parameter and initial-condition uncertainty through the hybrid model could also be used to generate prediction intervals around the scenario trajectories. — This would communicate the range of plausible outcomes under each climate/nutrient scenario, which can be useful for management decisions that depend on the confidence in projected hypoxia risk.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35733249
Paper: Deyle ER, Bouffard D, Frossard V, Schwefel R, Melack J, Sugihara G (2022). "A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures." PNAS 119(26):e2102466119. PMID 35733249 · PMCID PMC9245694 · DOI 10.1073/pnas.2102466119
Code: https://github.com/SugiharaLab/Geneva_Hybrid @ commit
91739e31e3ea448cebea97d77e23e90fab4de693 (resolved 2026-06-19, latest HEAD).
Authors' own repo (P16 not relevant). 100% R.
Data: Zenodo 10.5281/zenodo.6587597 + data also shipped inside the repo.
Nature of the study
Empirical Dynamic Modeling (EDM: simplex, S-map, CCM, multivariate block_lnlp,
multiview via the legacy rEDM 0.7.x API — repo loads library("rEDM", lib.loc="./LIB"),
"the archived version of rEDM") applied to ~monthly limnological time series of
Lake Geneva (CARRTEL/SOERE monitoring), combined with a parametric hydrodynamic
model (Simstrat) to build a "hybrid" forecast of deep-water dissolved oxygen (DOB).
Not a bioinformatics pipeline — but the reproducible computational core is the EDM
analysis run with a published third-party/legacy R package on the paper's own data,
which the brief explicitly counts as equally valid.
IN SCOPE — pipeline-derived (EDM) results we attempt to reproduce
Pipeline = rEDM 0.7.x driven by the repo's Rmd notebooks
(PNAS_SI_notebook.Rmd, analysis_chl.Rmd, analysis_po4_epi.Rmd).
| id | result | where in paper | how computed |
|---|---|---|---|
| C1 | Hybrid DOB hindcast skill ρ = 0.89 (Obs vs Hybrid, 1981–2017) | Fig 4A, text | hybrid_model_v1() + EDM-simulated chl/PO4 + Simstrat physics; cor() in SI notebook L1200 |
| C2 | Hybrid DOB hindcast MAE = 0.94 mg/L | Fig 4A, text | mean(abs(Obs-Hybrid)) SI L1202 |
| C3 | Parametric (Simstrat/"Robert") DOB skill — comparison baseline | Fig 4A | cor(Obs, Parametric) SI L1201; directly checkable from two shipped CSVs |
| C4 | Table S1: CCM cross-map skill of DO_delta drivers (ρ, sorted) | Table S1 | ccm() per driver, E* by max ρ, tp=-floor(E*/2) |
| C5 | Fig S4 / Table S5: CCM skill from chl and PO4_epi | Fig S4 panels | ccm() as C4 |
| C6 | Fig 1D: multivariate EDM forecast skill (ρ vs θ) for DOB, sequential variable addition | Fig 1D | do_mEDM_models() over 5 embeddings, S-map θ sweep |
| C7 | mEDM "greedy" model for chl (embedding, θ, ρ, mae, rmse) vs baselines (univar/seasonal/multiview) | text/SI, supports hybrid | do_mEDM_greedy(), multiview(), s_map() |
| C8 | mEDM "greedy" model for PO4_epi vs baselines | SI | as C7 |
| C9 | Fig 2 state-dependent S-map coefficients: ∂CHL/∂TP sign change near TP≈40 µg/L; ∂DOB/∂CHL threshold | Fig 2, text | block_lnlp(..., save_smap_coefficients=T, theta=6) |
| C10 | Scenario hypoxia: % time DOB<4 mg/L = 55/20/55/85 % across TP×ΔT scenarios | Fig 4B/C, text | iterative hybrid_model_v1() over 45 TP × 3 ΔT (heavier) |
| C11 | Seasonal deep-O2 depletion range 0–0.017 mg/L/day from EDM scenarios | text | derived from scenario forecasts |
Priority / quick floor: C3 (CSV-only), C1, C2, C4, C6 are the low-hanging, clearly-specified outputs. C7–C11 are the harder "keep going" targets.
OUT OF SCOPE — not pipeline-derived (not attempted, or inputs-shipped only)
- Simstrat hydrodynamic runs (thermocline NRMSE 6.7 %, rmse 21 m; deep-temp
RMSE ~0.2 °C). This is the parametric half: a separate equation-based physics
model (Fortran/Windows binary), not EDM. Its outputs (
T_out.dat, scenario runs) are shipped in the repo, so the EDM/hybrid steps consume them as fixed inputs. We do NOT re-run Simstrat. (Could optionally cross-check shipped Simstrat temp vs observed, but that validates the parametric model, not the EDM pipeline.) - Field monitoring / wet-lab measurement (CARRTEL sampling of TP, SRP, CHL, DOB; MeteoSwiss met data; Rhône discharge). External data collection — out of scope.
- **Robert/ob
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.