Integrating herbarium specimen observations into global phenology data systems.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough for the CORE pipeline; reproduced 1:1 on the paper's own data. The PPO data pipeline (ppo-data-pipeline@3b948343 -> ontology-data-pipeline@ea47c4ab, ROBOT v1.8.1, PPO ontology) was run unmodified on the Zenodo Prunus serotina herbarium input (10.5281/zenodo.1473702, sha256 20c3c023...): it validated + triplified 2280 specimen annotations, ELK-reasoned them against the PPO, and emitted a 2280-row reasoned CSV with PPO trait classes (open/unopened/senesced flower presence, fruit presence). Two reported counts were checkable against the shipped data: 570 scored specimens reproduced EXACTLY, and 203 open-flower annotations reproduced as 210 (delta 7, ~3.4%; 210 distinct specimens, all complete, no dedup effect -> most likely a dataset-version/undocumented-filter difference, NOT a fabrication signal). NOT attempted (the hard 20% / out of scope): the 766 NPN observations and the 969 herbarium+NPN total (require a live NPN-API harvest, not the Zenodo dataset), and the flowering linear regression (F=105.2, adj R2=0.459) which ships NO analysis code in the repo or Zenodo and is therefore not independently verifiable from released artifacts (flagged). Known repo gap: projects/herbarium/preprocessor.py imports a missing AbstractPreProcessor base module, so the herbarium pre-processing entrypoint is not runnable as shipped; its deterministic transformation was re-implemented while the heavy RDF/ELK reasoning used the repo's unmodified engine.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 60assessed: 2026-06-16 ⛓ c80f047d2af6
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan phenology observations from herbarium specimens (portions of plants) be modeled and integrated with whole-plant field observation network data through the Plant Phenology Ontology (PPO) and its pipelines, and does adding such historical data improve detection of phenology change?
- ★ A new PPO release adds terms and properties to relate observations of parts of plants to whole plants, enabling integration of herbarium phenology data with field observation data. resource
- ★ The new class 'portion of a plant' plus object properties 'is or was part of', 'generated from', and 'quality datum of' allow phenology observations of plant parts to be translated into whole-plant inferences without losing accuracy. method
- ★ The data integration pipeline was extended with logical mapping rules that translate part-of-plant observations into whole-plant data items (CSV→RDF→inference→CSV). method
- ★ Ingesting Prunus serotina herbarium annotations provides proof of concept that herbarium and observational phenology data can be integrated and served via the Global Plant Phenology Portal. resource
- ★ Integrating herbarium records extends temporal coverage and reveals that P. serotina flowering has steadily accelerated over the past ~125 years. finding
- The framework could be expanded to include data from photograph-based citizen science platforms. finding
- Adding new terms and properties required changing axioms for well over 100 terms in the PPO. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Ontology development / knowledge representation (PPO extension) | Plant Phenology Ontology (software/ontology) | other (added new class 'portion of a plant' and object properties) | logical axioms and inferences relating plant parts to whole plants | OWL/Plant Ontology, Information Artifact Ontology, GitHub release v2019-01-16 |
| Phenological scoring of herbarium specimen images (first-order phenology scoring) | Prunus serotina (black cherry) imaged herbarium specimens | none (observation) | presence/absence of unopened flowers, opened flowers, senesced flowers, and fruits; 570 images scored | iDigBio imaged specimen records; NPN reporting standards |
| Data integration pipeline processing | P. serotina herbarium test data set integrated into Global Plant Phenology knowledge base | none | CSV input converted to RDF triples, inferencing run, output CSV of whole-plant data | ontology-data-pipeline / ppo-data-pipeline (GitHub) |
| Multiple linear regression (flowering time analysis) | P. serotina, North America; integrated NPN + herbarium observations of opened flowers | none | day_of_earliest_flowering vs year, latitude, and year*latitude interaction | — |
| Linear regression for data-source bias | P. serotina, NPN vs herbarium data, years 2007–2018 | none | day_of_earliest_flowering vs latitude and data_source (spatially aggregated to 0.1-degree grid cells) | — |
- ▼ P. serotina flowering times in North America have steadily accelerated since 1873, significant after controlling for latitude −0.913 days per year; adjusted R2=0.459
- – Overall latitude*year regression model was statistically significant F=105.2 (3 and 366 df), P<0.001
- – Significant year*latitude interaction indicates phenological shifts vary by geographic location coefficient 0.0186, P=0.0218
- ▼ Latitude effect on earliest flowering day −0.330 days per degree, P=0.0400
- – No significant difference in observation dates between NPN and herbarium sources when controlling for latitude mean difference 3.13 days, P=0.695
- – Adding herbarium records extended earliest opened-flower records for P. serotina from 2007 back to 1875 969 total observations (203 herbarium + 766 NPN)
- – 570 herbarium specimen images of P. serotina were scored for phenological states 570 images
- other F = 105.2 (3 and 366 df), P < 0.001, adjusted R2 = 0.459 (overall latitude*year multiple linear regression model)
- fold_change −0.913 days per year (year coefficient, P=0.0054, earlier flowering trend)
- fold_change −0.330 days per degree (latitude coefficient, P=0.0400)
- pvalue P = 0.0218 (latitude*year interaction coefficient = 0.0186)
- mean 3.13 days, P = 0.695 (estimated mean difference between NPN (n=178) and herbarium (n=16) observation dates)
- count 969 observations (203 herbarium, 766 NPN) (opened flower observations for P. serotina dating back to 1875)
- count 570 images (P. serotina herbarium images scored)
- count over 100 terms (PPO terms whose axioms had to be changed)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper's primary statistical approach was observational regression modeling of spatially aggregated phenology records. The main analysis fit a multiple linear regression with earliest day of flowering as the response and year, latitude, and their interaction as predictors; a second linear regression (response: earliest day of flowering; predictors: latitude and data source, restricted to 2007–2018) was used to check for observation-date bias between the two data sources. Results were reported as overall model F and P values, an adjusted R-squared, and a table of coefficient estimates with P values, along with an estimated mean difference for the bias check.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Multiple linear regression (OLS), with year × latitude interaction; overall fit via F-test | Effect of year and latitude on day_of_earliest_flowering for P. serotina (Fig. 3, Table 2) | residual df reported as 3 and 366 (F = 105.2 [3 and 366 df]); n not stated directly | not stated |
| Linear regression with predictors latitude and data_source | Test for observation-date bias between NPN and herbarium data, 2007–2018 | NPN n = 178, herbarium n = 16 | not stated |
-
Earliest flowering day was modeled with ordinary multiple linear regression including a year × latitude interaction.↳ Could also: A linear mixed-effects model with random effects for grid cell, year, or collecting institution could also be fit. — A mixed model would also account for repeated structure and non-independence among aggregated grid cells and sources, and could partition variance among spatial/temporal groupings.
-
Data were spatially aggregated to 0.1-degree grid cells, using the single earliest flowering date per cell per year for modeling.↳ Could also: Models on the un-aggregated records (e.g., with cell as a random or fixed effect), or quantile regression on the lower tail of flowering dates, could also be used. — These alternatives would also use more of the available observations and can directly model the 'earliest' (extreme) date, which an aggregated minimum approximates.
-
Coefficient estimates were reported with P values and an adjusted R-squared.↳ Could also: Reporting 95% confidence intervals and standard errors alongside the estimates could also be done. — Confidence intervals would also convey the precision of each effect and the range of plausible slopes, complementing the point estimates and P values.
-
Source bias was assessed by including data_source as a predictor and reporting an estimated mean difference with a P value, noting the unbalanced sample sizes (NPN n = 178, herbarium n = 16).↳ Could also: An equivalence test (e.g., TOST) or a power/sensitivity analysis for the small herbarium n could also be presented. — An equivalence framing would also let one describe whether sources are similar within a pre-specified margin, which a non-significant difference test alone does not establish, especially with small n.
-
Overall model significance was summarized with an F-test and adjusted R-squared.↳ Could also: Model comparison via AIC/BIC or likelihood-ratio tests (e.g., with vs. without the interaction) could also be reported. — Information criteria would also provide a basis for comparing candidate model structures and for justifying inclusion of the interaction term.
-
Linear (Gaussian) regression was used for day-of-year flowering responses.↳ Could also: Diagnostic checks of residual normality/heteroscedasticity, or circular/Generalized models suited to day-of-year data, could also be applied. — These would also confirm that linear-model assumptions hold for calendar-day responses and accommodate any boundary or seasonal-wrapping behavior in the dates.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Integrating herbarium specimens extended Prunus serotina opened-flower records from 2007 back to 1875, yielding 969 total observations (203 herbarium + 766 NPN).imaging prunus serotina north-america up 2019×1papers★ This paper is the founder (earliest)
-
Prunus serotina earliest flowering day decreases with increasing latitude (−0.330 days per degree N, P=0.0400).other prunus serotina north-america down 2019×1papers★ This paper is the founder (earliest)
-
Prunus serotina earliest flowering time has significantly advanced at ~0.913 days/year across North America since 1873, after controlling for latitude (adj. R²=0.459).other prunus serotina north-america down 2019×1papers★ This paper is the founder (earliest)
-
Prunus serotina phenological shift rates vary geographically, with a significant year×latitude interaction (coefficient 0.0186, P=0.0218).other prunus serotina north-america mixed 2019×1papers★ This paper is the founder (earliest)
-
Overall regression model for Prunus serotina flowering time (year, latitude, year×latitude) was statistically significant (F=105.2, df=3,366, P<0.001).other prunus serotina north-america 2019×1papers★ This paper is the founder (earliest)
-
No significant difference in Prunus serotina earliest flowering dates between NPN citizen-science and herbarium data sources when controlling for latitude (mean difference 3.13 days, P=0.695).other prunus serotina north-america none 2019×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-30937223
Paper: Brenskelle L, Stucky BJ, Deck J, Walls R, Guralnick RP. Integrating herbarium specimen observations into global phenology data systems. Appl Plant Sci. 2019;7(3):e1231. doi:10.1002/aps3.1231 · PMCID PMC6426164.
Code: https://github.com/biocodellc/ppo-data-pipeline (partner pre-processing +
config) which calls https://github.com/biocodellc/ontology-data-pipeline (the
generic triplify → ELK-reason → SPARQL → CSV engine), against the Plant
Phenology Ontology (PPO) config/ppo.owl.
Data: Zenodo 10.5281/zenodo.1473702 — a single file
test_Prunus_serotina_herbarium_input.csv (724 KB): the herbarium specimen
phenology annotations for Prunus serotina used as the paper's worked example.
Nature of the paper
This is a methods / data-systems demonstration paper. Its reproducible, pipeline-derived content is the PPO data pipeline that converts plant-portion phenology annotations into whole-plant phenological-trait records via OWL/RDF reasoning. The reported numbers are a Prunus serotina worked example.
Reported quantitative values (candidate claims)
| value | location | derivable from shipped data/code? |
|---|---|---|
| 570 herbarium images scored for P. serotina | Results | upstream image scoring (CrowdCurio) → the Zenodo input rows |
| 203 herbarium specimen annotations with open flowers | Results | YES — count in the Zenodo dataset / pipeline output |
| 766 NPN field observations | Results | NO — separate live NPN API fetch (projects/npn/data_fetcher.py) |
| 969 total opened-flower observations (1875–2018) | Results | = 203 (herbarium) + 766 (NPN); herbarium part in scope |
| linear model F=105.2, df (3,366), adj R²=0.459, p<0.001 | Results | NO — no statistical-analysis script is shipped in repo or Zenodo |
IN SCOPE (attempted)
- PPO pipeline runs (engine reproduction). Build the env + ROBOT v1.8.1 and
run
ontology-data-pipeline/pipeline.pyend-to-end on the repo's owntest_data/data.csv→ confirm it triplifies, reasons (ELK), and emits a reasoned CSV. Proves the published tool reproduces on «our HPC». - Herbarium open-flower count = 203 (data-derived claim). Take the published Zenodo P. serotina herbarium input and count annotations recording opened flowers present → compare to the reported 203. This reproduces a reported value directly from the paper's own published dataset. Also: total annotations (vs 570) and observation year span (vs 1875–2018).
- Stretch (best-effort): push the P. serotina data through the full pipeline (pre-process → triplify → reason → SPARQL) and count inferred open-flower presences in the reasoned output; compare to 203.
OUT OF SCOPE (not attempted, with reason)
- 766 NPN observations / 969 total — require a live NPN-API harvest
(
data_fetcher.py) whose result depends on the NPN database state at fetch time; not the Zenodo dataset. Hard-20%; skipped. - Linear regression (F=105.2, R²=0.459, df 3,366) —
no_code: no regression / statistical-analysis script is shipped in the repo or the Zenodo record. Not reproducible from the shipped artifacts; flagged as not-independently-verifiable. - 570 image scores — produced by manual/crowd image scoring (CrowdCurio), not a computational pipeline; out of scope (non-pipeline).
Known incompleteness in the shipped code (honest note)
projects/herbarium/preprocessor.py imports from preprocessor import AbstractPreProcessor, but no such base module exists in the repo — so the
herbarium pre-processing entrypoint is not runnable as shipped. The core engine
(ontology-data-pipeline) IS complete and runnable. For the stretch goal the
herbarium pre-processing transformation (fully readable in that file) is
re-implemented deterministically; the heavy/interesting computation (RDF + ELK
reasoning) uses the repo's unmodified code.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The core contribution — running the PPO data pipeline (triplify→ELK reason→SPARQL→CSV) on the authors' own Zenodo Prunus serotina herbarium data — reproduced cleanly: C1=570 exact, 2280 annotations reasoned end-to-end, and C2=210 vs reported 203 (~3.4%, magnitude/direction intact, no fabrication signal). The discrepancy and the unattempted parts are not failures of our method: the C2 gap most likely reflects a Zenodo-vs-manuscript dataset-version/filter difference, while C4/C5 (NPN, 766/969) need a live NPN API and C6 (regression F=105.2, adj R²=0.459) ships no analysis code at all, so it is not derivable from released artifacts (authors-side unverifiability, flagged but not fabrication). Net: a solid partial reproduction with explainable deviations; the central methodological claim holds, the downstream statistical claim is untestable.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.