The indirect effect of mRNA-based COVID-19 vaccination on healthcare workers' unvaccinated household members.
Part of the results reproduced; minor but material deviations remained.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Any deviation was negligible
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
This paper has a computational component, but its primary data is legally or ethically access-restricted — identifiable patient cohorts, rare-disease genomes, or controlled-access biobanks that cannot be openly shared. The reproduction therefore could not be attempted. That is a neutral verdict: it does not mean the result is wrong or that the authors fell short — only that, for legitimate privacy reasons, it cannot be independently checked from public data. We deliberately do NOT assign a 0–100 score here, because a low number would wrongly read as a failed reproduction.
▸Reproduction agent’s raw note
DROP / data_restricted. The paper (Nat Commun 2022, Saaksvuori et al.) is an observational cohort study estimating direct & indirect mRNA COVID-19 vaccine effectiveness using Finnish nationwide register microdata. Code is described well enough and is fully public: 33 Stata .do files on GitHub (commit b9f0aa6), entrypoint main.do, also mirrored on Zenodo 5905898 (which is the CODE, 43 KB, NOT data despite being the listed 'data' DOI). BUT the analysis is un-runnable: every input is controlled-access individual-level Finnish national-register data (FOLK/Statistics Finland; TTR & AVOHILMO vaccination/care registers/THL; income register; encrypted person/household ID linkage), keyed on shnro/shetu and obtainable only via a Findata data-permit in a secure remote environment. Repo README states verbatim: 'Data used in the project is not publicly available, but can be requested and applied for.' No public or synthetic dataset is shipped, so 0 of 3 headline claims can be regenerated. No «our HPC» compute was submitted because there is nothing to run against. NOT attempted: data-permit application (out of scope/months-long), and no fabricated substitute data was created. This is the politically salient 'data only on request' class: public code does NOT make the result reproducible. No fabrication assessment possible (reported estimates are not independently checkable, but absence of shipped data is a transparency limitation, not evidence of fabrication).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-15 ⛓ 9e2811fbbc83
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusDo mRNA-based COVID-19 vaccines reduce SARS-CoV-2 infections not only among vaccinated individuals (direct effect) but also among their unvaccinated household members (indirect effect), and does this indirect protection extend to unvaccinated children?
- ★ mRNA-based COVID-19 vaccination of healthcare workers is associated with reduced SARS-CoV-2 infections among their unvaccinated adult household members (indirect effectiveness). finding
- ★ Vaccination provides substantial direct effectiveness in reducing SARS-CoV-2 infections among vaccinated individuals, increasing after the second dose. finding
- ★ The indirect protective effect within households is smaller for unvaccinated children/adolescents than for adults and is statistically insignificant. finding
- ★ Indirect vaccine effectiveness is estimated by comparing treatment vs control groups defined by a close contact's vaccination status, mimicking an RCT design, with randomly assigned follow-up start dates and week fixed effects to control for calendar time. method
- ★ Linking individual health records (PCR-confirmed infections, vaccination records) with full-population administrative datasets containing household IDs, occupation, and demographics enables estimation of household spillover effects. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Observational cohort study (log-binomial regression on PCR-confirmed SARS-CoV-2 infections) | Vaccinated healthcare workers in Finland | mRNA vaccine (BNT162b2 or mRNA-1273), first dose | Direct vaccine effectiveness (relative risk reduction) of PCR-confirmed SARS-CoV-2 infection by follow-up week | Finnish National Infectious Diseases Register; national vaccination and administrative full-population datasets |
| Observational cohort study (log-binomial regression) | Unvaccinated partners of vaccinated healthcare workers (same household) | Household exposure to a vaccinated adult (first dose) | Indirect vaccine effectiveness (relative risk reduction) of PCR-confirmed SARS-CoV-2 infection by follow-up week | Finnish National Infectious Diseases Register; administrative household datasets |
| Observational cohort study (log-binomial regression) | Unvaccinated children/adolescents aged 3–18 years of healthcare workers (same household) | Household exposure to a vaccinated adult (first dose) | Indirect vaccine effectiveness (relative risk reduction) of PCR-confirmed SARS-CoV-2 infection, weeks 2–5 stacked and by week | Finnish National Infectious Diseases Register |
| Observational cohort study (log-binomial regression, second-dose sample) | Double (fully) vaccinated healthcare workers and their unvaccinated partners and children | mRNA vaccine, second dose | Direct and indirect vaccine effectiveness (relative risk reduction) by week after second dose | Finnish National Infectious Diseases Register |
- ▼ Direct effectiveness in healthcare workers after second dose: 82.7% at 2 weeks and 82.7% at 8 weeks 82.7% (95% CI: 76.0%-87.5%) at 8 weeks
- ▼ Indirect effectiveness for unvaccinated partners after second dose: 39.1% at 2 weeks and 39.0% at 8 weeks 39.0% (95% CI: 18.9%-54.0%) at 8 weeks
- ▼ Direct effectiveness after first dose increases over time: 44.4% at 4 weeks to 63.0% at 12 weeks 63.0% (95% CI: 56.3%-68.7%) at 12 weeks
- ▼ Indirect effectiveness for unvaccinated partners after first dose: 16.7% at 4 weeks, 23.0% at 12 weeks 23.0% (95% CI: 6.2%-36.9%) at 12 weeks
- – Indirect effectiveness for unvaccinated children aged 3–18 years is small and statistically insignificant -16.3% (95% CI: -65.8% to 18.4%) at 2-5 weeks; 6.8% at 12 weeks
- – Indirect effectiveness for children 3–12 years not significantly different from zero -37.1% (95% CI: -127.9% to 17.5%) at 2-5 weeks; 8.1% at 12 weeks
- – Indirect effectiveness for adolescents 13–18 years statistically insignificant 4.8% (95% CI: -47.9% to 38.7%) at 2-5 weeks; 4.5% at 12 weeks
- count 265,326 healthcare workers; 128,952 unvaccinated partners; 169,148 unvaccinated children (Total study sample sizes)
- count 1471 (0.55%), 782 (0.61%), 820 (0.48%) (PCR-confirmed SARS-CoV-2 infections in healthcare workers, partners, children)
- other 82.7% (95% CI: 65.4% to 91.3%) (Direct effectiveness 2 weeks after second dose)
- other 39.1% (95% CI: -7.1% to 65.3%) (Indirect effectiveness for partners 2 weeks after second dose)
- other 44.4% (95% CI: 30.4% to 55.6%) (Direct effectiveness 4 weeks after first dose)
- count 112,496 (42.4%) received at least one dose; 63,986 (24.1%) double vaccinated (Vaccinated healthcare workers Dec 27 2020 – Apr 25 2021)
- mean 44 (SD 14), 45 (SD 12), 11 (SD 5) years (Mean ages of healthcare workers, partners, children)
- other foreign background 6.3% of sample but 23.7% of infections; practical nurses 73.3% of infections (Overrepresentation of subgroups among infections)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is an observational cohort study using nationwide Finnish administrative registers to estimate the direct and indirect (household spillover) effectiveness of mRNA COVID-19 vaccines, comparing the cumulative incidence of PCR-confirmed SARS-CoV-2 infections between treatment (vaccinated healthcare workers / their unvaccinated household members) and control groups, where controls were assigned a randomly chosen follow-up start date. Vaccine effectiveness (relative risk reduction) was estimated with covariate-adjusted log-binomial regression models on individual-week data, reported by follow-up week with 95% confidence intervals clustered at the individual level. Estimates were adjusted for calendar time (week fixed effects), age, sex, occupation, household size, ethnicity, and geographic area.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Log-binomial regression model estimating relative risk reduction (vaccine effectiveness) | Direct effectiveness in vaccinated healthcare workers (Fig. 1a, Fig. 3a) | 265,326 healthcare worker observations (first dose); 216,557 (second dose) | not stated |
| Log-binomial regression model estimating relative risk reduction (vaccine effectiveness) | Indirect effectiveness in unvaccinated partners (Fig. 1b, Fig. 3b) | 128,952 partner observations (first dose); 110,426 (second dose) | not stated |
| Log-binomial regression model estimating relative risk reduction (vaccine effectiveness) | Indirect effectiveness in unvaccinated children/adolescents aged 3-18 and by age subgroups (Fig. 2, Fig. 3c,d) | 169,148 child/adolescent observations (first dose); subgroups 105,186 (3-12y) and 63,962 (13-18y) | not stated |
-
Vaccine effectiveness (relative risk reduction) was estimated using log-binomial regression on individual-week collapsed data.↳ Could also: A Poisson or modified-Poisson regression with robust (sandwich) variance, or a Cox proportional-hazards / pooled logistic survival model on the time-to-infection data. — Poisson with robust variance is a common alternative for estimating relative risks that converges more reliably than log-binomial, while survival models would directly use the timing of infection events and censoring; reporting these alongside could show robustness of the effect estimates.
-
Statistical significance was communicated through 95% confidence intervals that exclude or include zero.↳ Could also: Reporting exact p-values alongside the confidence intervals. — Exact p-values provide a continuous measure of evidence strength that complements interval estimates, which some readers find useful for interpreting borderline results.
-
Multiple effectiveness estimates were reported across follow-up weeks and demographic subgroups, each with its own 95% CI.↳ Could also: A pre-specified family-wise or false-discovery-rate adjustment (e.g., Bonferroni or Benjamini-Hochberg) for the set of subgroup/time-point estimates. — A multiplicity adjustment would account for the number of comparisons made and is one standard way to characterize the chance of spurious findings when many estimates are presented.
-
Confidence intervals were clustered at the individual level using an endpoint transformation of the originally estimated intervals.↳ Could also: Household-level clustering of standard errors or a multilevel/mixed-effects model with household random effects. — Because the design involves household members, clustering at the household level or modeling within-household correlation directly is another standard way to reflect the data's dependence structure.
-
Estimates were adjusted for covariates within a regression framework using the observational sample.↳ Could also: Propensity-score matching, weighting (IPTW), or doubly-robust estimation in addition to regression adjustment. — Propensity-based methods provide an alternative route to covariate balance and, when combined with outcome regression, offer doubly-robust estimates that can corroborate the regression-adjusted results reported in the covariate-balance tables.
-
For children, conclusions of statistical insignificance were drawn from estimates with wide confidence intervals owing to few infection events.↳ Could also: A formal power/precision analysis or Bayesian estimation reporting the posterior probability of a meaningful effect. — These approaches help distinguish 'evidence of no effect' from 'no evidence of an effect' when event counts are small, quantifying how much uncertainty remains in the children's estimates.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
mRNA COVID-19 vaccination (first dose) provides increasing direct effectiveness against SARS-CoV-2 infection in Finnish healthcare workers, reaching 63.0% at 12 weeks.other human healthcare-worker down 2022×1papers★ This paper is the founder (earliest)
-
mRNA COVID-19 vaccination (second dose) provides 82.7% direct effectiveness against SARS-CoV-2 infection in Finnish healthcare workers at 8 weeks post-dose.other human healthcare-worker down 2022×1papers★ This paper is the founder (earliest)
-
Household exposure to a vaccinated adult does not significantly reduce SARS-CoV-2 infection risk in unvaccinated adolescents aged 13–18 years.other human unvaccinated-adolescent-13-18 none 2022×1papers★ This paper is the founder (earliest)
-
Household exposure to a first-dose-vaccinated adult provides 23.0% indirect protection against SARS-CoV-2 infection in unvaccinated adult partners at 12 weeks.other human unvaccinated-adult-partner down 2022×1papers★ This paper is the founder (earliest)
-
Household exposure to a doubly vaccinated adult provides 39.0% indirect protection against SARS-CoV-2 infection in unvaccinated adult partners at 8 weeks post second dose.other human unvaccinated-adult-partner down 2022×1papers★ This paper is the founder (earliest)
-
Household exposure to a vaccinated adult does not significantly reduce SARS-CoV-2 infection risk in unvaccinated children aged 3–12 years.other human unvaccinated-child-3-12 none 2022×1papers★ This paper is the founder (earliest)
-
Household exposure to a vaccinated adult does not significantly reduce SARS-CoV-2 infection risk in unvaccinated children aged 3–18 years.other human unvaccinated-child-3-18 none 2022×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope assessment — pmid-35246536
Title: The indirect effect of mRNA-based COVID-19 vaccination on healthcare
workers' unvaccinated household members.
Venue: Nat Commun 2022 · DOI 10.1038/s41467-022-28825-4 · PMCID PMC8897446
Code: https://github.com/covidhealth/indirect-effects-covidvaccinations
(pinned commit b9f0aa60e031860a1058ccfc1b1841ea8afd9e19, 2022-01-26)
"Data" DOI: zenodo:10.5281/zenodo.5905898 — see note below.
What kind of study is this?
An observational cohort / epidemiological study estimating the direct and indirect effectiveness of mRNA COVID-19 vaccines using Finnish nationwide administrative register microdata linked by encrypted household & person IDs. The "pipeline" is an econometric estimation pipeline in Stata (event-study / linear probability + logit regressions around vaccination dates), not a bioinformatics pipeline.
Pipeline-derived results (would be in scope)
The headline causal estimates are entirely pipeline-derived (they come from the
Stata .do estimation scripts), so in principle they are in scope:
| result | producing script(s) |
|---|---|
| Indirect effectiveness for adults, 2 wks & 8 wks post 2nd dose (abstract) | linear_main_and_spouse_estimation.do, logit_main_and_spouse_estimation*.do |
| Children indirect effect (smaller, n.s.) | linear_children_estimation.do, logit_children_estimation*.do |
| Descriptive / sample-size tables | descriptive_stats*.do, sample_sizes.do |
| Weekly-infection & cumulative figures | weekly_infections*.do, cumulative_graph_revised.do, coefplots_appendix_lpm.do |
Everything else (study design, ethics, data-access governance) is non-computational and out of scope.
Why this cannot be reproduced — DATA IS RESTRICTED (the blocker)
The pipeline above is un-runnable because none of its inputs are obtainable:
-
No data is published. The repo README states verbatim:
"Data used in the project is not publicly available, but can be requested and applied for. Correspondence should be addressed to Lauri Sääksvuori."
-
The Zenodo DOI is the CODE, not data. zenodo:5905898 is titled "… : Code" and contains a single 43 KB file
indirect-effects-covidvaccinations-v.1.0.0.zip— a snapshot of the same GitHub repo. There is no data deposit anywhere. -
Inputs are protected individual-level national registers. The
.dofilesuse/appendnamed register extracts keyed on encrypted personal/household identifiers (shnro,shetu):folk_20112019_tua_perus20tot_2— FOLK module, Statistics Finlandttr2504_new— national infectious-disease register (THL / TTR)avoh_rokotus_*,avohilmo_*— vaccination & primary-care register (THL / AVOHILMO)Tulorek82020— income register (Tulorekisteri), occupation codesshetushnrolinkki*— encrypted ID linkage keys
These are controlled-access Finnish health/administrative microdata released only under a Findata / Statistics Finland data-permit application with a secure-remote-access environment. They are not downloadable; obtaining them requires a research permit (months-long process, project-bound, non-shareable).
Verdict
DROP — drop_reason = data_restricted. Code is fully public and the analysis
is clearly described, but every input is controlled-access registry microdata
available only on permit application. No public/synthetic test dataset is shipped,
so 0 of the pipeline outputs can be regenerated. No «our HPC» compute is warranted:
there is nothing to run the Stata scripts against. We do not fabricate
substitute data.
This is the politically salient class the study tracks explicitly: data "behind a paywall or only available on request." The code's existence does not make the result reproducible.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a textbook data_restricted drop: the analysis code is fully public (33 Stata .do files on GitHub @b9f0aa6, mirrored on Zenodo 5905898 — which is the code, not data), but every input is permit-only Finnish national-register microdata, so 0 of 3 headline claims (adults 39.1% / 39.0%, children smaller & n.s.) can be regenerated. The limitation lies on the data-provision side, not our methodology and not a technical seed/version issue. Per the fair-handling rule, inability to compare due to restricted access is not an authors' defect or fabrication signal — q5/q7 stay yellow (not assessable / untested) and there is no evidence of fabrication, yielding an overall yellow transparency-limited verdict.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.