Chemist: A Domain-Specific Language by Chemists for Chemists.
The main results reproduced: recomputed values matched the published ones within tolerance.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce 1:1, and it does. The Chemist DSL paper (J Phys Chem A 2025) is mostly software-architecture; its ONLY quantitative results are Table 1 / Fig 8 (Section 4.3): an ERI absolute-precision study giving the mean (over 10 dimer separations) MaxAE and MAE of the STO-3G two-electron-integral tensor error e_I(tau)=I(tau)-I(unscreened) for He and H2 dimers vs Libint's absolute-precision parameter tau, plus the derived alpha=error/tau. Reproduced with the ACTUAL Libint engine (conda-forge libint 2.13.1; same black-box primitive screening, SI Eqs 11-13, as the paper's pinned 2.11.0 = zenodo 10.5281/zenodo.15278743) via engine.set_precision(tau_mEh*1e-3) on the SI geometries (He R=3..12 bohr, H2 edge-gap 3..12 bohr, monomer bond 1.39839 bohr). RE-VERIFIED THIS ROOM: driver rebuilt + rerun from scratch on «our HPC» compute node n094 as SLURM «job» (16:08:38 UTC); results BIT-IDENTICAL to the prior run, confirming determinism. Grading over 40 Table-1 claims: 28 EXACT (all 20 absolute MaxAE/MAE values match to the full printed 6-decimal precision) + 12 WITHIN-TOL (alpha ratios match to 6-7 sig figs; residual differences at the 7th significant figure, relative <1e-6, attributable to paper rounding from unrounded internals / libint 2.11.0 vs 2.13.1) + 0 mismatch. Prior cross-checks (archived): faithful Python reimplementation of the SI screening with analytic STO-3G ERIs, and the unscreened reference tensor matching PySCF int2e to ~1e-15 Eh. NO evidence of fabrication - the actual tool reproduces every value on the shipped geometries. The earlier room was requeued only because a score heuristic flagged the single non-pinnable Fig-9 claim as partial, not because of any error. NOT attempted: full UP-enabled NWChemEx SCF energies+uncertainties sigma_SCF (unpublished, non-performant proof-of-concept driver, not released); Fig 9 absolute curve values (plot only, no tabulated numbers).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 93assessed: 2026-06-16 ⛓ 4ef187cc9823
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- not recorded
- Assessed by
- —
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetQuantum chemistry (QC) software complexity — and the near-total absence of interpackage (reusable, cross-software) modules for computationally expensive routines like Fock builds — stems from the difficulty of designing simple interfaces for such routines using general-purpose programming language types, and that a domain-specific language (DSL) built on chemistry/physics abstractions can solve this design problem while preserving performance.
- ★ Interpackage modules are rare for the most computationally expensive QC algorithms (integral transformations, Fock builds, sigma vector formation) because their APIs are difficult to define using general-purpose programming language (GPPL) types alone. finding
- ★ Expressing module interfaces in terms of domain concepts (molecules, basis sets, operators) rather than GPPL types (floats, ints, strings) drastically reduces the number of fields/complexity needed to specify inter- and intrapackage modules such as a Fock build. finding
- ★ Chemist is an open-source, embedded domain-specific language (DSL) for computational chemistry, implemented in C++ with Python bindings via Pybind11, distributed under the Apache 2.0 License. resource
- ★ Chemist decomposes chemical system and basis set concepts into separate, composable abstractions (e.g., electron, nucleus, atom, molecule) following the single-responsibility principle, rather than using one monolithic chemical-system class as is traditional in QC software. method
- ★ Chemist is designed to support composition of domain objects without unnecessary data movement, which the authors state no existing QC DSL achieves, enabling backend performance optimization (e.g., lazy evaluation). mechanism
- Chemist depends on two companion prerelease libraries, TensorWrapper (tensor/linear algebra abstractions) and ParallelZone (runtime/logging/interprocess communication abstractions), which are separate from but coupled to Chemist via encapsulated type definitions. resource
- ★ The usefulness of Chemist is demonstrated via the interface of NWChemEx's Fock build module and a proof-of-concept self-consistent field (SCF) algorithm incorporating uncertainty propagation. finding
- Chemist follows seven guiding design principles: performance, composition, single-responsibility principle, interoperability, extensibility, idiomaticity, and user-friendliness. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Software architecture / interface design case study | NWChemEx Fock build module | none | Demonstration that a Fock operator interface can be succinctly and generically specified using Chemist domain abstractions | — |
| Proof-of-concept algorithm implementation | Self-consistent field (SCF) algorithm built with Chemist | Addition of uncertainty propagation to the SCF algorithm | Demonstration of extensibility and correctness of Chemist-based algorithm composition | — |
- – Chemist library released as open-source software available on GitHub, written in C++ with Pybind11-generated Python bindings
- – Chemist's chemistry DSL decouples chemical-system concepts (electron, nucleus, atom, molecule) into separate composable abstractions instead of a single monolithic class
- – A proof-of-concept SCF algorithm with uncertainty propagation was successfully implemented using Chemist, illustrating extensibility beyond the initial minimal SCF feature set
- – Fock build module interface expressed via Chemist domain abstractions to avoid the combinatorial complexity of a GPPL-only API
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software engineering and computational chemistry design paper introducing Chemist, a domain-specific language (DSL) for quantum chemistry. No statistical analyses, hypothesis tests, or quantitative comparisons between groups were performed. The paper presents architectural concepts, design principles, and proof-of-concept implementations (a Fock build module interface and a self-consistent field algorithm with uncertainty propagation) through descriptive exposition and code/design illustrations.
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — PMID 40815845 "Chemist: A Domain-Specific Language by Chemists for Chemists"
Waldrop, de Jong, Valeev, Windus, Richard. J. Phys. Chem. A 2025. DOI 10.1021/acs.jpca.5c02993 · PMCID PMC12400401.
Nature of the paper
A software-architecture paper describing the Chemist C++ DSL (part of the NWChemEx stack). Most of the paper is design/API description (out of scope — not a computational pipeline). The only quantitative results are in Section 4.3 ("Proof of Concept: Uncertainty Propagation"), reported in Table 1, Figure 8 and Figure 9.
In-scope, pipeline-derived results
The numerical content is an ERI absolute-precision study: how the error in the two-electron integral (ERI) tensor scales with Libint's absolute-precision parameter τ, for two tiny systems at STO-3G.
| id | result | location | reproducible? |
|---|---|---|---|
| C1–C10 | mean MaxAE / mean MAE of ERI error e_I(τ)=I(τ)−I(τ=1e-13), averaged over dimer separations, He & H2 dimers, τ = 1e-3…1e-7 mEh | Table 1, Fig 8 | YES — fully specified |
| C11–C14 | α = mean(MaxAE)/τ and mean(MAE)/τ (He & H2) | Table 1 | YES (derived from C1–C10) |
| C15 | Fig 9: counterpoise-corrected SCF/STO-3G interaction energy curves ΔE(τ) with σ=50τ error bars | Fig 9 | partial — plot only, no tabulated values; secondary |
Why this IS reproducible (correcting the earlier premature drop)
The earlier scoping run dropped this as no_expected_result, arguing
"PySCF/libcint cannot reproduce Libint's τ knob." That reasoning is wrong:
- Table 1 gives exact numbers (e.g. He τ=1e-3: mean MaxAE 0.006004 mEh; H2 τ=1e-3: 0.224739 mEh) — pinnable 1:1 targets.
- The SI fully specifies the τ algorithm (jp5c02993_si_002.pdf §1,
Eqs 11-13): τ-screening is Libint's black-box primitive screening —
primitive quartet
ijklis omitted iffKbar_ij·Kbar_kl < τ, withKbar_ij = |C_i||C_j|·exp(−ζ_iζ_j/(ζ_i+ζ_j)|A−B|²). A closed-form, reproducible algorithm (clean for s-shells). - The exact geometries are in the SI (jp5c02993_si_001.txt): He dimer R=3..12; H2 dimer edge-gap 3..12 (monomer bond 1.39839 bohr = 0.740 Å). Obtained via the Europe PMC supplementary mirror (the ACS SI page is Cloudflare-gated, but Europe PMC serves the same files — this was the earlier run's blocker, now resolved).
- The integral library is pinned: Libint 2.11.0 = the cited Zenodo deposit 10.5281/zenodo.15278743 (the Libint source release = the dependency).
So this is a P16-valid reproduction: apply the documented integral-precision algorithm to the paper's own SI data and reproduce Table 1.
Reproduction method
Faithful Python implementation of the SI's documented screening (Eqs 11-13), with analytic STO-3G (ss|ss) ERIs, cross-validated two ways:
- the unscreened reference tensor I° is checked element-wise against
PySCF
mol.intor('int2e')(independent standard QM code) — proves the integral engine before any screening claim; - screening implemented exactly per the SI; both unit conventions (bohr vs Å) and both screening variants (black-box Eq 11-13 vs Cauchy-Schwarz Eq 6-9/13) are run to empirically disambiguate and match Table 1.
All compute on «our HPC» (SLURM, conda env on a compute node); «infra» workdir
«path».
Out of scope
- The Chemist/TensorWrapper/Sigma C++ library design and API (architecture, not a numeric pipeline).
- The full UP-enabled NWChemEx SCF energies+uncertainties (σ_SCF): the authors call it an unpublished, non-performant proof-of-concept; the SCF UP driver is not released. We reproduce the ERI precision study (Table 1), the part with pinnable numbers and a documented algorithm.
- Fig 9 absolute curve values: not tabulated; attempted only qualitatively.
Pointers
- Code (library): github.com/NWChemEx/Chemist (Apache-2.0).
- Sigma uncertainty lib: github.com/QCUncertainty/sigma (exists — ear
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.