vibe-qcDocumentation

Basis-set optimizer engine: design

Status: design for vibeqc.basisopt, increment BOE-01 (#1051) of hub #1050. Decided 2026-11-09. The decisions themselves are recorded on #1050; this page turns them into layers, types and rules that the later increments implement.

What the engine is

vibeqc.basisopt designs Gaussian basis sets: it changes exponents, contraction coefficients and shell topology so that an objective, evaluated with vibe-qc’s own methods, improves. It is a clean reimplementation. It replaces two code bases, vibe-basis/ and python/vibeqc/basis_optimization/, which are frozen (bug and licence fixes only) and retire at BOE-10 (#1059).

Rules that hold everywhere:

  • Native evaluation. Every energy and gradient comes from vibe-qc in process. External programs (CRYSTAL, ORCA, GPAW) appear only as references for validation, run out of process through vq. They are never an objective.

  • No dependency on the frozen code. Nothing in vibeqc.basisopt imports vibe_basis or vibeqc.basis_optimization. tests/basisset_dev/test_basisopt_layering.py enforces it.

  • Exact inputs. Structures and bases pass as numbers, never through formatted text. Rounding a relaxed cell by 3e-7 Angstrom once changed one SCF by 46,126 kJ/mol (#224).

  • Hartree and bohr inside. Other units appear only at the edges: input readers and reports.

  • No silent stand-ins. A failed or invalid point is reported to the optimizer as failed, with its reason. It is never replaced by a finite number.

Layers

Each subpackage may import the subpackages above it in this table, vibe-qc’s core and numerical libraries. It may not import those below it.

Subpackage

Holds

Increment

model

Element bases, shells, ECP binding, content digests, native BasisSet construction

BOE-02 (#1052)

params

Parametrisations with Jacobians, bounds, constraints, topology edits

BOE-02 (#1052)

systems

Atoms, molecules, crystals, counterpoise clusters, training sets, registry adapters

BOE-03 (#1053)

evaluate

Evaluation requests and records; one evaluator per route

BOE-04 (#1041)

objectives

Objective terms, composition, derivatives, guards

BOE-05 (#1054)

optimize

The optimizer interface, owned algorithms, library adapters, OptResult

BOE-06 (#1055)

run

Cache, journal, local and vq execution, failure handling, identity

BOE-07 (#1056)

workflows

Staged pipelines, topology search, contraction, element sweeps

BOE-08 (#1057)

report

Reports, export, provenance through vibeqc.output

BOE-09 (#1058)

Two placements follow from dependencies rather than from the data flow:

  • optimize sits below objectives. It sees an objective only through a small callable protocol: a value, an optional gradient, or a rejection with its reason. The algorithms therefore never know about bases or systems.

  • run sits below optimize. A campaign drives optimizers and evaluators both. The optimizer itself never schedules work; it asks for values, and run decides whether they come from the cache, a local process or a fleet job.

The CLI (vb) moves to vibeqc.basisopt.cli at BOE-10.

Data flow

x (parameters, log exponents)
  -> params.Parametrisation.to_model(x)  -> model.BasisModel (+ Jacobian dmodel/dx)
  -> model.native_basis(system)          -> vibeqc.BasisSet (in memory)
  -> evaluate.Evaluator(request)         -> evaluate.Evaluation (energy, components, gradient?)
  -> objectives.Objective                -> value, gradient dvalue/dx (chain rule), or rejection
  -> optimize.minimize                   -> next x, until OptResult

run wraps the evaluator step: it computes the cache key, looks the record up, and otherwise evaluates locally or submits a fleet job.

Core types

These sketches fix names and responsibilities. Field lists are settled in each increment.

  • model.Shell: angular momentum, exponents, contraction coefficients, pure or Cartesian. Immutable.

  • model.ElementBasis: an element’s ordered shells, plus an optional EcpRef (name and core-electron count). The ECP is bound to the element and never optimized.

  • model.BasisModel: element to ElementBasis, with a canonical content digest that is exact to the exponent bit. The model builds a vibeqc.BasisSet for a given molecule or cell, through the existing BasisSet(molecule, shells, name, coefficients_pre_normalized) constructor. Routes that accept a basis only by name get a scoped in-memory registration (BOE-02).

  • params.Parametrisation: x0, bounds, to_model(x) and jacobian(x). Implementations: free log exponents, even-tempered, well-tempered, shared exponents, contraction coefficients and frozen subsets. Constraints such as ordering and a minimum ratio between neighbours are declared with it.

  • systems.System: one of Atom, Molecule, Crystal or GhostCluster, each with a canonical digest. A crystal carries cell vectors and fractional positions as floats, a k-mesh and a magnetic state. A TrainingSet holds weighted systems and a recorded training and validation split.

  • evaluate.EvaluationRequest: system, basis model, method key, route and settings. evaluate.Evaluation: energy and components, convergence and validity flags, conditioning numbers, the gradient when the route provides one, wall time and code identity.

  • objectives.Objective: a weighted composition of terms, each of which turns evaluations into a value and, when it can, a gradient.

  • optimize.OptResult: final parameters and value, gradient norm, evaluation count, history and the reason it stopped.

Evaluation routes

Each route calls one vibe-qc entry point. A method key and a route together select it, and both enter the cache key.

Route

Methods

vibe-qc entry point

molecular

HF (RHF, UHF, ROHF), DFT

run_rhf, run_uhf, run_rohf, run_rks, run_uks on an in-memory BasisSet

molecular

MP2, CCSD(T)

run_mp2, run_ump2, run_rohf_mp2, run_ccsd, run_uccsd

molecular

DLPNO-MP2, DLPNO-CCSD(T)

vibeqc.dlpno (run_dlpno_mp2, run_dlpno_ump2, the DLPNO-CCSD drivers)

bipole

Periodic HF and DFT

run_periodic_job(..., jk_method="bipole")

gdf

Periodic HF, then DLPNO-CCSD(T)

run_periodic_job(..., jk_method="gdf"), run_periodic_dlpno_ccsd

aiccm:<variant>

Cluster-model MP2, CCSD, DLPNO

run_periodic_job(method="aiccm", variant=..., correlation=...)

A route that is missing is refused with its name and reason; BIPOLE with correlation is the standing example. Each refusal is recorded on #1041 as a requirement for that method’s owner, not as engine work.

Cache key and identity

An evaluation’s key is a digest over:

  • the basis model digest;

  • the system digest;

  • the method key, route and every setting that can change the number (k-mesh, cutoffs, convergence thresholds, frozen-core convention);

  • the code identity: vibe-qc version, source commit and native build digest.

A cached record is reused only under an identical key. A campaign refuses to mix code identities unless it is told to. A changed native build is then never silently merged with older numbers.

Failures and validity

Kind

Examples

Handling

Transient

Network loss, queue or daemon unavailable

Retry within an outage window; never cached

Resource limit

Time, memory or starvation limit of a job

Stop the campaign and name the limit; never cached

Scientific

No convergence, stalled SCF, linear dependence, unstable UHF

Cached with its reason; the optimizer sees a rejection

Input

Missing element, refused route

Raised before any evaluation

A converged flag is not sufficient. Each route adds its own validity checks: stall detection (convergence declared on a cycle that left the density unchanged), the S or S(k) conditioning guard, and UHF stability for open-shell atoms.

Derivatives

  • Analytic where vibe-qc provides them:

    • molecular RHF, UHF and ROHF, ECP atoms included (#1046);

    • closed-shell RHF on the BIPOLE route without ECPs (#1046). The two-electron and nuclear terms are a difference of BIPOLE’s own energy functional at the frozen converged density, replaying the driver call that run_periodic_job records on its result;

    • they are chained through the parametrisation Jacobian.

  • Otherwise, central finite differences in log exponents, with a step rule and an error estimate.

  • Every objective value records which of the two it used.

Output, provenance and export

BOE-09 (#1058) settles how a run leaves the engine.

  • Report. report.write_report declares an OutputPlan of job kind basisopt and writes it under an OutputWriter:

    • {stem}.out and {stem}.basisopt.json (schema vibeqc.basisopt-report.v1) hold, per stage and element, the objective history, the final and term values, the guards that fired and the cost;

    • {stem}.bibtex and {stem}.references hold the citations;

    • the {stem}.system manifest records every file, the citations and the exported store records. It records no hostname by default.

  • Citations. report.run_citations fires the routes table of database.toml for every method, route and reference the run evaluated, the parent basis and ECPs, and adds the optimizers’ and procedures’ keys. Natural-orbital contraction cites Widmark, Malmqvist and Roos (1990), Section 2, and Ye and Berkelbach (2022), Section II.A. The conditioning guard cites Ye and Berkelbach, Section II.B. A combination the table cannot cite is a warning in the report.

  • Export. report.export_basis writes a vibe-basis record through basis_store.import_record, so the basis resolves by name. Its provenance holds the training-set digest (Objective.usage), the methods, the code identity, the parent basis with its citation routes restricted to the exported elements, and the release status.

  • Unpublished families (#1038, D7) export only to store= or $VIBEQC_BASISOPT_UNPUBLISHED_STORE. A location inside a Git checkout is refused, and there is no default.

Audit of the frozen code bases

The two packages hold about 21,000 lines. Each concept is kept as an idea, moved into vibe-qc’s core, or dropped.

Moved into core unchanged (decided 2026-11-09). This is method code, not optimizer code. The frozen package keeps re-exports, so nothing there changes behaviour, and the clean engine calls these modules as vibe-qc methods. BOE-11 (#1061) carried out the move.

Frozen module

Lines

Proposed home

basis_optimization/energy_gradient.py

940

vibeqc.basis_derivatives.molecular

basis_optimization/periodic_energy_gradient.py

1014

vibeqc.basis_derivatives.periodic

basis_optimization/gradients.py

367

vibeqc.basis_derivatives.overlap_penalty

basis_optimization/ld_diagnostics.py

517

vibeqc.basis_derivatives.conditioning

basis_optimization/counterpoise.py

334

vibeqc.counterpoise

basis_optimization/phonons.py

974

vibeqc.bipole_phonons

Kept as ideas and reimplemented.

Concept

Frozen source

What the new design keeps

Parametrisation

parametrise.py, topology.py

Log exponents, topology edits as pure functions, the d and f scan

Objectives

objective.py (both), pipeline.py

Cohesive energies with counterpoise atoms; per-system weights; validation-only terms

Optimizers

drivers.py, optimize.py, bdiis.py

One result type; BDIIS; a global pre-pass followed by a local polish

Cache and campaigns

cache.py, campaign.py, transports/ledger.py

Content-keyed append-only journal; submission ledger adopted on restart; outage windows

Failure handling

engine.py failure modes

The three kinds above; resource limits stop the run

Export

export.py (both), io/library_bridge.py

Basis-store records with provenance

Dropped.

  • The CRYSTAL deck writer and output parser (backends/, engines/crystal23.py, recipes/run_d12.py), about a quarter of vibe-basis. CRYSTAL stays a reference program, driven by separate validation scripts.

  • Temporary-file basis libraries (io.py’s TempBasisLibrary). The model builds in-memory bases.

  • The field-by-field transcription between vibe_basis.topology and vibeqc.basis_crystal. There is one basis model.

  • Direct file writes. Every file goes through vibeqc.output.

  • The CRYSTAL-specific recipes (recipes/crystal_stage*.py, crystal_objective.py).

Missing in both, and supplied by the new engine.

  • In-memory bases on every route, including those that look bases up by name.

  • Energy components (reference, correlation, triples) in every evaluation record.

  • Correlated evaluators for atoms and solids (#1041).

  • The S(k) conditioning guard and stall detection on the BIPOLE route.

  • A code identity in every cached record.

Open questions left to later increments

  • BOE-02: the in-memory registration for name-only routes (the periodic runner, AICCM), and whether any route still needs a file.

  • BOE-05, settled: the conditioning guard caps S or S(k) at a condition number of 1e10 by default, after Ye and Berkelbach (doi:10.1021/acs.jctc.1c01245). #1043 may set its own cap for the wavefunction families.

  • BOE-06, settled: the bounded quasi-Newton method is projected BFGS with an active set, where variables at a bound with the gradient pushing outward are held. The steps use Armijo backtracking and a trust radius.

  • BOE-07: how many evaluations to batch in one fleet job, against queue overhead.