Basis-set optimizer engine: design¶
Status: design for vibeqc.basisopt, increment BOE-01 (#1051) of hub #1050.
Decided 2026-11-09. The decisions themselves are recorded on #1050; this page
turns them into layers, types and rules that the later increments implement.
What the engine is¶
vibeqc.basisopt designs Gaussian basis sets: it changes exponents,
contraction coefficients and shell topology so that an objective, evaluated
with vibe-qc’s own methods, improves. It is a clean reimplementation. It
replaces two code bases, vibe-basis/ and python/vibeqc/basis_optimization/,
which are frozen (bug and licence fixes only) and retire at BOE-10 (#1059).
Rules that hold everywhere:
Native evaluation. Every energy and gradient comes from vibe-qc in process. External programs (CRYSTAL, ORCA, GPAW) appear only as references for validation, run out of process through
vq. They are never an objective.No dependency on the frozen code. Nothing in
vibeqc.basisoptimportsvibe_basisorvibeqc.basis_optimization.tests/basisset_dev/test_basisopt_layering.pyenforces it.Exact inputs. Structures and bases pass as numbers, never through formatted text. Rounding a relaxed cell by 3e-7 Angstrom once changed one SCF by 46,126 kJ/mol (#224).
Hartree and bohr inside. Other units appear only at the edges: input readers and reports.
No silent stand-ins. A failed or invalid point is reported to the optimizer as failed, with its reason. It is never replaced by a finite number.
Layers¶
Each subpackage may import the subpackages above it in this table, vibe-qc’s core and numerical libraries. It may not import those below it.
Subpackage |
Holds |
Increment |
|---|---|---|
|
Element bases, shells, ECP binding, content digests, native |
BOE-02 (#1052) |
|
Parametrisations with Jacobians, bounds, constraints, topology edits |
BOE-02 (#1052) |
|
Atoms, molecules, crystals, counterpoise clusters, training sets, registry adapters |
BOE-03 (#1053) |
|
Evaluation requests and records; one evaluator per route |
BOE-04 (#1041) |
|
Objective terms, composition, derivatives, guards |
BOE-05 (#1054) |
|
The optimizer interface, owned algorithms, library adapters, |
BOE-06 (#1055) |
|
Cache, journal, local and |
BOE-07 (#1056) |
|
Staged pipelines, topology search, contraction, element sweeps |
BOE-08 (#1057) |
|
Reports, export, provenance through |
BOE-09 (#1058) |
Two placements follow from dependencies rather than from the data flow:
optimizesits belowobjectives. It sees an objective only through a small callable protocol: a value, an optional gradient, or a rejection with its reason. The algorithms therefore never know about bases or systems.runsits belowoptimize. A campaign drives optimizers and evaluators both. The optimizer itself never schedules work; it asks for values, andrundecides whether they come from the cache, a local process or a fleet job.
The CLI (vb) moves to vibeqc.basisopt.cli at BOE-10.
Data flow¶
x (parameters, log exponents)
-> params.Parametrisation.to_model(x) -> model.BasisModel (+ Jacobian dmodel/dx)
-> model.native_basis(system) -> vibeqc.BasisSet (in memory)
-> evaluate.Evaluator(request) -> evaluate.Evaluation (energy, components, gradient?)
-> objectives.Objective -> value, gradient dvalue/dx (chain rule), or rejection
-> optimize.minimize -> next x, until OptResult
run wraps the evaluator step: it computes the cache key, looks the record up,
and otherwise evaluates locally or submits a fleet job.
Core types¶
These sketches fix names and responsibilities. Field lists are settled in each increment.
model.Shell: angular momentum, exponents, contraction coefficients, pure or Cartesian. Immutable.model.ElementBasis: an element’s ordered shells, plus an optionalEcpRef(name and core-electron count). The ECP is bound to the element and never optimized.model.BasisModel: element toElementBasis, with a canonical content digest that is exact to the exponent bit. The model builds avibeqc.BasisSetfor a given molecule or cell, through the existingBasisSet(molecule, shells, name, coefficients_pre_normalized)constructor. Routes that accept a basis only by name get a scoped in-memory registration (BOE-02).params.Parametrisation:x0,bounds,to_model(x)andjacobian(x). Implementations: free log exponents, even-tempered, well-tempered, shared exponents, contraction coefficients and frozen subsets. Constraints such as ordering and a minimum ratio between neighbours are declared with it.systems.System: one ofAtom,Molecule,CrystalorGhostCluster, each with a canonical digest. A crystal carries cell vectors and fractional positions as floats, a k-mesh and a magnetic state. ATrainingSetholds weighted systems and a recorded training and validation split.evaluate.EvaluationRequest: system, basis model, method key, route and settings.evaluate.Evaluation: energy and components, convergence and validity flags, conditioning numbers, the gradient when the route provides one, wall time and code identity.objectives.Objective: a weighted composition of terms, each of which turns evaluations into a value and, when it can, a gradient.optimize.OptResult: final parameters and value, gradient norm, evaluation count, history and the reason it stopped.
Evaluation routes¶
Each route calls one vibe-qc entry point. A method key and a route together select it, and both enter the cache key.
Route |
Methods |
vibe-qc entry point |
|---|---|---|
|
HF (RHF, UHF, ROHF), DFT |
|
|
MP2, CCSD(T) |
|
|
DLPNO-MP2, DLPNO-CCSD(T) |
|
|
Periodic HF and DFT |
|
|
Periodic HF, then DLPNO-CCSD(T) |
|
|
Cluster-model MP2, CCSD, DLPNO |
|
A route that is missing is refused with its name and reason; BIPOLE with correlation is the standing example. Each refusal is recorded on #1041 as a requirement for that method’s owner, not as engine work.
Cache key and identity¶
An evaluation’s key is a digest over:
the basis model digest;
the system digest;
the method key, route and every setting that can change the number (k-mesh, cutoffs, convergence thresholds, frozen-core convention);
the code identity: vibe-qc version, source commit and native build digest.
A cached record is reused only under an identical key. A campaign refuses to mix code identities unless it is told to. A changed native build is then never silently merged with older numbers.
Failures and validity¶
Kind |
Examples |
Handling |
|---|---|---|
Transient |
Network loss, queue or daemon unavailable |
Retry within an outage window; never cached |
Resource limit |
Time, memory or starvation limit of a job |
Stop the campaign and name the limit; never cached |
Scientific |
No convergence, stalled SCF, linear dependence, unstable UHF |
Cached with its reason; the optimizer sees a rejection |
Input |
Missing element, refused route |
Raised before any evaluation |
A converged flag is not sufficient. Each route adds its own validity checks: stall detection (convergence declared on a cycle that left the density unchanged), the S or S(k) conditioning guard, and UHF stability for open-shell atoms.
Derivatives¶
Analytic where vibe-qc provides them:
molecular RHF, UHF and ROHF, ECP atoms included (#1046);
closed-shell RHF on the BIPOLE route without ECPs (#1046). The two-electron and nuclear terms are a difference of BIPOLE’s own energy functional at the frozen converged density, replaying the driver call that
run_periodic_jobrecords on its result;they are chained through the parametrisation Jacobian.
Otherwise, central finite differences in log exponents, with a step rule and an error estimate.
Every objective value records which of the two it used.
Output, provenance and export¶
BOE-09 (#1058) settles how a run leaves the engine.
Report.
report.write_reportdeclares anOutputPlanof job kindbasisoptand writes it under anOutputWriter:{stem}.outand{stem}.basisopt.json(schemavibeqc.basisopt-report.v1) hold, per stage and element, the objective history, the final and term values, the guards that fired and the cost;{stem}.bibtexand{stem}.referenceshold the citations;the
{stem}.systemmanifest records every file, the citations and the exported store records. It records no hostname by default.
Citations.
report.run_citationsfires the routes table ofdatabase.tomlfor every method, route and reference the run evaluated, the parent basis and ECPs, and adds the optimizers’ and procedures’ keys. Natural-orbital contraction cites Widmark, Malmqvist and Roos (1990), Section 2, and Ye and Berkelbach (2022), Section II.A. The conditioning guard cites Ye and Berkelbach, Section II.B. A combination the table cannot cite is a warning in the report.Export.
report.export_basiswrites avibe-basisrecord throughbasis_store.import_record, so the basis resolves by name. Its provenance holds the training-set digest (Objective.usage), the methods, the code identity, the parent basis with its citation routes restricted to the exported elements, and the release status.Unpublished families (#1038, D7) export only to
store=or$VIBEQC_BASISOPT_UNPUBLISHED_STORE. A location inside a Git checkout is refused, and there is no default.
Audit of the frozen code bases¶
The two packages hold about 21,000 lines. Each concept is kept as an idea, moved into vibe-qc’s core, or dropped.
Moved into core unchanged (decided 2026-11-09). This is method code, not optimizer code. The frozen package keeps re-exports, so nothing there changes behaviour, and the clean engine calls these modules as vibe-qc methods. BOE-11 (#1061) carried out the move.
Frozen module |
Lines |
Proposed home |
|---|---|---|
|
940 |
|
|
1014 |
|
|
367 |
|
|
517 |
|
|
334 |
|
|
974 |
|
Kept as ideas and reimplemented.
Concept |
Frozen source |
What the new design keeps |
|---|---|---|
Parametrisation |
|
Log exponents, topology edits as pure functions, the d and f scan |
Objectives |
|
Cohesive energies with counterpoise atoms; per-system weights; validation-only terms |
Optimizers |
|
One result type; BDIIS; a global pre-pass followed by a local polish |
Cache and campaigns |
|
Content-keyed append-only journal; submission ledger adopted on restart; outage windows |
Failure handling |
|
The three kinds above; resource limits stop the run |
Export |
|
Basis-store records with provenance |
Dropped.
The CRYSTAL deck writer and output parser (
backends/,engines/crystal23.py,recipes/run_d12.py), about a quarter of vibe-basis. CRYSTAL stays a reference program, driven by separate validation scripts.Temporary-file basis libraries (
io.py’sTempBasisLibrary). The model builds in-memory bases.The field-by-field transcription between
vibe_basis.topologyandvibeqc.basis_crystal. There is one basis model.Direct file writes. Every file goes through
vibeqc.output.The CRYSTAL-specific recipes (
recipes/crystal_stage*.py,crystal_objective.py).
Missing in both, and supplied by the new engine.
In-memory bases on every route, including those that look bases up by name.
Energy components (reference, correlation, triples) in every evaluation record.
Correlated evaluators for atoms and solids (#1041).
The S(k) conditioning guard and stall detection on the BIPOLE route.
A code identity in every cached record.
Open questions left to later increments¶
BOE-02: the in-memory registration for name-only routes (the periodic runner, AICCM), and whether any route still needs a file.
BOE-05, settled: the conditioning guard caps S or S(k) at a condition number of 1e10 by default, after Ye and Berkelbach (doi:10.1021/acs.jctc.1c01245). #1043 may set its own cap for the wavefunction families.
BOE-06, settled: the bounded quasi-Newton method is projected BFGS with an active set, where variables at a bound with the gradient pushing outward are held. The steps use Armijo backtracking and a trust radius.
BOE-07: how many evaluations to batch in one fleet job, against queue overhead.