Fragment and embedding framework for very large molecules¶
Status: proposed architecture, 2026-11-03. Hub issue FRAG-00 (#861),
increments #862-#870. F0 landed as the experimental vibeqc.fragment
package (bookkeeping only, no fragment physics). The first F2 slice,
closed-shell DC-HF (vibeqc.fragment.dc.run_dc_rhf) and DC-KS
(run_dc_rks, local and global hybrids), energy only, is on main; open
shells remain on #865. The first F1
slice, closed-shell projection embedding
(vibeqc.fragment.projection.run_projection_embedding, HF/LDA/PBE low levels,
HF or KS high levels, Huzinaga and level-shift exclusion, energy only), is on
main with the native embedding source term of conventional libint RHF/RKS;
correlated high levels are F1b (#864). This adds no runtime capability,
dependency, default or release gate, and claims no accuracy or speedup. It
extends the Fock/operator contract, the
Fock unification plan, the
MANTA engine, the GPU programme, the
massively parallel architecture and the
tensor engine without changing their scope or ownership.
The literature survey of 2026-11-03 added 57 records to the configured
library (catalog keys frag20261103_*). The equations quoted below were read
from the primary papers; project notation replaces the authors’ symbols.
Objective¶
Large molecules should be computed from fragments: detect symmetry, split the system along bonds automatically, solve each fragment in the electronic field of its neighbours (a density-level environment, not point charges), and iterate until the fragments and their environment are mutually consistent. One theory should serve almost every vibe-qc method and run on the unified Fock operator, MANTA, GPUs and MPI.
The workflow itself is established prior art. The research problem is making partitioning, energy accounting, convergence and error control reliable across different electronic structures. The survey supports three formal starting points and no single universal scheme:
Overlapping divide-and-conquer SCF (DC) couples the whole molecule through one global density and Fock matrix. This is the speedup layer.
Projection/Huzinaga embedding treats active regions at a higher level inside a lower-level density. Its same-level limit is exact, which makes it the correctness backbone.
Screened generalized many-body expansion (GMBE) assembles fragment energies by inclusion-exclusion. This is the parallel correlated layer and an independent energy-based baseline.
Bootstrap embedding (BE) is the strongest later correlated extension. It needs a bath, transformed two-electron integrals, reduced density matrices and matching constraints, so it is not a one-body operator on the Fock matrix.
The design below gives these models one shared infrastructure: partitions, frames, environment epochs, embedding source terms, assemblers, symmetry orbits and a fragment task graph. The chemistry-specific energy reconstruction stays in each model’s assembler.
Not supported by the literature, and therefore not claimed here: universal linear scaling, chemical accuracy by construction, or one Fock-only theory covering all correlated methods.
Primary sources and the equations used¶
Source |
Passage read |
Used for |
|---|---|---|
Section 2.2, Eqs. (9)-(16); Figure 2 |
DC subsystem Fock blocks, partition matrix, common Fermi level, exact full-buffer limit |
|
Section 2, Eqs. (1)-(3) and the energy expression after Eq. (3) |
Embedded core Hamiltonian |
|
Section II A, Eqs. (4)-(11) |
Huzinaga operator, the two-level energy (7), its first-order corrections (9) and (10), Boughton-Pulay orbital selection |
|
Eqs. (7), (8), (16)-(18) |
Intersection-corrected m-body energies over overlapping fragments |
|
Section II A, Eqs. (9)-(14) |
FMO2 many-body expansion of the Fock matrix with separated ESP terms |
|
Critical review; precision analysis |
Truncation, counterpoise and numerical-precision requirements of MBE energies |
|
DC-CCSD, DC-CCSD(T) |
Correlation energies on a DC reference |
|
Welborn et al. 2016; Cho et al. 2025 (QuEmb), sections 2.3.1-2.3.6 and SI |
BE matching and implementation |
Requirements of the later BE layer |
Steinmann et al. 2012 (FragIt); Kundu et al. 2026 (GAMA), SI Tables S1-S2 |
Automatic cutting rules; low-level correction sensitivity |
Rule-based cutting and why the whole-system low level matters |
Not yet in the library: Davis and Kahan 1970 (doi:10.1137/0707001) for the subspace perturbation bound used in certified reuse, Anderson 1965 (doi:10.1145/321296.321305) for fixed-point mixing, and the Kobayashi exact DC-SCF gradient (Nakai 2023, ref. 113). They must go through the library intake before their slices land.
The common fixed point¶
All three models are instances of one map. At epoch k:
Sigma_k = GlobalBuild(D_k) # unified Fock, MANTA, GPU
Pi_s(Sigma_k) = (O_s, h + V^emb_s[Sigma_k], Q_s[Sigma_k], N_s or mu)
R_s = Solve_s(Pi_s) # any admitted method
(D_{k+1}, E_k) = Assemble({R_s}, Sigma_k)
They differ in four declared choices:
Choice |
DC-SCF |
Projection/Huzinaga |
Embedded GMBE |
|---|---|---|---|
Inner self-consistency in a fragment |
None: one diagonalization of a block of |
Yes: SCF of the active subsystem in fixed |
Yes, per fragment, in the low-level environment |
Exclusion |
None; locality comes from truncating |
|
Huzinaga, against the environment’s occupied orbitals |
Electron constraint |
One common Fermi level for the whole molecule |
Fixed |
Fixed per fragment |
Assembly |
Weighted density, global energy of the assembled density |
Two-level energy difference |
Inclusion-exclusion over fragment energies plus a whole-system low level |
Each model has an exact limit that becomes an internal regression test at
about 1e-8 Eh: DC with L(s) equal to the whole basis, projection at the
same level for A and B, and GMBE with one fragment equal to the whole system
or with identical high and low levels. These limits test the infrastructure,
not the approximation.
Model 1: divide-and-conquer SCF (the speedup layer)¶
Following Nakai 2023, Eqs. (9)-(16): build the global Fock matrix of the
assembled density, F_k = h + G[D_k] (Eq. 9). For every fragment solve the
local generalized eigenproblem on its block,
F_s = E_s^T F_k E_s, S_s = E_s^T S E_s, F_s C_s = S_s C_s eps_s (Eq. 11)
and form the local finite-temperature density
D_s = C_s f_beta(eps_F - eps_s) C_s^H (Eq. 15). The global density is the
partition-weighted sum D_{k+1} = sum_s P_s o (E_s D_s E_s^T) (Eq. 14), with
P_s equal to 1 on core-core elements, 1/2 on core-buffer elements and 0
otherwise (Eq. 13). The single Fermi level eps_F solves
N = sum_s sum_p f_beta(eps_F - eps_{s,p}) w_{s,p},
w_{s,p} = sum_{mu nu} (P_s)_{mu nu} C_{s,mu p} C*_{s,nu p} S_{nu mu} (Eq. 16)
Contract rules:
Symmetric coverage. For atoms
xinC(a)andyinC(b)the weights add to one only ifyinB(a)exactly whenxinB(b). Otherwise a retained element is silently weighted 1/2. That forces buffers to be unions of whole neighbouring cores with a symmetric neighbour relation.FragmentPartition.from_atom_bufferscloses requested atom buffers that way (never dropping a requested interaction), and the tests checksum_s (P_s)_{mu nu} in {0, 1}.Fixed-point residual. The assembled
Dis not idempotent and is not a function of one global diagonalization, soF D S - S D Fdoes not vanish at DC convergence. Convergence is judged on the fixed-point residualr_k = D_{k+1}[F(D_k)] - D_k, the electron count and the energy change. Acceleration is Pulay/Anderson mixing of that residual, not commutator DIIS.Energy. For HF,
E = E_nuc + Tr(D h) + 1/2 Tr(D G[D])on the assembledD(Eq. 10). For KS, the global builder evaluatesE_J[D] + E_xc[rho_D]on the assembled density. With smearing the free energy, entropy and internal energy stay separate quantities (Mermin, as in the Fock plan). The DC energy is not variational inD, so Hellmann-Feynman forces are wrong. The model refuses analytic gradients until the coupled-perturbed treatment exists (see Derivatives).Exact limit. If
L(s)is the whole basis for everys, eachD_sis the finite-temperature density of the globalF, the weights sum to one, and DC reduces to conventional HF/KS at the chosen temperature (Nakai 2023, end of section 2.2). This is the F2 regression test.Cost boundary. Each epoch still needs one global
G[D]build. The fragmentation makes the eigensolve local. It does not make J, K or XC linear-scaling. That comes from screened, fitted or seminumerical builders (MANTA RI-J, COSX, sparse density). This is why DC sits on the MANTA and tensor programmes rather than replacing them.
DC applies to HF, KS (pure and global hybrids admitted on the global builder) and DFTB-type effective Hamiltonians (Nakai 2023, section 2.7). The correlated DC energies of Kobayashi and Nakai partition the correlation energy by core ownership on top of this reference.
Model 2: projection/Huzinaga embedding (the correctness backbone)¶
Start from a converged low-level state D = D_A + D_B (Manby Eq. 1; Hegely
Eq. 6), localize its occupied orbitals (IBO or Pipek-Mezey, localise.py),
and assign them to A or B. Use Boughton-Pulay completeness 0.985 by default,
which Hegely et al. found more robust than Mulliken thresholds for conjugated
systems. The embedded core Hamiltonian for A is
h^emb_A = h + G_low[D_A + D_B] - G_low[D_A] (Manby Eq. 3)
and the active orbitals solve the Huzinaga equations
(F_A - S D_B F_A - F_A D_B S) C_A = S C_A E_A (Hegely Eq. 11)
which keep A orthogonal to the frozen B orbitals without the mu
dependence of F + mu S D_B S. The level-shift form stays available as the
Manby variant for comparison. The total energy is
E = E_low[D_A + D_B] + E_high[A in B] - E_low[D_A] - Tr(D_A dE_low/dD_A) (Hegely Eqs. 7, 10)
with the first-order correction term of Eq. (9) for DFT-in-DFT. For KS low levels, the XC nonadditive corrections of Manby’s energy expression are explicit components of the record, not folded into a trace.
As implemented in F1, E_high[A in B] is the embedded SCF energy with
h^emb_A, so it already contains Tr(D~_A V_emb); Eqs. (9) and (10) then
reduce to the same four components, E_low[D] - E_low[D_A] - Tr(D_A V_emb) + E_high^emb[D~_A]. For the level-shift variant mu S G_B S sits in h^emb_A
(Manby Eq. 3), which leaves the SCF energy low by O(1/mu). Manby’s
first-order term mu Tr(D~_A S G_B S) is added once more as an explicit
correction and leaves an O(1/mu^2) error (ethanol HF-in-HF, 6-31G*:
2e-6 Eh at mu = 1e3, 2e-8 Eh at 1e4). Orbitals are assigned to A when
their Boughton-Pulay domain contains an active atom. Hegely’s separate
pi-orbital rule is not implemented.
Contract rules:
Same-level exactness. With the same method and full basis for A and B,
D~_A = D_AandE = E_low[D]to SCF precision. The F1 regression test requires agreement with the full calculation at about1e-8Eh, and with matched ORCA for the full calculation.Solver generality. Any method that runs on a molecule with an extra one-body operator and a frozen excluded occupied space can be the high level: HF, KS, MP2/RI-MP2, CCSD(T), DLPNO, CASSCF/NEVPT2 (through the
Hamiltonian(h1e, h2e, e_core)seam), and TDDFT for excitations localized in A. Each method declares whether it admits embedding source terms. Unadmitted methods refuse.Approximations stay named. WF-in-DFT error, basis truncation for A and active-region choice are separate, recorded controls. None of them is part of the exact-limit test.
Native hook. The SCF drivers need one admitted way to add a one-body source term and an exclusion operator in a declared frame. This belongs in Fock source composition (#472/#594), not in a fragment-private SCF copy.
Automatic fragmentation along bonds¶
Automatic fragmentation is prior art: rule-based cutting goes back to FragIt (2012), and QFRAGS, GAMA and graph-based DMET add other objectives. None guarantees accuracy for arbitrary cuts. The proposal is a checked pipeline whose every decision is recorded and reproducible:
Low-level picture. Starting density from SAD, GFN2/DFTB or a coarse DC run. Compute IAO Wiberg bond orders and delocalization indices (
bond_analysis.py), IAO charges and IBOs.Cuttable bonds. Only single sigma bonds with bond order below a threshold, outside conjugated domains and aromatic rings, and not dative or strongly polar. The FragIt-style protection rules are data, not code branches. A conjugated domain larger than the fragment budget is reported as a stress case requiring large buffers, never cut silently.
Partition. Balanced graph partitioning minimizing total cut cost under a per-fragment cost model (about
n_AO^3for DC, the method’s scaling for correlated fragments). Use greedy region growing from seeds plus Kernighan-Lin refinement, implemented in-house so that no new hard dependency is needed. The partition is constrained to unions of symmetry orbits.Buffers. A distance radius plus coupling closure: add atom
btoB(s)whenmax |D_{mu nu}|ormax |F_{mu nu}|betweenC(s)andbexceeds a threshold at the current epoch. Then apply the symmetric-coverage closure.Adaptive refinement. The estimator
eta_sis the change of the fragment’s core energy and core density when its buffer grows by one shell (DC), or the next-order increment (GMBE). Refine whereeta_s > tau / N_fragfor the declared budgettau. This ties fragmentation to an observable error, which the survey identified as the most defensible opportunity.
Changing the partition changes the energy surface. Geometry optimization and dynamics therefore run at fixed topology. Topology changes are recorded events, and discontinuities at cutoff crossings are reported (EXESS discusses these), not smoothed silently.
Symmetry and fragment reuse¶
Reuse distinguishes a candidate match from certified equivalence.
Candidates. Canonical labelled bond graph plus an aligned geometry (RMSD under a point operation
gwith atom permutation) propose that fragmenttequalsg s.Certified exact reuse. With AO action
U_g(symmetry_ao.build_molecular_space_action), reuse requires the same basis, charge/spin, method options, weights and electron constraint, and an embedded-operator residual|| U_g^T (h^emb_t + Q_t) U_g - (h^emb_s + Q_s) || <= tau_symat the current epoch. ThenD_t = U_g D_s U_g^T. For a molecule whose partition is invariant under its point group, this is exact whenever the converged state keeps the symmetry. One fragment per orbit is solved. The residual is checked at every epoch because a symmetry-broken SCF must fail closed.Approximate reuse (repeat units, crystals). Allowed only with an error budget. In the orthonormalized local frame, the Davis-Kahan bound
|| sin Theta || <= || Delta f || / delta(withdeltathe occupied-virtual gap) bounds the occupied-subspace error from the operator difference. For a fragment energy stationary in its own density, the energy error is second order in that change. This does not hold for the DC weighted assembly, which must use the density bound directly. Ends, defects, fields and differently polarized neighbours fail the residual and are recomputed.
Identical connectivity alone never certifies identical embedded Hamiltonians.
Method coverage¶
Method family |
DC reference |
Projection high level |
GMBE fragment |
Notes |
|---|---|---|---|---|
HF, KS (LDA/GGA/meta/global hybrid) |
yes |
yes |
yes |
Range-separated and VV10 only where the global builder admits them |
RO/UHF, RO/UKS |
after closed shell |
yes |
yes |
Independent spin densities and Fermi levels per spin are declared |
MP2, RI-MP2, double hybrids |
DC-MP2 partition |
yes |
yes |
Frozen-core and the excluded-occupied space compose |
CCSD, CCSD(T), DLPNO-CCSD(T) |
DC-CC partition |
yes |
yes |
Compare with existing DLPNO, do not assume it is superseded |
CASSCF, NEVPT2/CASPT2, DMRG, selected CI |
no |
yes, through |
yes |
Active space inside A |
TDDFT/TDA |
no |
localized excitations |
exciton-coupling extension |
Needs its own theory (FMO-TDDFT), later |
DFTB, GFN, NDDO |
DC-DFTB |
as low level |
as low level |
Model-specific SCC terms stay with the model |
Periodic routes |
later |
embedded cluster (existing AIMP research) |
crystal MBE |
Separate image/k/long-range convergence |
Execution on the unified Fock operator, MANTA, GPU and MPI¶
Two layers with different shapes:
Global layer, one per epoch. G[D_k] on the assembled, block-sparse,
possibly fractional density: J by MANTA RI-J, K by the admitted exact,
COSX or screened route, and XC on the grid, through FockBuild providers.
This is the dominant cost and inherits all MANTA, tensor and GPU work
unchanged. The assembled density’s sparsity pattern is known from the
partition and is passed to screening. The fragment framework adds no
competing integral or contraction engine.
Fragment layer, many independent tasks.
DC: gather
F_s = E_s^T F_k E_sblocks, run batched dense eigensolves (CPU threads, or batched GPUsyevdunder the GPU programme), and scatter-addP_s o D_s. The Fermi level needs only scalars: each task returns(eps_{s,p}, w_{s,p}), and the bisection or Newton solve foreps_Fis one small allreduce per step. No matrix moves for it.Projection and GMBE: each fragment problem is a task with a cost estimate, scheduled across MPI process groups (communicator split), with threads or GPU inside each solver. This is the hierarchical model demonstrated by GDDI and EXESS, built on the #804 descriptors and leases for block gather/scatter, not on a second planner.
Determinism: energy and density reductions run in fixed fragment order for a given layout. Every task result is keyed on
(epoch identity, fragment identity, method options). That key gives restart and symmetry reuse from the same cache.
No speedup is claimed. The measurement contract of massive_parallel_benchmarks.md applies: end-to-end time, peak memory, transfers and scaling at identical scientific settings.
Derivatives¶
Converged fragment solves do not give the derivative of the assembled energy.
All models start energy-only and refuse analytic gradients.
Finite differences of the assembled energy at fixed topology are the internal check for every later analytic gradient (
<= 1e-5Eh/bohr agreement). Fragmentation error of forces is assessed separately against the full calculation.DC: Lagrangian with coupled-perturbed response of all subsystems (Kobayashi exact DC gradient, to acquire). Projection: Lagrangian with low-level CPSCF for the environment’s response (Lee, Miller and co-workers, to acquire). GMBE:
sum_alpha c_alpha dE_alphaplus the response of each fragment’s embedding to the low-level density.The 2026 BE force results use approximate response. That is not an exact conservative gradient, and vibe-qc labels such modes accordingly if it adopts them.
Proposed code layout¶
An experimental package python/vibeqc/fragment/, imported explicitly and
not re-exported at top level until accepted:
Module |
Content |
|---|---|
|
Cores, buffers, coverage closure, weights, AO selectors, identities |
|
Fragment frames and injections on |
|
Immutable environment epochs over |
|
Embedding source terms: |
|
DC assembler, Fermi solve, fixed-point mixing |
|
Orbital selection, two-level energy record |
|
Fragment families, intersection lattice, count-once coefficients, screening |
|
Rule data, bond-order graph, partitioning, buffers, adaptive refinement |
|
Orbits, candidate matching, certified reuse |
|
Task graph, cost model, MPI groups, result cache |
User-facing reports go through vibeqc.output (a formats/ module for
fragment tables). Each model connects to output/citations/database.toml.
New tests are classified through scripts/test_gate/update_suite_manifest.py.
Increments and acceptance¶
Each increment lands on main on its own, with its focused check. Numerical
acceptance goes through vq against ORCA where it has the method.
Increment |
Deliverable |
Acceptance |
|---|---|---|
F0 (#862) |
Partition, frames, epochs, weights, count-once lattice, task records; no new physics |
Unit identities: symmetric coverage, partition of unity, count-once, one-fragment identity, frame injection round trip |
F1 (#863) |
Embedding source term in molecular HF/KS SCF (through #594), Huzinaga and level-shift variants, projection HF-in-HF and PBE-in-PBE |
Same-level limit |
F1b (#864) |
MP2-, CCSD(T)- and DLPNO-in-DFT/HF |
Manby 2012 ethanol OH case and Hegely 2016 cases against their reported tables; convergence to full ORCA CCSD(T) with growing A |
F2 (#865) |
DC-HF/DFT molecular with Fermi level and Anderson mixing |
Full-buffer limit |
F3 (#866) |
Embedded GMBE corrections on the DC reference, capped and LMO units |
Water clusters and S66x8 against full ORCA: interaction MAE target |
F4 (#867) |
Automatic fragmentation, adaptive buffers, symmetry orbits, certified reuse |
Reproducible partitions; reuse residuals; saved solves counted; conformer/barrier errors |
F5 (#868) |
MPI task groups, batched GPU local eigensolves, MANTA global build in DC |
Identical results across layouts at fixed order; measured end-to-end time and memory under the benchmark contract |
F6 (#869) |
Analytic gradients: projection, then DC |
Finite-difference agreement |
F7 (#870) |
Bootstrap embedding on the |
Separate plan once F0-F3 are accepted |
The targets in this table are proposed acceptance targets, not results. F0 and F1 depend only on the accepted conventional libint RHF/RKS slices of the Fock bridge. F2 needs smearing occupations and a global build on fractional, non-idempotent densities. F5 consumes the MPI, GPU and MANTA programmes as they land and does not block F0-F4.
Decisions¶
Recorded on #861, 2026-11-03:
F1 and F2 both start after F0. They share only F0 and may run in parallel.
The tracker has one hub issue plus one issue per increment.
Still open: the public surface. Keep vibeqc.fragment experimental and
explicit-import until F3 acceptance, then decide on
run_job(fragmentation=...).