Massively parallel molecular and periodic execution¶
Status: proposed, unscheduled architecture, 2026-11-02. This adds no runtime capability, dependency, release gate or measured speedup. The work tracker assigns bounded implementation slices and the measurement contract separates source, scientific and scaling acceptance. It extends the existing MPI strategy, tensor engine, GPU programme and Fock/operator contract.
The #804 descriptor/completion ADR specifies portable identities, bounded leases, publication and synthetic complex fixtures. The bounded CPU binding/completion slice now has portable byte envelopes and real 1/2/3/empty-rank checks through the opt-in #820 transfer adapter. GPU and physical distributed consumers remain open; no scaling result is claimed.
Objective and starting evidence¶
Optimize time to a specified scientific result within an explicit resource budget. Minimum latency, campaign throughput, capacity beyond one node, and node/GPU-hours are different objectives. Report their tradeoffs; no single configuration is presumed optimal. Energy claims require measured power with a stated accounting boundary. Independent jobs remain scheduler work, not another queue inside the numerical engine.
The configured library collection massive-parallel-vibe-qc-2026-11-02, study
commit ebecf6db919f3459d2e1224da4f8142818f57cb7 and handoff commit
07e96d336fff744ef075c1230ca1ac81b88172fb, supplies 31 evidence rows, methods
locators, models and acquisition provenance. Six missing main papers were
filed there. This design reuses that reading, with primary anchors below;
it does not claim to have reproduced those papers’ benchmarks.
Source rechecked at 37e53157:
Existing component |
Actual boundary |
Consequence |
|---|---|---|
|
Optional activation, serial identity, launcher checks and at least FUNNELED thread support; k and lattice complete-output partitions |
Extend ownership and collective protocols; do not recreate activation or allow native worker MPI calls |
|
q-group factor construction and bra-row exchange work; |
Work distribution is useful but factor capacity remains replicated |
|
Slab collocation, complete-grid Allreduce, replicated FFT-Poisson and projected AO reduction |
Distributed FFT and grid capacity remain new work |
|
Initial checked CPU RI-J and complex local streaming slices |
Reuse views, domains, leases and budgets. Their current tracked-allocation boundary is not a whole-process memory guarantee |
#472/#594, #580/#581, #617/#639 |
Physical Fock/state, local source/factor admission and MANTA numerical kernels |
Distribution changes execution, not their scientific authority |
One-node measurements and source tests do not establish multi-node scaling. Existing #96 GDF scaling and #144 experimental chi output-farming acceptance remain open on their own terms. A new plan neither closes them nor takes their ownership. CPU RI-J/stream source landing under #798/#799 is not blanket scientific acceptance of distributed consumers.
Architecture decisions¶
Keep Python as coarse orchestration and C++17 as the numerical core. Extend TENS-11 (#804) for distributed placement and execution and TENS-10 (#803) for device transport. #793 owns common tensor semantics and resources; #800 owns bounded planning. No second tensor planner, Fock engine or integral engine.
Separate four facts: the logical scientific object, its authoritative storage, where an operation executes, and the lifetime of each consumer. Dense, block-sparse, ragged, packed, streamed and factorized representations remain valid. An opaque specialized kernel may participate if its effects, sources, resources and completion are declared. Dense materialization is never a requirement merely to enter the interface.
Initial execution policy:
Preserve the fastest qualified single-node path and explicitly admitted replicated small-state execution. Multiple nodes do not imply a speed win.
Use stable cost-weighted complete-output/grid/shell tasks before fine remote updates. Cost includes screened work, message volume and maximum task size.
Distribute factors, grids and matrices when capacity or measured latency requires it. Whole-object allgather is an explicit small-object capability, with a preflight bound, not a fallback for unsupported distribution.
Use a fixed logical reduction order when requested; ordinary strict FP64 permits documented rounding changes. Neither mode promises bitwise equality across BLAS, devices or process counts without a qualified accumulator.
Keep approximations method-owned. Placement cannot change fitting bases, screening, domains, precision, exchange model, Hamiltonian or convergence.
Execution contexts and communicator roles¶
The logical hierarchy is job/method, spin/k pool, tile or q/pair team, and node/NUMA/device placement. These roles need not form one fixed Cartesian tree: exchange may require a redistribution between k and q layouts. Construct a small number of persistent communicators collectively. Do not create a communicator per tile, multiply ranks by every axis, or use rank as object ID.
The initial MPI owner is the initializing/main thread of each rank. Existing FUNNELED admission remains; native OpenMP workers perform MPI-free computation. The owner progresses bounded requests between submitted compute batches and backend events. A blocking native call can prevent useful overlap. Measure that cost before changing the model; a progress thread or native MPI requires an explicit higher-thread-support design and qualification, not a hidden fix.
Each phase defines participant communicator, sequence and epoch, reductions, and next collective before workers start. Empty ranks participate. Collective admission exchanges compact status after local preflight and before payload communication. Recoverable provider errors are reported at agreed phase boundaries. A transport failure that prevents agreement terminates the affected job according to the MPI implementation’s supported policy. Portable MPI cannot promise bounded recovery from a disappeared process. ULFM is separate research.
MPI request cancellation, especially for collectives, is not a general rollback mechanism. Do not free an active buffer or communicator to handle an exception. Drain only operations whose completion can safely be established; otherwise terminate without publishing partial output as a valid epoch.
Distribution and completion contract¶
These are proposed fields and operations, not a new public ABI:
Contract |
Required contents or effect |
|---|---|
Distribution descriptor |
Tensor/schema ID, ordered global domain IDs and revisions, extents/block map, dtype/conjugation/representation, scientific source/frame hash, partition revision, ownership and allowed replica policy |
Block key |
Global domain coordinates, source epoch and structural identity, independent of rank and local address; ragged extent and reconstruction rules supplied by provider |
Owner map |
One authoritative writer, or declared partial-contribution owners plus exactly one reduction destination; deterministic mapping or bounded directory |
Replica |
Read-only source epoch, placement and invalidation rule; stale replicas cannot satisfy a newer operation |
Read/write lease |
Local owner/provider generation, access exclusion, reservation and remote consumers; no borrowed pointer serialized as data |
Task descriptor |
Logical inputs/outputs, required source epochs, contributor IDs, cost/work bounds, communication credits and completion dependencies |
Transfer ticket |
Packing reservation, producer event, MPI request, receive validation, consumer event and release action |
Publication |
All required contributions validated and combined once; atomic transition from private partial output to readable epoch |
The lifecycle is reserve, acquire leases, pack/produce, transfer, validate, compute/reduce, publish, release. A send completion permits reuse of its send buffer; it does not mean a remote GPU consumer has finished. A remote lease is released only after its last declared consumer acknowledges completion. No eviction, epoch reuse or retry may duplicate a contribution. Track provider failures and delayed completions as well as the successful path.
Validate checksummed envelopes/schema lengths, checked count/offset arithmetic, MPI count-width limits and chunking before allocation. Structural zero, screened omission, unavailable block and transport failure are distinct states. Retries use bounded IDs/epochs and exactly-once accounting; the initial design may refuse rather than implement retry. Avoid replicated directories whose metadata scales like the entire global tensor.
Resource admission and placement¶
Use the existing resource ledger with separate rank, node/shared and device inventories. Reserve simultaneous owned blocks, replicas, packs, intermediates, solver state, requests, backend workspace, caches, transfer buffers and any checkpoint overlap. Node-shared allocations count once per node. Enforce a bounded number of in-flight blocks and receive credits before sending payloads.
Distinguish tracked allocation caps from measured process RSS, device residency and scheduler limits. MPI, BLAS and allocator memory need measured headroom or documented external reservations. Strict admission refuses unknown unbounded workspace; it cannot turn the current CPU ledger into an OS memory guarantee. Measure live concurrency, not the sum of unrelated phase minima. Budget failure reduces tiles/concurrency, streams or recomputes within the same method, or refuses collectively. It must not loosen scientific thresholds.
Begin with static weighted placement and reusable layouts. Bounded coarse stealing is a later response to measured imbalance, with provider/owner locality in the cost. Compare recompute against network, disk and cache opportunity cost. One rank per NUMA domain or GPU is a candidate, not a universal default. Account for OpenMP, BLAS and backend thread teams together. Calibration remains optional, bounded and keyed by actual shape, topology and software identity.
First complete molecular and periodic routes¶
Molecular HF/DFT¶
MPP-05 (#823) first distributes fixed-density, already whitened RI-J panels with replicated small density/output state, then a complete admitted RHF calculation. #805 retains the native RI-J registration; #804 supplies placement and lifetime. The complete path must join the native SCF loop through a coarse builder seam; do not duplicate SCF in Python or perform a Python callback per integral. If that seam is missing, implementing it is the first bounded prerequisite.
Metric construction and whitening remain globally coupled. Independent raw auxiliary-row solves do not implement the existing factorization. Subsequent slices cover screened direct J/K, DF-K, grid XC and seminumerical exchange with their own scientific identity. Keep local matrix accumulation and coarse reduction initially; direct remote updates for every quartet are not the plan. Large AO density/Fock/DIIS storage later needs distributed assembly, solver and consumers together. Merely distributing integrals does not solve that limit.
Multi-k SCF and occupations¶
MPP-03 (#821) starts with one existing restricted multi-k GDF HF route, keeping small state explicitly replicated where necessary. Pools own selected k outputs; the method defines full/irreducible maps, spin constraints and residuals. The first slice integrates eigenproblems, global occupations, density rebuild, energy, convergence, terminal physical-state output and restart association. It is not a claim that every existing Fock operation is distributed.
For weights normalized to one, the electron count per cell is
N(mu) = sum(k,spin,i) w[k] f(eps[k,spin,i] - mu), with spin degeneracy included
exactly once. Each occupation-root evaluation reduces local counts, and slopes
if used, over the declared global constraint. Constrained-spin methods may have
separate declared chemical potentials. Pool-local Fermi levels are invalid.
Metallic/smeared slices need sufficient eigenstates or an occupation-tail bound.
Keep internal energy, entropy and free energy distinct. Reduce the method’s
weighted residual and an explicit maximum diagnostic; an average cannot hide a
failed k block. Final output uses one coherent evaluated state, not a mixture
of pre-update energy and post-update orbitals.
q/pair-owned GDF factors¶
MPP-04 (#822) replaces one selected allgathered cache with a provider whose whitened factor blocks have q/pair ownership and bounded panel access. Start with the existing complex exchange contraction at fixed densities. Keep full-BZ ket sums when only IBZ bra outputs are requested. Distribute output reductions or move bounded factor/density panels according to the measured contraction shape. Cache policy includes partner factors and metric storage, not only the panels visible to the contraction kernel. Capacity acceptance includes construction.
Mesh shifts, k/q maps, reciprocal-lattice identity, Bloch phases, complex adjoints, symmetry/star multiplicities, variable metric ranks, Coulomb q=0 and exchange-divergence conventions travel as method-owned identities. A q index from another mesh is not interchangeable. Do not assume q/-q reuse or real Gamma arithmetic without source admission. Lossless redistribution preserves the scientific source and must pass round-trip reconstruction checks.
GPW grids and distributed FFT¶
MPP-06 (#824) owns one Gamma GPW Hartree route: screened collocation to owned grids, layout conversion, distributed FFT-Poisson, inverse conversion and projection. The grid descriptor contains cell, global mesh, local box/strides, normalization, boundary/kernel identity and communicator. Every transpose/pack workspace is budgeted. No complete grid is assembled for a distributed-capacity claim.
Evaluate optional heFFTe against the current FFT using pinned API/build and license evidence; keep a numerical-library-neutral layout boundary. The first prototype may remain separate until that dependency choice is resolved. A declared small-grid replicated fallback is valid but cannot pass capacity acceptance. GAPW augmentation/compensation, atom-local grids and derivatives remain #604/#811 work and need their own communication and scientific checks.
Solvers and matrix capacity¶
MPP-07 (#825) compares direct ELPA with the admitted alternatives under #472, #594 and
python/vibeqc/solvers/eigensolver.py. Include matrix assembly/redistribution,
overlap analysis, rank decision, orthogonalization, standard solve, backtransform
and redistribution to density/response consumers. Distributed vectors alone
do not remove a replicated Fock or DIIS history limit. Keep small replicated
Eigen/LAPACK execution where it wins. Optional-library adoption requires a
concrete build/license/API decision; none is made by this design.
Full spectra, extremal subspaces and matrix-free actions are different requests.
ChASE/block iteration is a measured alternative for compatible Hermitian
subspaces; metals, large occupied fractions and required virtuals may eliminate
its advantage. Preserve physical/effective Fock roles and overlap rank. Compare
FC - SC eps, C^H S C - I, density and subspace projectors near degeneracies.
Non-Hermitian response is not a Hermitian Fock solve. Density-matrix solvers,
mixed precision and lossy compression remain separate scientific experiments.
Correlation, response and remaining consumers¶
Existing owner |
Distributed first increment |
Required later coverage and limit |
|---|---|---|
#806 canonical MPn/CC/triples |
Fixed-source DF-MP2 occupied-pair contributions, then complete MP2 |
Bounded triples worklists and persistent CC amplitude/intermediate layouts; fixed-amplitude residual checks before converged comparisons; no global rank-six temporary |
#807 local correlation |
One coupled pair-residual neighborhood across ranks |
PNO/PAO transforms, ghost amplitude epochs and pair/triple multiplicities; noncanonical MP2 and local CC are not independent pair-energy sums |
#808 CI/CAS/MR/DMRG |
Fixed selected determinant sigma action with exact duplicate aggregation |
RDM/perturber partitions, CAS macroiterations, symmetry-block MPS/operator ownership; do not force dense CC storage or change selected spaces/bond cutoffs |
#809 response/properties |
One Hermitian TDA block action plus global dots |
Distributed orthogonalization, root tracking and EOM/non-Hermitian admission; state-property farming alone does not distribute the solver |
#810 derivatives |
One existing analytic derivative of an accepted primal route |
Response/Pulay/grid/metric terms, Hessians, strain/stress and k/cell derivatives; energy parity does not qualify derivatives |
#811 periodic overlays |
Common k/q/source identities for MPP-03 (#821)/04/06 |
Ewald, COSX, GAPW, BIPOLE and experimental CCM/chi consumers retain their own source domains; #144 acceptance is reused only where applicable |
#812 semiempirical |
One supported SCC charge/energy reduction over coarse blocks |
Tiny irregular kernels and declared spin/parameter/cell routes; existing AICCM/CCM and SECCM experimental labels and refusals remain |
#803 GPU transport |
One bounded real/complex transfer plus registered operation |
Explicit producer/consumer events, staged and direct transports, pinned/device caps and actual progress; no inferred CUDA/HIP/Metal parity |
These are distribution extensions to existing owners, not a new ownership tree for their scientific migrations. Unsupported physics is recorded as unsupported. Implemented routes awaiting distribution keep a next slice and owner. Distribution qualification is additional to local migration, not retroactive invalidation of already accepted CPU work. No unrelated family must finish before a useful slice.
Restart, output and failure¶
MPP-08 (#826) builds on #613 and vibeqc.output, including existing checkpoint/state
schemas. Initially checkpoint a completed SCF iteration or other declared safe
method boundary. Preserve scientific inputs/source hashes, physical evaluated
state, global block keys, precision/schema, solver history and validity rules.
Write chunks with hashes, then an atomic completeness manifest. An interrupted
write never replaces the last committed checkpoint. Scratch is not automatically
durable. Admission includes I/O concurrency and staging memory.
Restart at a different rank count validates compatibility and repartitions
global keys; it must not require old rank IDs or gather the complete state on
rank zero. Recompute state only if the schema explicitly declares that safe.
Historical schema/API compatibility is preserved or versioned deliberately.
Rank zero emits normal progress and scalar summaries through vibeqc.output;
large outputs use declared chunk/stream writers instead of an implicit gather.
Record actual topology/backend/source identity using public-safe result metadata.
Portable checkpoint/relaunch is the committed design target. In-place recovery from rank loss, ULFM, spare ranks and GPU reconstruction are optional later work requiring an explicit failure model and exactly-once contribution restoration.
Research decisions and evidence limits¶
The table gives catalog keys, primary reading locations and the limits of adaptation. Version/API/license snapshots live in the library, not product deployment files. Recheck the selected dependency when implementing it.
Anchor |
Design use and material departure |
|---|---|
|
Separate lattice Fock, k/solver and distributed storage. Use actual smallest-feasible baselines, not extrapolated small-allocation timings; modern vibe-qc scaling remains unmeasured |
|
Adapt screened cost weighting, not its scientific model or fixed-density Fock timing as complete SCF evidence |
|
Persistent layouts, bounded DAGs and replication tradeoffs; keep vibe-qc’s C++17 contracts, no mandatory external runtime or imported chemistry |
|
Topology-aware mapping must repay redistribution under the same memory cap; retain all timing samples rather than silently trim slow ones |
|
Preserve metric, q and singularity conventions; compare regeneration with transport/storage rather than assume it is cheaper |
|
Reuse spin/k solver roles and subspace ideas; Gaussian overlap/source admission stays local and solver alternatives remain optional |
|
Distributed grid/transpose design; Gaussian collocation and GAPW augmentation are additional work, not FFT capabilities |
|
Coupled neighborhoods and solver-memory accounting; do not substitute another PNO model, regularization or truncation for exact scheduling |
|
Include response and differentiated ownership; SOS-MP2 is not canonical MP2 and unavailable SI is not reproduced |
|
Exact determinant dedup and symmetry-block operator ownership; no neural selection or bond-truncation change, no capacity-to-speedup inference |
|
Communicator repair does not restore scientific state; initial restart is relaunch-based, not fault-tolerant MPI |
Remaining library gaps: foundational DBCSR PDF, Bussy SI, GauXC SI,
unidentified DMRG SI, and unverified HIP-DFPT full text. The collection’s
pending-sources.json and si-intake.json retain intake status. These are not
read sources and do not block unrelated planning or implementation.
Rank/thread/GPU layout, replication crossover, progress, persistent-layout amortization, direct ELPA versus ELSI, FFT backend and checkpoint granularity remain bounded measurement/ADR decisions. The plan permits baseline and protocol work now, with no dependency adoption or performance guarantee.