vibe-qc/Documentation

CPU tensor slice ADR and measured source record

Date: 2 November 2026. Implementing issue: TENS-05 (#798), with bounded capabilities from #794-#797. This records the first C++17 implementation, not completion of those issues or scientific acceptance of a runtime.

Subsequent source evidence: the bounded planner ADR records #800 RI-J lowering and structural schedule reuse, plus unchanged complex-kernel regression checks. The historical measurements and limitations below describe this earlier slice.

Inventory and method boundary

Route	Actual storage and operation	State and next owner
Native DensityFitting::build_J in cpp/src/df.cpp	B_per_P_ is a vector of independently allocated column-major FP64 matrices. Each has strides (1,nAO). gamma[P]=sum(B[P]*D) is a bilinear Frobenius reduction; J=sum(P,B[P]*gamma[P]) accumulates in ascending P.	Uses the common checked index program and admitted registered Eigen kernel. #798 owns complete runtime acceptance and the following quarter AO transform.
Native construction and retained owners	Construction solves L B=T, then unpacks per-P factors. L_, optional cached V_inv_, factor storage, vector capacity and the DF object are live during J. D is caller-owned.	Inventoried; construction and whitening unchanged. The J adapter includes these retained DF owners, D, output, gamma and metadata.
Screened J, RI-K, transforms, gradients and public Python DF	build_J_screened, build_K_density, mo_transform, gradient helpers and python/vibeqc/density_fitting.py retain their existing algorithms.	Inventoried entry points, not migrated. #798 owns the bounded AO extension; #805/#810 own the other routes. Broader #794 inventory remains open.
Gaussian local panel	PeriodicGaussianLocalOrbitalFactorPanel is complex FP64 row-major [auxiliary,left,right], strides (left*right,right,1). Rectangles have independent left/right selections, q and auxiliary slices.	Interface inventory only for #799. Preserve producer identity, finite-image source, HF state, whitener and gauge, dependent occupied/virtual selections, compensation, work caps and node inventory.
Local factor consumer	PeriodicCorrelationLocalFactorBlock is complex row-major [auxiliary,virtual]; OV and VO are independent. The contraction uses conj(L_ai)*L_jb.	#799 must bind dependent domains and explicit conjugation. No q/-q shortcut, real projection, source substitution or full local-factor materialization is authorized by this slice.

AO and auxiliary domains are minted within the synchronous method call and shared explicitly with its density and provider. The existing matrix API cannot certify a caller’s physical density source from dimensions alone; that association remains the method’s responsibility. There is no persistent cross-call plan cache or invented Fock/source adapter.

Decisions implemented

cpp/include/vibeqc/tensor/core.hpp provides runtime-rank FP64 and complex FP64 views. Rank zero is scalar; any zero extent is empty and does not form an element pointer. Checked products, signed stride spans, origin, alignment, byte ranges and work products precede execution. Reversal, permutation and slicing are explicit strides plus method-supplied domains. Broadcast is read-only; writable overlap is conservatively refused. Unsupported reshape and automatic domain transport have no API in this increment.

A domain binds an opaque space/revision, an optional parent space/revision and coordinate, and an ordered affine or selected coordinate map. Equal extents cannot establish compatibility. Selected maps are immutable, charged and retained. These are coordinate semantics, not chemical source certificates.

Owned allocations survive through views and leases. Native imports need an explicit BorrowScope on the same ledger. Expired scopes fail before access. An importer promises that backing storage remains unchanged and live during that synchronous scope, including through other raw aliases. Read/write leases reject conflicts on the same imported allocation; write acquisition advances its version and invalidates old input bindings. Import each physical allocation once and derive views to share identity and avoid double charging. Separate overlapping imports are conservatively recognized by byte intervals for output alias rejection. Async imports and untracked external mutation are not supported safety guarantees.

IndexProgram checks explicit input/output labels, repeated input labels (diagonals), free/batch/reduction domains and checked work counts. The binary executor supports scalars, arbitrary runtime rank and explicit conjugation. It performs no algebraic rewrite, implicit dtype conversion or reduced precision. Alpha zero skips numerical input evaluation; beta zero does not read output. Input metadata/lifetime checks still apply. Empty reductions have additive identity zero. General reductions follow the index program’s first-input encounter order; there is no cross-backend bitwise promise.

ri_j.hpp registers cpu.eigen.ri_j.per_p.v1 against two instances of the same index checker. A logical [P,mu,nu] provider schema never becomes an affine rank-three allocation. The first capability is stable resident, replayable FP64 panels with zero provider workspace. Single-pass providers and unknown/nonzero workspace refuse this schedule. Null evaluated payloads and unavailable tiles fail; only an explicit payload-free exact_zero tile can be omitted. All tiles are checked before numerical work. Failure releases scratch and never publishes a partial J.

The registered kernel preserves the predecessor’s Eigen Frobenius reduction and ascending-P output accumulation, including the ten-million-term OpenMP threshold and independent output-column ownership. It suppresses nested teams and never changes global BLAS settings. General strided D uses an explicitly admitted column-major pack. Eigen coefficient-wise reductions and updates have no dynamic backend packing workspace. The default path does not pack B. There is no mandatory backend, compiler stack, GPU/MPI or standard upgrade.

Resource boundary

One shared Budget uses atomic-under-mutex reservations and retains tickets through owner/plan/result lifetimes. The ledger counts its fixed control, logical owner/view/IR/provider objects, copied axis/label metadata, selected maps, allocation extents, factor-vector capacity, caller-declared additional live inventory, tile snapshots, gamma, density packing and private output. Fixed executor control includes a 256-byte allowance for kernel view/lease objects. Caller schema-vector capacities are conservatively charged alongside the retained copies. No unbounded planning search or global factor pack occurs.

A complete execution preflight happens before provider visits. Each allocation then holds its own reservation; competing admissions can still cause a clean refusal before computation. There is no wait while holding partial resources. The snapshot needs O(P) metadata for the already resident P allocations, not metadata for every logical element. This schedule does not claim streaming of a logical dataset larger than resident storage.

This is a ceiling on tracked requested bytes, not process RSS. Allocator headers, shared-pointer implementation overhead, compiler stack frames, OpenMP/BLAS runtime caches and OS pages are outside that ceiling. The packed GEMM benchmark has opaque BLAS workspace and is not admitted as a strict production kernel. All numerical and variable metadata owned by the slice are accounted; process-global runtime overhead is reported separately.

The legacy build_J(D) API has no caller memory-cap parameter and explicitly uses the maximum representable tracked limit. It retains bounded scratch O(P+nAO squared), checks overflow and reports df.j_tensor_tracked_peak_bytes through component timing. Native callers using Budget can impose finite limits. The surrounding SCF/basis/integral inventory is outside this J execution scope; this diagnostic is not a whole calculation memory cap. A future finite SCF cap must import that external inventory explicitly. Returned J transfers to the existing caller inventory.

Focused source verification

The native fixture in tests/native/tensor_engine.cpp checks scalar/empty and ranks 1/4/8/12/40, unequal complex contractions, conjugation, diagonals, reversed/sliced/transposed/broadcast views, overflow/alignment, aliases, revised/dependent/ordered domains, owner retention, expired borrows, stale bindings, conflicting leases, provider failure, exact structural absence, non-replayable refusal and exact/one-byte-below caps. Small RI-J cases use independent explicit sums; the 257-by-199-by-197 case exercises two workers. Tracked reservations return to their expected live baseline on failures.

Portable focused invocation with an installed compiler and Eigen pkg-config:

python -m pytest --noconftest tests/test_tensor_engine.py -q

The isolated invocation intentionally compiles these current headers and does not import the QC extension or load repository-wide chemistry fixtures. Four cases passed. The same native cases passed AddressSanitizer and UndefinedBehaviorSanitizer; the OpenMP explicit-sum case passed. The actual changed df.cpp translation unit compiled with the existing CMake flags and dependencies. No full CI was requested.

The existing local Python environment could not run test_df_storage_contract: its basis overlay lacks a required ECP, and its October native extension lacks build_jk_gamma_physical required by current Python. No stale-runtime override, old-runtime scientific claim or environment replacement was used.

Measurements and next acceptance

The reproducible harness is benchmarks/tensor_ri_j.cpp. It compares the predecessor kernel, repeated use of the checked plan, full import/plan/execute, and explicit pack plus two GEMMs with one density RHS. Inputs use fixed trigonometric formulas and asymmetric rectangular controls. The predecessor and registered result agreed exactly for all measured cases. Packed GEMM maximum absolute differences were at most 8.89e-16.

Measured on Apple M5 Max, Apple Clang 21.0.0, Eigen 5.0.1 and Accelerate, C++17 -O3 -DNDEBUG -DEIGEN_USE_BLAS, no fast-math. BLAS used one thread; OpenMP limits were one and two. No CPU affinity pinning was used. Warm predecessor/new timings are medians of 30 alternating paired calls after five warmups; other timings are 25-call medians. First plan execution is a single sample, with validation excluded; it does not establish cold hardware caches. Frequency/placement variation is visible; these are local microbenchmarks, not complete-method speedups.

m x n, P	OMP limit	Plan us	First execute us	Legacy us	Warm us	Full us	Pack allocate + copy us	GEMM us	Tracked engine peak bytes
24 x 24, 84	1	0.583	15.541	13.000	13.208	13.833	6.750	1.250	461697
31 x 47, 127	1	0.500	54.542	53.041	53.166	54.209	25.624	4.667	1642993
97 x 97, 193	1	0.625	606.958	585.417	586.666	580.166	238.000	88.625	14991553
257 x 257, 521	1	0.792	10957.250	10933.959	10935.000	10998.041	4581.916	3488.083	278564993
24 x 24, 84	2	0.542	14.625	12.375	12.458	13.625	6.709	1.875	461697
31 x 47, 127	2	0.542	54.083	52.250	52.583	53.000	23.791	4.417	1642993
97 x 97, 193	2	0.584	580.833	575.708	576.459	574.458	240.667	77.375	14991553
257 x 257, 521	2	0.958	15212.875	14868.583	15012.584	14762.625	4529.042	3335.542	278564993

Engine factor packing is zero in these dense-D cases. Explicit-pack baseline additional tracked numerical payloads are 396960, 1504640, 14679584 and 276353184 bytes respectively, excluding opaque BLAS workspace and allocator cost. Its observed pack-allocation-plus-copy cost amortizes within one use against the measured per-P path in every case. This is evidence for #800 to investigate bounded packing, not permission to double production factor storage. A general strided-D pack is covered by correctness and budget tests; its separate performance crossover has not been measured.

Warm matched-policy overhead is at most 1.7% in these runs; no engine speedup is claimed. Two OpenMP workers are slower than one at the largest shape on this machine for both predecessor and new kernel. One worker is the fastest qualified measured existing policy for that case. The inherited threshold is preserved; thread/backend selection is unfinished #800 work. Full-call and first-use measurements also include frequency/placement variation. Whole benchmark process RSS was about 586 MB with both the original and packed factors live. It is not the engine’s tracked peak or an incremental workspace measurement. Raw timing/resource records remain outside the checkout.

Complete H2O RHF and pure-PBE calculations through vq, fixed-density physical-factor checks, predecessor comparison and matched ORCA references are pending a current built runtime. The local extension predates this change and is incompatible with current Python, so no scientific job was submitted with it. The numerical verdict is pending, not PASS. Reuse the water/def2-SVP/def2-universal-jkfit geometry and roundoff bound in tests/test_df_storage_contract.py; the native coordinates are in bohr. Before observing a new complete-method result, require predecessor energy agreement within 1e-9 Eh and density/Fock maximum differences within 1e-8, with unchanged charge/spin, integral thresholds, SCF convergence and PBE grid. Record convergence residuals and iteration counts. ORCA comparison must first match geometry/units, basis/ECP, auxiliary metric/rank, RI-J versus RI-JK, functional/grid and convergence; establish its reference tolerance from that matched fixture rather than comparing unmatched approximations. Actual Python and native source/digest identities must accompany those future results.

Evidence reused and remaining capabilities

Reuse the completed library study’s primary reading: mdspan mapping/accessor design, 10.1109/P3HPC49587.2019.00011; TAMM III-IV, 10.1063/5.0142433; Matthews sections 4-7 and 9.1, 10.1137/16M108968X; and GETT sections 3-5 and 7-8, 10.1145/3157733. The departures are runtime rank rather than mdspan’s template rank, a small synchronous C++17 contract rather than TAMM’s distributed runtime, and an existing Eigen path rather than a packing-aware replacement. No publication’s performance measurements are presented as vibe-qc measurements.

Still open: exhaustive inventory (#794), split/fuse and transport APIs (#795), einsum parsing/general multi-expression DAG (#796), transient providers and async events (#797), quarter AO transform and scientific acceptance (#798), complex local-factor migration (#799), bounded pack/GEMM and thread crossover (#800), other representations and all remaining method migrations. Parent #793 remains open. Source landing and these remaining acceptance tasks are distinct states.

Standalone portable benchmark build (Eigen’s default backend):

c++ -std=c++17 -O3 -DNDEBUG -Icpp/include \
  $(pkg-config --cflags eigen3) benchmarks/tensor_ri_j.cpp -o /tmp/tensor-ri-j
/tmp/tensor-ri-j 1

To reproduce the measured backend, additionally define EIGEN_USE_BLAS, link Accelerate, and enable/link the installed OpenMP runtime. Use VECLIB_MAXIMUM_THREADS=1 and OMP_NUM_THREADS=1 or 2, passing the same OpenMP limit to the executable. The portable Eigen-only command does not reproduce the reported Accelerate GEMM performance.