vibe-qcDocumentation

Tensor engine architecture review for coupled-cluster and multireference work

Date: 14 November 2026. Parent: TENS-00 (#793). Source baseline: main d54ee9fb2. Line numbers below refer to that commit. This review supersedes the planning and backend sections of the semantic proposal where the two disagree. It keeps the proposal’s semantic invariants. Measured tables in the existing ADRs stay as historical evidence and are not rewritten.

Two things changed on main while the review ran. The virtual-pair ladder landed in 6a821317a, and a claim for one ring contraction is open on #806. Both continue the one-plan-per-contraction approach. Their effect on the findings is stated where it matters.

This document creates no approval, release or CI gate. Production changes proceed through the bounded tasks in the migration plan below.

Verdict

Amend substantially. Keep the semantic core. Replace the planning layer, which exists only as prose. Change the backend and resource policy, which cause the measured losses.

Layer

Verdict

One-line reason

Semantic core: spaces, domains, views, leases, checked arithmetic

Retain

Sound, small and tested; the checks cost microseconds

Provider and stream protocol, dependent traversal

Retain, narrow

Correct for RI and local factor panels; not a general sparse layer

Operation IR

Replace above the binary node

One binary contraction exists; no graph, sum, permutation or reuse

Planner and schedule cache

Replace

A two-kernel RI-J choice and a one-entry cache; no order, layout or lifetime planning

Per-operation plan classes

Change into registered kernels

Each builds an index program that execution never consults

Resource ledger

Amend

An operation-phase cap fed by a hand-written inventory; no iteration or method limit

Kernel and backend policy

Change

Strict admission excludes the vendor GEMM and costs a factor of 4 to 9

Distributed descriptors

Amend

A second descriptor vocabulary for one rank-two operation

Python submission

Retain, re-baseline

The remaining NumPy gap has the same backend cause

Multireference scope (#808)

Re-scope

No multireference code uses the engine; most of it is not tensor algebra

The engine can check and admit a single dense contraction. It cannot yet optimize a complete coupled-cluster iteration, and it has no representation that the implemented multireference code would use. The memory guarantee that justifies the slower kernels is narrower than the name suggests: it bounds the tracked bytes live during one operation, using a list of matrices that the method author maintains by hand.

What exists and what is prose

The headers under cpp/include/vibeqc/tensor/ total about 3,200 lines.

Design claim

State at d54ee9fb2

Opaque spaces, domains, selections, signed strides, checked offsets

Implemented in core.hpp

Allocation identity, borrow scopes, read/write leases, value versions

Implemented, synchronous only

Binary indexed contraction with diagonals, batches, alpha and beta

Implemented as a scalar reference loop (Contraction::execute, core.hpp:493). No production caller

Einsum frontend

Not implemented

Multi-expression DAG, SSA values, common-subexpression reuse

Not implemented. core.hpp:411 states that no reassociation or reuse is performed

Add/scale, permutation, cast, gather/scatter, reshape nodes

Not implemented as IR nodes

Planner: order, layout, fusion, slicing, lifetimes

Not implemented. MoTransformPlan picks left-first or right-first; RiJPlan picks one of two kernels

Plan cache

One mutex-protected entry holding a kernel tag (structure.hpp:84)

Reservation ledger

Budget, one per operation call

Provider protocol, replay and assembly policies

Implemented for resident RI factors and one dependent affine fiber

Sparse assembly, packed symmetry, factor graphs, block sparsity

Not implemented

Asynchronous completion

A single-threaded fake executor and a CPU transfer session for one rank-two complex product

GPU adapter

Not implemented

Production call sites are RI-J, the DF AO-to-MO transform, the DF-MP2 and DF-UMP2 OV products, the complex periodic local panel and pair leaf, and one CCSD contraction at the baseline (two after 6a821317a). Each has its own plan class with a hand-written kernel.

Workload trace

Coupled cluster

Route

Implementation

Structure the engine must express

Restricted CCSD, native

ccsd.cpp compute_residuals, lines 637 to 1062. Pair-flattened column-major matrices. About 40 terms: three one-body intermediates, Wmnij, three ring intermediates, two ladders, a nine-term half and a pair symmetrizer

Multi-output graph (R1 and R2) with shared F_me, F_ae, F_mi, tau, taut; scaled sums of permuted operands (2t - t, 2J - K); output symmetrizer; amplitude epochs

Particle-particle ladder

In-core vv_vv when it fits 2 GiB, else tiles regenerated from B_vv (lines 924 to 987)

A provider-fused tile loop feeding a consumer, with the tile size chosen from the budget

CC2, LCCD, CEPA, QCISD

The same residual evaluated 2, 4 or 15 times per iteration on a stencil (lines 1081 to 1266)

Term selection on one graph instead of numerical extraction

Unrestricted CCSD

uccsd.cpp: spin-orbital loops over one dense antisymmetrized array of size n to the fourth. Its header calls it a small-molecule pilot

Spin blocks aa, bb and ab as distinct spaces. Not present today

Perturbative triples

Native, batched over occupied triples with per-thread cubes and a real memory planner (lines 1493 to 2852)

A registered kernel with declared per-thread scratch. Already has admission. Leave specialized

CC3, CCSDT

Python determinant dictionaries, benchmark scale

Not tensor algebra. Out of engine scope

Lambda, densities, gradients

Python reverse-mode sweep through the einsum residual (dlpno/_ccsd_cs.py, lines 88 to 253; gradient/ri_ccsd.py). All labelled experimental

Vector-Jacobian products of the same graph

EOM-CC

None

Speculative

DLPNO-CCSD

dlpno_ccsd.cpp: one PairData per pair with ragged blocks and overlap maps S = U_p^T U_q

Keyed block families and explicit transport operands

The same closed-shell or spin-orbital equations are transcribed at least eight times: native spatial, native spin-orbital, two Python einsum references, two native PNO solvers and two periodic variants. One checked contraction in one of those eight is the current coupled-cluster migration.

Multireference

Route

Implementation

Relation to a tensor engine

CAS-CI sigma

casci.cpp:144 to 240. Gather through a CSR string table, one dense GEMM against the pair-flattened integrals, scatter

An index-map gather and scatter around a dense kernel

1- and 2-RDM

Native, spin-summed, dense

Reductions through the same maps

3-RDM

Dense n_act to the sixth, hash-map states (nevpt2_ops.cpp:321)

Consumed by an experimental CASPT2 engine and gradients

4-RDM

Python, dense n_act to the eighth. Only a test oracle consumes it

No production requirement

CASSCF

Full in-core MO integrals; every orbital step repeats ap,bq,cr,ds,abcd->pqrs in NumPy (_casscf.py:323)

A staged dense transform, which the engine already does well

SC-NEVPT2, IC-CASPT2, MRCI

Sparse determinant dictionaries keyed by bit masks; dense overlap metrics with eigh and a rank threshold

Operator-on-state actions and small dense linear algebra. Not index contraction

Selected CI

Native hash map and incremental sparse Hamiltonian

Sparse matrix-vector products. Not index contraction

DMRG

_dmrg_contract.py: a dense full-Fock diagonalization limited to six spatial orbitals. No MPS, MPO or environment code runs

Nothing to migrate

v2RDM, transcorrelated

A full-CI stub and a placeholder

Nothing to migrate

No multireference file includes a tensor header. The design document does not mention determinants, RDMs or matrix product states.

Existing, near-term and speculative

  • Existing and worth serving now: restricted CCSD residual graph, DF tile regeneration, DLPNO pair blocks, staged integral transforms, the CAS sigma dense kernel and its buffers.

  • Near-term: unrestricted CCSD in spin blocks, reverse-mode products of the residual graph for Lambda, a single source for the transcribed equations, root-batched CI vectors.

  • Speculative: EOM-CC, matrix product states with quantum-number blocks, SU(2) coupling, high-rank RDM tensors as engine values. None has a caller. The engine should not grow a schema for them now.

Findings

Correctness

  1. The checked IR is decorative in every production plan. OccupiedLadderPlan builds an IndexProgram for mnij,mnab->ijab (ccsd_ladder.hpp:49) and then calls a GEMM with hand-written operand orders (lines 58 to 72). MoTransformPlan and OvProductPlan do the same. Nothing derives the kernel call from the program, so a wrong layout argument is caught only by a numerical test. The check covers label ranks and domain equality.

  2. The memory inventory is a hand-written list. The ladder call imports 26 matrices by name (ccsd.cpp:892 to 895) plus DIIS history and outer objects (lines 3480 to 3483). A new intermediate that is not added to the list is silently outside the cap. Eigen expression temporaries in the other terms are never in it.

  3. Exact-byte admission depends on the standard library. Control reservations use std::vector::capacity() (core.hpp:332 and 428) and sizeof of control structs. The exact-cap and one-byte-below tests are therefore specific to one ABI. A byte cap that admits a run on one platform can refuse it on another.

  4. Space identity is created at the point of use. CCSD creates its occupied, virtual and frame spaces inside the solver (ccsd.cpp:3435 to 3438). The identity check then compares tokens that the same function minted. It becomes a real guard only when the orbital-space owner under #611 hands out the identities.

  5. Summation-order contracts live in duplicated source. The Python residual spells out two branches to keep bitwise results (_ccsd_cs.py:400 to 427). The engine has a per-plan numerical mode but no way to state the order of a sum of terms.

Scalability

  1. There is no iteration-level or method-level admission. At the baseline the CCSD cap covers the instant of one contraction. The in-core Wabef is allocated later in the same iteration (ccsd.cpp:956 to 965) and doubles the nv to the fourth storage, so a run could pass the cap and then need twice the memory. Commit 6a821317a closes that case by adding a second hand-written reservation for M and W. The structure is unchanged: each phase is admitted when it is reached, and half, expression temporaries and every later phase stay outside until someone adds them.

  2. A common subexpression costs live memory. In the prototype below, a schedule that shares intermediates raised the transient peak from 12 to 16 amplitude-sized arrays and added integral-epoch copies of 1.4 times the resident integrals. Sharing must be decided against a memory limit.

  3. Unrestricted CCSD has no spin blocks and stores a dense spin-orbital array. It cannot be migrated term by term; it needs the graph and spaces.

  4. The distributed operation is fixed at rank two with its own Descriptor, DomainRef and Token types (distributed.hpp:239). A graph with several operations has no distributed form.

Performance

  1. The loss to vendor GEMM is a kernel policy, not abstraction cost. In the existing tables the checked path matches the raw leaf and the Eigen-only predecessor. The gap to the vendor library is the gap between Eigen’s generic kernel and the vendor kernel. Strict admission excludes the vendor kernel because its workspace is not declared. The virtual ladder record repeats the pattern: 363 us checked against 85 us for the predecessor GEMM at (4,19), and 2,204 us against 630 us for a (8,31) tile.

  2. The fixed 64 by 64 pack tile costs another 10 to 40 percent against the same Eigen kernel with cache-derived block sizes and supplied buffers. Both are equally strict, because both sizes are known before the call.

  3. Lowering matters more than planning. The Python closed-shell residual uses unoptimized np.einsum. Routing each term to a matrix product is 18 to 50 times faster. Sharing intermediates adds a factor of 1.0 to 2.2.

  4. Per-call planning is unaffordable at small sizes. Searching a path for each term on every call cost about 25 ms per iteration against 2.5 ms of execution at water size. A graph planned once cost 18 ms in total.

Maintainability

  1. One contraction cost 479 inserted lines across 13 files (commit a0e7bc6df), for no speedup. The restricted residual has about 40 terms and exists in eight transcriptions.

  2. The kernels depend on Eigen::internal in 21 places, guarded by version checks. A new Eigen release needs a new audit before the faster paths are used.

  3. The structural cache key has about 30 fields and one slot. It protects a choice between two kernels. It is not a base for a planner.

  4. Including the ladder header adds about 2 s to a translation unit that already includes Eigen (8.5 s against 6.6 s, syntax-only, -O2). This is acceptable. A template-heavy expression layer would not be.

Evidence

All measurements below are bounded disposable prototypes run for this review on an Apple M5 Max with Apple Clang 21, Eigen 5.0.1 and Accelerate, one thread. The host was heavily loaded by other work, so each figure is the minimum over interleaved samples. Ratios are meaningful; absolute times are not. No calculation through vq was run and no scientific result is claimed.

A. Restricted CCSD residual fragment

Real fragment (_ccsd_cs.py:289 to 319 and 393; native twin at ccsd.cpp:675 to 767):

F_me = f_ov + t1[nf] * (2 ovov[menf] - ovov[mfne])
F_ae = f_vv - 1/2 f_ov[me] t1[ma] + t1[mf] * (2 ovvv[mfae] - ovvv[meaf])
            - taut[mnaf] * (2 ovov[menf] - ovov[mfne])
F_mi = f_oo + 1/2 t1[ie] f_ov[me] + t1[ne] * (2 ooov[mine] - ooov[nime])
            + taut[inef] * (2 ovov[menf] - ovov[mfne])
R1  += t1[ie] F_ae[ae] - F_mi[mi] t1[ma] + (2 t2 - t2^T)[imae] F_me[me] + ...
Fh_be = F_ae - 1/2 t1[mb] F_me[me]      (consumed by R2)

Current architecture. The engine can express each binary product, one at a time, after the method has formed 2 ovov - ovov^T or evaluated the two halves separately. It cannot express the sum, the shared operand, the two consumers of F_me, or that 2 ovov - ovov^T changes once per solve while 2 t2 - t2^T changes once per iteration. Each term would need its own plan class, call-site inventory and test, as the ladder did.

Recommended architecture. The method declares the fragment once as an expression set with value epochs:

epoch integrals:  L[menf]  = 2 ovov[menf] - ovov[mfne]
epoch amplitudes: t2as     = 2 t2 - t2^T
outputs:          R1, R2   (R2 symmetrized under ij,ab exchange)

The schedule compiler lowers it to registered kernels, places L in the integral epoch only if the iteration budget admits it, and frees taut and t2as after their last consumer.

Measured, complete residual without the particle-particle ladder, real shapes for (occupied, virtual):

Arm

Contractions per iteration

(4,19) warm, ms

(8,40) warm, ms

(12,72) warm, ms

Transient peak, units of one amplitude array

A. Repository code, one np.einsum per term

54

6.1

1920

40600

10 to 11

A’. Each term lowered alone, path searched per call

54

2.5

36

2220

12 to 13

B. One planned schedule, shared operands, cached paths

41

1.2

18 to 61

1180 to 2560

16

Arm B agrees with arm A to 5e-13. A second interleaved run gave B against a cached-path A’ as 2.2, 1.0 and 1.1 times faster at (8,40), (12,72) and (16,100). Arm A’ spent 778 ms searching paths over 31 calls at (4,19). Arm B spent 18 ms once. Arm B holds 8.4 MB of integral-epoch copies at (8,40) against 5.9 MB of resident integrals.

Reading: lowering to a matrix product is worth 18 to 50 times. A graph is worth 1 to 2 times in speed, removes per-call planning, and makes lifetimes visible. Its naive form uses more memory, which is the adversarial case in section D.

B. CAS-CI sigma action

Real kernel (casci.cpp:176 to 232), for n_det determinants, n active orbitals and a chunk of w determinants:

D[(q,p), j] += sign * c[target(I_j, p, q)]        gather, CSR string table
F[:, 0:w]    = 1/2 * eri[(pq),(rs)] * D[:, 0:w]   dense GEMM, n^2 by n^2 by w
sigma[target(I_j, p, q)] += sign * F[(p,q), j]    scatter

Storage: c and sigma of n_det; eri of n to the fourth; D and F of n^2 by 8192 per thread; one full sigma accumulator per thread (casci.cpp:162); Davidson vectors of n_det by 24.

Current architecture. Nothing applies. IndexProgram requires every index to be a dense domain axis. The string table is a sparse map between two spaces, and the scatter has duplicate targets with signs.

Recommended architecture. Do not model determinants as tensor axes. Register the sigma action as an operator kernel with a declared resource record, and let the engine own what it is good at:

  • the dense GEMM through the backend capability layer;

  • admission of D, F and the per-thread accumulators, which are the hidden cost: 18 threads and two million determinants need 288 MB of accumulators that no estimate currently names;

  • an IndexMap operand (offsets, targets, signs) with declared additive assembly, reused by the RDM reductions.

High-rank RDMs stay method objects. The 4-RDM has no production consumer and must not become an engine value.

C. DLPNO pair domains and the absent tensor network

Real storage (dlpno_ccsd.cpp:62 to 97): each pair holds U of nv by n, T of n by n, Bvv of naux by n^2, a list of Bov rows and mixed projections JA1 to JA4 whose column space is another pair’s domain. The amplitude of pair (m,n) is moved into pair p as S T S^T with S = U_p^T U_q (lines 629 to 663).

Current architecture. DependentStream iterates one affine child fiber of one parent and refuses selected slicing. It cannot name a family of blocks keyed by pair, a block whose two axes belong to different pairs, or the transport S.

Recommended architecture. A BlockFamily keyed by a method-supplied key (here the pair id), where each block carries its own domains. A domain of pair p and a domain of pair q are different spaces even at equal rank. S_pq is an explicit rank-two operand from space q to space p, supplied by the method. The projection is then an ordinary three-operand term per key. The planner sees per-key extents, so it can batch equal shapes and bound the workspace by the largest block. No padding is involved.

Tensor networks. There is no MPS, MPO or environment contraction in the repository. The “DMRG” route is an exact dense solver with a six-orbital limit. The engine must not carry bond, sector or SU(2) schemas until a method exists. A block family keyed by sector labels is the hook when one does; the method would supply the fusion rule.

D. Adversarial cases

  1. A cap that passes and a run that still needs twice the memory. Restricted CCSD with 100 virtuals and in-core integrals at the baseline: vv_vv is 0.8 GB and is in the ladder inventory. Wabef, another 0.8 GB, is allocated after the checked operation, and ccsd_contraction_peak_bytes reports the smaller number. The virtual ladder commit now reserves W. The same gap reopens for any allocation that follows the last checked phase.

  2. A forgotten matrix. Add one o^2 v^2 intermediate to compute_residuals and omit it from the list at ccsd.cpp:892. Every test still passes and the cap is wrong by that amount.

  3. Sharing raises the peak. Arm B above.

  4. A byte-exact cap moves between platforms. Finding 3.

  5. A strict policy that excludes a kernel whose workspace is one small allocation. Section E.

E. Backend prototype

C(m,n) += A^T B with depth k, FP64, one thread. “Strict 64” is the repository leaf. “Strict wide” is the same Eigen kernel with Eigen’s own block sizes and caller-supplied packs.

Shape (m, n, k)

Accelerate, ms

Eigen default, ms

Strict 64, ms

Strict wide, ms

Strict wide packs, MB

Accelerate workspace, MB

Occupied ladder, (12,40): 144, 1600, 144

1.02

4.10

4.49

4.06

1.1

0.29

Occupied ladder, (20,80): 400, 6400, 400

108

827

536

470

7.4

1.38

Virtual tile, (12,40): 1600, 1600, 144

16.7

145

318

225

2.0

0.29

Square 1024

316

860

1000

747

2.2

2.10

Square 2500

244

5970

2210

1970

5.4

8.65

The square 2500 Accelerate time is lower than the square 1024 time. That is a load artefact of this host and is reported as measured.

The vendor workspace was observed by interposing the allocator during the call. Every call made exactly one heap allocation and no virtual-memory allocation. Its size grew with the problem, from 0.29 MB to 8.65 MB, and was 17 percent of one operand at the largest shape. This is one library version, one thread and five shapes. It is an observation, not a contract.

Reading: the strict path is 4 to 9 times slower than the vendor kernel at coupled-cluster shapes. Of that, 10 to 40 percent is the tile choice and the rest is the kernel. The vendor workspace is not zero and not unbounded. It can be measured and charged.

Answers to the review questions

1. Is the abstraction boundary right? The boundary between method and engine is right: methods own equations, spin algebra, spaces and error controls. The boundary inside the engine is missing a layer. Today a method jumps from hand-written loops straight to a per-operation plan class. The layers should be: method equations; a declared expression set; the numeric operation IR; representations; a schedule; kernels. The expression set is the missing one. Symbolic second-quantized algebra stays outside. A generator may emit expression sets later.

2. Can the IR express complete reusable computations? No. It has one binary node. It needs multi-output sets, sums of products, linear combinations of permuted operands, an output symmetrizer, reductions, explicit accumulation and value epochs. A graph has the lifetime of a solve and depends only on spaces. An execution has the lifetime of one epoch and binds values. Binary wrappers prevent four things: sharing an operand, merging terms that share a factor, choosing which intermediates to keep under a memory limit, and admitting an iteration as a whole.

3. Are spaces and representations sufficient? Spaces and dense affine views are sufficient for canonical CC. Missing and needed now: spin-block spaces, keyed block families with transport operands, a tile provider fused into a consumer, and an index-map operand for determinant actions. Packed antisymmetry is optional: the native code stores full pair-flattened matrices and no measurement here shows that packing would pay. Matrix product bonds and SU(2) coupling have no consumer. High-rank RDMs must stay out of the engine.

4. Can planning optimize actual workloads? Not today. The information that must be visible before execution is: the whole term list of an iteration, the epoch of every value, the extents of every space, the resident integral inventory and the memory limit. With that, a planner can choose shared operands, order, layout, tile sizes and frees jointly. Search must run once per solve. The measured search cost of 18 ms for 41 terms is negligible against 15 to 50 iterations. Per-call search is not.

5. Are the memory and lifetime guarantees useful and achievable? Leases, versions and alias refusal are useful and cheap. The byte cap is useful only when the engine allocates what it counts. Three limits must be separate names: the operation phase (exists), the iteration schedule (missing) and the method (missing). Process RSS stays a measured telemetry value. Two execution policies are warranted and one more is honest: strict runs only kernels with caller-supplied workspace; declared_margin runs a vendor kernel and charges a qualified, measured workspace bound; untracked is an explicit opt-out that reports its workspace as unknown. A requested policy is never weakened and unknown workspace is never recorded as zero.

6. Is the backend strategy sound? The safety reasoning is sound. The outcome is not: the default path gives up a factor of 4 to 9. The immediate fixes are budget-sized packs for the strict path and a qualified vendor path. TBLIS or BLIS remain candidates for a strict path with a competitive kernel; neither was built for this review, and their pool allocation needs its own audit before a workspace claim. The dependence on Eigen internals should shrink to one adapter file. FP64 and complex FP64 stay the only types. Kernel selection stays a deterministic rule; timing is opt-in.

7. Will the design extend to GPU and MPI without another runtime? The lease, epoch, placement and wire envelope pieces are reusable. The operation descriptor is not: it duplicates the local vocabulary and fixes one operation. A schedule step should be the unit that is placed, transferred and completed. The capability record needs a placement field and transfer bytes in the ledger. Hardware performance is unmeasured.

8. Does it deliver practical value? Checked execution: yes, for six routes. Optimization: no measured method-level gain so far. The value available is concrete: one source for equations that exist eight times, BLAS lowering of the Python residual (18 to 50 times), a vendor-speed kernel under an honest policy (4 to 9 times), and an iteration memory limit that the method cannot get wrong by omission.

Revised architecture

Layers

L0  Method            equations, spin algebra, spaces, epochs, error controls
L1  ExpressionSet     declared sums of products with shared, epoch-scoped values
L2  Operation IR      contract, combine, symmetrize, gather/scatter, tile loop,
                      registered operator
L3  Representation    dense affine, tile provider, block family, index map
L4  Schedule          ordered steps with allocations, frees and peak bytes
L5  Kernel backends   capability records and execution policy

L0 and L1 meet at a builder call. L1 lowers to L2 by a fixed, inspectable pass. L4 is compiled once per solve from L2, L3 extents and a memory limit. L5 is selected per step by a deterministic rule.

L1: expression set

ExpressionSet rccsd(spaces);                  // occupied, virtual from #611
auto ovov = rccsd.input("ovov", {o,v,o,v}, Epoch::integrals);
auto t1   = rccsd.input("t1",   {o,v},     Epoch::amplitudes);
auto t2   = rccsd.input("t2",   {o,o,v,v}, Epoch::amplitudes);

// A linear combination of permuted operands is one value, not two terms.
auto L    = rccsd.combine("L", {{2.0, ovov, "menf"}, {-1.0, ovov, "mfne"}}, "menf");
auto t2as = rccsd.combine("t2as", {{2.0, t2, "imae"}, {-1.0, t2, "miae"}}, "imae");

auto F_me = rccsd.value("F_me", {o,v});
F_me += f_ov;
F_me += term(1.0, t1("nf"), L("menf")) >> "me";

auto R1 = rccsd.output("R1", {o,v});
R1 += term(1.0, t2as("imae"), F_me("me")) >> "ia";
auto R2 = rccsd.output("R2", {o,o,v,v}, Symmetrize::pair_exchange("ij","ab"));

Rules:

  • Every operand names its spaces. Equal extents do not match.

  • An epoch states when a value changes: integrals, amplitudes, or a method-defined level. A derived value takes the latest epoch of its inputs.

  • A sum of terms carries a numerical mode. ordered fixes the term order and forbids merging. reassociated_fp64 permits merging and sharing.

  • Term selection (CC2, linearized variants) is a tag on terms, not a numerical stencil.

  • The set is plain data. It can be printed, compared and differentiated. A reverse-mode product for Lambda is a transformation of the set.

L2: operation IR

The existing IndexProgram remains the checked binary contraction node. New nodes: Combine (scaled sum of permutations), Symmetrize, Reduce, GatherScatter over an IndexMap, TileLoop (a provider enumeration with a body) and RegisteredOperator (an opaque action with a declared read set, write set, workspace and work estimate). A registered kernel declares the index pattern and layouts it implements. The lowering pass matches it against the node, so the kernel call is derived from the program instead of written beside it.

L3: representations

Representation

Contract

First consumer

Dense affine view

Existing

All canonical routes

Tile provider

Existing provider protocol, plus a body that the planner sizes from the budget

Virtual ladder from B_vv

Block family

Key type from the method; per-key domains; enumeration without a global table; per-key extents visible to the planner

DLPNO pairs

Transport operand

A rank-two value between two named spaces, supplied by the method

S_pq projections

Index map

Offsets, targets, signs; additive assembly with a declared order

CAS sigma and RDM reductions

Packed symmetry, block-sparse sectors and factor graphs beyond RI stay unimplemented until a consumer and a measurement justify them.

L4: schedule

Schedule s = compile(rccsd, extents, Limits{iteration_bytes, policy});
// s.steps: alloc, combine, contract(kernel), free, in a fixed order
// s.peak_bytes(Epoch::amplitudes): the admitted iteration peak
// s.epoch_bytes(Epoch::integrals): retained shared operands
for (iter...) { s.bind(t1, t2); s.run(Epoch::amplitudes); }
  • Compilation is deterministic and bounded: a greedy pass with a fixed tie-break. No timing.

  • The objective is work subject to the limit. A shared operand is kept only if the peak stays within the limit; otherwise it is recomputed or the two halves are evaluated separately.

  • The schedule allocates every intermediate through the ledger. The method registers its resident owners once, by handle. There is no list of names.

  • peak_bytes is the iteration limit. Admission happens before the first iteration, not at the first checked operation.

  • A step that fails leaves its outputs invalid and later steps unrun.

L5: kernel capabilities and policy

struct KernelCapability {
    Pattern pattern; Layouts layouts; DType dtype; bool conjugation;
    Workspace workspace;      // exact(n) | bounded(f(shape)) | measured(f, qualification) | unknown
    Determinism determinism;  // fixed order | fixed at thread count | none
    Threads threads; Placement placement;
};
enum class ExecutionPolicy { strict, declared_margin, untracked };

Policy

Kernels admitted

Ledger entry for workspace

Recorded in output

strict

exact and bounded only

The exact or bounded bytes

Policy and kernel

declared_margin

Also measured

The qualified bound, labelled measured

Policy, kernel, backend version, qualification id

untracked

All

None; reported as unknown, never zero

Policy and an explicit untracked flag

A measured capability comes from a qualification run that interposes the allocator over a shape grid and thread counts for one backend version. A version without a qualification record is unknown. The default stays strict until a qualification exists for the linked backend; the user or the method selects another policy explicitly.

Resource names

Name

Scope

Source

Operation-phase bytes

One step

Existing Budget

Iteration bytes

One epoch replay

Schedule::peak_bytes

Method bytes

Resident owners, epoch-retained values, solver history, iteration peak

Method ledger

Process RSS

The process

Measured telemetry only

Control-structure bytes are charged at a fixed documented size per object, not through capacity() or sizeof, so a cap means the same on every platform.

Rejected alternatives

  • Keep per-operation plan classes and migrate term by term. About 480 lines per contraction, no shared operands, no iteration limit.

  • An einsum-string call per term. Planning per call costs ten times the execution at small sizes and hides lifetimes.

  • Adopt TiledArray or TAMM as the runtime. C++20, heavy builds and, for one of them, a GPL distribution decision. Unchanged from the study.

  • Put second-quantized algebra in the engine. Chemistry would leak into the numeric core. A generator can target the expression set.

  • Timing-based schedule search. Rejected by the 13 November determinism decision on #800.

  • Model determinants, RDMs or MPS bonds as engine tensors now. No consumer; high risk of a schema that the eventual method cannot use.

Keep, change, retire

Item

Action

core.hpp spaces, domains, views, allocations, leases, checked arithmetic

Keep. Change control accounting to fixed sizes

Contraction scalar loop

Keep as the reference oracle and test fallback

IndexProgram

Keep as the L2 binary node; kernels must be matched against it

RiJPlan, MoTransformPlan, OvProductPlan, ladder plans, stream plans

Keep their kernels and tests. Re-register as kernels with capability records. Retire the pattern of one plan class per contraction for new work

mo_detail::product and the 64 by 64 tile

Change: size packs from the budget. Move all Eigen::internal use into one adapter

StructuralKey, StructuralCache

Keep for RI-J. Do not extend. Schedules are cached on the expression set

RiJSelection::rule

Keep. It is the model for deterministic selection

DependentStream

Keep for local factor panels. New ragged work uses block families

distributed.hpp leases, epochs, envelope, wire

Keep. Retire the separate operation descriptor once a schedule step can be serialized

CCSD LadderContext and ladder_matrix_bytes

Retire when the residual runs from a schedule. Until then it stays, labelled as an operation-phase inventory

CPU, stream, transform, MP2, representation, Python ADRs

Keep as records of landed slices and their evidence

Planner ADR

Keep as the RI-J kernel record. Its title overstates its scope; the planner is defined here

CCSD ADR

Keep. The term-by-term approach it starts is superseded for remaining residual terms

Distributed ADR

Keep for identity, lease and publication semantics

Semantic proposal

Keep the invariants. Its planning, backend and budget sections are superseded here

Accepted scientific behaviour does not change. Every landed route keeps its kernel and results. There is no flag-day rewrite: the graph is introduced beside the existing residual and takes over one group of terms at a time.

Migration plan

Each task is bounded and lands on main with a focused check. Scientific acceptance through vq follows the existing rules and is owned by the Validator role.

Order

Task

Issue

Depends on

Acceptance

1

Expression set, lowering and schedule for a dense fragment, beside the existing code

#796

None

The real restricted one-body fragment (F_me, F_ae, F_mi, R1) reproduces compute_residuals to 1e-12 at arbitrary amplitudes; the schedule reports iteration peak bytes; compile time and warm replay are measured at three sizes

2

Backend capability record, execution policies, budget-sized packs, vendor qualification harness

#800

None

Strict path within 10 percent of Eigen with supplied wide packs; declared_margin within 15 percent of the vendor GEMM on the ladder and OV shapes; policy and kernel recorded; determinism at fixed thread count

3

Iteration and method ledger; fixed control accounting; owners registered by handle

#797

1

A residual with a deliberately unregistered intermediate fails a test; the same cap admits or refuses identically across two standard libraries

4

Restricted CCSD residual from the schedule, including the tiled virtual ladder as a tile loop

#806

1, 3

Residual traces match the predecessor; the admitted iteration peak is at least the measured tracked peak; H4 and frozen-core water through vq against matched ORCA

5

Single source for the closed-shell equations: Python reference and Lambda sweep read the same set

#806

4

The Python residual and its vector-Jacobian product agree with the native schedule; the unoptimized einsum path is retired from production use

6

Block families and transport operands for DLPNO pair blocks

#801, consumer #807

1

One PNO residual term group runs from a keyed schedule with unchanged pair energies and bounded per-pair workspace

7

Spin-block unrestricted CCSD on the same set

#806

4

Matches the spin-orbital pilot on small open-shell cases; storage scales as the spin blocks

8

CAS sigma as a registered operator with admitted buffers and an index-map operand; CASSCF integral rotation through the staged transform

#808

2

Root energies and RDM traces unchanged; per-thread accumulators appear in the estimate; the four-index rotation uses quarter transforms

9

Schedule step as the distributed unit; retire the separate operation descriptor

#804

4

The existing rank-two transfer test passes through a serialized step

Tasks 1 and 2 are independent and can start together. Task 1 is the first implementation task: it is new files only, it does not touch the CCSD source that the open ring claim on #806 is editing, and every later task needs it.

Contractions already claimed on #806 finish under their claims. New term-by-term claims for the remaining residual terms should stop there: task 4 replaces them, and each one adds a plan class, a call-site inventory and a kernel that runs several times slower than the predecessor.

Not planned: an MPS or MPO layer, packed antisymmetric storage, a compiler lowering pilot (#813 stays deferred until task 4 gives a profile), and any approximation or precision change (#814).

Handoff prompt for the implementation chat

Implement task 1 of docs/design/tensor_engine_architecture_review.md in
vibe-qc: a dense expression set, its lowering and a deterministic schedule.

Follow AGENTS.md. Fetch first. Read the review, core.hpp and
mo_transform.hpp, and the latest notes on #796 and #806. Open claims on
#806 edit cpp/src/ccsd.cpp and the tensor/ccsd_*.hpp plans; do not touch
them.
Claim your files on #796 before editing.

Scope: new header cpp/include/vibeqc/tensor/graph.hpp, a native test and a
benchmark. C++17, FP64, no new dependency, no template expression layer.

Build:
- ExpressionSet with inputs, combined operands (scaled sums of permuted
  tensors), intermediate values, outputs, value epochs and a numerical mode
  per sum. Operands name tensor::Space identities.
- A lowering pass to IndexProgram binary nodes and Combine nodes, executed
  through existing checked views and the supplied-pack GEMM leaf.
- compile(set, extents, limit) -> Schedule with ordered alloc, combine,
  contract and free steps; a greedy deterministic rule; shared operands kept
  only when the peak stays within the limit; peak_bytes per epoch.
- All intermediates allocated through tensor::Budget by the schedule.

Evidence: declare the real restricted CCSD one-body fragment (F_me, F_ae,
F_mi and the R1 terms of cpp/src/ccsd.cpp lines 675 to 767) and compare with
an independent explicit-loop oracle at arbitrary asymmetric amplitudes for
(occupied, virtual) = (2,5), (5,2), (9,11), (1,7) and empty spaces, at
1e-12. Test a limit that forces recomputation instead of sharing, a stale
epoch, an alias and a domain mismatch. Benchmark compile once and warm
replay against the per-term evaluation at (4,19), (8,40), (12,72); report
losses.

Do not wire it into production CCSD; that is task 4. Classify the new test
through scripts/test_gate/update_suite_manifest.py. Land on main with
[skip ci] and git push -o ci.skip, record the result once on #796, and stop.