vibe-qc/Documentation

Distributed execution measurement and acceptance

This is a proposed protocol for the architecture and work tracker, not completed measurements. Reuse qualified inputs and earlier evidence where applicable. No new CI or fleet acceptance gate is introduced; ordinary development skips CI.

Four objectives and honest baselines

Objective	Baseline and reported quantity
Latency	Fastest qualified one-node run at identical scientific settings and requested outputs; full elapsed speedup and iteration counts
Throughput	Same total resource budget running independent jobs, including duplicated setup, I/O and failed/restarted work; completed jobs per time
Capacity	Smallest feasible allocation for the same oversized input; explain why one node cannot fit and report maximum rank/node/device memory
Resource cost	Node-hours and GPU-hours alongside latency; measured energy only with synchronized power and explicit host/device/network/storage boundary

For N nodes and elapsed T_N, one-node speedup is T_1/T_N and efficiency is (T_1/T_N)/N. If the smallest feasible baseline is N_0, use (T_N0/T_N)/(N/N_0) for efficiency. Core/GPU metrics need their own explicit resource denominator. Never create a one-node baseline by extrapolating an oversized run. Kernel speedup, fixed-density Fock timing and partial SCF cycles do not establish complete-method speedup.

Strong scaling holds inputs, accuracy and outputs fixed. Weak scaling records exactly what grows, expected work and unique output counts; atoms or determinant candidates alone need not imply linear work. Test ordinary Ethernet and HPC networks when available, reporting unavailable hardware as unmeasured.

Measurement record and predeclared targets

MPP-01 (#819) defines a portable record with scientific input fingerprint, resolved route, output request, source and actual Python/native identity; compiler, BLAS/MPI/backend versions; nodes/ranks/threads/devices and affinity; cap/precision policy; scalar/matrix/force units; warm-state and checkpoint state. Raw site paths, job details and logs stay outside product checkouts.

Measure setup, planning/screening, factors/whitening, transforms, J/K/XC, solver, reductions/waits, correlation/response, derivatives, I/O and full wall time. Include SCF/CC/root iterations, restarts, rejected steps and failure status. Track payload and metadata bytes, maximum in-flight requests, message/collective bytes/time, packing and host/device transfers, maximum/mean task time, owned and replicated state, workspace and peak rank/node/device usage. State uninstrumented MPI/allocator overhead separately from the tracked memory cap.

Use at least three timed repeats where affordable, publish all samples and median/range, and separate cold setup from warm reuse. If repeats are too expensive, label single-shot evidence and its uncertainty. Power, clock and allocation differences are part of the evidence, not silent outlier filters.

Before the optimized run, each implementation issue records its fixture and tolerance IDs, baseline samples, target metric, resource cap, allowable regression/noise envelope and minimum useful gain. Select those numerical targets from the baseline/pilot, not after inspecting a favorable result. Numerical tolerances come from method qualification, never performance wishes. An implementation can land as opt-in without a speed claim; automatic selection requires demonstrated benefit exceeding the declared noise envelope and no regression on the admitted small-case set. A negative crossover result is valid evidence for retaining single-node execution.

Capacity acceptance must show bounded live state throughout construction, execution and output/restart. Include an input exceeding one node’s admitted memory and no hidden full-object gather. Report the residual replicated floor; do not claim ideal 1/P memory from payload partitioning alone.

Cost models to instrument

Engineering estimates, not measured performance:

T >= T_serial + max(rank work) + exposed communication + layout + I/O + sync. Only actually progressed communication can overlap compute. For a message, measure effective latency alpha and inverse bandwidth beta at intended concurrency; alpha + beta * bytes is a model, not an exact collective cost.

M_live = owned + replicas + packs + intermediates + solver + requests + backend + caches + transfers + checkpoint + reserved headroom. Evaluate simultaneous liveness. Count node-shared allocations once and record RSS separately; do not sum duplicated shared RSS as physical ownership.

For AO count n, auxiliary rank a, full mesh K, grid size g, history h and scalar size b, replicated SCF state grows with b*K*n*n*(1+h) and spin/ solver copies. Dense GDF factor size is approximately b*K*K*a*n*n, or the actual sum of variable-rank pair extents. Owned factors may shrink with ranks, but metric, replicas and in-flight panels remain. FFT grids can scale as b*g/P_fft plus transforms/packs; dense solves as b*n*n/P_solver plus vectors/panels. Sparse locality and screening require actual censuses rather than these dense worst-case models. CC/local methods must include amplitudes, ghost neighborhoods, DIIS and multiple live intermediates.

Workload and scientific matrix

Family	Required controls and complete-method evidence
Molecular HF/DFT	Small RHF control then admitted RKS/UHF/UKS/RO slices; direct, DF and seminumerical routes kept distinct; compact/diffuse and high-angular-momentum work; charge/spin, J/K/XC components, final physical Fock/density/residual and iterations; matched ORCA
Multi-k	Gamma, uniform/shifted meshes, full-BZ/IBZ and symmetry on/off; electron count/global chemical potential, spin/star weights, entropy/free energy, maximum and weighted residuals; matched CRYSTAL or justified same-source evidence
Metals/open shells	Small gaps, fractional occupations, uneven pools, sufficient-state/tail controls and genuine complex matrices; compare density/subspaces near degeneracy; no independent pool occupations or unadmitted time reversal
GDF	Variable metric ranks, q=0, full-BZ ket partners, unequal local domains, streamed/fitting and oversized inputs; same-source fixed-density J/K then complete admitted HF/MP2; no global factor cache
GPW	Insulating Gamma cell ladder, skew cell/awkward grid decompositions; analytic Fourier-mode FFT normalization, Hartree/J/charge/energy and complete SCF; augmentation qualifies separately
Solvers	Real/complex, conditioned/near-dependent overlap, full/partial spectra and cold/warm states; FC-SC eps, C^H S C-I, rank, occupations and projectors; full transformation and redistribution cost
MP2/CC/triples	Frozen-core/spin/denominator/source identity; fixed-amplitude residuals, correlation/triples energies and iterations; full preparation cost; matched ORCA or justified small-system reference
Local correlation	Skewed pair domains, dominant pair, noncanonical coupling, translated pairs and threshold ladders; remote amplitude epochs, raw coupled residuals and energies; fixed method/model
Response/CI/MR/DMRG	Roots, symmetry, selected-space/bond controls, determinant duplicates and global orthogonalization; actions, residuals, normalization, RDM checks and observables, not ground-state energy alone
Derivatives/stress	Supported analytic versus converged finite differences at controlled steps/topology; response, Pulay, grid/metric, cell/k terms and translation/rotation invariants; matched references where available
Semiempirical/CCM	Only supported source/parameter/element/spin/cell domains; SCC residual, charges, energy and requested observables; experimental AICCM/CCM/SECCM restrictions unchanged
Restart/failure	Safe-boundary restart at same/different ranks, missing/corrupt chunks, stale epochs and partial manifests; coherent refusal, no published partial result; process-loss recovery excluded from portable claim

Complete scientific runs go through vq, with actual source/native identity containing the change. Match geometry/cell, method, basis/auxiliary/ECP, charge/spin, k mesh, approximations, convergence, units and tolerances. Use ORCA/CRYSTAL when available; otherwise give justified literature/limits or independent same-source sums and state the uncertainty. Existing valid results can be reused. No old runtime may qualify a new distributed implementation.

Focused source and protocol checks

Use real 1/2/3-rank runs plus ranks exceeding tasks; uneven and empty partitions; serial import without MPI initialization; rectangular real/complex blocks with asymmetric sentinels; transpose versus adjoint; ragged/packed/streamed sources; count/offset overflow; ownership/epoch mismatch and exact contribution coverage. Check exact/below-cap admission and delayed backend/transport completion, provider refusal and shared resource release. Use bounded harness timeouts for recoverable protocol tests; do not claim arbitrary rank-loss recovery.

GPU checks distinguish host staging and direct device buffers, supported collectives, explicit stream events and progress. CPU-only builds stay usable. Checkpoint checks inject interruption before and after manifest commit. New test files use the curated manifest generator. Documentation-only planning needs text/link/privacy checks, not runtime or chemistry suites.

Three independent status records

Each route records source landed, scientifically accepted, and scaling measured, with evidence, limitation and next action for each. Dev completes focused checks, main landing and own cleanup; Validator owns complete calculation/reference acceptance and may be the same worker. A performance experiment cannot waive scientific acceptance. Parent completion requires the declared route inventory to be reconciled, with unsupported and deferred routes explicit; optional compression/compiler/ULFM research adds no gate to otherwise completed exact execution work.

MPP-01 portable record, version 1

benchmarks/mpp_measurement.py provides an MPI-free, standard-library record validator, repeat summary, allocation-hour calculation and adapter for the existing native_fock_capture.py worker. This is a benchmark evidence interface, not a production output format or a replacement for vibeqc.output. It returns data to the caller; raw JSON, runtime paths and scheduler receipts stay outside the checkout. Importing it does not load a native extension or initialize MPI.

The schema ID is vibeqc.mpp-measurement.v1. A quantity has value, unit, scope and reason; unknown values are JSON null with a nonempty reason. Zero means measured absence, not unavailable instrumentation. Counts and byte values are nonnegative signed-64-bit integers; times are finite seconds, energies are Hartree, resource costs are node hours and GPU hours. NaN, infinity, booleans as counts, inconsistent uniform topology and count-product overflow are refused. Nonuniform topology needs a future version rather than an invented average ranks-per-node value.

Records bind scientific settings and requested outputs with canonical sorted JSON SHA-256 fingerprints. Identity fields describe the actual recorded source, Python/native runtime, compiler and numerical libraries. Unknown versions remain explicit. The two versioned fixture manifests in benchmarks/fixtures/ pin existing entry points, source revisions, file hashes, geometries, units, options, default definitions and qualification limits. These pins make historical settings reproducible; they do not establish that a current runtime has the same basis bytes or resolved fit route. Capture those resolved identities before a new run. An input fingerprint is not proof of scientific equivalence.

Each sample distinguishes its measured boundary from full launch-through-output elapsed time. Native phase inclusive values may overlap; only disjoint exclusive wall intervals can be added, and their sum must fit the enclosing boundary. Do not feed rank-summed CPU timers into that check. Unattributed overhead is allowed; missing timers are not zero. Repeats retain original order and all samples, with independent cold/warm min, median and max. First native call after imports/basis construction is not a cold filesystem or fresh process measurement.

Memory fields separate whole-calculation owned, replicated, workspace, metadata, in-flight, tracked-live/cap, process RSS and device residency. Optional snapshots sum simultaneous owned, replicated, workspace and other bytes within one stated scope. Separate phase high-water marks are never added. A captured-array subset is recorded separately, not promoted to total live owned memory. Tracked cap checks require the same ledger scope. RSS includes allocator/runtime overhead and may include shared pages; it is neither the ledger nor device residency. Communication records retain message/payload/metadata counts, collective/wait/ packing times, transfer bytes and in-flight counts, including explicit unknowns.

Latency, independent-job throughput, capacity and allocation cost remain separate experiments. resource_cost uses full elapsed time and actual allocated node/device counts, never a Fock timer. Unknown time or topology produces unknown hours. A provisional feasible run is not the best feasible one-node baseline. A qualified one-node baseline needs measured selection evidence; a capacity baseline needs smallest-feasible allocation evidence and an explanation of one-node infeasibility. No extrapolated one-node time is accepted.

Before optimization, freeze targets with the baseline sample fingerprint, full-run metric, noise and allowed-regression fractions, minimum useful gain, resource cap and rationale. Useful gain must exceed noise. The current records leave targets null because their full-run/noise evidence is insufficient; this blocks a performance claim, not source landing. No optimized run is part of this increment. Repeat at least three times in each cold/warm class before relying on noise estimates, or document the sampling limitation and conservative rationale.

Focused source check, using the existing environment without chemistry fixtures:

.venv/bin/python -m pytest --noconftest tests/test_mpp_measurement.py -q

For an external single-record JSON, python benchmarks/mpp_measurement.py EXTERNAL_RECORD.json validates and prints its timing summary. The native_fock_record(case, fixture, identity, topology) adapter accepts one existing benchmark worker, a fixture manifest and verified portable identity/ topology metadata. It deliberately does not copy raw paths or platform strings. Validation checks consistency, not the truth of caller-supplied evidence.

First fixture evidence and next increment

benchmarks/mpp_baseline_evidence.json retains all nine water/STO-3G samples from the existing #615 cost result: three complete SCFs per off/capture/export worker, one first call and two warm calls per mode. The accepted matched ORCA energy and source/native digest are retained in the fixture and records. This is reused scientific acceptance of the existing conventional route, not of a new distributed implementation. Cold evidence is single-shot and two warm samples do not establish a robust noise envelope. Imports, molecule/basis setup and final output are outside the per-SCF timers, so full-run latency, throughput and allocation-hour claims remain pending. No duplicate scientific run was made.

At the first increment, the H2/STO-3G two-k-point GDF fixture reused the existing 12-bohr cubic debug case but was not yet scientifically qualified: the debug reference uses different convergence settings and no matched current complete vq receipt was identified. #96’s historical effective-thread measurements describe a GDF stage on other inputs, not this fixture, full-run latency or multi-node speedup. The #794 RI-J/complex inventory remains with its existing owners; these fixtures neither migrate those operations nor qualify their tensor implementations.

The first increment left the following work, partly addressed in the pilot below: resolve and hash the GDF fit/basis/default settings on an available containing runtime, obtain matched complete vq and reference evidence, and collect end-to-end cold/warm samples with actual hardware, live memory and communication instrumentation. Then compare feasible one-node configurations and predeclare numeric targets from their noise. Source implementation, scientific qualification and measured scaling remain separate status fields. #819 remains open; #820 runtime and #804 descriptor interfaces can be consumed incrementally when available.

This treatment follows the primary evidence limits: Orlando et al., 10.1002/jcc.23072, Table 1 and Figure 3, used extrapolated small allocations for X10 despite its 256-core minimum; Williams-Young et al., 10.1063/5.0151070, section III and Figures 1-2, timed Fock construction from preconverged densities. Neither supplies a measured full-run one-node baseline for these fixtures.

Qualified two-k GDF pilot (second MPP-01 increment)

The registered H2/12-bohr cubic/STO-3G, def2-svp-jk, (2,1,1) fixture now has matched complete vq evidence. The historical debug name says compcell, but its current default resolves to native SR/LR shared-q GDF. The fixture retains the actual executed fit descriptor, contracted basis records/file hashes, metric cutoff, reciprocal support and BvK-Ewald exchange convention. Production settings were not changed. Actual runtime source is ddcdf9adda4a3698f2be737287c569affb06bdc3; its GDF Python driver hash matches the registered source and actual native SHA-256 is 8bdb60e59e5d54c3a155971bf29d1bee7212bcee0bbf8cf6a5223d52ad0c2693.

The independent reference is external PySCF 2.6.2 KRHF/RSGDF, reading exactly the same contracted orbital and auxiliary Gaussian data. Both calculations use the same geometry/cell, charge/spin, Gamma-containing full mesh/weights, Coulomb interaction, Ewald exchange correction, energy convergence 1e-8 Eh, residual convergence 1e-6, normalized-overlap cutoff 1e-7, metric cutoff 1e-9, and SR/LR split 0.4 inverse bohr. The reference uses 1e-10 integral precision and no auxiliary exponent deletion. Integral support algorithms remain engine-specific; agreement is tested, not assumed. The registered native thresholds and physical approximation are unchanged. Initial guess and SCF acceleration implementations differ; no cross-code timing claim is made.

Before reading the reference result, qualification required energy agreement within 1e-6 Eh, overlap within 1e-9, electron count within 1e-8 and maximum commutator residual within 1e-6, with both runs converged. All 18 subsequent baseline calculations satisfy these gates. Maximum energy difference from reference -1.1201398815641401 Eh is 6.90e-10 Eh; maximum overlap, density and Fock differences are 2.22e-16, 1.55e-15 and 8.04e-10. K points and weights agree exactly. Native runs take three SCF iterations and their energy spread is 1.78e-15 Eh. This qualifies this fixture and runtime, not other cells, gradients, distributed methods or a newer native implementation.

CRYSTAL direct HF would remove the finite auxiliary approximation and is not substituted for a same-model GDF reference. This distinction follows Ye and Berkelbach, 10.1063/5.0046617, equations 3-12: fixed-auxiliary fitting error is distinct from integral precision, and q=0 treatment belongs to the declared physical convention. The existing reference runner script and all full raw receipts remain in external calculation storage linked from #819; the portable fixture records the exact reference settings and basis coefficients, and the evidence records its script digest.

benchmarks/gdf_baseline.py is byte-identical to the executed benchmark payload. Submit it through vq on a containing runtime, providing --source, --native and --driver from the fixture’s runtime object and --output outside the checkout. It refuses identity mismatches. Default --threads 1 2 4 --repeats 3 runs nine fresh worker processes, each with one cold and one warm full calculation. Each warm calculation recreates cell, basis and fit; no SCF density, factor cache or checkpoint is reused. OpenMP dynamic teams are disabled and BLAS remains at one thread. The worker writes energy, convergence, iterations, electron count, residual, executed source, matrices and k metadata before the parent observes completion. Setup and output are included in each elapsed time.

Cold means interpreter launch through closed result/state files; warm means previous completion through a new complete calculation and closed result/state files. Filesystem caches are not flushed. The parent runtime launcher/bundle staging, scheduler wait and disk durability are outside those sample boundaries. Whole-command elapsed time is reported separately; it is not divided by sample count to fabricate individual timings. No result excludes a slow sample.

One Intel Xeon Platinum 8468 allocation supplied four scheduler-bound logical CPUs and a 4096 MiB scheduler limit throughout. Compiler was GCC 13.3.0, Eigen 3.4.0, Python 3.12.3 and NumPy 2.5.4. The three policies ran in alternating order across repeats on the same node. All samples are in benchmarks/mpp_gdf_baseline_evidence.json.

OpenMP threads	Cold median (range), seconds	Warm median (range), seconds
1	3.418039 (3.417179-3.445519)	2.356143 (2.353139-2.359820)
2	3.143963 (3.141949-3.146210)	2.080467 (2.079166-2.085454)
4	2.374487 (2.251396-2.881479)	1.160781 (1.160173-1.161107)

Four threads is the fastest tested one-node policy for this bounded CPU allocation, even using its slowest observed cold sample. This is not an exhaustive whole-node/hardware search. Whole-command time for all 18 samples was 53.16 seconds, or 0.0147667 node hours and zero GPU hours; queue wait is excluded. Largest-process RSS was 287629312 bytes. Per-worker high-water RSS and returned-array bytes remain distinct from unmeasured whole live owned, replicated and workspace bytes. The scheduler limit is not a tracked allocation cap. Distributed communication, GPU residency and independent-job throughput remain unmeasured; no multi-node, oversized-capacity or energy claim is made.

Targets frozen before optimization

predeclare_targets requires three measured positive full runs in each class. It uses all samples: noise is the worst cold/warm (max-min)/median, minimum useful gain is max(5%, 2*noise), and the allowed-regression envelope is max(2%, noise). It refuses an envelope too large to yield a meaningful target. Targets bind the exact baseline sample fingerprint and resource cap.

For the four-thread baseline, cold noise is 26.5356% and warm noise is 0.0805%. The frozen overall full-run target is therefore 53.0711% minimum useful gain, with 26.5356% noise/regression envelope, inside the same 4294967296-byte scheduler boundary and scientific/output contract. Compare cold and warm separately and require both for an overall claim. This conservative target was recorded on #819 before any optimized experiment. It is not a statistical confidence interval, a production dispatch policy or a claim that such a gain is achievable. Small warm gains cannot be advertised as overall cold/full-run gains. Do not revise the envelope after observing a favorable optimization.

Next: investigate first-call variability or preregister paired measurements before testing small cold gains, instrument live GDF allocation/communication boundaries, then consume #820/#804 interfaces for a separately qualified multi-node pilot. Existing source landing, this scientific acceptance and the one-node thread measurements have independent status records. #819 remains open for the wider inventory and distributed performance work.

Preregistered paired cold-start and live-census diagnostic

The third MPP-01 increment is an observer experiment, not a kernel optimization. benchmarks/gdf_paired_protocol.json freezes six adjacent control/observer pairs in AB, BA, AB, BA, AB, BA order. Each member launches a fresh interpreter and runs one cold and one fresh-complete warm calculation of the qualified two-k H2 fixture. A is an unobserved control; B enables the benchmark observer. Both use four OpenMP threads, one BLAS thread and the same pinned runtime and four-CPU/4096-MiB allocation. There is no cache flush, checkpoint reuse, frequency intervention, outlier deletion or timing-driven rerun.

benchmarks/gdf_paired.py records parent-observed interpreter/readiness time, full completion time, setup and SCF times, driver wall/process CPU time and RSS. Readiness includes import and initialization; the residual between these stages and full completion includes identity checks, Python validation, serialization, output and observation delays. It must not be called isolated I/O time. Process CPU aggregates worker threads and is not wall time. Cold and warm retain their previously declared completion boundaries.

analyze_gdf_pairs retains all six signed B-minus-A differences and B/A ratios for each class, with medians and complete ranges. Control dispersion and stage ranges are secondary diagnostics. The preregistered attribution criterion is that a candidate phase’s range account for at least half the full elapsed range and its paired direction agree; failure of that criterion leaves the cause unresolved. Correlation alone does not prove a scheduler, filesystem, CPU clock or allocator cause. Observer overhead is measured rather than subtracted, and the prior 53.0711% performance target is unchanged.

Scope of live memory and communication observation

benchmarks/gdf_observer.py profiles selected existing GDF call, return and native-call boundaries without editing production code. It visits arrays on active GDF Python frames and returned factor/cache records, follows NumPy base ownership and unions overlapping address spans. Aliased views are counted once; non-NumPy external-buffer spans are marked. Arrays are inspected without copies, and only scalar counts are retained, so the observer does not extend their lifetimes. Traversal and event counts have explicit bounds and fail rather than silently truncate a census.

Factor, state and workspace categories describe observed variable roles, not a global distributed owner map. The highest-priority factor/state role wins when the same storage has several aliases. Peak snapshots retain simultaneous category totals; separate category maxima are never summed. This is a sampled lower bound on visible live array storage, not a whole-allocator peak. Native opaque buffers, allocations between observation boundaries, library/allocator metadata, device residency and asynchronous lifetime accounting remain unknown. External-buffer spans can include padding; no physical RSS equivalence is claimed. Existing memory caps, factor-build reservations and solver workspace estimates are reported in a separate reservation record, never relabeled as actual allocation. RSS remains the process high water, including imports and previous calls.

Communication observation counts selected existing Python MPI API calls, elapsed API time and logical array payload spans. These spans exclude Python serialization and message metadata, and do not measure network traffic. The observer currently admits only a checked single-rank identity: inter-rank bytes for those calls are then zero by execution topology. Serialization size, wait/progress and unobserved transport remain unknown. Multi-rank observation requires actual ownership and communicator evidence from the #820/#804 owners; this instrument does not claim it, initialize another runtime or replace their interfaces. An existing profiler is never overwritten; normal and exceptional exits restore the prior profiling state.

Measured diagnostic result

The corrected paired experiment completed all 24 calculations. The collector, observer, protocol and underlying baseline files are byte-identical to their executed payloads. Source/native/driver and basis identities remain those of the qualified fixture. Every sample passes the existing matched-reference gates; maximum energy difference is 6.90e-10 Eh. D/F/S and other returned-array hashes are identical between A and B in all twelve cold/warm pairs, as is the executed fit identity. No production formula, threshold or owner interface changed.

Class	A median (range), seconds	Median paired B/A	Median paired B-A, seconds
Cold	2.208206 (2.203487-2.222107)	1.724704	1.600204
Warm	1.157752 (1.155143-1.160185)	2.315255	1.523318

The cold control range/median is 0.8432%, warm 0.4355%. The earlier large cold spread did not recur under this protocol, but this is not a paired comparison with the previous harness or allocation and does not establish an improvement. Cold readiness range was 0.015396 seconds out of a 0.018620-second full range; however, its paired direction agreed in only two of six pairs. The preregistered dominance criterion is therefore not met: the original first-call variability remains causally unresolved. Warm SCF range accounted for 0.004882 of 0.005042 seconds and aligned in all six pairs, localizing this run’s warm variation to the broad SCF scope, not an individual kernel.

Observation itself adds approximately 72.5% cold and 131.5% warm elapsed time. The Python profiler is suitable only for diagnostics on this fixture. The reported callback time excludes global dispatch/filter overhead and cannot be subtracted to recover an unobserved performance result. All signed differences, ratios, stage values and outliers are retained in benchmarks/mpp_gdf_paired_evidence.json; earlier performance targets remain unchanged. No optimized experiment was run.

Each observed calculation produced 896 selected census events. The sampled visible peak is 240984 bytes, simultaneously comprising 4608 factor bytes, 128 state bytes and 236248 workspace bytes. Separate category maxima are 9216, 384 and 236248 bytes; their sum is not a simultaneous peak. At the selected peak, 34760 bytes belong to externally backed spans, already included in the total. Reported factor-build reservation is 67481616 bytes, and native workspace cap is 67108864 bytes. These are admission quantities, not measured allocation. The difference from the visible census cannot be interpreted as free or wasted memory. Per-sample RSS and the whole-command largest-process RSS are separate fields; opaque native buffers and allocation intervals remain unmeasured.

Each observed calculation recorded four selected MPI API calls: one mpi_allgather with no ndarray payload, and three allgather_ordered calls with 128 logical array bytes each. The checked serial identity makes actual inter-rank traffic zero for these calls. The first call may still carry Python scalar/status metadata; that serialization is unmeasured. API times include instrumentation and local work, not isolated network time or progress.

The accepted command took 68.76 seconds (0.0191 node hours) on one four-CPU allocation with no GPU. An earlier completed run is preserved separately: a nested-list role test caught state arrays labeled workspace, without changing total visible bytes. The corrected experiment was explicitly recorded before inspecting timing results, using the identical protocol. This measurement-defect rerun was not an outlier filter. Both receipts remain external; only the corrected run enters the primary analysis. No existing jobs or other owners’ work were cancelled or changed.

Next: replace diagnostic profiling with explicit low-overhead lifetime and allocator events at an agreed GDF provider boundary, then qualify actual transport counters with the existing #820/#804 interfaces. A new attribution experiment must be preregistered before changing cold-start execution policy. The bounded diagnostic source, scientific parity and these measurements are complete; broad live-native-memory and distributed scaling work keeps #819 open.

Provider lifetime and completed transport records

benchmarks/gdf_lifetimes.py replaces global profiling with four explicit provider-return hooks: shared-q batches, SR three-center sources, completed factor caches and exchange outputs. The shared-q hook belongs to aux_basis; the other references belong to periodic_k_gdf. Hook installation validates all targets before changing any, refuses nested ownership, and restores only its own replacements on exit. Production defaults and numerical code are unchanged. Select gdf_paired.py --observer lifetimes with --protocol gdf_lifetime_protocol.json through the same qualified vq fixture.

The ledger registers ndarray root spans when a provider returns them and releases each registration through a weak reference when that ndarray dies. It never retains the array, copies numerical data, walks interpreter frames or changes a reservation. Storage views inherit their root lifetime; overlapping external spans count once. Factor, state and workspace roles are explicit provider classifications. A simultaneous peak snapshot is separate from independent category maxima. Counts are bytes with signed-64 limits and bounded live-root/provider-result traversal. Retained metadata grows with live roots, not the number of SCF iterations.

This observes exported storage lifetimes, not every allocation inside a provider. A buffer first appears when returned; internal scratch and temporaries that die before return are absent. An external buffer can outlive its ndarray wrapper, so the counter tracks observed storage rather than that external owner’s allocation. Native workspace, allocator metadata, device residency and distributed ownership remain unknown. RSS is a separate process high water. The smaller coverage is intentional and must not be presented as a memory reduction from the previous interpreter census. Event callback time also omits wrapper/traversal overhead; end-to-end paired timing is the overhead test.

The new protocol freezes the same six alternating AB/BA pairs, scientific settings, output requests and runtime identity. The instrumentation target is median B/A at most 1.05 in each cold/warm class, declared before measurement. All samples and signed differences remain in the evidence; no overhead is subtracted. This target concerns measurement usability, not a new chemistry optimization or a revision of earlier performance targets.

Completed transport and native Budget accounting

benchmarks/transport_measurement.py consumes successful, retired BoundedTransferPhase tickets from #820. It refuses active/refused tickets, inconsistent rank/count/request inventories, missing time fields and invalid units/counts. It records application payload sends and receives separately, plus the two-int64 release receipt in the opposite direction. Same-owner transfers post neither payload nor receipt; a zero-byte remote transfer still posts a payload message and a receipt. Summing sends gives application traffic; adding receives would count the same bytes twice. Collective wire traffic, MPI headers/algorithms and internal storage remain explicit unknowns.

Tracked protocol inventory is 232 bytes per rank plus one payload on each endpoint (one copy for same-owner transfers). It is an admitted buffer inventory, not bytes still live after ticket retirement or whole-allocator memory. Exposed wait includes payload/receipt waits; admission and terminal agreement have their own timings. These are local API timings, not isolated network latency or measured useful overlap.

The #804 Budget adapter accepts explicit live/peak/cap and imported transport inventory. Imported bytes are already included in Budget live storage, so they must not be added again. A native core contract check exercises exact cap, one-byte-over refusal and release to the baseline. This qualifies the existing Budget interface and accounting conversion; the newly landed CPU adapter is qualified separately below. Neither check measures native physical allocation or distributed chemistry.

benchmarks/transport_probe.py qualifies real local 1/2/3-rank execution with three repeats each of cross-owner, same-owner and zero-byte transfers. Receiver contents and aggregate send/receive balances pass, including an inactive third rank. The primary record uses an external snapshot of the landed #820 module; source hashes distinguish a supplemental probe that encountered a concurrent owner’s unpublished edits. The existing native extension was imported by package setup but not exercised; its stale-core override applies only to Python/MPI functional checks. MPI is Open MPI 5.0.11 with mpi4py 4.1.2. Launch-inclusive cost, physical wire volume and distributed scaling remain unmeasured.

Portable results are in benchmarks/mpp_lifetime_evidence.json. Raw logs, receipts, the snapshot and analysis generator remain outside product Git. An initial GDF run failed on the shared-q module lookup before a valid pair; its evidence is retained separately. The corrected hook and regression were recorded before timing analysis, with the same experiment and acceptance target.

Qualified lifetime-hook result

All 24 corrected calculations pass the prior matched-reference gates; maximum energy difference is 6.90e-10 Eh. All twelve A/B cold/warm pairs have identical returned-array and executed-fit hashes. The executed collector, provider hooks, protocol and baseline match the recorded payload hashes. The source/native/driver identity is unchanged from the qualified fixture; no new native chemistry is claimed.

Class	Control median (range), seconds	Median B/A (range)	Median signed B-A, seconds
Cold	2.252778 (2.242723-2.299717)	1.001294 (0.986582-1.014224)	0.002904
Warm	1.169063 (1.164276-1.178529)	1.004669 (0.994596-1.013248)	0.005474

Both medians satisfy the preregistered 5% instrumentation limit. Observed median increases are about 0.13% cold and 0.47% warm, below control range/median noise of 2.53% and 1.22%. These are six paired samples, not precise intrinsic overhead estimates or a chemistry speedup. No sample is removed and no overhead is subtracted. The earlier performance targets remain unchanged.

Each observed calculation registers and releases thirteen root lifetimes over seven provider calls. Current tracked storage returns to zero before reporting. The simultaneous observed peak is 18744 bytes: 9216 factor bytes and 9528 workspace bytes, with no state at that instant. The independent state maximum is 128 bytes. Externally backed spans contribute 9216 bytes already included in the peak. This does not supersede the larger previous census as an allocator measurement; the two instruments observe different boundaries. Native internal workspace is still unknown. Whole-command time was 50.96 seconds, or 0.014156 node hours on one four-CPU allocation, with zero GPU hours. Largest-process RSS was 282865664 bytes, separate from all live-storage counters.

The bounded source and existing-fixture parity pass. Transport acceptance is functional local MPI accounting; no measured multi-node scaling or distributed scientific acceptance is inferred. Next, add explicit allocation events inside native GDF providers and preserve separate accounting for imported transport storage, owned factors and native workspace before a distributed GDF pilot.

Newly available CPU-session integration

During this increment, #804 landed its portable CPU envelope and real MPI adapter. benchmarks/tensor_transport_probe.py consumes that interface directly, using the existing isolated native binding and independent integer-pair oracle. No owner file is edited. Three repeats of both transpose and adjoint pass at 1/2/3 local ranks. At most two ranks own input, so the three-rank probe includes an empty participant. Each contraction records its manifest transfers and three ordered contribution tickets, together with native Session snapshots at admission, produce, consume, retire and safe release.

The native adapter imports 376 control bytes plus its maximum transport payload capacity into the same Budget. This includes both sum and transfer controls; provider inventory is separate within that Budget. Snapshot conversion uses the actual Session live/peak and the exact admitted cap. All snapshots remain within cap; imported transport is zero after safe release. Example transpose admission is 102594 bytes on the output/input owner, 99514 on the other input owner and 31697 on the empty rank. These remain tracked inventories, not allocator or RSS measurements. Exported results can retain native storage after safe release.

Per-sample application send/receive and receipt totals balance. The portable record retains every repeat, admission/release snapshots, boundary maxima and sums of local ticket time scopes; raw per-ticket/per-boundary logs remain external with hashes. Summed ticket times omit other contraction work and must not be read as full-run latency. The actual isolated native binary and source hashes identify this probe, independently of the older scientific runtime. This is functional transport/accounting acceptance, not distributed chemistry, useful overlap or scaling. Future routing changes need their own bounded qualification; #819 does not change #804 placement or #820 runtime semantics.