Python CPU ownership slice and measured source record¶
Date: 3 November 2026. Implementing issue: TENS-09 (#802), parent #793. This is the first synchronous local Python source increment, not completion of TENS-09 or scientific acceptance. The API manual specifies accepted layouts, ownership, budget boundaries and examples. The subsequent mutable-input performance increment below supersedes the initial per-P-only choice for the actual FP64 Python consumer. Historical measurements and source qualifications are retained.
Reused contracts and actual consumer¶
The implementation uses core.hpp Space/Domain, Allocation/View, BorrowScope,
Lease, IndexProgram and Budget; ri_j.hpp ResidentFactors/RiJPlan; and
stream.hpp SourceStamp/CompensatedPairPlan. local.hpp supplies retained
coarse handles around those contracts. It does not add descriptors, an
allocator, a planner, MPI initialization or a wire protocol. The existing
#804 binding informed owned snapshots and immutable owner-retaining array
publication; its completion/transport authority is unchanged. Current main
includes dependent traversal commit 72303c81.
python/vibeqc/density_fitting.py:DensityFitting.build_J is the real consumer.
Its constructed whitened B[P,mu,nu] and caller density are snapshotted on
every FP64 call, then contracted by the registered single-worker RI-J path.
The method owns the source and AO association; fresh opaque tokens represent
that call, not an external physical-source certificate. The public arrays
remain mutable and each call sees their current contents. The public writable
return, RI equations, whitening, basis, thresholds and other DF operations
are unchanged. The former non-FP64/complex-density NumPy path is retained;
no implicit dtype cast broadens the native provider’s contract.
The second operation is the existing compensated complex [p,a] by [p,b]
leaf, with explicit independent conjugation and full source/domain binding.
Its native registered implementation remains the executor. This is a useful
coarse binding, not another physical local-method migration: #580/#581/#617
still own physical source, reference, frame and finite-source qualification.
No q/-q shortcut or normalization change is introduced.
Decisions and evidence sources¶
Snapshots hold the GIL, prove allocation bounds through a bounded NumPy base chain, and copy under native admission. Persistent inputs have no Python owner or mutable exported view. Before releasing the GIL, execution establishes input read leases and previews admitted execution storage; the native executor retains its own tickets and checks. Session concurrency is refused rather than waiting with the GIL. Each execution creates private output and publishes only on success. Previous publications survive later refusals and invalidation.
Retained owners are charged once. The input root is charged during its copy, then released from the engine inventory. Python object headers, caller data outside the import, allocator overhead and RSS are distinct. There is no claim of an inventory of the full Python DF object’s metric, Cholesky or three-center arrays. The legacy method has no finite-cap parameter; finite admission belongs to the additive Session API. Controls and rank/base-chain enumeration are bounded. Complex execution uses the existing conservative stream control envelope for its preflight. Results expose an admitted writable export separately from immutable owner-retaining views.
Read primary evidence: Hollman et al.,
mdspan, design/layout/accessor
and performance discussion; Mutlu et al.,
TAMM, sections III-IV, explicit index spaces
and coarse scheduling. Their mapping and scheduling ideas are reused; neither
paper supplies Python alias lifetime or this snapshot policy. The configured
study’s feedback, ownership/Python evidence and software/benchmark sections
were read. Installed pybind11 is 3.1.0; the matching
NumPy documentation
and GIL documentation
were consulted. The binding uses py::array without force-cast, explicit
gil_scoped_release, and a native shared result as the NumPy base owner.
Focused source verification¶
The test fixture compiles the actual binding translation unit and current headers with C++17/O2 into an isolated module, using installed pybind11 3.1.0 and Eigen headers. It never substitutes the installed chemistry extension. Eight new cases pass on the final binding source (42.94 s, including compilation). Eight existing native core/stream cases also pass; the combined earlier run passed all 16 in 34.14 s. Those unchanged native checks are reused.
Coverage includes independent asymmetric real sums and four complex
conjugation combinations; the native compensation residual; C/F, transpose,
slice, reverse, broadcast, scalar and empty layouts; dtype/conversion and
unsupported-owner refusal; source/metric/frame/partition and domain/parent
revision mismatch; mutation, shared Python aliases, deletion and immutable
publication; repeat execution and stale invalidation; exact and one-byte-below
snapshot/RI-J/complex admission; non-finite execution refusal with cleanup;
and progress of another Python thread during native execution. The real
Python DensityFitting.build_J source is loaded with unrelated integral
constructors stubbed and supplied numerical factors. It matches independent
sums, sees subsequent B/D mutations, preserves writable returns and records
all cost fields. These fixtures are not Gaussian or SCF evidence.
The ordinary pytest startup still refuses the pre-existing missing
pob-dzvp-rev2.ecp basis overlay before collection. The focused invocation is:
.venv/bin/python -m pytest --noconftest tests/test_tensor_local.py tests/test_tensor_engine.py tests/test_tensor_stream.py -q
Local cost measurements¶
benchmarks/tensor_local.py consumes the compiled current-source module and
reports 15-call medians in microseconds, one BLAS/OMP thread, C++17/O2. Inputs
and numerical policy match the direct native registered per-P reference in
the test module. Direct timing excludes factor construction and import. The
Python predecessor is the original two optimized NumPy einsums. Mutable
consumer calls include snapshots and a writable export. Immutable-plan reuse
is a separate workload and must not replace mutable public B/D semantics.
The initial general index-copy loop cost 8,363 us at (180,48,48). Replacing
per-element division/checked arithmetic with validated stride traversal and
a contiguous copy path reduced that C-layout copy to about 217 us in the
recorded run. Numerical kernels and ownership did not change.
Measured final values are recorded below at source delivery. First-call time includes first binding/NumPy owner setup after the module is loaded; it is not a cold process/import benchmark. Phase timings exclude Python dispatch and are not expected to sum exactly to wall time. Tracked bytes include the external import root and admitted native controls, not process RSS.
Operation / layout |
Old NumPy |
Mutable consumer |
Snapshot API |
Reused plan |
Direct native |
Tracked peak bytes |
|---|---|---|---|---|---|---|
RI-J (7, 3, 5) |
23.08 |
n/a |
11.04 |
1.79 |
0.33 |
9,504 |
RI-J (64, 24, 24) |
26.17 |
56.71 |
47.58 |
19.25 |
15.38 |
596,096 |
RI-J (180, 48, 48) |
48.62 |
460.96 |
480.21 |
178.25 |
169.67 |
6,641,792 |
RI-J (180, 48, 48) producer |
94.25 |
512.46 |
560.58 |
211.46 |
174.12 |
6,641,792 |
complex pair (257, 17, 23) |
n/a |
n/a |
354.50 |
356.79 |
339.92 |
266,080 |
Operation / layout |
First call |
Import metadata |
Snapshot copy |
Bind |
Plan |
Native execution |
Export |
Direct peak bytes |
|---|---|---|---|---|---|---|---|---|
RI-J (7, 3, 5) |
483.08 |
1.50 |
0.29 |
0.00 |
0.75 |
0.58 |
1.38 |
7,192 |
RI-J (64, 24, 24) |
57.54 |
1.71 |
17.83 |
0.04 |
0.79 |
19.62 |
1.71 |
313,432 |
RI-J (180, 48, 48) |
491.79 |
2.17 |
255.42 |
0.04 |
1.17 |
208.79 |
3.29 |
3,370,424 |
RI-J (180, 48, 48) producer |
776.58 |
4.12 |
309.83 |
0.04 |
2.58 |
207.67 |
5.38 |
3,370,424 |
complex pair (257, 17, 23) |
391.17 |
2.58 |
3.79 |
0.04 |
1.33 |
339.75 |
0.71 |
180,824 |
This run used arm64 Darwin 27, Apple clang 21, Eigen package 5.0.1,
NumPy 2.5.3 and pybind11 3.1.0. The isolated extension SHA-256 is
b6518fb09eefd3bc39a682946b459495a1c191137cd3500c8456fefa9afbae93.
This identifies a source-check artifact, not a delivered chemistry runtime.
The workstation was shared and timing variance was material: earlier
(180,48,48) producer-layout medians were 981-1,745 us for the snapshot API
and 103-357 us for the old NumPy path. The table is one complete run, not
an isolated hardware crossover or a speedup qualification. The producer layout
matches the auxiliary-contiguous matrix returned by the Python triangular
solve and its subsequent three-axis reshape. Complex direct/native timings
use the same compensated registered leaf and conjugation; a BLAS Hermitian
product is only an additional numerical oracle, not a reduction-policy-matched
performance reference. Complex export is an immutable view; RI-J export is
the writable copy needed by the existing consumer.
There is no presumed speedup. The existing registered per-P specialization is retained; generic indexed lowering is not substituted. Snapshot conversion of the producer’s auxiliary-contiguous layout and the per-P kernel are slower than the old borrowed NumPy path at the larger fixture. Numeric caching would violate public mutation semantics. Further packing/backend policy work belongs to #800/#802 and must include these import costs, not just kernel timing.
Runtime and remaining work¶
The inspected vq registry has a scheduler-only dev entry and an available 0.18.7 release. Neither contains this unlanded binding. Existing jobs and runtime owners are preserved; no duplicate build, fleet update or stale-runtime calculation is used to claim acceptance. Complete affected-method calculations through vq with containing Python/native identities and matched molecular references remain pending runtime delivery and Validator acceptance. The local basis-overlay refusal also remains. No ORCA/CRYSTAL or complete-method PASS is claimed by these source checks.
Keep #793/#802 and related unfinished issues open. Remaining capabilities include a second physical local consumer, selected non-affine domain binding, zero-copy contracts, async/device handshakes, serialized structural rebinding, full interpreter-mode qualification, and performance work for mutable inputs. Dev completion is focused checked source on main and its own cleanup; runtime delivery and scientific acceptance do not prolong finished source work.
Mutable-input performance increment¶
Date: 3 November 2026. Bounded follow-up under #800/#802, based on main
30967e2a, which contains 44817206 and 9b6479d0. This optimizes the
actual DensityFitting.build_J, not only an immutable native microkernel.
Diagnosis and implemented choice¶
Reproduction confirmed both costs: one pilot measured 709 us for the large C-layout consumer and 1261 us for the producer layout, with 275/749 us copying and 223/230 us native execution. In a fuller alternating baseline run, those consumer calls measured 735/1060 us. Shared-workstation timing variance is material; these observations are not universal shape thresholds.
ResidentFactors now optionally supplies an owner-retaining checked affine
View. RiJPlan validates its full domains, dtype, readable access, ledger,
bounds, value/pattern revisions and one of four dense physical matrix orders.
The existing two IndexPrograms still check the equations. Native providers
without this explicit capability retain their prior tile-only behavior.
The new ri_j snapshot layout preserves dense C, F and the actual triangular
solve’s auxiliary-contiguous order. Other mappings copy directly into a
supported matrix order, with contiguous or contiguous-row copies where
possible. There is no factor transpose followed by another factor pack.
The explicitly reassociated affine schedule reads that resident storage with
two audited Eigen BuiltIn GEMVs, on one caller thread and zero dynamic backend
workspace, even with EIGEN_USE_BLAS. Optional density and output reorder
buffers and full gamma are preflighted and ticketed by the existing Budget.
No opaque BLAS workspace is assumed free. Scalar IEEE execution is the
supported older-backend and non-finite-result fallback. The original ordered
per-P kernel and compensated complex paths are unchanged.
The Python method explicitly selects this numerical mode, just as its previous
optimized NumPy products allowed different associations. No basis, whitening,
fitting rank, threshold, density coefficient or scientific approximation
changes. Native default RiJPlan and public Session defaults remain ordered.
The source/domain/resource/thread key includes the affine View and mode;
there is no timing calibration or numeric cache in the Python method. Offline
full-call comparisons select this bounded capability rather than adding
first-call calibration that needs many SCF iterations to amortize.
A coarse method-owned entry point uses the same Session, snapshots, Input, Plan, leases and execution with one Python submission. Fresh call-local source and AO tokens retain the original method-owned association. Python callbacks and round trips are removed from phase setup, not from validation. Both B and D are snapshotted every time; inputs are released after execution and before writable export. A result remains owned independently. Diagnostics include all input/plan phases, packing bytes, audited backend workspace and the explicit reduction policy, through the existing output record.
Verification and performance protocol¶
The actual production binding TU is rebuilt independently of the old chemistry extension. Expanded native cases compare all four dense matrix orders and reversed signed densities against long-double explicit sums. They test exact and one-byte-below caps, stale native factors, invalid provider domains, provider/allocator failure, the non-finite scalar IEEE fallback, and allocation-free backend execution. Python cases exercise the real consumer, C/F/producer/sliced/reversed/broadcast inputs, odd/skewed shapes, cancellation and empty axes, mutation/deletion, stale bindings, writable publication, dtype refusal and GIL progress. The coarse GIL case mutates original B/D from another Python thread during native execution and verifies the result uses the completed snapshots. Existing ordered and compensated complex regressions remain in the focused run.
benchmarks/tensor_local.py --production measures the actual square-AO
Python method. Producer factors are obtained through SciPy solve_triangular
and its actual reshape, with strides recorded, rather than assuming a layout.
It alternates 31 samples of the old two-einsum expression, the same expression
with copy(order="K") snapshots of both inputs, and the actual consumer.
The candidate also times its coarse entry separately; paired differences
estimate Python facade/record overhead. Direct native timings execute the
same registered affine plan, numerical policy and one-thread allocation.
Those direct timings exclude imports; immutable prepared execution is not
used as the public mutable-input baseline.
Three sequences per length record 1/10/100/1000 complete mutable calls; the 19 MB factor fixtures stop at 100 calls. No factor values are cached between calls. First-call timing follows facade/binding imports and includes first operation setup, not a cold operating-system or dependency-import measurement. Phase sums omit some native object setup/cleanup and Python dispatch; the residual is labelled uninstrumented overhead, not a pure Python measurement. All BLAS/OMP ceilings are fixed in the benchmark process environment before import; the library never changes global BLAS settings. Process maximum RSS is reported separately from tracked bytes and includes Python, SciPy, benchmark arrays and allocator state; it is a cumulative process measure, not exact backend workspace.
Final source checks and before/after tables are recorded below. Raw runs, source/artifact identities and runtime/job inspection stay outside the public checkout. The inspected vq dev registration remains scheduler-only and the available release is 0.18.7. No inspected runtime contains this new source. Complete affected-method vq calculations with unchanged settings and matched references remain pending a containing runtime/Validator acceptance. Existing jobs and updater ownership are preserved; no old-runtime scientific PASS or duplicate managed build is claimed.
Final measured evidence¶
The final focused run passed 23 cases in 17.66 s, including compiling the
actual binding. The added exact/below-cap coarse-call case also passed against
that same artifact. Five planner cases passed with ASan/UBSan. The numerical
case passed with EIGEN_USE_BLAS and Accelerate linked, including both affine
orders with Eigen and global heap allocation disabled during the backend.
The isolated binding SHA-256 is
1d87934e1c3f443cceca77aba64db2359a4b0908796d3e8e70c57d90b45facef.
This is source evidence, not a delivered chemistry runtime. The additional
scalar IEEE fallback assertion passed in the sanitizer and BLAS-profile
numerical cases. Only the installed Eigen version is qualified here; the
older-backend scalar implementation is covered directly as an oracle, without
an older-runtime performance claim.
Final matched cohort: microseconds, 31 alternating samples per case, arm64
Darwin 27, Apple clang 21, C++17/O2, Eigen package 5.0.1, NumPy 2.5.3,
SciPy 1.18.1 and pybind11 3.1.0. The pre-change extension and Python facade
were preserved before editing and executed in separate processes; both used
one-thread environment ceilings. The stronger snapshot matmul reference
also copies both inputs but explicitly flattens the physical AO order for two
NumPy products, avoiding einsum planning and unnecessary F-layout conversion.
This prevents counting a win over Python einsum overhead as parity with the
fastest semantically equivalent baseline. The minimal output collector is
matched before/after; active calculation-file logging is not measured.
P x m x n / layout |
Old consumer |
New consumer |
Old NumPy expression |
NumPy with snapshots |
Snapshot matmul |
Direct native |
Before/new ratio |
|---|---|---|---|---|---|---|---|
7x3x3 / C |
14.46 |
8.29 |
14.17 |
15.17 |
2.54 |
0.21 |
1.74 |
64x24x24 / C |
44.88 |
22.08 |
15.75 |
23.88 |
11.25 |
7.75 |
2.03 |
180x48x48 / C |
364.79 |
165.04 |
38.58 |
97.62 |
80.79 |
95.12 |
2.21 |
180x48x48 / producer |
376.42 |
182.58 |
46.83 |
107.79 |
91.29 |
95.17 |
2.06 |
127x31x31 / F |
120.71 |
58.54 |
109.42 |
130.25 |
30.38 |
28.96 |
2.06 |
521x3x3 / producer |
22.96 |
10.54 |
14.92 |
17.46 |
4.71 |
1.08 |
2.18 |
257x97x97 / C |
2808.04 |
1000.33 |
329.38 |
605.29 |
571.67 |
599.42 |
2.81 |
257x97x97 / producer |
3697.92 |
1148.33 |
302.67 |
600.04 |
569.54 |
772.54 |
3.22 |
All eight recorded production cases improve over the previous engine consumer in this cohort, by 1.74-3.22 times. Residual cost remains: the new consumer is still slower than the layout-aware NumPy snapshot-matmul baseline. Small calls pay ownership/validation/dispatch and output-record overhead; larger calls also pay the slower audited Eigen backend compared with NumPy’s BLAS. No opaque BLAS workspace is waived to erase that difference. Qualifying another backend/workspace contract is remaining #800 work, not a claimed speedup here.
P x m x n / layout |
Import |
Snapshot copy |
Bind |
Plan |
Execute |
Export |
Facade overhead |
Input/export copy bytes |
Tracked peak bytes |
|---|---|---|---|---|---|---|---|---|---|
7x3x3 / C |
0.79 |
0.12 |
0.00 |
0.54 |
0.25 |
0.79 |
3.83 |
576 / 72 |
8,232 |
64x24x24 / C |
0.96 |
4.88 |
0.04 |
0.58 |
8.12 |
1.08 |
4.25 |
299,520 / 4,608 |
596,096 |
180x48x48 / C |
1.54 |
57.42 |
0.04 |
1.00 |
93.29 |
2.17 |
7.29 |
3,336,192 / 18,432 |
6,641,792 |
180x48x48 / producer |
1.54 |
60.79 |
0.04 |
1.08 |
108.00 |
2.25 |
7.00 |
3,336,192 / 18,432 |
6,641,792 |
127x31x31 / F |
0.96 |
17.04 |
0.04 |
0.62 |
31.67 |
1.46 |
4.79 |
984,064 / 7,688 |
1,959,024 |
521x3x3 / producer |
0.88 |
1.08 |
0.00 |
0.50 |
1.46 |
0.83 |
4.00 |
37,584 / 72 |
81,296 |
257x97x97 / C |
3.46 |
333.17 |
0.04 |
2.25 |
636.67 |
6.00 |
21.12 |
19,420,176 / 75,272 |
38,696,080 |
257x97x97 / producer |
3.37 |
322.08 |
0.08 |
2.08 |
794.54 |
6.17 |
19.75 |
19,420,176 / 75,272 |
38,696,080 |
Factor packing and dynamic backend workspace are zero. Density/output reordering remains included in execution and in tracked admission; its byte counters are available in diagnostics. The metadata/import and explicit copy phases are separate. Facade overhead is the median paired difference between the actual method and the same coarse native entry plus writable export. Phase medians do not sum exactly to wall time. The operation’s complete input copy is made on every call, including all sequence samples.
P x m x n / layout |
First operation |
1 call |
10 calls |
100 calls |
1000 calls |
Before cohort median range |
After cohort median range |
|---|---|---|---|---|---|---|---|
7x3x3 / C |
3086.46 |
7.33 |
5.92 |
5.82 |
5.82 |
14.46-32.25 |
8.29-15.46 |
64x24x24 / C |
55.12 |
20.17 |
19.27 |
18.81 |
19.69 |
44.88-71.54 |
22.08-42.00 |
180x48x48 / C |
201.92 |
169.17 |
167.99 |
161.91 |
158.42 |
364.79-667.96 |
165.04-378.04 |
180x48x48 / producer |
480.54 |
177.96 |
177.38 |
176.26 |
172.30 |
376.42-1120.88 |
182.58-394.17 |
127x31x31 / F |
75.88 |
50.21 |
48.85 |
55.08 |
54.34 |
120.71-283.42 |
58.54-141.46 |
521x3x3 / producer |
26.25 |
8.88 |
8.10 |
7.84 |
8.12 |
22.96-41.62 |
10.54-19.79 |
257x97x97 / C |
1164.92 |
981.17 |
999.33 |
938.51 |
not run |
2808.04-6288.79 |
1000.33-2009.25 |
257x97x97 / producer |
2217.50 |
1138.29 |
1169.12 |
1225.46 |
not run |
3697.92-6073.92 |
1148.33-3312.17 |
Sequences report median average us/call across three repetitions, including all repeated imports and exports. There is no calibration to amortize or hidden immutable preparation. First operation follows facade/binding imports; it is not an OS-cold or dependency-import measurement. Shared-workstation variance is large, as the three-cohort ranges show; the ratios are observations, not universal thresholds. The largest factors occupy 19,344,904 bytes. Maximum discrepancy versus the NumPy oracle in the final cohort is 8.59757e-13. The asymmetric long-double native oracle uses a 2e-13 absolute bound; cancellation, mutation and ordered/complex checks remain independent. No complete-method energy/density acceptance follows from these fixtures.
Whole-process maximum RSS in the final cohort was 155,828,224 bytes before and 135,708,672 bytes after. It includes Python/SciPy/BLAS, benchmark arrays, allocator reuse and the sequence collector. It is cumulative and separate from the per-session tracked peaks; it is not an estimate of backend workspace or a full DF method memory guarantee. The recorded performance and source increment leave broader #800/#802 and scientific acceptance open.