vibe-qc/Documentation

MPI runtime protocol: bounded sum phase

Accepted for the first runtime-only increment of MPP-02 (#820). This implements one opt-in substrate operation, not scientific dispatch or distributed tensor storage. See the architecture and measurement contract.

Decision and interface

vibeqc.mpi.BoundedSumPhase() borrows the validated MPIWorld communicator (COMM_WORLD, including a launched singleton). Ordinary serial construction uses no communicator and neither imports nor initializes MPI. Construct one persistent instance collectively during job setup, before local provider work. Control-storage allocation failure during construction is a job setup failure, not a supported collective refusal. No communicator is duplicated or freed. Future pool communicators require their own explicit collective construction and lifetime design; this API does not accept arbitrary communicator roles.

Only the Python main thread, also verified as the MPI initializing thread at world binding, owns MPI. The activation, launcher consistency and provided THREAD_FUNNELED checks remain. Cached world access rejects worker threads before they call MPI. Native/OpenMP workers remain MPI-free. Direct raw communicator use is the caller’s responsibility; this is not a wrapper around all mpi4py entry points. No thread-level or rank/thread default is changed.

Call phase.sum(prepare, phase_id=..., epoch=..., count=..., max_elements=...) on every rank in the same order. IDs and epochs are nonnegative signed-64 integers. Count and cap additionally fit signed-32 MPI counts. The caller assigns common phase IDs and source epochs; the instance owns an increasing sequence, consumed on success and collective refusal. Epochs identify source state but need not increase, allowing repeated work on one source epoch. This boundary does not validate tensor identity or leases.

prepare() performs bounded MPI-free local preflight/provider computation and returns a one-dimensional native float64 NumPy array with count elements. Empty contributors return zeros of that common shape; a globally empty phase uses count zero everywhere. Uneven task assignment remains with the caller, for example the existing KPointPartition. No native call with an ambiguous empty-task list is made. Providers own and must bound their source/workspace allocation; max_elements caps transport buffers, not provider memory or RSS.

Ordered protocol and resource ownership

1. Run local preparation, shape/type/count validation and allocate two owned contiguous payload buffers. Catch local exceptions as a one-bit refusal. Local exception details remain local; no strings or pickled objects travel.

2. All participants enter exactly two fixed Allreduce operations, MIN then MAX, on six int64 fields: sequence, phase ID, epoch, count, cap, refusal. Unequal first five fields or any refusal rejects payload posting everywhere. Control storage is three 48-byte arrays, independent of rank count.

3. On admission, post one Iallreduce(SUM) on the owned float64 send/receive buffers. The owner calls Test once and Wait if incomplete. There is no application callback while the request is active. No tags are allocated; at most one request and 144 + 16*count tracked buffer bytes are live.

4. Publish the complete replicated output on every rank only after completion. The instance clears its request/send references; the caller owns the returned result. Rank zero owns normal human/file output through vibeqc.output. The protocol itself performs no I/O.

Do not interleave other collectives or another phase instance on this borrowed communicator. Sequence agreement detects divergent calls that reach the same admission operation; it cannot rescue missing participation or mismatched raw collective ordering. Local reentry is rejected. All ranks must remain alive and reach each boundary. The application may retry a collectively refused phase with corrected input at the next sequence; partial output is never published.

Failure and shutdown

A CollectiveRefusal is recoverable only because no payload was posted and all participants completed admission. Shape, cap, provider and contract errors use this path. A slow participant is supported without an application timeout; tests enforce an external launcher timeout. A hung provider, rank loss, signal inside MPI, or failed transport has no portable bounded-recovery guarantee.

An exception during MPI posting/progress poisons the instance. Its request, communicator and buffers remain strongly referenced, including a module-owned reference if the caller drops the instance. Do not reuse it or proceed to normal finalization; the launcher/application must terminate the affected job under its MPI policy (for example owner-thread COMM_WORLD.Abort). No request cancellation, buffer release, communicator free or collective rollback is attempted. A retained poisoned instance is intentional fatal-path retention, not a resource reclamation strategy. Ordinary successful/refused phases have no active requests when they return; finish all worker work before normal MPI finalization on the initializing thread. The library does not finalize MPI.

Progress and measurement boundary

This first slice deliberately has no compute/communication overlap. A nonblocking call followed by owner progress is not evidence of overlap. It works with weak progress; blocking native work would delay owner progress. Tests delay both local preparation and one rank’s payload posting, then check completion, subsequent reuse and release of provider arrays.

The #819 measurement adapter is maintained separately. phase.metrics is a minimal compatible local success record: sequence/phase/epoch, rank/ranks, prepare/admission/total seconds, test_seconds, exposed_wait_seconds, payload bytes, tracked buffer bytes, and maximum in-flight requests. Test timing covers one owner Test call; wait includes imbalance/progress, not pure network time. Total also includes Python validation, posting and timer overhead. Metrics are local, not rank aggregates; no implicit telemetry collective is added. The last successful record is not a refusal record. Callers must record failure status separately.

The landed MPP-01 adapter uses quantities with value, unit, scope and reason. An adapter can map total seconds to boundary elapsed, admission and Test/wait to named phase intervals, exposed wait to communication.wait, payload bytes to communication.payload, request credit to communication.max_inflight, and tracked bytes to memory.tracked_live. Use seconds (s), bytes (byte) and counts (count), scoped to one rank and one call. Inclusive intervals must not be added to their children. Full-run time, actual network bytes and process memory remain null with reasons; this microbenchmark cannot populate a complete scientific-run record. No dependency on the measurement adapter is introduced.

For evidence, attach source and actual Python/native identity, MPI version and provided thread level, rank/thread/BLAS topology, three raw samples and their median/range to the compatible external record. Missing affinity, RSS, MPI internal allocations, network traffic, power and multi-node behavior are unmeasured, not zero. Buffer accounting excludes provider source/workspace, Python objects and MPI internal memory. No speed threshold, automatic selector or scaling claim is introduced. Scientific and scaling acceptance remain separate from this synthetic runtime acceptance.

Primary basis and next increment

MPI 5.0 sections 3.9, progress and 12.6.2, thread clarifications require explicit progress and ordered collective ownership; section 7 defines collective matching. This protocol uses standard MIN/MAX agreement over live participants, not fault-tolerant consensus.

Zhou et al., arXiv:2401.16551, section 2, defines MPIX threadcomm creation/activation and thread endpoints. That MPICH extension is not adopted. Losada et al., 10.1016/j.future.2020.01.026, sections 4-6, separates communicator repair from application data recovery. Open MPI ULFM documentation adds build, launch and component restrictions. No ULFM dependency or arbitrary process-loss recovery follows from this increment.

#804 can consume the exact prepare -> admission -> completed sum interface, with its own descriptor/source/lease admission inside preparation. Remote providers and leases remain its responsibility. The paired transport below adds a callback consumer and bounded tickets with measured compute-batch progress. Existing periodic collective helpers and scientific dispatch remain unchanged; physical consumers require their own numerical qualification.

Second increment: paired transport and admitted consumer

vibeqc.mpi_transfer.BoundedTransferPhase adds a separate W=1 byte transport. The sum API above is unchanged. All world ranks collectively construct the phase on the initializing/main thread. It duplicates the validated world once, so payload tag 0 and receipt tag 1 cannot collide with other phases or raw world traffic. Construction/control allocation failures are fatal setup failures. All ranks call close() collectively when idle; it frees only this duplicate. Close refuses active/poisoned phases. There is no destructor or exception handler that cancels requests or frees communicators. Native workers remain MPI-free.

phase = BoundedTransferPhase()  # collective setup
value = phase.run(
    prepare, consume, compute_batch,
    source=0, destination=1, nbytes=panel_bytes, max_bytes=panel_cap,
    phase_id=operation_id, epoch=source_epoch, batches=4,
)
phase.close()  # collective normal shutdown, after every ticket retires

This is an opt-in admitted callback consumer, qualified with a byte-validation and checksum fixture. It is not an automatic scientific consumer or a tensor adapter. prepare() runs only on source and returns a one-dimensional uint8 array. Transport copies it into an owned contiguous buffer. consume(view) runs only on destination with a read-only view after payload completion and the pre-consumption agreement. Its return value remains private until terminal agreement; all other ranks return None. A returned alias becomes caller-owned storage after retirement. Providers, computation and output allocations remain caller-budgeted, outside the transport cap. Callbacks must not publish external side effects, call MPI, reenter the phase, or retain failed partial outputs. No arbitrary callback can be made transactional by MPI.

compute_batch(index) runs on the owner between posting and finishing, once per admitted batch, with progress() after each successful batch. Work must be independent of the active payload and MPI-free, finite and bounded by the caller. Empty ranks still enter every control collective; their callback can be a no-op. This driver introduces no progress thread or native MPI and changes no defaults.

For adapters needing explicit scheduling, begin(prepare, **contract) returns a TransferTicket; ticket.progress() reports local payload completion; ticket.finish(consume, local_error=exception_or_none) drains and publishes or refuses. Exactly one ticket is live per phase. A completed send does not retire that ticket. Finished tickets reject progress/finish, and active phases reject another begin or close. The phase owns the ticket even if its caller drops it. A module-owned live registry roots posted phases until retirement, so abandoning both handles cannot let cyclic garbage collection reclaim active buffers or communicators. Callers of the explicit API must finish every admitted ticket or terminate the job on fatal failure; abandoning a ticket cannot invoke safe collective cleanup.

Ordering, receipts and failure

All participants execute the following sequence, including ranks with no data:

1. Prepare, reserve transport storage and agree with fixed MIN/MAX Allreduces on nine int64 fields: sequence, phase ID, epoch, source, destination, nbytes, cap, batches and local refusal. Invalid ranks/counts/caps, provider exceptions or differing contracts raise CollectiveRefusal before posting. Sequence is consumed on refusal. IDs/counts are nonnegative int64; bytes/cap also fit signed-32 MPI counts. Batches are a finite nonnegative int64 callback bound.

2. Source posts receipt Irecv and payload Isend. Destination posts payload Irecv. The same-rank case uses the owned local copy; zero-byte transfers are allowed. Other ranks post no point-to-point requests.

3. Poll between compute batches. Then complete local payload requests and enter a fixed status MAX agreement. A compute failure on any rank skips consumption everywhere. Stop remaining local callbacks after the first exception.

4. Destination consumes or records refusal, then sends a two-int64 receipt containing ticket sequence and success/refusal. Source validates the sequence after receiving it. Both complete their receipt request. This is a transport generation, not a tensor source identity or a remote lease token.

5. All ranks enter terminal status MAX agreement. Only then return the consumer value or raise collective refusal, clear request/buffer references and permit the next ticket. A callback exception is recoverable because all requests have completed, including the failure receipt. No partial output is returned.

The three control arrays occupy 216 bytes/rank, receipt storage 16 bytes/rank, and source/destination each own at most max_bytes payload bytes. Same-rank transfer uses one copy; inactive ranks have no payload. At most two requests are live on source (payload plus receipt) and one on destination; no dynamic tag allocation, unbounded ticket queue or per-panel communicator is created. The counted bound is 232 + nbytes on data ranks and 232 elsewhere, excluding Python objects, producer/consumer workspaces and MPI internals. Receipts have independent storage/credit, so releasing a data slot never needs another data slot.

Transport/posting/Test/Wait/agreement failures poison and retain the phase, communicator, ticket, requests and buffers in a module-owned fatal list. They require owner/launcher job termination, not retry or normal finalization. No active resource is freed on exception cleanup. Lost ranks, hung callbacks, unmatched participation, process signals inside MPI and arbitrary process-loss recovery remain unsupported. ULFM is not used. The isolated communicator avoids tag interference; it does not make incorrectly ordered collective construction or crossed operations on several communicators safe.

Measurements and #804 boundary

Each ticket and phase.metrics expose the latest local prepare/admission, compute, cumulative progress-call, payload/receipt wait, consumer, status-agreement and total seconds, executed batches/polls, bytes, request credit and terminal status. Preflight refusal has a minimal sequence/rank/status/total record; transport errors mark status fatal. The completed_during_batches flag reports observed local payload completion from polling; it does not prove a faster full operation, independent asynchronous progress, remote consumption, or lease retirement. Immediate-wait comparisons must include the same independent computation in their full boundary time. First-call and later reuse samples remain separate; rank maxima describe the collective critical path. No performance-based automatic selection is added.

Use the landed #819 value/unit/scope/reason quantities: elapsed for full boundary, named phases for disjoint timings, communication.wait for exposed wait, communication.max_inflight for request credit, and memory.tracked_live for counted transport buffers. Actual wire traffic and MPI/RSS allocations remain unknown; do not equate the payload count to network bytes or sum inclusive total time with its child intervals. Runtime records are not complete QC runs.

#804 owns descriptor authentication, contributions, source/replica epochs, Budget reservations, tensor lease meaning and its fake/native completion state machine. Its adapter can supply packed bytes and use consumer callback completion as the point at which this transport sends a release receipt. It must separately validate its own epochs/contribution and retain leases until its own obligations are met. No descriptor or lease ABI is introduced here, and no direct GPU buffer, remote-provider implementation or scientific dispatch is added. The next joint increment can bind #804’s admitted CPU adapter to this transport with that owner’s numerical and resource accounting. Broader scheduling across multiple communicators and GPU completion remain later work.

Compact refusal provenance

Transfer refusals now raise vibeqc.mpi_transfer.TransferRefusal, a subclass of CollectiveRefusal. Existing callers can keep catching the base class. The exception and phase.metrics expose failure_stage and failure_rank; the exception also exposes sequence. Stages are preflight, contract, compute, consumer and receipt. A contract mismatch without a local refusal reports rank None, since no rank has been established as authoritative. Local exceptions remain chained only on their originating rank; their text and type are never sent to peers. Do not parse exception messages for control flow.

The existing status int64 carries zero for success or 8*rank + stage_code, where preflight/compute/consumer/receipt codes are 1/2/3/4. Existing MAX agreements select the highest failing rank and then the highest stage code on a rank tie. This is deterministic representative provenance, not a list of every failure and not a claim about which failure happened first. A local preflight failure takes precedence over a simultaneous contract mismatch. On preflight refusal, sequence reports the maximum presented sequence, so even sequence disagreement produces identical diagnostics on every participant. This reports the mismatch; it does not repair the caller’s sequence stream or authorize replay.

The same packed code travels in the existing receipt status slot. Source accepts only the agreed compute refusal or the destination’s consumer outcome; an unexpected code or generation becomes a receipt refusal attributed to source, where validation failed. All requests still drain before terminal agreement and credit release. Invalid non-exception local_error arguments become compute refusals, rather than raising a rank-local Python chaining error after agreement. Successful metrics reset both provenance fields to None. Fatal transport metrics identify the local detecting rank and stage transport only; they are not a collectively agreed diagnosis or evidence that resources can be released.

No control/receipt buffers, collectives, tags or request credits were added; the 232-byte control/receipt census and payload cap are unchanged. All participants must use the same runtime revision, as with the existing private wire protocol. This diagnostic extension does not change #804’s tensor or lease ownership, scientific dispatch, process-loss limitations or fatal retention rules.