Compatibility and user-evaluation protocol¶
Protocol revision 1 is preparation for #68–#71. It does not freeze the 1.0 contract, certify a platform, report participant results or authorize publication. The milestone exits and maintainer decisions in the roadmap still apply. Run against one named candidate; retain failed attempts as well as passes.
Inputs and evidence¶
Before a run, record the source commit, immutable artifact filename and SHA-256, package version, OS/architecture, browser/Electron version, optional companion versions and corpus revision. Record the processor and memory for performance; exclude machine names, personal paths, credentials and participant identities. Keep operational logs and coordination in private operations storage. Publish only reviewed, public-safe fixtures and aggregate findings.
A case record needs its protocol revision, fixture ID/hash, interface, action, expected outcome, observed outcome, pass/fail/blocked status, evidence reference and linked defect. Missing dependencies, untested paths and manual tasks left undone are blocked, never passes. Reuse evidence only when its exact artifact, platform, fixture and affected code match; state why it still applies.
Performance and accessibility (#68)¶
Reuse corpus version 1 at tests/data/corpus/manifest.json and the frozen
performance budgets. Do not regenerate fixtures or
relax budgets to make a candidate pass. The existing harness is:
python scripts/make_corpus.py --check
python scripts/benchmark.py --browser --output perf.json
python scripts/benchmark.py --check perf.json
Retain cold and warm observations and memory records. Report non-reference hardware as comparative evidence, not reference-budget acceptance. A deliberate format refusal is an explicit outcome; it cannot stand in for a missing run. Keep graph, stereo and export fidelity checks alongside timing measurements.
The remaining manual review records screen reader, version and platform for keyboard-only drawing or its documented text alternative, import, save, export, submission and cancellation. Check accessible names, visible focus, logical focus order, status/error announcements and recovery from modal controls. Use a synthetic disposable job when a queue action is needed. A trapped focus, unannounced blocking error or task with no usable accessible alternative blocks that workflow’s acceptance. The existing browser checks support this review; they do not replace a person using assistive technology.
Packaged recovery (#69)¶
The source transaction evidence for e65b95a701a3763801def19e2852637e6c0f280d
is reusable for its bounded file-open/bind/save repairs. It does not establish
packaged recovery on a new platform. Use disposable directories and copies;
never inject failures into a user’s documents or an active runtime update.
For each platform declared supported by the candidate, record these cases:
Case |
Required outcome |
|---|---|
Interrupted startup; missing optional program/runtime |
Actionable recovery; original input unchanged; retry after restoring dependency succeeds. |
External replacement/deletion; stale revision; file/parent symlink retarget |
Save refuses the stale destination; original and external bytes remain intact; reopening or Save As permits deliberate recovery. |
Interrupted save |
Reopen yields the complete original or complete committed document; temporary residue cannot silently replace either. |
Interrupted update |
Use the release owner’s existing isolated update fixture; retain the previous usable artifact or documented recovery path. |
Malformed/oversized document or archive |
Bounded, explained refusal; subsequent valid open/save still works. |
Fresh install, update and uninstall |
Declared file associations and launch routes work; removing the app preserves user documents. |
Capture before/after file hashes, chosen destinations and recovered graph/stereo. Do not infer Windows/Linux/macOS acceptance from another platform. The release owner supplies exact artifacts, supported matrix and safe interruption points; this protocol does not authorize a new build or live update failure injection.
Compatibility corpus (#70)¶
Build an inventory from every shipped document-producing release, including patches with document changes. For each original VSK/QVF fixture, record producer version, original artifact hash, fixture hash, schema ID, features covered and expected retained/refused content. Preserve original bytes and producer metadata; a document recreated by current code is not a historical fixture. An absent release sample is an explicit coverage gap. The performance corpus is useful coverage but is not a complete historical compatibility corpus.
Read each fixture through SDK, CLI, REST and MCP where that interface supports the action. Save to a new destination, reopen, and compare the full supported content: canvas, ordered atoms/bonds/stereo, species IDs/labels, reactions, conformers, jobs/results, recipes and provenance. Compare meaning where a writer normalizes JSON ordering; preserve unknown fields or refuse explicitly according to the documented format contract. Test deprecated imports/commands separately from canonical entry points. Invalid/future schemas must produce explicit migration/refusal guidance without modifying the original. Silent field loss, changed chemistry, broken supported aliases or an unexplained refusal blocks compatibility acceptance. Record intentional format limitations as such.
Use the document model, formats, Python API, CLI, REST and MCP manuals as the proposed interface inventory. The maintainer reviews the supported surface and migration rules before a freeze; a test passing does not approve a breaking change.
Historical readback implementation¶
Corpus revision 4 at tests/data/compatibility/manifest.json inventories the
12 changelog release tags through v0.6.5. It preserves 18 distinct original VSK
fixtures, deduplicated by content, with per-tag source commit/path/blob receipts,
SHA-256 and unchanged producer provenance. Reproduction uses git show TAG:PATH
or git cat-file blob BLOB from those receipts; tests need no Git history or
network. tests/data/compatibility/README.md documents collection boundaries.
Run python -m pytest tests/test_historical_compatibility.py -q to verify
inventory, original fields, SDK/CLI/REST/MCP read/save/reopen and unsupported
schema/field refusal. Additive defaults are allowed, but every original field
must survive and reopened documents must match the entire normalized document.
The tests also exercise current-writer QVF round trips. Revision 2 adds 13
unchanged historical QVF originals from the authenticated 0.6.0 scientific-data
revision 2 archive: its SDK fixture and 12 calculation results. The manifest
binds archive and member SHA-256, sizes, public source URL and producer identity;
ordinary tests remain offline. Nine geometry-only figure files are explicitly
excluded from whole-document coverage. Result imports retain original conformers
and append the recorded result geometry; checks require that exact addition and
full normalized document preservation, including REST result receipts.
This is tagged source-fixture evidence. Original producer metadata can predate the release carrying an example. Revision 3 authenticates exact source-archive fixture membership for 0.6.4 and 0.6.5 against their published checksums; ten older public source URLs returned 404. Revision 4 additionally authenticates retained original release sdists for those ten versions; they contain zero saved-document members, so release build identity and tagged-fixture membership stay distinct. Six unchanged retained development-acceptance outputs add DNA/RNA/peptide return and generic/ R-group/nested-stereo readback coverage, labelled separately from released files. Its reproducible saved-field inventory distinguishes present job/recipe/conformer/sequence/generic/R-group examples from missing surface, highlight, nanotube, solid, capping and half-reaction originals. Remaining older tagged-VSK packaged-membership evidence, historical QVF versions beyond this archive and feature-complete producer coverage remain explicit manifest gaps; the tests do not count them as passed. The initial strict readback exposed four CLI/REST failures for two originals. Approved #181 policy now preserves validated saved content on whole-document open, resolving these failures without changing original fixture bytes or relaxing comparisons. Derivation remains part of edits. Extending the release inventory requires reviewed originals and updated receipts, never newly forged old-version output.
User evaluation and release decision (#70–#71)¶
Give participants the same candidate, public-safe inputs and task instructions. Record whether each task succeeded unaided, needed help, failed or was blocked, with elapsed time and an anonymized finding; keep consent and identities private. Do not invent results or count agent automation as human user evaluation.
Workflow |
Task and observable result |
|---|---|
Author |
Import/draw a stereochemical reaction, edit labels, save/reopen and export a publication figure; compare chemistry and legibility with the supplied target. |
Computational chemist |
Prepare a supported job, inspect explicit method/options, submit to the designated test queue, cancel a disposable job and inspect a completed result; no silent option substitution. |
Agent |
Discover capabilities, create/edit through the public API, handle a revision conflict and an unsupported request, then save/export and independently reopen. |
Any data loss, silent chemistry/option change, privacy boundary failure or inability to finish a promised workflow is a blocker. Link each finding to a defect, fix and rerun the affected task. Report task success/help counts and unresolved findings; do not set an unsupported numerical usability threshold. The maintainer reviews the protocol, participant coverage and findings before contract freeze and explicitly approves 1.0 publication under #71. Contributor policy changes remain maintainer decisions; evaluation participation grants no permission for outside code contributions.