diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md new file mode 100644 index 00000000..7d5379dc --- /dev/null +++ b/docs/design/tableau-data-structure.md @@ -0,0 +1,413 @@ +# Contiguous tableau data structure + +Status: design sketch + +## Purpose + +The tableau should be a specialized bit-matrix data structure. It should not +be represented as `Vec>`, and its internal rows should not +affect the shared PPVM trait system. + +This design separates: + +- the logical stabilizer-tableau model; +- its physical, contiguous memory layout; +- orientation changes used to accelerate different operations; and +- structural hashing used when a tableau is a classical-mixture key. + +Gate, noise, measurement, reset, and `Indexable` traits observe tableau +behavior. They do not expose matrix blocks, row values, strides, alignment, or +orientation. + +## Goals + +- Store the X/Z tableau bits in contiguous, aligned, bit-packed memory. +- Make one- and two-qubit column operations efficient. +- Permit temporary transposition for row-oriented elimination and collapse. +- Keep phases and per-qubit loss state independently addressable and hashable. +- Hash the logical tableau independently of its current physical orientation. +- Avoid parameterizing the trait system by a row type or Pauli-word storage. +- Leave room for SIMD-width and padding changes without changing public + behavioral traits. + +## Non-goals + +- Reusing the standalone `PauliWord` representation for tableau rows. +- Exposing borrowed tableau rows as the primary public interface. +- Standardizing a general-purpose matrix trait in `ppvm-traits-2`. +- Maintaining both row-major and column-major copies before benchmarks show + that the memory and synchronization cost is worthwhile. +- Deciding in this document whether PPVM should store a forward or inverse + tableau. Orientation and inversion are independent design choices. + +## Logical model + +For `n` qubits, a stabilizer/destabilizer tableau has `2n` generators. Each +generator contains an X bit and a Z bit for every qubit, plus a phase: + +```text + qubit + 0 1 2 ... n-1 +generator 0 x/z x/z x/z +generator 1 x/z x/z x/z + ... +generator 2n-1 x/z x/z x/z + +phase one value per generator +``` + +The logical state consists of: + +- an X matrix of shape `(2n, n)`; +- a Z matrix of shape `(2n, n)`; +- a phase plane of length `2n`; and +- a per-qubit loss plane of length `n`. + +This model does not imply a physical row object. + +## Physical storage + +The first prototype should use one aligned contiguous allocation for the bit +planes, divided by computed offsets: + +```rust +pub struct TableauData { + blocks: AlignedVec, + x_offset: usize, + z_offset: usize, + phase_offset: usize, + loss_offset: usize, + major_stride: usize, + n_qubits: usize, + orientation: Orientation, +} +``` + +`Block` is an internal implementation choice such as `u64` or a SIMD-width +block. It is not an associated type of a public tableau trait. Offsets and +strides account for alignment and padding, while logical accessors enforce the +actual `(2n, n)` dimensions. + +Keeping all planes in one allocation improves cloning and locality and avoids +one allocation per generator. It also lets a mixture branch copy a single +contiguous region. If benchmarks favor separate aligned allocations for X and +Z, that change remains internal to `TableauData`. + +Padding must either be kept zero or excluded from equality and hashing. Zeroed +padding is preferable because it permits bulk comparison and hashing of +canonical ranges. + +## Loss ownership + +The existing concrete `Tableau` always owns the per-qubit loss plane. This is a +capability of the same tableau type, not a `LossyTableau` variant or a +`Tableau` parameter: + +```rust +pub struct Tableau { + data: TableauData, + lost_count: usize, + // hash caches and RNG +} + +pub struct GeneralizedTableau { + tableau: Tableau, + coefficients: S, + // threshold and measurement record +} +``` + +`GeneralizedTableau` therefore has no separate `is_lost: Vec`. Its gate, +noise, and measurement algorithms query and mutate the loss plane owned by the +inner tableau. + +`lost_count` is derived metadata used to preserve a fast +`lost_count == 0` path. It is excluded from equality and hashing; debug builds +should verify that it equals the population count of the logical loss plane. +When no qubit is lost, gate kernels enter their existing lossless bulk path. +When loss is present, a one-qubit gate skips a lost target and a two-qubit gate +skips the operation if either target is lost, matching the current generalized +tableau semantics. Batch kernels should mask or skip lost targets without +allocating filtered target vectors. + +This ownership enables a pure Clifford-plus-loss simulation with the same +`Tableau`. Loss events use the pure Clifford collapse procedure before marking +the affected qubit lost. Generalized loss events still use the +coefficient-aware generalized measurement procedure before marking the loss; +moving the bit does not move that algorithm. The pure path covers loss models +whose conditional trajectory remains a stabilizer state; faithful +non-Clifford survival back-action remains generalized. + +## Column-major orientation + +The default orientation should make the generator dimension contiguous for a +fixed qubit. In other words, the X and Z planes are column-major with respect +to the logical `(generator, qubit)` matrix: + +```text +qubit 0: X bits for generators 0..2n, then padding +qubit 1: X bits for generators 0..2n, then padding +... + +qubit 0: Z bits for generators 0..2n, then padding +qubit 1: Z bits for generators 0..2n, then padding +... +``` + +This layout makes a selected qubit column contiguous. Single-qubit gates can +load the X and Z columns, update them with bitwise operations, and update the +affected phase bits. Two-qubit gates operate on two pairs of contiguous +columns. Measurement can scan the selected anticommutation column without +stepping across separately allocated row objects. + +Stim uses aligned SIMD bit tables and documents its tableau layout as +column-major, with output-observable iteration following contiguous memory. It +also provides an explicit quadrant transpose and a transposition guard for +operations that need the opposite orientation: + +- [Stim `Tableau`](https://github.com/quantumlib/Stim/blob/main/src/stim/stabilizers/tableau.h) +- [Stim `TableauSimulator`](https://github.com/quantumlib/Stim/blob/main/src/stim/simulators/tableau_simulator.h) +- [Stim `simd_bit_table`](https://github.com/quantumlib/Stim/blob/main/src/stim/mem/simd_bit_table.h) + +Column-major storage is only one part of Stim's performance strategy. PPVM +should benchmark its own gate, measurement, and sampling workloads instead of +assuming the same total performance from layout alone. + +## Temporary transposition + +Row multiplication and elimination need long generator rows and therefore +prefer the opposite orientation. The initial design should transpose the X/Z +quadrants temporarily instead of storing two permanent copies: + +```rust +pub enum Orientation { + ColumnMajor, + RowMajor, +} + +pub struct TransposedTableau<'a> { + tableau: &'a mut Tableau, +} +``` + +Creating the guard transposes the bit matrices and marks the physical +orientation. Dropping it restores the canonical column-major orientation: + +```rust +impl Drop for TransposedTableau<'_> { + fn drop(&mut self) { + self.tableau.restore_column_major(); + } +} +``` + +Operations that require row-major access receive the guard, making the +orientation precondition explicit. Public methods return with the tableau in +canonical orientation. Panic safety must be preserved by the guard's `Drop` +implementation. + +Transposition is a physical reordering of the same logical bits. It does not +invalidate structural hashes or change equality. Hashing should occur only +through logical access or while the tableau is in its public canonical state. + +Maintaining both orientations with dirty flags remains a future alternative +for workloads that switch frequently enough to amortize the doubled storage. + +## Tableau API boundary + +The public API exposes logical operations instead of storage slices: + +```rust +impl Tableau { + pub fn n_qubits(&self) -> usize; + + pub fn h(&mut self, qubit: usize); + pub fn cnot(&mut self, control: usize, target: usize); + pub fn measure_z(&mut self, qubit: usize) -> Option; + + pub fn x_bit(&self, generator: usize, qubit: usize) -> bool; + pub fn z_bit(&self, generator: usize, qubit: usize) -> bool; + pub fn phase(&self, generator: usize) -> Phase; + pub fn is_lost(&self, qubit: usize) -> bool; +} +``` + +Logical bit accessors are useful for tests, serialization, debugging, and +interoperability. They are not intended as the hot gate implementation path. +Bulk import/export methods may use a canonical serialized representation +without exposing the in-memory layout. + +There should be no `stabilizers_mut() -> &mut [Row]` or equivalent escape hatch +that bypasses hash invalidation and orientation invariants. Specialized +internal row operations operate through `TableauData` or a transposition guard. + +## Measurement algorithms + +The Rust behavioral boundary is one loss-aware trait: + +```rust +pub trait Measure { + fn measure(&mut self, qubit: usize) -> Option; + + fn measure_many(&mut self, targets: &[usize]) -> Vec> { + targets.iter().map(|&q| self.measure(q)).collect() + } +} +``` + +`Some(false)` and `Some(true)` represent computational-basis outcomes; `None` +represents a lost qubit. This is the existing core representation used by +`GeneralizedTableau`. The Python binding may continue mapping it to +`MeasurementResult::{ZERO, ONE, LOST}`. The old public split between +`Measure -> bool` and `LossyMeasure -> Option` is removed. The bare +boolean Clifford measurement routine becomes a private helper called only +after the public implementation has established that the target is present. + +The common trait and result type do not imply a common measurement algorithm: + +- `Tableau::measure` checks the loss plane and then uses the pure Clifford + stabilizer measurement procedure. It does not decompose against a sparse + generalized-state coefficient basis. +- `GeneralizedTableau::measure` checks the inner tableau's loss plane, then + decomposes the measured Pauli into stabilizers and destabilizers and updates + the sparse coefficients. This path is fundamentally \(O(n^2)\). + +The physical tableau must make both stabilizer and destabilizer generators +available to the generalized decomposition, but that requirement must not be +promoted into the `Measure` trait or force the generalized algorithm onto the +pure Clifford implementation. Pure and generalized measurement performance +must be benchmarked separately; Stim's inverse-tableau measurement +optimizations are not complexity promises for the generalized algorithm. + +## Gate access patterns + +The implementation should categorize mutations by physical access and hash +effect: + +| Operation | Preferred access | X/Z changed | Phase changed | +| --- | --- | --- | --- | +| Pauli `X`, `Y`, `Z` | column | no | possibly | +| `H`, `S` | one column pair | yes | possibly | +| `CNOT`, `CZ` | two column pairs | yes | possibly | +| Find measurement pivot | column | no | no | +| Row multiplication | transposed row | yes | possibly | +| Collapse/elimination | column scan + transposed rows | yes | yes | +| Physical transpose | bulk matrix | logically no | no | + +This table describes logical mutations. A gate may determine that no phase bit +changed and preserve the phase cache in that special case, but conservative +component invalidation is correct for the first implementation. + +## Structural hashing + +A tableau used in a classical mixture is an `Indexable` key. It owns its own +hasher and cache representations; neither is inherited from `PauliWord`. + +The structural hash is composed from independent logical components: + +```text +tableau hash = combine(xz hash, phase hash, loss hash) +``` + +The X and Z planes share an `xz_hash` cache because most Clifford mutations +update them together. The phase plane has a separate cache so Pauli +conjugations and sign changes do not force a matrix rehash. The independently +mutable loss plane uses a third cache. + +```rust +pub struct Tableau { + data: TableauData, + lost_count: usize, + xz_hash: OnceLock, + phase_hash: OnceLock, + loss_hash: OnceLock, + rng: SmallRng, +} +``` + +The cache fields are private representation choices made by the tableau +author. `Indexable` exposes only the associated build hasher; it does not name +cache types or expose invalidation. + +Equality and hashing include logical qubit count, generator order, all logical +X/Z bits, phases, and loss state. They exclude: + +- RNG state; +- allocation capacity; +- alignment padding; +- cache values and validity flags; and +- current physical orientation. + +The component invalidation rules are: + +| Mutation | X/Z cache | Phase cache | Loss cache | +| --- | --- | --- | --- | +| Pauli `X`, `Y`, `Z` | preserve | invalidate if changed | preserve | +| Direct phase change | preserve | invalidate | preserve | +| `H`, `S`, `CNOT`, `CZ` | invalidate | invalidate if changed | preserve | +| Row multiplication | invalidate | invalidate if changed | preserve | +| Toggle a loss bit | preserve | preserve | invalidate | +| Physical transpose | preserve | preserve | preserve | +| RNG update | preserve | preserve | preserve | + +The current `ppvm-tableau-sum` split between `word_fingerprint` and +`phase_loss_hash` is evidence that component hashing matters. The new tableau +owns X/Z, phase, and loss components directly. + +## Cloning and mixture use + +Classical-mixture branching can clone tableaus frequently. Contiguous backing +storage makes cloning a bulk memory copy. A clone may copy valid hash caches +because it initially has identical logical contents. Subsequent mutation of +the branch invalidates only the affected components. + +Copy-on-write backing storage may be evaluated later, but it should not be part +of the initial design: most branches are mutated immediately, which may turn +reference counting and deferred copying into overhead. + +An indexable tableau must not be structurally mutated while stored as a map +key. Mixture storage removes or clones a key before applying gates and inserts +the resulting tableau under its updated hash. + +## Sampling implications + +Column-major X/Z planes make fixed-qubit queries and bit-parallel gate updates +efficient. They are also compatible with scanning many generators during a +measurement. Temporary transposition makes elimination and row products +contiguous when required. + +This layout should be evaluated separately from higher-level sampling +algorithms. Stim also uses an inverse tableau and reference-frame sampling; +those algorithmic choices are not consequences of column-major storage and do +not belong in the PPVM trait system. + +## Prototype validation + +The prototype should include: + +- property tests comparing gates and measurements with the existing tableau; +- differential tests for the pure Clifford and generalized measurement + algorithms, including lost targets; +- tests that gates skip lost targets and retain the `lost_count == 0` fast + path; +- round-trip tests for column-major -> row-major -> column-major transpose; +- equality and hash tests across physical orientations; +- tests proving phase-only changes preserve the X/Z hash cache; +- tests proving padding never affects equality or hashing; +- benchmarks for one- and two-qubit Clifford gates; +- separate benchmarks for pure Clifford and generalized deterministic and + random measurement paths; +- benchmarks for lossless gates, sparse loss, and Clifford-plus-loss sampling; +- benchmarks for clone-and-mutate mixture branching; and +- benchmarks comparing permanent row-major, permanent column-major, and + temporary-transpose variants on representative circuits. + +## Open questions + +1. What block width and alignment should the first implementation use? +2. Should phases occupy one or two bits per generator in the tableau model? +3. Which operations should receive a transposition guard versus performing + column-strided work directly? +4. Does a dual-orientation representation outperform temporary transposition + for PPVM's measurement-heavy workloads? +5. Should PPVM ultimately store a forward or inverse tableau? diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md new file mode 100644 index 00000000..2709ea04 --- /dev/null +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -0,0 +1,527 @@ +# `ppvm-traits-2`: type composition, indexable values, and cached hashing + +Status: design sketch + +## Motivation + +The current `ppvm_traits::Config` bundles choices from several unrelated +layers: + +- coefficient type; +- packed Pauli storage; +- Pauli-word representation; +- key hasher; +- truncation strategy; and +- concrete map implementation. + +This makes the foundational configuration specific to `PauliSum`, even though +the gate and noise traits are shared by other algorithms. In particular, a map +and a truncation strategy are not properties of quantum data. They are choices +made by a particular algorithm. + +The second trait-system experiment should separate: + +1. the coefficient type, passed directly as a generic parameter; +2. concrete quantum-data representations; +3. the hashing contract of values that can be used as keys; and +4. algorithm-specific storage and policy choices. + +The redesign should remain compile-time generic. It should not introduce +runtime trait objects or runtime dispatch for these choices. + +This proposal was compared against the current definitions in +`ppvm-traits/src/config.rs`, `traits/map.rs`, `traits/strategy.rs`, and +`traits/word_trait.rs`; `ppvm-pauli-sum/src/sum/data.rs`; the concrete word +types in `ppvm-pauli-word`; and `ppvm-tableau-sum`'s `EntryStore`, `VecStorage`, +and `MapStorage`. Existing names are retained below unless the abstraction or +responsibility changes. + +## Type-composition layers + +### Coefficient type + +There is no algorithm-agnostic `Config` trait. Algorithms use the coefficient +type directly: + +```rust +pub struct SomeAlgorithm { + // ... +} +``` + +A trait containing only `type Coeff` adds indirection without providing useful +composition. Maps, policies, Pauli-word storage, Pauli-word +implementations, and hashers are also selected independently rather than being +collected into a replacement global configuration trait. + +### Representation types + +There is no separate global storage configuration, and representation storage +does not appear as an associated type. A concrete value encapsulates its own +fields; generic algorithms use behavioral methods instead of naming or +inspecting the backing memory. + +`Word` is the common concept for an indexed algebraic monomial. The old +Pauli-word operations are not Pauli-specific: every supported word has a site +alphabet, an indexed extent, site access and mutation, and a weight: + +```rust +pub trait Word { + type Site; + + fn n_sites(&self) -> usize; + fn get(&self, index: usize) -> Self::Site; + fn set(&mut self, index: usize, site: Self::Site); + fn weight(&self) -> usize; +} +``` + +`Site` selects the operator alphabet without introducing `PauliWord` or +`FermionWord` subtraits. For example, the relevant concrete types may be: + +```rust +pub enum Pauli { + I, + X, + Y, + Z, +} + +pub enum LossySite { + Present(S), + Lost, +} + +pub struct FermionSite { + pub mode: usize, + pub action: FermionAction, +} +``` + +Thus an ordinary packed Pauli word implements `Word`, a +concrete packed lossy word implements `Word>`, and a +future ordered fermionic product can implement `Word`. A +fermionic word's index denotes factor order; `FermionSite` carries the physical +mode. For a dense Pauli word, the index is the qubit and `n_sites()` is its +width. +`weight()` is the number of non-identity factors according to the concrete site +alphabet; an ordered representation that stores no explicit identities may +therefore have `weight() == n_sites()`. Implementations of `set()` preserve +their representation invariants and invalidate the affected hash components. + +The concrete `LossyPauliWord` stores packed X, Z, and loss planes directly and +provides loss mutation and `loss_weight()` as inherent methods. Loss channels +and loss-specific truncation specialize directly on that concrete type; there +is no one-implementation `LossyPauliWord` capability trait. A generic loss +wrapper should be reconsidered only after a second real word representation +needs the same composition. + +Concrete packed Pauli, lossy, phased, and hash-cache layouts are described in +[`word-data-structures.md`](word-data-structures.md). None of those layouts is +visible through `Word`. + +A tableau is an independent concrete representation. It does not contain a +public row type, implement `Word`, or select an associated word +implementation. Its X/Z matrices, phases, orientation, and contiguous backing +allocation are private implementation details described in +[`tableau-data-structure.md`](tableau-data-structure.md). + +### Behavioral traits + +Shared traits describe operations, not representation layout. Clifford gates +need no coefficient parameter. Operations that consume numeric parameters use +the coefficient type directly: + +```rust +pub trait Clifford { + fn h(&mut self, qubit: usize); + fn cnot(&mut self, control: usize, target: usize); + // ... +} + +pub trait RotationOne { + fn rx(&mut self, qubit: usize, theta: C); + // ... +} + +pub trait PauliError { + fn pauli_error(&mut self, qubit: usize, probabilities: [C; 3]); +} + +pub trait Measure { + fn measure(&mut self, qubit: usize) -> Option; + + fn measure_many(&mut self, targets: &[usize]) -> Vec> { + targets.iter().map(|&q| self.measure(q)).collect() + } +} +``` + +The same concrete tableau may implement a numeric trait for every supported +coefficient type without storing that coefficient type. Measurement and reset +traits likewise expose behavior and result types without exposing tableau +rows, packed matrix blocks, or matrix orientation. `Measure` is loss-aware for +both `Tableau` and `GeneralizedTableau`: `Some(false)` and `Some(true)` denote +computational-basis outcomes and `None` denotes a lost qubit. The former +`Measure -> bool` and `LossyMeasure -> Option` split is removed. +Python may continue translating this Rust representation to its +`MeasurementResult` enum. + +Sharing the result type does not share the measurement algorithm. `Tableau` +uses the pure Clifford measurement procedure. `GeneralizedTableau` performs +its coefficient-aware stabilizer/destabilizer decomposition and update, which +is always \(O(n^2)\). The shared behavioral trait must not force the latter +algorithm or its scratch state into the concrete tableau. + +There is no shared `TableauStorage` trait in the first design. If multiple +tableau implementations are later useful, each concrete type can implement +the same behavioral traits. A storage abstraction should only be introduced +after two implementations demonstrate a common interface. + +### Algorithm and storage parameters + +An algorithm should take its independent choices as direct type parameters. +An associated-type bundle is not useful merely because it replaces two type +parameters with one. In particular, there is no `PauliSumAlgorithm` trait that +bundles a term map with a policy: storage layout and policy are orthogonal +choices. + +The reusable sparse-sum shape is: + +```rust +pub struct OperatorSum +where + C: Coefficient, + W: Word + Indexable, + S: SumStorage, + P: Policy, +{ + storage: S, + policy: P, + n_sites: usize, +} +``` + +Here `C`, `W`, `S`, and `P` respectively mean coefficient domain, algebraic +word, concrete sparse-sum storage engine, and algorithm policy. Each +parameter has an independent meaning. Propagation methods select their algebra +through the site type, for example `W: Word` or +`W: Word`. + +`Policy` is the proposed name for the current `Strategy` concept. It retains +the current responsibilities: predicting initial capacity and truncating the +sum. Existing concrete strategies become policies without otherwise changing +their meaning; `NoStrategy` and `CombinedStrategy` become `NoPolicy` and +`CombinedPolicy`, while `MaxPauliWeight` and `CoefficientThreshold` keep their +established names: + +```rust +pub trait Policy: Default + Clone + Copy +where + W: Word + Indexable, + C: Coefficient, +{ + fn capacity(&self, n_sites: usize) -> usize; + + fn truncate(&self, map: &mut M) + where + M: ACMap; +} +``` + +`Policy` and its concrete implementations belong to the sparse-sum crate. This +removes the current split where the `Strategy` trait lives in `ppvm-traits` but +its concrete strategies live in `ppvm-pauli-sum`; the policy is not an +algorithm-agnostic `ppvm-traits-2` concern. + +`ACMap` remains the name of the associative coefficient-map capability already +implemented by `HashMap`, `IndexMap`, and `DashMap`. Its generic signature can +be simplified after `PauliStorage` and the separately supplied build hasher are +removed, but coefficient accumulation, iteration, insertion, retention, and +consumption are the same concept. `ACMap` moves with the sparse-sum engine (the +existing `ppvm-pauli-sum` initially) rather than being renamed or kept in the +algorithm-agnostic trait crate. Its existing capability names—such as +`ACMapBase`, `ACMapIter`, `ACMapAddAssign`, `ACMapInsert`, `ACMapRetain`, and +`ACMapConsume`—should also remain unless implementation work shows that two +capabilities should actually be merged or split. + +A `SumStorage` is a new abstraction extracted from the fields currently owned +directly by `PauliSum`: its maps and reusable workspace. It is an actual value, +not a marker configuration: + +```rust +pub trait SumStorage: Clone +where + W: Word + Indexable, + C: Coefficient, +{ + type Map: ACMap; + + fn data(&self) -> &Self::Map; + fn data_mut(&mut self) -> &mut Self::Map; + + fn map_insert(&mut self, f: F) + where + F: Fn(&W, &mut C) -> Option<(W, C)>; + + fn map_insert_multiple(&mut self, f: F) + where + F: Fn(&W, &mut C) -> Option>; + + fn map_add(&mut self, f: F) + where + F: Fn(&W, &C) -> (W, C); + + fn consume(&mut self); +} +``` + +The exact closure bounds and support for multiple produced terms remain an +implementation detail for the prototype. The important boundary is that the +trait preserves the current semantic operation names without exposing physical +auxiliary maps or scratch buffers. + +`SumStorage` owns the semantic whole-map operations and its reusable workspace. +It delegates to the lower-level `ACMap` batch kernels without restoring the +removed map-to-map `ACMapInsert::map_insert` method: + +```text +SumStorage::map_insert -> ACMapInsert::map_insert_vec +SumStorage::map_insert_multiple -> ACMapInsert::map_insert_multiple +SumStorage::map_add -> ACMapAddAssign::map_add_assign +``` + +This boundary is compatible with +`refactor/shrink-internal-trait-surface`: the higher-level sparse-sum operation +remains, while the dead low-level primitive stays removed. + +The generalized engine may be named `OperatorSum`, but the Pauli specialization +retains the existing domain-facing `PauliSum` name. This is a new internal +generalization, not a requirement to rename Pauli call sites. + +A classical tableau mixture follows the same principle and takes its +`EntryStore` directly rather than introducing a one-associated-type +`TableauMixtureAlgorithm` bundle: + +```rust +pub struct GeneralizedTableauSum +where + C: Coefficient, + T: Indexable, + S: EntryStore, +{ + entries: S, +} +``` + +`GeneralizedTableauSum` and `EntryStore` retain their current names because the +proposal does not change their underlying roles. `GeneralizedTableauSum` and +`OperatorSum` are both sparse linear combinations of indexable keys, so they +may eventually share an implementation. This iteration deliberately keeps them +separate: their mutation, branching, normalization, and storage requirements +have not yet been reduced to a proven common interface. The next design +iteration should look for the smallest useful common factor and merge only that +factor, rather than assuming that the two complete algorithms are identical. + +Every keyed store must use the build hasher associated with its key type. + +### Compatibility with current names + +The redesign is not a vocabulary reset. The following names are retained or +changed according to whether their underlying responsibility changes: + +| Current implementation | Proposal | Rationale | +| --- | --- | --- | +| `Config` | removed | The bundle itself is removed; this is not a rename. | +| `PauliWordTrait` | `Word` plus `Indexable` where used as a key | Word operations are generalized through `Word::Site`; hashing becomes a separate capability. | +| `n_qubits`, `get`, `set`, `weight` | `n_sites`, `get`, `set`, `weight` | Only the Pauli-specific extent name changes; the other operation names stay. | +| concrete `PauliWord` | `PauliWord` | The packed X/Z word is the same domain concept. | +| concrete `LossyPauliWord` | `LossyPauliWord` | The packed X/Z/loss representation remains concrete and flattened. | +| `PhasedPauliWord` | `PhasedPauliWord` alias over non-indexable `Phased` | The wrapper is generic over ordinary and lossy words but is not a production map key. | +| `rehash` | private cache invalidation | Recalculation changes from eager mutation-time work to lazy demand-time work without exposing cache mechanics through `Indexable`. | +| `Strategy` | `Policy` | Intentional terminology change requested for this redesign. | +| `ACMap` | `ACMap` | The associative coefficient map has the same role. | +| `PauliSum::data`, `map_insert`, `map_add` | same method names on `SumStorage` | These semantic operations already match the proposed boundary. | +| `PauliSum` map pair and `scratch` fields | `SumStorage` | A new abstraction is extracted from currently unnamed storage state. | +| `PauliSum` | `PauliSum` over generalized `OperatorSum` machinery | Pauli-facing code keeps its established name; `OperatorSum` names the new cross-algebra engine. | +| `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | +| `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | +| `BuildHasher` | associated with `Indexable` | Hasher ownership moves from `Config` to each indexable key type. | +| `HashFinalize` | removed from the shared contract | Concrete keys may finalize or compose hashes privately. | +| `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | + +Names such as `Word`, `Indexable`, `SumStorage`, and `OperatorSum` are +therefore new because they denote abstractions that do not +exist in the current implementation, not because the existing API is being +renamed wholesale. + +### Trait admission rule + +A proposed trait belongs in the design only when generic code consumes it and +there are multiple meaningful implementation families, or when it is an +established behavioral boundary implemented by different backends. A trait is +not justified merely to name inherent methods on one generic struct. + +This rule keeps: + +- `Indexable`, consumed by keyed stores and implemented by hash-enabled words + and tableaus; +- `Word`, consumed by the shared sparse-sum engine and propagation algorithms; +- gate and noise traits, implemented across propagation and tableau backends; +- `SumStorage` and `ACMap`, implemented by genuinely different storage engines + and collections; +- `EntryStore`, already implemented by `VecStorage` and `MapStorage`; and +- `Policy`, implemented by independent capacity and truncation behaviors. + +It rejects the removed global `Config`, `PauliSumAlgorithm`, +`TableauMixtureAlgorithm`, and `TableauStorage` traits, as well as word +subtraits named `PauliWord`, `FermionWord`, or `LossyPauliWord`. Their +distinctions are expressed by `Word::Site` or by concrete types instead +of one-alphabet subtraits; the concrete `PauliWord` and `LossyPauliWord` type +names remain available. + +### Sparse-sum branch staging + +A propagation rule can turn one term into multiple terms. For example, a +Pauli rotation may produce: + +```text +c P -> c cos(theta) P + c sin(theta) P' +``` + +The existing entry can be updated while the active map is traversed, but a new +key cannot generally be inserted into that same map during mutable iteration. +New terms must therefore be staged and merged after the traversal. New keys +may collide with each other or with existing keys, so the merge accumulates +their coefficients. + +Only one staging mechanism is required for correctness. The current engine +uses two because they serve different performance paths: + +- an auxiliary map supports whole-map rewrites, combines output collisions as + they are produced, and retains its allocation across operations; and +- a reusable `Vec<(W, C)>` scratch buffer stages additional terms when existing + entries remain in place, avoiding an auxiliary-map insertion followed by a + second map probe during the merge. + +Conceptually: + +```text +whole-map rewrite: active map -> auxiliary map -> swap +in-place branching: active map + scratch buffer -> merge into active map +``` + +Neither physical mechanism is part of the public storage contract. A default +`SumStorage` implementation may privately contain both maps and the reusable +vector, matching the current `PauliSum` layout. A simpler backend may implement +both `map_insert` and `map_add` using only an auxiliary map; another backend +may use per-thread buffers. Whether retaining both mechanisms is worthwhile is +a benchmark decision, not a type-system requirement. + +## Indexable values + +Hash-enabled `Word` values and `Tableau` can both be expensive, mostly-stable +map keys. Their hashing contract should be expressed independently of any map. + +The common capability is intentionally minimal: + +```rust +pub trait Indexable: Clone + Eq + Hash { + type BuildHasher: std::hash::BuildHasher + Clone + Default; +} +``` + +The important points are: + +- `BuildHasher` retains the current associated-type name and moves from + `Config` to the key type; +- the build hasher is associated with the key type, not with a configuration + bundle; +- cache layout and invalidation are private representation invariants of the + concrete type; and +- equality and hashing cover only structural key identity, never cache fields + or incidental runtime state. + +No generic consumer needs to name a cache type or request invalidation. +Structural mutation already occurs through `&mut self`, so each mutator can +clear the affected private cache as part of maintaining its concrete +invariants. + +### Lazy hashing and interior mutability + +Rust's `Hash::hash` receives `&self`, so shipped indexable words and tableaus +use private component `OnceLock` caches. `Hash::hash` may populate a cache +through shared access, while structural mutators clear affected cells through +their exclusive `&mut self` access. This preserves `Send + Sync` for the +shipped representations. `Indexable` itself does not require either +concurrency bound. + +### Key mutation invariant + +An indexable value must not be structurally mutated while it is stored as a map +key. Cache invalidation makes a value correct for its next insertion or lookup; +it cannot make in-place mutation of an existing map key valid. + +Structural fields should therefore be private in the new representations. +Mutation must go through operations that invalidate the affected cache, or +through mutation guards that invalidate on completion. + +## Concrete word hashing + +Every concrete `Word` owns its `BuildHasher`, private cache representation, +structural hash algorithm, and invalidation logic. Pauli words hash +their X/Z content, lossy words compose Pauli and loss components, and future +fermion words hash their ordered factors. Factor order is part of fermionic +identity. + +The trait layer does not expose packed storage to support hashing. Concrete +implementations hash their private fields and may apply hasher-specific +finalization internally. Detailed layouts and component invalidation rules are +in [`word-data-structures.md`](word-data-structures.md). + +## Tableau indexability + +A tableau may itself be used as a key by a classical-mixture algorithm, so the +concrete tableau implements `Indexable` directly and owns a tableau-specific +hasher and cache representation. This does not imply that a tableau is a +`Word`; they only share the `Indexable` key capability. + +The tableau's structural hash is composed from its logical X/Z matrix, phase +plane, and per-qubit loss plane. It excludes RNG, padding, cache state, and +physical matrix orientation. Separate component caches allow phase-only +changes to avoid rehashing the X/Z matrix. Physical transposition is a layout +change, not a logical mutation, and does not invalidate the structural hash. + +The concrete memory layout, component invalidation table, and canonical hash +order live in [`tableau-data-structure.md`](tableau-data-structure.md) so they +do not leak into the shared trait-system design. + +## Expected generic composition + +The intended composition is explicit: + +```rust +OperatorSum +Tableau +GeneralizedTableauSum +``` + +Domain-specific aliases or wrappers can preserve `PauliSum` and introduce +`FermionSum` without rebuilding a monolithic configuration trait. + +## Non-goals for the first prototype + +- Migrating the existing crates to `ppvm-traits-2` immediately. +- Merging `GeneralizedTableauSum` and `OperatorSum` in this iteration; only a + smaller proven common factor should be considered in the next iteration. +- Defining one collection interface shared by all algorithms. +- Requiring every sparse-sum storage backend to physically contain both an + auxiliary map and a scratch buffer. +- Exposing cache representation or invalidation through `Indexable`. +- Preserving `Copy` at the expense of correct lazy caching. +- Adding runtime dispatch for storage, hashing, or algorithm policies. + +## Open design questions + +1. Do benchmarks justify retaining both the auxiliary-map and vector-staging + fast paths in the default sparse-sum storage backend? diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md new file mode 100644 index 00000000..30d87f22 --- /dev/null +++ b/docs/design/word-data-structures.md @@ -0,0 +1,304 @@ +# Word data structures + +Status: design sketch + +## Purpose + +This document describes concrete data structures for standalone algebraic +words. The shared trait design is specified in +[`traits-2-configuration-and-hashing.md`](traits-2-configuration-and-hashing.md). + +The trait-level `Word` is representation-free but defines the common indexed +product operations: + +```rust +pub trait Word { + type Site; + + fn n_sites(&self) -> usize; + fn get(&self, index: usize) -> Self::Site; + fn set(&mut self, index: usize, site: Self::Site); + fn weight(&self) -> usize; +} +``` + +There is no `Word::Storage` associated type. Packed arrays, loss masks, ordered +factors, hash fields, and validity flags are private fields of concrete types. +Generic propagation and collection code uses behavioral traits and never names +the backing memory. + +`Word` does not extend `Indexable`. A word used as an `ACMap` key implements +both traits, while a mutable intermediate such as `Phased` can implement +`Word` without pretending to be a valid map key. This separation replaces the +current `REHASH = false` use case with an explicitly non-indexable type. + +This document initially focuses on: + +- packed Pauli words; +- lossy Pauli words; +- phased-word composition; and +- lazy structural hash caches. + +Fermion-word storage will receive a separate design when fermionic propagation +is implemented. It will use the same `Word` interface with a different `Site`: +the word index records product order, while the fermionic site value records +the physical mode and creation or annihilation action. + +`weight()` counts non-identity factors according to the selected site type. +For a representation that stores only non-identity factors, it may equal +`n_sites()`. `set()` is the shared structural mutation boundary and must +preserve concrete invariants and invalidate the affected hash components. + +## Logical Pauli model + +An ordinary Pauli word is a fixed-width tensor product over `I`, `X`, `Y`, and +`Z`. Each site is represented by two logical bits: + +| Pauli | X bit | Z bit | +| --- | --- | --- | +| `I` | 0 | 0 | +| `X` | 1 | 0 | +| `Z` | 0 | 1 | +| `Y` | 1 | 1 | + +A lossy Pauli word adds a `Lost` state. Loss is exclusive with the four Pauli +states; a lost site has canonical bits `(x, z, lost) = (0, 0, 1)`. + +The exact Pauli alphabet and its lossy extension are site types rather than +word subtraits: + +```rust +pub enum Pauli { + I, + X, + Y, + Z, +} + +pub enum LossySite { + Present(S), + Lost, +} +``` + +An ordinary word implements `Word`. `LossyPauliWord` implements +`Word>` and returns `Lost` for marked sites. This keeps +the word interface independent of the chosen operator alphabet. + +## `PauliWord` packed representation + +The initial packed implementation stores parallel fixed-size X and Z arrays: + +```rust +pub struct PauliWord { + xbits: BitArray, + zbits: BitArray, + nqubits: usize, + hash_cache: OnceLock, + _hasher: PhantomData H>, +} +``` + +`A` is an implementation parameter such as `[u8; N]` or `[usize; N]`, and `H` +is the build hasher associated through `Indexable`. Neither is exposed through +`Word`. The implementation validates that `nqubits` fits the arrays and +ignores or canonicalizes unused high bits. + +The structural identity is: + +```text +(nqubits, logical X bits, logical Z bits) +``` + +Equality and hashing exclude unused capacity, cache contents, cache validity, +and the `PhantomData` marker. + +All fields are private. Mutations go through methods that preserve unused-bit +invariants and invalidate the hash when a logical Pauli site changes. + +## Lossy Pauli word + +The first prototype keeps the established lossy Pauli word as a flattened +packed representation: + +```rust +pub struct LossyPauliWord { + xbits: BitArray, + zbits: BitArray, + lbits: BitArray, + nqubits: usize, + xz_hash_cache: OnceLock, + loss_hash_cache: OnceLock, + _hasher: PhantomData H>, +} +``` + +Inlining all three planes avoids wrapper nesting and keeps the lossy hot path +and component hashes direct. A generic loss wrapper is not introduced until a +second real word representation demonstrates that it needs the same +composition. `A` and `H` remain private implementation parameters from the +perspective of `Word`. + +### Canonical loss invariant + +A lost site must contain identity in its X/Z planes: + +```text +lost[q] = 1 => xbits[q] = 0 and zbits[q] = 0 +``` + +`set_lost(q)` first clears the X/Z bits, then sets the loss bit. +`set(q, LossySite::Present(p))` clears the loss bit and then writes `p`. This +prevents multiple physical encodings from representing the same logical lossy +word. + +The structural identity is: + +```text +(nqubits, logical X bits, logical Z bits, logical loss bits) +``` + +`weight()` counts `X`, `Y`, `Z`, and `Lost`; `loss_weight()` counts only lost +sites. + +### Loss-specific behavior + +Generic lossy Pauli propagation sees `LossySite::Lost` through `Word` and +preserves or skips the site according to the operation's semantics. Operations +that create, clear, or count loss use inherent `LossyPauliWord` methods: + +```rust +impl LossyPauliWord { + pub fn is_lost(&self, qubit: usize) -> bool; + pub fn set_lost(&mut self, qubit: usize); + pub fn clear_loss(&mut self, qubit: usize); + pub fn loss_weight(&self) -> usize; +} +``` + +Loss channels and maximum-loss-weight truncation specialize directly on the +concrete lossy word: + +```rust +impl LossChannel + for OperatorSum, S, P> +{ + // ... +} +``` + +There are no traits named `PauliWord`, `LossyPauliWord`, or `FermionWord`. +`PauliWord` and `LossyPauliWord` remain concrete domain type names. Algorithms +select the algebra through `Word::Site`; loss-only operations remain inherent +to `LossyPauliWord`. + +## Phased words + +Phase is another orthogonal wrapper: + +```rust +pub struct Phased +where + W: Word, +{ + word: W, + phase: Phase, +} +``` + +`Phased` is the generalized wrapper implementation. Pauli-facing code retains +the established name through an alias: + +```rust +pub type PhasedPauliWord = Phased; +``` + +For Pauli use, `Phase` represents `+1`, `+i`, `-1`, and `-i`. The wrapper may +wrap both ordinary and lossy Pauli words. It may also be useful for other word +algebras, but algebra-specific multiplication is implemented only under the +appropriate specialized word bound. + +Loss and phase compose without a new combined representation: + +```rust +PhasedPauliWord> +``` + +No phased word is a production map key in the first prototype, so `Phased` +does not implement `Hash` or `Indexable` and stores no hash mode or cache. + +## Hash ownership + +Every hash-enabled word implements `Indexable`, associates its build hasher, +and privately owns the fields and algorithm used to cache its structural +hash. Cache representation and invalidation are not exposed through +`Indexable`. + +The shipped indexable words use component `OnceLock` caches. `Hash::hash` +can populate them through `&self`; structural mutators clear affected cells +through `&mut self`. This preserves `Send + Sync` without imposing either +bound on the `Indexable` trait. + +## Component hashes + +Hash composition follows the logical wrappers: + +```text +packed Pauli hash = hash(nqubits, X bits, Z bits) +lossy hash = combine(Pauli hash, loss hash) +``` + +`combine` must be ordered and domain-separated. It must not be an +unqualified XOR of arbitrary component digests. + +Loss masks may be large, so `LossyPauliWord` caches the loss component +separately from the X/Z component. A loss-only mutation then avoids rehashing +X/Z. `Phased` is absent from this composition because it is not indexable. + +## Invalidation rules + +| Mutation | X/Z component | Loss component | +| --- | --- | --- | +| Change ordinary Pauli site | invalidate | preserve | +| Mark identity site lost | preserve | invalidate | +| Mark nonidentity site lost | invalidate | invalidate | +| Clear loss to identity | preserve | invalidate | +| Replace loss with Pauli | invalidate if nonidentity | invalidate | + +Constructors leave caches empty. Cloning may copy a valid cached value because +the clone initially has identical structural contents. + +## Ordering and serialization + +`Eq`, `Ord`, `Hash`, display, parsing, and serialization must agree on logical +identity: + +- Pauli sites compare in a documented order. +- Loss participates after the underlying Pauli content or through an explicit + `LossySite` ordering. +- Phase participates in equality and serialization for `Phased`, but not in + map-key hashing because the wrapper is not indexable. +- Unused bits and cache state never participate. + +Serialization uses logical symbols and lengths, not raw native-word memory, so +it remains stable across storage widths and platforms. + +## Prototype validation + +The prototype should include: + +- round-trip parsing tests for ordinary and lossy symbols; +- property tests comparing packed operations with a simple symbol vector; +- tests enforcing `lost => X/Z identity` after every mutator; +- equality/hash agreement tests for ordinary and lossy indexable words; +- tests showing loss-only changes preserve the X/Z hash component; +- equality and serialization tests for ordinary and lossy `Phased` values; +- tests proving unused high bits do not affect identity; +- `Send`/`Sync` assertions for shipped indexable words; and +- benchmarks comparing uncached structural hashing with the private lazy + component caches. + +## Open questions + +1. Should the loss plane use the same packed array width as the X/Z planes, or + a separately selected private width?