FFT symmetrization tool + review fixes and doc accuracy pass

README.md

@@ -70,7 +70,8 @@ subtraction...` (`Ctrl+Shift+B`).
 **Performing an FFT.** `Measurements → FFT viewer...` (`Ctrl+Shift+F`)
 shows the spectrum with q-axes in nm⁻¹, intensity controls, and a radial
 profile; the tabs fit a reciprocal lattice, correct drift distortion,
-suppress mains pickup, and reconstruct a filtered image by inverse FFT.
+suppress mains pickup, reconstruct a filtered image by inverse FFT, and
+symmetrize the image by rotational averaging.
 
 **Finding features.** `Measurements → Feature finder...` detects maxima or
 minima with threshold, spacing, and smoothing controls, then exports the
@@ -100,9 +101,11 @@ suite. What it does today:
 - **FFT tools** (the FFT viewer) — inspect the magnitude and radial profile with
   q in nm⁻¹; overlay a draggable reciprocal-lattice grid and apply an affine
   lattice correction; show Bragg-shell rings for a known structure; predict and
-  notch out **mains pickup** (50/60 Hz); and use the **inverse-FFT /
+  notch out **mains pickup** (50/60 Hz); use the **inverse-FFT /
   Fourier-reconstruction** tool to select circle/ellipse features, *remove* or
-  *keep* them, preview the reconstructed image and the residual, then apply.
+  *keep* them, preview the reconstructed image and the residual, then apply;
+  and **symmetrize** an image by n-fold (optionally mirrored) rotational
+  averaging, with the removed residual always shown alongside.
 - **ROIs and measurements** — rectangle, ellipse, polygon, freehand, line, and
   point ROIs; ROI-scoped processing; line profiles, periodicity, ROI
   statistics, step heights, distances and angles, feature points, point-mask
@@ -112,10 +115,17 @@ suite. What it does today:
   molecule segmentation, counting, few-shot classification, template-match
   counting, lattice extraction, and reproducible step-edge exclusion. Requires
   the `features` extra (OpenCV + scikit-learn).
+- **Particle Statistics** *(optional)* — test detected point patterns against
+  spatial null models (random, hard-core, measured-feature) with simulation
+  envelopes, plus a guided tutorial and free-play simulations. Model runs
+  require the `adstat` extra; treat verdicts as exploratory (see the
+  [guide](docs/adstat_user_guide.md)).
 - **Spectroscopy** — inspect single traces or overlays / waterfalls. Smoothing,
   derivative, normalization, outlier masking, and offsets operate on derived
   display data; the raw loaded arrays are left intact.
-- **Convert** Createc `.dat` scans to Nanonis-compatible `.sxm` (and PNG).
+- **Convert** Createc `.dat` scans to Nanonis-compatible `.sxm` (and PNG), or
+  export them as raw / physical NumPy `.npy` bundles with header and
+  provenance sidecars (`dat2npy`).
 - **Export with context** — PNG, PDF, CSV, JSON, SXM, and optional Gwyddion
   `.gwy`. Many exports also write a JSON provenance sidecar (source file and
   channel, display settings, processing state, ROIs, and warnings). It is not a
@@ -133,6 +143,7 @@ suite. What it does today:
 | Input | Nanonis `.dat` | Point spectroscopy |
 | Input | RHK `.sm4` | STM/SPM image scan |
 | Output | `.sxm` | Converted or processed scan data |
+| Output | `.npy` | Raw / physical NumPy array bundles (with header + provenance sidecars) |
 | Output | `.png`, `.pdf` | Figure / image export |
 | Output | `.csv`, `.json` | Numerical data, metadata, or provenance |
 | Output | `.gwy` | Optional Gwyddion export when `gwyfile` is installed |
@@ -143,18 +154,23 @@ are in [docs/createc_dat_reader.md](docs/createc_dat_reader.md).
 ## Installation notes
 
 Python 3.11 or newer is required. The core install pulls in numpy, scipy,
-Pillow, PySide6, matplotlib, and shapely.
+Pillow, PySide6, matplotlib, shapely, and scikit-image.
 
 Optional extras:
 
 ```bash
 python -m pip install -e ".[features]"   # particle/feature + lattice tools (OpenCV, scikit-learn)
+python -m pip install -e ".[adstat]"     # Particle Statistics engine (adstat)
+python -m pip install -e ".[clip]"       # CLIP encoder for few-shot classification (torch)
 python -m pip install -e ".[gwyddion]"   # Gwyddion .gwy writer (gwyfile)
-python -m pip install -e ".[dev]"        # test + lint tooling
+python -m pip install -e ".[dev]"        # test tooling (pytest, vulture)
+python -m pip install -e ".[all]"        # features + adstat + gwyddion + pytest (no CLIP)
 ```
 
 The Feature Counting and lattice-extraction tools are inactive until the
-`features` extra is installed; everything else works with the core install.
+`features` extra is installed, Particle Statistics model comparisons need the
+`adstat` extra, and the CLIP feature encoder needs the `clip` extra;
+everything else works with the core install.
 
 ## Using ProbeFlow from Python
 
@@ -195,7 +211,7 @@ dzdv = numeric_derivative(spec.x_array, z_smooth)
 ```bash
 python -m pip install -e ".[dev,features]"
 pytest                          # run the test suite
-ruff check probeflow tests      # lint
+ruff check probeflow tests      # lint (install ruff separately or via pre-commit)
 ```
 
 Repository layout:

docs/createc_dat_reader.md

@@ -82,7 +82,11 @@ ProbeFlow interprets it as the one-based next Y position, so completed rows are
 `ImageYPosMax - 1`. If that value is inside the declared image height, the scan
 is marked partial and decoded arrays are trimmed to the completed rows.
 
-If `ImageYPosMax` is absent or outside the declared height, ProbeFlow falls back
-to the existing channel-0 row heuristic. `is_partial_scan`, `image_y_pos_max`,
+If `ImageYPosMax` indicates that every declared row completed
+(`ImageYPosMax - 1 >= Num.Y`), the stack is returned untrimmed and the
+channel-0 row heuristic is skipped — the heuristic could otherwise silently
+drop genuine trailing rows whose DAC values happen to be zero. Only when
+`ImageYPosMax` is absent (or nonsensical, below 1) does ProbeFlow fall back to
+the channel-0 row heuristic. `is_partial_scan`, `image_y_pos_max`,
 `original_Ny`, and `trimmed_Ny` are recorded in the decode report so callers can
 distinguish a completed smaller image from an interrupted acquisition.

docs/gui.md

@@ -84,6 +84,12 @@ axes. The tabs below cover the common reciprocal-space tasks:
 * **Mains** — detect and suppress mains-frequency pickup streaks.
 * **Inverse FFT** — mask regions of the spectrum and reconstruct the
   filtered image.
+* **Symmetrize** — enforce an n-fold (optionally mirrored) symmetry by
+  averaging the image with its rotated copies. Rotated copies are
+  registered back onto the original automatically, so the symmetry axis
+  need not sit at the image centre. Always check the **Residual**
+  preview: it holds everything symmetrization removed — noise, but also
+  real defects and domain boundaries.
 
 **Focus FFT** and the zoom buttons home in on the spectral content near
 the origin, and the **Export** menu saves the spectrum or filtered image.
@@ -135,9 +141,9 @@ and the developer-facing [AdStat integration](adstat_integration.md) contract.
 
 ## Beyond the basics
 
-* **ROIs** — draw rectangles, ellipses, and lines from the ROI tab; ROIs
-  restrict background fits, FFTs, and statistics, and are saved as
-  sidecar files next to the scan.
+* **ROIs** — draw rectangles, ellipses, polygons, freehand outlines,
+  lines, and points from the ROI tab; ROIs restrict background fits,
+  FFTs, and statistics, and are saved as sidecar files next to the scan.
 * **Measurements** — distance and angle measurements, line profiles, ROI
   statistics, step heights, pair correlation.
 * **Spectroscopy** — `.VERT` and Nanonis spectroscopy files open in a

docs/review_status.md

@@ -1,6 +1,6 @@
 # ProbeFlow Review Status
 
-**Updated**: 2026-06-12
+**Updated**: 2026-07-03
 
 This is the single, consolidated record of ProbeFlow's code-review history. The
 detailed per-angle review files (`docs/reviews/2026-05-27-*.md`) and the earlier
@@ -201,6 +201,102 @@ the per-finding detail.
 
 Suite grew from ~2,260 to 2,449 tests across the campaign; main is green.
 
+## Scheduled deep review, 2026-07-03 (new-code pass: .dat→.npy converter, feature bank, CLIP classify, contrast fix)
+
+Targets: everything merged since the adversarial campaign that earlier passes
+had not covered — the Createc `.dat`→`.npy` converter (+ the #46 scan-range
+fix), the feature bank Phases 1–2, the CLIP classify encoder, and the
+viewer-black contrast fix (8a2950d). Particle Statistics (#45) was itself a
+review and was not re-reviewed.
+
+### Fixed in this pass (uncommitted, suite green: 2508 passed / 3 skipped)
+
+1. **Feature-bank silent data loss (the one real find)** —
+   `analysis/feature_bank.py`: a corrupt/unparseable bank file made
+   `load_bank` silently return an empty bank, so the next "Add to bank"
+   overwrote the user's entire cross-scan labelled-sample collection with just
+   the new entries. `append_entries` now loads strict (`ValueError`, surfaced
+   on the GUI status bar, original file untouched) and writes atomically
+   (tmp + `os.replace`), closing the truncated-write corruption source too.
+   Read path (classification) stays permissive. New `tests/test_feature_bank.py`
+   (the module previously had **zero** tests).
+2. **Createc complete-scan trim hazard** — `io/readers/createc_dat.py`
+   `_trim_createc_stack`: when `ImageYPosMax` confirmed the scan completed
+   (`= Num.Y + 1`), the code still fell through to the legacy channel-0
+   nonzero heuristic, which silently drops genuine trailing rows whose DAC
+   values are zero (and drops the last row if the bottom-right pixel is 0/NaN).
+   Header-confirmed-complete now returns untrimmed. Pinned in
+   `tests/test_createc_dat_decode.py`.
+3. **Converter `main()` partial-kwarg crash** —
+   `io/converters/createc_dat_to_npy.py`: programmatic calls that passed only
+   some keywords (e.g. `basis=` without `src=`) skipped the CLI arg-parse
+   branch and crashed on `Path(None)`. Defaults now resolved unconditionally.
+4. **Mojibake log line** — `gui/workers.py` NumPy-conversion log printed
+   `──` (double-encoded box-drawing chars) in the convert dialog.
+
+### Verified sound (no change needed)
+
+- `decoded_scan_range_m` (#46): per-pixel step from the programmed frame ×
+  decoded pixel count — physically correct for first-column removal and
+  partial-scan trims; metadata path consistent (`shape` (Ny, Nx) + decoded range).
+- CLIP classify + bank integration (`analysis/features.py`): bank-dimension
+  guard, zero-norm cosine guard, bank-only degenerate sample stack, sharpness
+  z-norm padding for bank entries, encoder gating in the GUI controller.
+- Contrast revalidation (8a2950d): sound; known tradeoff — a deliberate manual
+  window capturing <2% of finite data is reset by the next processing op.
+- Previously-flagged gaps now confirmed fixed: shear ROI/mask invalidation
+  (`_on_shear_applied`), quick-selection menu sync after transform-drop.
+
+### Larger items — outlined fixes (not done; ordered by value)
+
+1. **Feature-bank scale blindness** (tracked TODO in `feature_bank.py`): crops
+   are fixed 48 px, so a banked embedding's physical field of view depends on
+   the source scan resolution and cross-scan matches are scale-blind.
+   *Plan:* bump `BANK_SCHEMA_VERSION` → 2; `make_entry` gains
+   `pixel_size_m` (or crop physical size in nm) threaded from the features
+   panel's scan; `bank_to_samples` returns it; `classify_particles` warns when
+   bank-entry crop size differs from the current crop's physical size by more
+   than ~2× (later: resample crops to a canonical nm/px before embedding).
+   Old v1 entries: treat as unknown scale, warn once.
+2. **Background-dominated CLIP embedding** (carried from UniMR review):
+   similarities crush to ~0.998–1.0 because the 48 px crop is mostly flat
+   background, so outlier rejection is unreliable in general. *Plan:* mask or
+   weight the crop by the particle contour before embedding (contours already
+   exist on `Particle`); re-verify `_threshold_similarities` guards afterwards
+   — the spread guard may become unnecessary.
+3. **Bundle-provenance pixel-key convention**: in the `.npy` bundle sidecars,
+   `original_shape`/`decoded_shape` are `(Nx, Ny)` while each plane's `shape`
+   is `(rows, cols)` — same word, opposite conventions in one JSON (e.g.
+   A250407: `decoded_shape: [255, 9]` vs plane `shape: [9, 255]`). *Plan:*
+   rename the bundle keys to `original_pixels`/`decoded_pixels` (matching
+   `scan_pixels`) while the format is young; regenerate the committed sample
+   bundles under `test_data/output_raw_npy/`; grep for external consumers first.
+4. **Bank rotation-augmentation asymmetry**: in-image samples get 36 rotated
+   copies under `rotate_augment`; bank vectors are single unrotated embeddings,
+   so banked classes are disadvantaged for rotated instances. *Plan:* either
+   bank all 36 rotated embeddings per sample (36× file size) or store the raw
+   crop in the entry and re-embed with rotations at classify time (slower,
+   smaller file). Decide with real usage data.
+
+### Next-run candidates (unexplored, carried forward)
+
+- `processed_export.py` / provenance replay **without** canonical sidecars, and
+  a CLI replay smoke test of new-format sidecars.
+- `image_canvas.py` selection internals (~1500 lines, only ever grepped).
+- Spectroscopy display wiring (`gui/spec_viewer/`), `io/readers/createc_vert.py`
+  + `nanonis_spec.py` decoders, `analysis/spec_plot.py`.
+- Partial `.sxm`/`.dat` thumbnail decode in browse (open since the browse
+  loading review).
+- Feature-bank GUI flows under Qt (add-to-bank dialog, `refresh_bank_status`)
+  have no GUI-level tests.
+
+Environment note for future runs: the full suite needs
+`QT_QPA_PLATFORM_PLUGIN_PATH=$(python -c "import PySide6,pathlib; print(pathlib.Path(PySide6.__file__).parent/'Qt'/'plugins'/'platforms')")`
+alongside `QT_QPA_PLATFORM=offscreen` in sandboxed shells, or `qapp` creation
+aborts (SIGABRT) at `test_feature_finder.py`; `test_fft_viewer_utils.py` and
+`test_lattice_grid.py` still segfault under offscreen Qt on anaconda py3.13
+(environmental, pre-existing).
+
 ## Deferred (not in code-review scope)
 
 - A true Python 3.11/3.12 test matrix (only the local interpreter was available).

probeflow/analysis/feature_bank.py

@@ -23,6 +23,7 @@
 from __future__ import annotations
 
 import json
+import os
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Sequence
@@ -41,16 +42,33 @@ def _empty_bank() -> dict:
     return {"schema_version": BANK_SCHEMA_VERSION, "encoder": "clip", "entries": []}
 
 
-def load_bank(path) -> dict:
-    """Load a bank file, or an empty bank when it is missing / unreadable."""
+def load_bank(path, *, strict: bool = False) -> dict:
+    """Load a bank file, or an empty bank when it is missing / unreadable.
+
+    ``strict=True`` raises :class:`ValueError` instead of returning an empty
+    bank when the file exists but cannot be parsed as a bank. Read-only
+    consumers (classification) want the permissive default; writers must use
+    strict so a corrupt file is never silently replaced, destroying every
+    previously banked sample.
+    """
     p = Path(path)
     if not p.exists():
         return _empty_bank()
     try:
         data = json.loads(p.read_text(encoding="utf-8"))
-    except (json.JSONDecodeError, OSError, ValueError):
+    except (json.JSONDecodeError, OSError, ValueError) as exc:
+        if strict:
+            raise ValueError(
+                f"feature bank {p} exists but is not readable as JSON ({exc}); "
+                "refusing to overwrite it — repair or remove the file first"
+            ) from exc
         return _empty_bank()
     if not isinstance(data, dict):
+        if strict:
+            raise ValueError(
+                f"feature bank {p} does not contain a JSON object; "
+                "refusing to overwrite it — repair or remove the file first"
+            )
         return _empty_bank()
     data.setdefault("schema_version", BANK_SCHEMA_VERSION)
     data.setdefault("encoder", "clip")
@@ -106,7 +124,9 @@ def append_entries(path, new_entries: Sequence[dict]) -> dict:
     # Materialise up front: a generator would be consumed by the loop below,
     # making the ``skipped`` count in the summary wrong (negative).
     new_entries = list(new_entries)
-    bank = load_bank(p)
+    # strict: a corrupt existing bank must abort the append, not be silently
+    # replaced by an empty bank plus the new entries.
+    bank = load_bank(p, strict=True)
     seen = {
         (e.get("source_path"), e.get("particle_index"))
         for e in bank["entries"]
@@ -120,7 +140,11 @@ def append_entries(path, new_entries: Sequence[dict]) -> dict:
         seen.add(key)
         added += 1
     p.parent.mkdir(parents=True, exist_ok=True)
-    p.write_text(json.dumps(bank, indent=2), encoding="utf-8")
+    # Atomic replace: an interrupted write must never leave a truncated bank
+    # in place of the previous good one.
+    tmp = p.with_name(p.name + ".tmp")
+    tmp.write_text(json.dumps(bank, indent=2), encoding="utf-8")
+    os.replace(tmp, p)
     return {
         "added": added,
         "skipped": len(new_entries) - added,

probeflow/core/processing_state.py

@@ -36,6 +36,7 @@
     "periodic_notch_filter",
     "mains_pickup_suppression",
     "inverse_fft_filter",
+    "symmetrize_fft",
     "linear_undistort",
     "affine_lattice_correction",
     "arithmetic",

probeflow/gui/dialogs/fft_viewer.py

@@ -22,6 +22,7 @@
 from probeflow.gui.dialogs.fft_viewer_lattice_mixin import FFTViewerLatticeMixin
 from probeflow.gui.dialogs.fft_viewer_mains_mixin import FFTViewerMainsMixin
 from probeflow.gui.dialogs.fft_viewer_reconstruct_mixin import FFTViewerReconstructMixin
+from probeflow.gui.dialogs.fft_viewer_symmetrize_mixin import FFTViewerSymmetrizeMixin
 from probeflow.gui.no_wheel import install_no_wheel_spinboxes
 from probeflow.gui.viewer.display_range import DisplayRangeController
 from probeflow.gui.viewer.display_sliders import DisplaySliderController
@@ -83,6 +84,7 @@ class FFTViewerDialog(
     FFTViewerLatticeMixin,
     FFTViewerMainsMixin,
     FFTViewerReconstructMixin,
+    FFTViewerSymmetrizeMixin,
     QDialog,
 ):
     """Side-by-side real-space / FFT inspection window."""
@@ -171,6 +173,8 @@ def __init__(
         self._fft_selection_overlay = None
         self._reconstruct_tab_index = -1
         self._reconstruct_preview_active = False
+        self._symmetrize_tab_index = -1
+        self._symmetrize_preview_active = False
         self._new_image_fn = new_image_fn
 
         self._fft_drs = DisplayRangeController(clip_low=0.0, clip_high=100.0, parent=self)
@@ -806,6 +810,8 @@ def _build_fft_column(self) -> QVBoxLayout:
             self._build_mains_tab(), "⚡ Mains")
         self._reconstruct_tab_index = self._tab_widget.addTab(
             self._build_reconstruct_tab(), "Inverse FFT")
+        self._symmetrize_tab_index = self._tab_widget.addTab(
+            self._build_symmetrize_tab(), "Symmetrize")
 
         self._fft_splitter = QSplitter(Qt.Vertical)
         self._fft_splitter.addWidget(fft_top)