Spike: .NET sidecar deserialiser for EPU .dm/.xml (Option C) using NetDataContractSerializer

[vredchenko]

Summary

Spike a .NET sidecar deserialiser for EPU .dm/.xml files that loads EPU's own
assemblies and uses NetDataContractSerializer to rehydrate the original object graph,
emitting clean JSON to the Python agent - instead of hand-rolling lxml XPath against the
serialiser's internal structures.

This is Option C from the investigation of the EPU serialisation format. The goal of this
issue is a proof-of-concept plus a go/no-go decision, not a committed rewrite.

Background

EPU (ThermoFisher, closed-source, Windows .NET Framework 4.x) does not write a bespoke
interchange schema. The .dm files are NetDataContractSerializer dumps of EPU's in-memory
CLR object graph, and the per-image .xml sidecars are DataContractSerializer output. Full
write-up: smartem-devtools/docs/decision-records/epu-data-structures.md (section "File
Serialisation Format").

Because it is a runtime object-graph dump, the payload is entangled with serialiser plumbing:

• Data-bearing collections are serialised only in their ...Efficient (ConcurrentDictionary)
  form; the plain domain lists are i:nil. Verified: a 1.8 MB GridSquare_*.dm with 534 foil
  holes has a nil TargetLocations list and all 534 entries under
  TargetLocationsEfficient/m_serializationArray. Atlas tiles behave the same via
  TilesEfficient/_items.
• Per-entry element names carry an unstable generated type-hash suffix
  (KeyValuePairOfintTargetLocationXml...).

The current agent parser (src/smartem_agent/fs_parser.py, e.g. the foil-hole read at
TargetLocationsEfficient/m_serializationArray, and the atlas-tile read at TilesEfficient/_items)
handles this correctly today with namespace-aware XPath plus local-name() prefix matching, so
this is not a bug fix. The question this spike answers is whether delegating deserialisation
to the vendor's own code is a worthwhile robustness/maintainability improvement.

Proposal (Option C)

A small .NET Framework 4.x console helper, run on the EPU workstation next to the agent:

1. Loads EPU's own assemblies present on the workstation
   (Applications.Epu.Persistence.dll, Applications.SciencesAppsShared.*.dll, Fei.*.dll).
2. Calls NetDataContractSerializer.ReadObject() on a .dm file to rehydrate the exact
   EpuSessionXml / GridSquareXml / AtlasSessionXml / FoilHoleCollectionXml objects.
3. Walks the strongly-typed graph and serialises the domain fields we care about to a stable,
   flat JSON contract on stdout.
4. The Python agent invokes it per file (subprocess) and maps the JSON to the existing
   smartem_common.schemas pydantic models - the fs_watcher/orphan/queue machinery is unchanged.

The vendor's code owns refs, cycles, polymorphism and the Efficient dictionaries, so our side
never touches serialiser internals.

Explicit scope - what this does and does NOT solve

Removes:

• Coupling to serialiser plumbing (z:* object-graph attributes, m_serialization* bookkeeping,
  the KeyValuePairOfint... type-hash suffix, the TargetLocationsEfficient/TilesEfficient
  shapes).

Does not remove (out of scope - do not conflate):

• Filesystem sourcing. It still reads the .dm files; it does not consume from an upstream API.
• Liveness via fs watch. We still need watchdog events to know when to (re)read a file.
• Domain-schema coupling. If Thermo renames GridSquareXml.State, we still break - same as now.

If the objective is to escape the filesystem entirely, that is a different track (Smart EPU Open
API / Athena-DMP) and should be a separate issue - see Alternatives.

Dev-ready plan

Phase 0 - feasibility spike (timeboxed, ~1-2 days)

• Locate the EPU install on a DLS EPU workstation; confirm the persistence assemblies are
  present and loadable, and record their versions (dataset under test was built by EPU
  3.9.1.8206; assemblies stamp their own versions).
• Minimal C# program: load Applications.Epu.Persistence, NetDataContractSerializer.ReadObject()
  one real GridSquare_*.dm (use the large 534-foil-hole file), dump the foil-hole count and a
  couple of fields. Prove the graph rehydrates without our own assemblies.
• Decide runtime: native .NET Framework 4.x console (invoked as subprocess) vs pythonnet
  in-process. Default assumption: subprocess emitting JSON (clean process boundary, no
  interop pinning).

Phase 1 - JSON contract + parity

• Define the stdout JSON schema (mirror smartem_common.schemas: AcquisitionData,
  AtlasData+tiles+gridsquare_positions, GridSquareMetadata+foilhole_positions,
  FoilHoleData, MicrographManifest, MicroscopeData).
• Implement the graph walk for all root types: EpuSessionXml, AtlasSessionXml,
  SampleXml, ScreeningSessionXml, GridSquareXml (+ nested FoilHoleCollectionXml), and
  the MicroscopeImage sidecars.
• Parity harness: run both paths (current fs_parser and the .NET helper) over the
  bi37708-28 dataset and diff the resulting pydantic models. Zero diffs = pass.

Phase 2 - integration + decision gate

• Wire the helper behind a config flag in the agent intake path (fallback to fs_parser).
• Package the helper for the Windows agent build (.exe); confirm it runs where the agent runs.
• Write up findings and a recommendation on the ADR track; go/no-go on adopting it.

Acceptance criteria (for the spike)

• A working .NET helper rehydrates a real GridSquare_*.dm via NetDataContractSerializer using
  EPU's own assemblies, with no reimplementation of their types on our side.
• Byte-for-byte (or field-for-field) parity with the current parser across the bi37708-28
  dataset, demonstrated by the parity harness.
• A written recommendation covering the risks below, ending in go/no-go.

Risks / open questions

• NetDataContractSerializer is not in .NET Core/.NET 5+ (Framework-only; a Compat.Runtime.Serialization
  shim exists for modern .NET). The helper likely must target .NET Framework 4.x - acceptable since
  the agent already runs on Windows, but it adds a .NET runtime to the agent footprint.
• Security. NetDataContractSerializer instantiates arbitrary embedded .NET types (BinaryFormatter-class
  risk). Input here is EPU's own output on the same trusted box, but this must be stated explicitly.
• Licensing. Loading Thermo's proprietary assemblies at runtime - confirm this is acceptable for
  our deployment (we load DLLs already on the workstation; we do not redistribute them).
• Version coupling. The helper must load whatever assembly versions are on that workstation;
  behaviour across EPU upgrades needs a check (mitigated by loading the on-disk DLLs, not pinned copies).
• Interop cost. Adds a second process/runtime to a Python agent; weigh against the fact that the
  current parser already neutralises most plumbing brittleness.

Alternatives considered (context, not part of this issue)

• Option A - Smart EPU Open API (Thermo's supported plugin/automation API): the "proper" upstream,
  no filesystem, no XML. Requires Smart EPU enablement/licensing and a plugin host; strategic, separate track.
• Option B - Athena / DMP data platform: structured upstream, but the DMPSession block is i:nil
  across all sampled sessions, i.e. DLS runs EPU standalone. Needs infra/licensing, not agent code.
• Option D - read EPU process memory directly: rejected - closed-source heap layout, breaks on every
  EPU update.

References

• Format write-up: smartem-devtools/docs/decision-records/epu-data-structures.md
• Directory structure: same doc, "EPU Output Directory Structure"
• Current parser: smartem-decisions/src/smartem_agent/fs_parser.py; domain models:
  smartem-decisions/src/smartem_common/schemas.py
• Test dataset: testdata/datasets/bi37708-28-copy (EPU 3.9.1.8206, Titan Krios)