Unit conversion in pyAML

[gubaidulinvadim]

We've decided during a maintainer's meeting to open a discussion to advance on this topic and give necessary input/requirements for the development. One of the topics is whether we benefit from using pint for unit conversion in pyAML. There are some nice desirable features of pint

• Matplotlib integration
• Unit verification/checks for all calculations (this can help avoid typos and errors in the calculation scripts. But it is not clear if it it necessary for us)
• It can be very beneficial for the user to be able to do set(float) or set(pint.object). Then pint.object can be converted to SI units inside pyAML (and stripped of unit if we decide we don't need it internally for unit verification)
• Conversion between pyAML and control system can also be done with pint.

One option to integrate Pint could be to have it do conversions only in two places. At the user input and use everywhere SI units (either pint objects or floats/ints). At the moment of converting the SI unit of pyAML to the unit expected by the control system.

We've agreed that as an input for development direction we will provide.

Yoshi's experience and decisions with pint in PAMILA

In PAMILA, I adopted pint for unit handling. Here is a summary of what I did and why.

Two distinct types of "unit conversion" in PAMILA

PAMILA distinguishes two conceptually different types of unit conversion:

1. Universal unit conversion — "intra-dimensional" conversion where the physical dimension does not change (e.g., mA ↔ A, mm ↔ nm, GHz ↔ Hz). This is handled at the signal level using pint.
2. Representation (repr) conversion — "inter-dimensional" conversion that is device-dependent and typically changes the physical dimension (e.g., A ↔ mrad, A ↔ m⁻¹, digital counts ↔ A). This is handled at the device level using custom conversion functions, independently of pint.

How pint is used

pint is used to attach units to all values that users interact with. Specifically:

• All get() calls on a Middle Layer Variable (MLV) return a pint Quantity object (e.g., Q_("0.5 A"), Q_("-10 urad")).
• All put() calls on an MLV accept a pint Quantity object (or a list thereof for MIMO cases), which PAMILA internally converts to the low-level PV representation.
• Time-related device configuration parameters (e.g., wait times, tolerances) are also stored as pint Quantity objects.

Features of pint used in PAMILA

• UnitRegistry with caching — a single application-wide registry is created via set_application_registry() (needed for correct pickling/unpickling of Quantity objects).
• Matplotlib integration — ureg.setup_matplotlib() is called so that plots automatically display pint-aware axis labels.
• Serialization/deserialization — custom JSON serializers/deserializers were written for pint Quantity objects (e.g., for saving/loading configuration and data via tiled).
• Performance optimization — wrapper functions fast_create_Q() and fast_convert() were written to cache canonical unit name lookups, reducing repeated overhead from pint's unit string parsing.

A practical issue encountered

• One noteworthy issue: pint's cache file (stored at ~/.cache/pamila/pint/ in PAMILA) records the absolute path to pint's source data files inside the conda environment that was active when the cache was first created. When that conda environment is later deleted, pint raises a FileNotFoundError on startup because it can no longer find the recorded source path. I worked around it programmatically by catching the error and deleting the stale cache file automatically, so it gets regenerated fresh on the next startup, rather than requiring users to manually intervene.

Scope: where pint is not used

The device-level repr conversion functions (e.g., excitation curves for magnets, calibrated BPM data, MIMO ID corrector models) operate purely on plain Python float values, not pint objects. Users still receive properly unit-tagged results.

Compatibility with bluesky

pint objects flowing through bluesky plans required some monkey-patching to handle them correctly.

Diagrams

GET:

flowchart LR
  PV([PV]) -->|"float [PV unit]"| SIG([Signal])    
  SIG -->|"Q_ [LoLv repr unit]"| STRIP

  subgraph Device
      direction TB
      STRIP["strip unit (→ src_unit)"]
      CONV["repr conv func"]
      WRAP["wrap with dst_unit"]
      STRIP -->|floats| CONV
      CONV -->|floats| WRAP
    end

    WRAP -->|"Q_ [HiLv repr unit]"| USER([User])

Loading

PUT:

flowchart LR
  USER([User]) -->|"Q_ [HiLv repr unit]"| STRIP

  subgraph Device
      direction TB
      STRIP["strip unit (→ src_unit)"]
      CONV["repr conv func"]
      WRAP["wrap with LoLv repr unit"]
      STRIP -->|floats| CONV
      CONV -->|floats| WRAP
  end

  WRAP -->|"Q_ [LoLv repr unit]"| SIG([Signal])
  SIG -->|"float [PV unit]"| PV([PV])

Loading

Simple Examples

# Must pass a pint Quantity object Q_ — plain float raises TypeError
mlv.put(Q_("0.1 A"))

# get() returns a Q_ object
mlv.get()                   # e.g., Q_("0.1 A")
mlvs["C30_C1_x_angle_SP"].get()   # e.g., Q_("-10 urad") — different repr, no pint needed by user

# Extracting magnitude with .m (for display/print):
## 08_hla_disp_chrom.py
ksix = disp_chrom_data["chrom"]["x"][-2]
ksiy = disp_chrom_data["chrom"]["y"][-2]
print(f"Measured linear chromaticity = ({ksix.m:+.2f}, {ksiy.m:+.2f})")

# Arithmetic directly on Q_ objects:
## 04_mlvt.py
urad = Q_("1 urad")
cur_SP_val_d["scor"]["x"] += 5 * urad
cur_SP_val_d["QH1_K1"] -= Q_("1e-3 m^{-2}")

# MIMO put with a list of Q_:
## 09_setup_MIMO_magnet.py
mlv_I1_I2_SP.put([Q_("1 A"), Q_("2 A")])

# pint's .to() can be used for explicit unit conversion, but users usually
# don't need to call it directly — PAMILA handles unit conversion internally.
val = mlv.get()          # e.g., Q_("0.1 A")
val.to("mA")             # Q_("100.0 mA")   

Summary of decisions

Aspect	Decision
Unit tracking at user-facing API	Yes, mandatory pint Quantity objects
Unit tracking inside repr conversion functions	No — floats only, units converted based on config outside
Universal (intra-dimensional) unit conversion	Via pint
Device-specific (inter-dimensional) repr conversion	Custom Python functions, independent of pint
Matplotlib integration	Enabled via ureg.setup_matplotlib()
Serialization	Custom JSON serializer/deserializer for Quantity

Matlab Middle Layer handling of unit conversion (to be filled by @TeresiaOlsson and @marlibgin2, @gubaidulinvadim will ask Laurent Nadolski how it is done in SOLEIL MML)

Vadim recalls that in SOLEIL MML, there's some handling of unit conversion, one can choose to have RF frequency in Hz, MHz, kHz, etc. After the choice is made, this unit is displayed everywhere.
...

Ophyd/Bluesky community way of handling unit conversions (to be filled by @TeresiaOlsson)

...

[yhidaka]

In PAMILA, I adopted pint for unit handling. Here is a summary of what I did and why.

Two distinct types of "unit conversion" in PAMILA

PAMILA distinguishes two conceptually different types of unit conversion:

1. Universal unit conversion — "intra-dimensional" conversion where the physical dimension does not change (e.g., mA ↔ A, mm ↔ nm, GHz ↔ Hz). This is handled at the signal level using pint.
2. Representation (repr) conversion — "inter-dimensional" conversion that is device-dependent and typically changes the physical dimension (e.g., A ↔ mrad, A ↔ m⁻¹, digital counts ↔ A). This is handled at the device level using custom conversion functions, independently of pint.

How pint is used

pint is used to attach units to all values that users interact with. Specifically:

• All get() calls on a Middle Layer Variable (MLV) return a pint Quantity object (e.g., Q_("0.5 A"), Q_("-10 urad")).
• All put() calls on an MLV accept a pint Quantity object (or a list thereof for MIMO cases), which PAMILA internally converts to the low-level PV representation.
• Time-related device configuration parameters (e.g., wait times, tolerances) are also stored as pint Quantity objects.

Features of pint used in PAMILA

• UnitRegistry with caching — a single application-wide registry is created via set_application_registry() (needed for correct pickling/unpickling of Quantity objects).
• Matplotlib integration — ureg.setup_matplotlib() is called so that plots automatically display pint-aware axis labels.
• Serialization/deserialization — custom JSON serializers/deserializers were written for pint Quantity objects (e.g., for saving/loading configuration and data via tiled).
• Performance optimization — wrapper functions fast_create_Q() and fast_convert() were written to cache canonical unit name lookups, reducing repeated overhead from pint's unit string parsing.

A practical issue encountered

• One noteworthy issue: pint's cache file (stored at ~/.cache/pamila/pint/ in PAMILA) records the absolute path to pint's source data files inside the conda environment that was active when the cache was first created. When that conda environment is later deleted, pint raises a FileNotFoundError on startup because it can no longer find the recorded source path. I worked around it programmatically by catching the error and deleting the stale cache file automatically, so it gets regenerated fresh on the next startup, rather than requiring users to manually intervene.

Scope: where pint is not used

The device-level repr conversion functions (e.g., excitation curves for magnets, calibrated BPM data, MIMO ID corrector models) operate purely on plain Python float values, not pint objects. Users still receive properly unit-tagged results.

Compatibility with bluesky

pint objects flowing through bluesky plans required some monkey-patching to handle them correctly.

Diagrams

GET:

flowchart LR
  PV([PV]) -->|"float [PV unit]"| SIG([Signal])    
  SIG -->|"Q_ [LoLv repr unit]"| STRIP

  subgraph Device
      direction TB
      STRIP["strip unit (→ src_unit)"]
      CONV["repr conv func"]
      WRAP["wrap with dst_unit"]
      STRIP -->|floats| CONV
      CONV -->|floats| WRAP
    end

    WRAP -->|"Q_ [HiLv repr unit]"| USER([User])

Loading

PUT:

flowchart LR
  USER([User]) -->|"Q_ [HiLv repr unit]"| STRIP

  subgraph Device
      direction TB
      STRIP["strip unit (→ src_unit)"]
      CONV["repr conv func"]
      WRAP["wrap with LoLv repr unit"]
      STRIP -->|floats| CONV
      CONV -->|floats| WRAP
  end

  WRAP -->|"Q_ [LoLv repr unit]"| SIG([Signal])
  SIG -->|"float [PV unit]"| PV([PV])

Loading

Simple Examples

# Must pass a pint Quantity object Q_ — plain float raises TypeError
mlv.put(Q_("0.1 A"))

# get() returns a Q_ object
mlv.get()                   # e.g., Q_("0.1 A")
mlvs["C30_C1_x_angle_SP"].get()   # e.g., Q_("-10 urad") — different repr, no pint needed by user

# Extracting magnitude with .m (for display/print):
## 08_hla_disp_chrom.py
ksix = disp_chrom_data["chrom"]["x"][-2]
ksiy = disp_chrom_data["chrom"]["y"][-2]
print(f"Measured linear chromaticity = ({ksix.m:+.2f}, {ksiy.m:+.2f})")

# Arithmetic directly on Q_ objects:
## 04_mlvt.py
urad = Q_("1 urad")
cur_SP_val_d["scor"]["x"] += 5 * urad
cur_SP_val_d["QH1_K1"] -= Q_("1e-3 m^{-2}")

# MIMO put with a list of Q_:
## 09_setup_MIMO_magnet.py
mlv_I1_I2_SP.put([Q_("1 A"), Q_("2 A")])

# pint's .to() can be used for explicit unit conversion, but users usually
# don't need to call it directly — PAMILA handles unit conversion internally.
val = mlv.get()          # e.g., Q_("0.1 A")
val.to("mA")             # Q_("100.0 mA")   

Summary of decisions

Aspect	Decision
Unit tracking at user-facing API	Yes, mandatory pint Quantity objects
Unit tracking inside repr conversion functions	No — floats only, units converted based on config outside
Universal (intra-dimensional) unit conversion	Via pint
Device-specific (inter-dimensional) repr conversion	Custom Python functions, independent of pint
Matplotlib integration	Enabled via ureg.setup_matplotlib()
Serialization	Custom JSON serializer/deserializer for Quantity

[TeresiaOlsson]

My conclusion from looking into MML is that it only does hardware to physics conversions, no universal conversations. That is handled by a doc string and if you want to use an unit other than the one that comes in from the backend you need to implement the conversion yourself.

Below is an example of the relevant parts in the configuration for our BPMs.

• You define a native view for the device (AO.BPMx.Monitor.Units). In our case this is hardware but it could also have been physics.

• Then you define the unit for the hardware and the physics (AO.BPMx.Monitor.HWUnits and AO.BPMx.Monitor.PhysicsUnits and conversions between them (AO.BPMx.Monitor.HW2PhysicsParams and AO.BPMx.Monitor.Physics2HWParams.

• If it turns out that you want to use another unit inside MML than what you get from the backend, you need to implement the conversion. In our case the BPMs in the control system is in nm and we want to use mm in MML. We then need to switch to using the Special mode (AO.BPMx.Monitor.Mode = 'Special' and AO.BPMx.Monitor.SpecialFunctionGet = 'getBPMFromIOC') and inside the getBPMFromIOC we have the conversion between nm and mm.

AO.BPMx.FamilyName               = 'BPMx';
AO.BPMx.FamilyType               = 'BPM';
AO.BPMx.Monitor.Mode             = 'Special';
AO.BPMx.Monitor.SpecialFunctionGet       = 'getBPMFromIOC';
AO.BPMx.Monitor.DataType         = 'Scalar';
AO.BPMx.Monitor.Units            = 'Hardware';
AO.BPMx.Monitor.HWUnits          = 'mm';
AO.BPMx.Monitor.PhysicsUnits     = 'meter';

AO.BPMy.FamilyName               = 'BPMy';
AO.BPMy.FamilyType               = 'BPM';
AO.BPMy.Monitor.Mode             = 'Special';
AO.BPMy.Monitor.SpecialFunctionGet       = 'getBPMFromIOC';
AO.BPMy.Monitor.DataType         = 'Scalar';
AO.BPMy.Monitor.Units            = 'Hardware';
AO.BPMy.Monitor.HWUnits          = 'mm';
AO.BPMy.Monitor.PhysicsUnits     = 'meter';
....
AO.BPMx.Monitor.HW2PhysicsParams(ii,:)   = [1e-3 0];
AO.BPMx.Monitor.Physics2HWParams(ii,:)   = 1000;
AO.BPMy.Monitor.HW2PhysicsParams(ii,:)   = [1e-3 0];
AO.BPMy.Monitor.Physics2HWParams(ii,:)   = 1000;

[TeresiaOlsson]

In ophyd-async conversions can be done with derived signals. It allows to you define a transform and from some input signals get some output signals. The transformation is two-way and done on the device level.

Here is an example for a tune device I did as part of testing how it works. I tried to use it for the hardware to physics conversions since our tunes are read from the control system in frequency.

from pydantic import BaseModel
from typing import Optional, TypedDict
from ophyd_async.core import StandardReadable, derived_signal_r, Transform, DerivedSignalFactory
from ophyd_async.epics.core import epics_signal_r

class Tune1DConfig(BaseModel):
    name: Optional[str] = 'tune'
    read_pv: str
    settling_time: Optional[float] = 0

class TuneRaw(TypedDict):
    frequency: float

class TuneDerived(TypedDict):
    tune: float
              
class TuneTransform(Transform):
    
    revolution_frequency: float
    
    def derived_to_raw(self, *, tune: float) -> TypedDict:
        return TuneRaw(
            frequency = tune*self.revolution_frequency*1e-3
            )
    
    def raw_to_derived(self, *, frequency: float) -> TypedDict:
        return TuneDerived(
            tune = frequency*1e3/self.revolution_frequency
            )    

class Tune1D(StandardReadable):
    
    def __init__(self, config: Tune1DConfig, prefix: Optional[str] = None) -> None:
        
        self._settling_time = config.settling_time
        
        self._read_pv = f"{prefix}:{config.read_pv}" if prefix else config.read_pv
        
        revolution_frequency = 6.25e6 # This should not be hardcoded but read from somewhere else.
        
        frequency = epics_signal_r(float, self._read_pv, name='frequency')
                      
        self._factory = DerivedSignalFactory(
                TuneTransform,
                frequency=frequency,
                revolution_frequency=revolution_frequency,
            )
              
        tune = self._factory.derived_signal_r(float,name='tune')
        
        with self.add_children_as_readables():
            self.frequency = frequency
            self.tune = tune
            
        super().__init__(name=config.name)  

At the signal level there is no conversions yet. I think there is a way to read the unit from the control system (but I haven't tested it myself yet). As far as I have understood this works best for TANGO and the TANGO users seem less concerned about it than the EPICS users. There is however at the moment no functionality to verify that the unit from the control system is correct. For example, at many EPICS labs this field might be unmaintained and you shouldn't blindly trust it. There has been a discussion about implementing something on the signal level (including the option to use pint) but I think nothing has been done so far: bluesky/ophyd-async#657.