Skip to content

200 feature yellowpages standardized name mapping discovery#208

Merged
JeanLucPons merged 17 commits intomainfrom
200-feature-yellowpages-standardized-name-mapping-discovery
Mar 17, 2026
Merged

200 feature yellowpages standardized name mapping discovery#208
JeanLucPons merged 17 commits intomainfrom
200-feature-yellowpages-standardized-name-mapping-discovery

Conversation

@gupichon
Copy link
Contributor

@gupichon gupichon commented Mar 5, 2026

Pull Request

Description

This pull request introduces a dynamic YellowPages service for the Accelerator object.

The goal is to provide a unified discovery interface for arrays, tuning tools and diagnostics available in an accelerator configuration.
The service scans all ElementHolder instances associated with the accelerator controls and simulators and exposes the discovered objects through a simple API.

The implementation is fully dynamic:

  • no explicit registration is required
  • entries are discovered from ElementHolder at runtime
  • arrays, tools and diagnostics are resolved on demand

The service provides a simple identifier lookup API with two complementary behaviors:

  1. Named array lookup
sr.yellow_pages.get("BPM")
sr.yellow_pages.get("BPM", mode="live")

If the query exactly matches a discovered array name, the identifiers contained in this array are returned.

  1. Identifier search
sr.yellow_pages.get("OH4*")
sr.yellow_pages.get("re:^SH1A-C0[12]-H$")
sr.yellow_pages["OH4*"]

The search syntax is intentionally simple:

  • wildcard matching using fnmatch
  • regular expression matching using the re: prefix

The get() method always returns a list[str], and __getitem__ is an alias to it.

The __repr__ representation provides a compact overview of:

  • controls
  • simulators
  • discovered arrays, tools and diagnostics

including:

  • the Python module defining the object
  • array sizes
  • mode availability when an entry is not available everywhere

Example output:

Controls:
    live
    .

Simulators:
    design
    .

Arrays:
    BPM        (pyaml.arrays.bpm_array)             size=320
    HCorr      (pyaml.arrays.magnet_array)          size=288
    Skews      (pyaml.arrays.magnet_array)          size=288
    VCorr      (pyaml.arrays.magnet_array)          size=288
    .

Tools:
    DEFAULT_DISPERSION (pyaml.tuning_tools.dispersion)
    DEFAULT_ORBIT_CORRECTION (pyaml.tuning_tools.orbit)
    DEFAULT_ORBIT_RESPONSE_MATRIX (pyaml.tuning_tools.orbit_response_matrix)
    .

Diagnostics:
    BETATRON_TUNE (pyaml.diagnostics.tune_monitor)
    .

A specific effort was also made to preserve element ordering whenever possible:

  • discovery order is preserved across modes
  • array element order is preserved when iterating over arrays
  • global identifier collection avoids losing ordering through unordered set operations

Related Issue

Features/issues described there are:

  • new feature: implementation of a dynamic YellowPages discovery service
  • new feature: unified access to arrays, tools and diagnostics
  • new feature: query language for array identifiers using set operators and regex
  • new feature: human-readable accelerator overview via __repr__

Changes to existing functionality

The following changes were required to integrate YellowPages into the accelerator architecture:

  • Added a new module yellow_pages.py implementing the discovery service

  • Added yellow_pages property to the Accelerator class

  • Extended ElementHolder with generic accessors:

    • list_arrays()
    • list_tools()
    • list_diagnostics()
    • get_array()
    • get_tool()
    • get_diagnostic()

    These methods allow YellowPages to dynamically discover objects without knowing their concrete implementation.

  • Added a structured __repr__ representation showing available objects, types and sizes.


Testing

The following tests (compatible with pytest) were added:

  • discovery of arrays, tools and diagnostics from ElementHolder
  • has(), keys(), categories() behavior
  • array identifier resolution through get()
  • mode-restricted lookup with get(..., mode=...)
  • availability detection across modes
  • attribute access (yellow_pages.BPM)
  • wildcard search using fnmatch
  • regex search using re: syntax
  • error handling for invalid queries
  • validation of __repr__ output structure
  • validation that identifier ordering is preserved when possible

Verify that your checklist complies with the project

  • New and existing unit tests pass locally
  • Tests were added to prove that all features/changes are effective
  • The code is commented where appropriate
  • Any existing features are not broken (unless there is an explicit change to an existing functionality)

@gupichon gupichon linked an issue Mar 5, 2026 that may be closed by this pull request
2 tasks
@gubaidulinvadim

This comment was marked as outdated.

@JeanLucPons
Copy link
Contributor

JeanLucPons commented Mar 5, 2026

I do not understand why the ElementHolder is affected.
It goes a bit against the new plan for ElementHolderAPI.
How do you plan to integrate your mods 199 ?

@gupichon
Copy link
Contributor Author

gupichon commented Mar 5, 2026

I do not understand why the ElementHolder is affected. It goes a bit against the new plan for ElementHolderAPI. How do you plan to integrate your mods 199 ?

I think it will be fine. I might make some adjustments to the YellowPages, but I believe it is the easiest way to implement it. Basically, I will fetch all the necessary data from several sub-holders instead of the main ones.
This is why I would like to validate all the current PRs before starting the ElementHolder refactoring. That way, the other constraints to be respected will already be clearly established.

@JeanLucPons
Copy link
Contributor

OK so I would prefer to hide this internal functions from the API using _
To get the list of tools, you should do something like:

sr.live.tools.get()

@gupichon
Copy link
Contributor Author

gupichon commented Mar 5, 2026

OK so I would prefer to hide this internal functions from the API using _ To get the list of tools, you should do something like:

sr.live.tools.get()

Ok

@GamelinAl
Copy link
Contributor

Hi @gupichon-soleil I tested this PR with SOLEIL II configuration from last workshop and found a few issues:

It seems that all CFM are seen as a Arrays by the YellowPages. I am not sure if this is a configuration issue (on SOLEIL conf) or a YellowPage problem:

sr.yellow_pages
>>>
Controls:
    live
    .

Simulators:
    design
    .

Arrays:
    BPM        (pyaml.arrays.bpm_array.BPMArray)       size=180
    Cell1      (pyaml.arrays.element_array.ElementArray) size=121
    Cell10     (pyaml.arrays.element_array.ElementArray) size=121
    Cell11     (pyaml.arrays.element_array.ElementArray) size=121
    Cell12     (pyaml.arrays.element_array.ElementArray) size=76
    Cell13     (pyaml.arrays.element_array.ElementArray) size=116
    Cell14     (pyaml.arrays.element_array.ElementArray) size=76
    Cell15     (pyaml.arrays.element_array.ElementArray) size=120
    Cell16     (pyaml.arrays.element_array.ElementArray) size=136
    Cell17     (pyaml.arrays.element_array.ElementArray) size=76
    Cell18     (pyaml.arrays.element_array.ElementArray) size=116
    Cell19     (pyaml.arrays.element_array.ElementArray) size=76
    Cell2      (pyaml.arrays.element_array.ElementArray) size=76
    Cell20     (pyaml.arrays.element_array.ElementArray) size=121
    Cell3      (pyaml.arrays.element_array.ElementArray) size=116
    Cell4      (pyaml.arrays.element_array.ElementArray) size=76
    Cell5      (pyaml.arrays.element_array.ElementArray) size=121
    Cell6      (pyaml.arrays.element_array.ElementArray) size=137
    Cell7      (pyaml.arrays.element_array.ElementArray) size=76
    Cell8      (pyaml.arrays.element_array.ElementArray) size=116
    Cell9      (pyaml.arrays.element_array.ElementArray) size=76
    OCTF1_QCORROCT_157_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_158_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_159_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_160_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_161_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    ...

If I check, then I see clearly that this Array just corresponds to the different components of my CFM:

sr.yellow_pages._get("OH4_QCORROCT_11_QT",mode="live")
>>>
[Octupole(peer='TangoControlSystem:live', name='OH4_QCORROCT_11_QT_001.octupole', model_name='OH4_QCORROCT_11_QT_001', magnet_model=IdentityCFMagnetModel(multipoles=['B3', 'B1', 'A1'], powerconverters=None, physics=[Attribute(attribute='AN02-AR/EM-OCT/OH.01/strength', unit='1/m**3', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLN.01/strength', unit='1/m**2', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLT.01/strength', unit='1/m**2', range=None)], units=['1/m**3', '1/m**2', '1/m**2'])),
 Quadrupole(peer='TangoControlSystem:live', name='OH4_QCORROCT_11_QT_001.quadrupole', model_name='OH4_QCORROCT_11_QT_001', magnet_model=IdentityCFMagnetModel(multipoles=['B3', 'B1', 'A1'], powerconverters=None, physics=[Attribute(attribute='AN02-AR/EM-OCT/OH.01/strength', unit='1/m**3', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLN.01/strength', unit='1/m**2', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLT.01/strength', unit='1/m**2', range=None)], units=['1/m**3', '1/m**2', '1/m**2'])),
 SkewQuad(peer='TangoControlSystem:live', name='OH4_QCORROCT_11_QT_001.skewquad', model_name='OH4_QCORROCT_11_QT_001', magnet_model=IdentityCFMagnetModel(multipoles=['B3', 'B1', 'A1'], powerconverters=None, physics=[Attribute(attribute='AN02-AR/EM-OCT/OH.01/strength', unit='1/m**3', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLN.01/strength', unit='1/m**2', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLT.01/strength', unit='1/m**2', range=None)], units=['1/m**3', '1/m**2', '1/m**2']))]

There is also an issue with the design control mode, which seems not to be accessible:

sr.yellow_pages._get("Cell1").keys()
>>> dict_keys(['live'])

Why live and not design ?

In fact, design mode is not seen by yellow_page._get

sr.yellow_pages._get("BPM",mode="design")
>>>
---------------------------------------------------------------------------
YellowPagesError                          Traceback (most recent call last)
Cell In[52], line 1
----> 1 sr.yellow_pages._get("BPM",mode="design")

File [~/Python_dev/pyaml/pyaml/pyaml/yellow_pages.py:450](http://localhost:8888/home/gamelina/Python_dev/pyaml/pyaml/pyaml/yellow_pages.py#line=449), in YellowPages._get(self, key, mode)
    448 holder = self._acc.modes().get(mode)
    449 if holder is None:
--> 450     raise YellowPagesError(f"Unknown mode '{mode}'.")
    451 obj = self._try_resolve_in_holder(key, holder)
    452 if obj is None:

YellowPagesError: Unknown mode 'design'.

Somehow this is not working:

sr.yellow_pages["Cell1"]
>>> YellowPagesQueryError: Cannot tokenize near: 'ell1'
sr.yellow_pages["re{Cell*}"]
>>> [] 

But this is:

sr.yellow_pages._get("Cell1")
>>> Working array
sr.yellow_pages["BPM"]
>>>['BPM_001',
 'BPM_002',
 'BPM_003',
 'BPM_004',
 'BPM_005',
 'BPM_006',
 'BPM_007',
 'BPM_008',
 'BPM_009',
 ...]

Regular expression also don't work as expected:

sr.yellow_pages["re{OH4*}"]
>>> ['OH10_QCORROCT_112_QT_001',
 'OH10_QCORROCT_112_QT_001.octupole',
 'OH10_QCORROCT_112_QT_001.quadrupole',
 'OH10_QCORROCT_112_QT_001.skewquad',
 'OH10_QCORROCT_113_QT_001',
 'OH10_QCORROCT_113_QT_001.octupole',
 'OH10_QCORROCT_113_QT_001.quadrupole',
 'OH10_QCORROCT_113_QT_001.skewquad',
 'OH10_QCORROCT_134_QT_001',
 ...]

@gupichon
Copy link
Contributor Author

Hi @gupichon-soleil I tested this PR with SOLEIL II configuration from last workshop and found a few issues:

It seems that all CFM are seen as a Arrays by the YellowPages. I am not sure if this is a configuration issue (on SOLEIL conf) or a YellowPage problem:

sr.yellow_pages
>>>
Controls:
    live
    .

Simulators:
    design
    .

Arrays:
    BPM        (pyaml.arrays.bpm_array.BPMArray)       size=180
    Cell1      (pyaml.arrays.element_array.ElementArray) size=121
    Cell10     (pyaml.arrays.element_array.ElementArray) size=121
    Cell11     (pyaml.arrays.element_array.ElementArray) size=121
    Cell12     (pyaml.arrays.element_array.ElementArray) size=76
    Cell13     (pyaml.arrays.element_array.ElementArray) size=116
    Cell14     (pyaml.arrays.element_array.ElementArray) size=76
    Cell15     (pyaml.arrays.element_array.ElementArray) size=120
    Cell16     (pyaml.arrays.element_array.ElementArray) size=136
    Cell17     (pyaml.arrays.element_array.ElementArray) size=76
    Cell18     (pyaml.arrays.element_array.ElementArray) size=116
    Cell19     (pyaml.arrays.element_array.ElementArray) size=76
    Cell2      (pyaml.arrays.element_array.ElementArray) size=76
    Cell20     (pyaml.arrays.element_array.ElementArray) size=121
    Cell3      (pyaml.arrays.element_array.ElementArray) size=116
    Cell4      (pyaml.arrays.element_array.ElementArray) size=76
    Cell5      (pyaml.arrays.element_array.ElementArray) size=121
    Cell6      (pyaml.arrays.element_array.ElementArray) size=137
    Cell7      (pyaml.arrays.element_array.ElementArray) size=76
    Cell8      (pyaml.arrays.element_array.ElementArray) size=116
    Cell9      (pyaml.arrays.element_array.ElementArray) size=76
    OCTF1_QCORROCT_157_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_158_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_159_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_160_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    OCTF1_QCORROCT_161_QT (pyaml.arrays.magnet_array.MagnetArray) size=3
    ...

If I check, then I see clearly that this Array just corresponds to the different components of my CFM:

sr.yellow_pages._get("OH4_QCORROCT_11_QT",mode="live")
>>>
[Octupole(peer='TangoControlSystem:live', name='OH4_QCORROCT_11_QT_001.octupole', model_name='OH4_QCORROCT_11_QT_001', magnet_model=IdentityCFMagnetModel(multipoles=['B3', 'B1', 'A1'], powerconverters=None, physics=[Attribute(attribute='AN02-AR/EM-OCT/OH.01/strength', unit='1/m**3', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLN.01/strength', unit='1/m**2', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLT.01/strength', unit='1/m**2', range=None)], units=['1/m**3', '1/m**2', '1/m**2'])),
 Quadrupole(peer='TangoControlSystem:live', name='OH4_QCORROCT_11_QT_001.quadrupole', model_name='OH4_QCORROCT_11_QT_001', magnet_model=IdentityCFMagnetModel(multipoles=['B3', 'B1', 'A1'], powerconverters=None, physics=[Attribute(attribute='AN02-AR/EM-OCT/OH.01/strength', unit='1/m**3', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLN.01/strength', unit='1/m**2', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLT.01/strength', unit='1/m**2', range=None)], units=['1/m**3', '1/m**2', '1/m**2'])),
 SkewQuad(peer='TangoControlSystem:live', name='OH4_QCORROCT_11_QT_001.skewquad', model_name='OH4_QCORROCT_11_QT_001', magnet_model=IdentityCFMagnetModel(multipoles=['B3', 'B1', 'A1'], powerconverters=None, physics=[Attribute(attribute='AN02-AR/EM-OCT/OH.01/strength', unit='1/m**3', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLN.01/strength', unit='1/m**2', range=None), Attribute(attribute='AN02-AR/EM-COR/OH.01-CQLT.01/strength', unit='1/m**2', range=None)], units=['1/m**3', '1/m**2', '1/m**2']))]

There is also an issue with the design control mode, which seems not to be accessible:

sr.yellow_pages._get("Cell1").keys()
>>> dict_keys(['live'])

Why live and not design ?

In fact, design mode is not seen by yellow_page._get

sr.yellow_pages._get("BPM",mode="design")
>>>
---------------------------------------------------------------------------
YellowPagesError                          Traceback (most recent call last)
Cell In[52], line 1
----> 1 sr.yellow_pages._get("BPM",mode="design")

File [~/Python_dev/pyaml/pyaml/pyaml/yellow_pages.py:450](http://localhost:8888/home/gamelina/Python_dev/pyaml/pyaml/pyaml/yellow_pages.py#line=449), in YellowPages._get(self, key, mode)
    448 holder = self._acc.modes().get(mode)
    449 if holder is None:
--> 450     raise YellowPagesError(f"Unknown mode '{mode}'.")
    451 obj = self._try_resolve_in_holder(key, holder)
    452 if obj is None:

YellowPagesError: Unknown mode 'design'.

Somehow this is not working:

sr.yellow_pages["Cell1"]
>>> YellowPagesQueryError: Cannot tokenize near: 'ell1'
sr.yellow_pages["re{Cell*}"]
>>> [] 

But this is:

sr.yellow_pages._get("Cell1")
>>> Working array
sr.yellow_pages["BPM"]
>>>['BPM_001',
 'BPM_002',
 'BPM_003',
 'BPM_004',
 'BPM_005',
 'BPM_006',
 'BPM_007',
 'BPM_008',
 'BPM_009',
 ...]

Regular expression also don't work as expected:

sr.yellow_pages["re{OH4*}"]
>>> ['OH10_QCORROCT_112_QT_001',
 'OH10_QCORROCT_112_QT_001.octupole',
 'OH10_QCORROCT_112_QT_001.quadrupole',
 'OH10_QCORROCT_112_QT_001.skewquad',
 'OH10_QCORROCT_113_QT_001',
 'OH10_QCORROCT_113_QT_001.octupole',
 'OH10_QCORROCT_113_QT_001.quadrupole',
 'OH10_QCORROCT_113_QT_001.skewquad',
 'OH10_QCORROCT_134_QT_001',
 ...]

I'll look to it and add some tests

@JeanLucPons
Copy link
Contributor

Would it be possible to have a coherent syntax as in wildcard configuration ?
i.e:

sr.yellow_pages["OH4*"]  # fnmatch
sr.yellow_pages["re:^SH1A-C0[12]-H$"] # regexp

@gupichon
Copy link
Contributor Author

Would it be possible to have a coherent syntax as in wildcard configuration ? i.e:

sr.yellow_pages["OH4*"]  # fnmatch
sr.yellow_pages["re:^SH1A-C0[12]-H$"] # regexp

That’s a point I wanted to discuss. Which syntax do you prefer?
Obviously, I have my answer now! ;)

My initial plan was to test this syntax here and, if everyone liked it, implement it everywhere. However, it is quite too complex for the actual needs.

I’m changing it to the “classic” syntax used in the wildcard-based configuration. It will also solve the bugs that were observed, as the regex used to parse the query is too complicated anyway.

@gupichon
Copy link
Contributor Author

@GamelinAl, @JeanLucPons
I’m currently correcting the code. Just a few clarifications above.


sr.yellow_pages["re{OH4*}"]
>>> ['OH10_QCORROCT_112_QT_001',
 'OH10_QCORROCT_112_QT_001.octupole',
 'OH10_QCORROCT_112_QT_001.quadrupole',
 'OH10_QCORROCT_112_QT_001.skewquad',
 'OH10_QCORROCT_113_QT_001',
 'OH10_QCORROCT_113_QT_001.octupole',
 'OH10_QCORROCT_113_QT_001.quadrupole',
 'OH10_QCORROCT_113_QT_001.skewquad',
 'OH10_QCORROCT_134_QT_001',
 ...]

This is a correct result for the given regex, as it matches elements containing OH followed by zero or more occurrences of the character 4.


@JeanLucPons, it seems that the CFMs are added as arrays, is that correct?


Requesting sr.yellow_pages["BPM"] asks for a set of elements, not a single element. In any case, I’ve added this option, assuming that no elements share the same name as an array. I don’t think this is checked anywhere.


I’ll commit the modifications this morning.

…ssible. The get method is now public and returns an array of strings. The query syntax has been updated to match the one used in the configuration file. It is also possible to query an array by its name.
@gupichon
Copy link
Contributor Author

Everything should be OK now, I update the PR description.

@JeanLucPons
Copy link
Contributor

JeanLucPons commented Mar 13, 2026

Could you add your new module in apidoc/genapi.py ?
and add special member that needs to be documented as in the example below:

            if m in ["pyaml.arrays.element_array"]:
                # Include special members for operator overloading
                file.write(
                    "         :special-members: __add__, __and__, __or__, __sub__ \n"
                )

Thx

@JeanLucPons
Copy link
Contributor

One more remark:
I asked to have any methods in the element holder that goes against the new ElementHolder API (such as get_diagnostic() or list_tool()) to have _ ?

@JeanLucPons
Copy link
Contributor

JeanLucPons commented Mar 13, 2026

    def __init__(self, accelerator: Any):
        self._acc = accelerator

Could accelerator be typed to Accelerator rather than Any ?

@JeanLucPons
Copy link
Contributor

OK to what about renaming following function in ElementHolder ?

    def get_array(self, name: str):
    def get_tool(self, name: str):
    def list_tools(self) -> list[str]:
    def list_diagnostics(self) -> list[str]:

to

    def _get_array(self, name: str):
    def _get_tool(self, name: str):
    def _list_tools(self) -> list[str]:
    def _list_diagnostics(self) -> list[str]:

Thanks

@gupichon
Copy link
Contributor Author

Ok, I believe everything is done now. Could you confirm?

@JeanLucPons
Copy link
Contributor

Could you replace "Expected Accelerator interface" by "Accelerator interface" ?
And remove the ElementHolder stuff ?

image

@JeanLucPons
Copy link
Contributor

It would be nice to have link in the doctring, such as

:py:class:`~pyaml.accelerator.Accelerator`

when possible.
Thanks

@JeanLucPons
Copy link
Contributor

Small details (I presume)

image

@gupichon
Copy link
Contributor Author

Small details (I presume)

Wrong copy/paste...

Copy link
Contributor

@JeanLucPons JeanLucPons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for me

@JeanLucPons JeanLucPons merged commit 6b7e777 into main Mar 17, 2026
3 checks passed
@JeanLucPons JeanLucPons deleted the 200-feature-yellowpages-standardized-name-mapping-discovery branch March 17, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: YellowPages (standardized-name mapping & discovery)

5 participants