Improve MagnetArray class and implement generic ElementArray class

[GamelinAl]

The MagnetArray class needs to improved, it probably should inherit from a more generic class ElementArray which handles all kind of pyAML elements (BPMs, RF cavities, ...).

Most the features we need are detailed in section 2.3 of the User Interface Specification but I recall a few here:

I propose the following behavior for arrays (which I think match the User Interface Specification document):

• The array is composed of different types of pyAML elements, then it's a ElementArray and there is no access at the array level to the individual element specific properties (magnet strength, BPM reading, ...), only to the common properties (name, hardware_name, ...).
• The array is composed of the same pyAML type, then there is an array level access to the individual element specific properties.
• Specific case of CombinedFunctionMagnet. If an array is composed of Sextupole and CombinedFunctionMagnet with a sextupolar component, then the sextupolar strength is accessible at the array level.

1. Subset Handling:
   • Subsets (e.g., qcorr[0:5]) should return a MagnetArray, not a list.
     • Bug: qcorr[0:5].strengths → AttributeError: 'list' object has no attribute 'strengths'.
2. Support heterogeneous MagnetArray (different magnet types).
3. Implement element access by:
   • hardware_name or list of hardware_name (with wildcard type regex possible)
   • name or list of name (with wildcard type regex possible)
   • Element type filtering
   • a boolean array of len(ElementArray)
4. Implement element wise boolean operators
   • For example the & operator should return the intersection of two arrays:
   • bpm_cell8 = sr.cell8 & sr.bpm # return an ElementArray with the common element between the two arrays sr.cell8 and sr.bpm.

[JeanLucPons]

I don't see blocking point except:

CombinedFunctionMagnet are not supposed to be used in arrays.
A CombinedFunctionMagnet is supposed to export 1 single function Magnet per function that you can have in arrays.

Concerning array intersection, this might be a bit tricky:

• Is the operator is supposed to be commutative i.e sr.cell8 & sr.bpm == sr.bpm & sr.cell8 (I mean the order of returned elements may be different depending on the declaration sr.cell8 array ?).
  i.e.: Using set() to perform intersection does do preserve the order.

>>> b = [8,6,4,2]
>>> set(b)
{8, 2, 4, 6}
>>> 

• Returned type will be ElementArray which means that no methods (except get_name()) are common. An extra typing is of course possible but without warranty that methods/properties are available.

[GamelinAl]

CombinedFunctionMagnet are not supposed to be used in arrays.
A CombinedFunctionMagnet is supposed to export 1 single function Magnet per function that you can have in arrays.

I am not sure why you say that, in the current version it is what is done:

[CombinedFunctionMagnet(EtJmpy6H2l9ceFl), CombinedFunctionMagnet(K9DCoKC9KdqLgE3), 
CombinedFunctionMagnet(a8OgqVuDFCiUtHY),  Quadrupole(UwRF50cGEWVOsPz), Sextupole(RX2c4Al3f5WiGTR), 
....
 Quadrupole(Oy8YreBTelEaCwO), Sextupole(mUmaj7rXVCVOaQ2), CombinedFunctionMagnet(z8NPIKNaIZtlPVO)]
>>> type(cell1)
<class 'pyaml.arrays.magnet_array.MagnetArray'>

I would say that both case should be possible i.e. either to have CFM in arrays or to decompose directly in components in the array depending on how you define your array.

Concerning array intersection, yes I would say it's supposed to be commutative.
For the order, if we have a 'master array' of unique name of all pyAML elements (configured by file + dynamic addition), then it can be used to make this intersection and order the ouput.
The order would just be the definition order of the element.

Returned type will be ElementArray which means that no methods (except get_name()) are common. An extra typing is of course possible but without warranty that methods/properties are available.

I am not sure I follow. The return type should allow to use the common properties/methods. Either ElementArray should allow this or it should return a more specific type MagnetArray, BPMArray or something like this.

[JeanLucPons]

I am not sure why you say that, in the current version it is what is done:

[CombinedFunctionMagnet(EtJmpy6H2l9ceFl), CombinedFunctionMagnet(K9DCoKC9KdqLgE3), 
CombinedFunctionMagnet(a8OgqVuDFCiUtHY),  Quadrupole(UwRF50cGEWVOsPz), Sextupole(RX2c4Al3f5WiGTR), 
....
 Quadrupole(Oy8YreBTelEaCwO), Sextupole(mUmaj7rXVCVOaQ2), CombinedFunctionMagnet(z8NPIKNaIZtlPVO)]
>>> type(cell1)
<class 'pyaml.arrays.magnet_array.MagnetArray'>

Yes but CombinedFunctionMagnet allows nothing for the moment and it was supposed to have a strengths (or multipoles) attribute containing all strengths but it is not yet implemented.

I would say that both case should be possible i.e. either to have CFM in arrays or to decompose directly in components in the array depending on how you define your array.

For CombinedFunctionMagnet() array you will get a strengths attribute with a non constant length.
We can eventually add a flag to define the 'main strength' for a strength attribute but for instance at the ESRF we have combined function magnet without 'main strength' and depending on thier cabling they can even export different strengths.

Concerning array intersection, yes I would say it's supposed to be commutative. For the order, if we have a 'master array' of unique name of all pyAML elements (configured by file + dynamic addition), then it can be used to make this intersection and order the ouput. The order would just be the definition order of the element.

OK so the commutativity depends on the way the user configure its arrays. In a & b, you cannot say a is the master array or if you do so then it is equivalent to say that it is non commutative.

I am not sure I follow. The return type should allow to use the common properties/methods. Either ElementArray should allow this or it should return a more specific type MagnetArray, BPMArray or something like this.

Yes if you return a MagnetArray then you will have the magnet methods but your arrays is not supposed to contains only magnets.

[GamelinAl]

Yes but CombinedFunctionMagnet allows nothing for the moment and it was supposed to have a strengths (or multipoles) attribute containing all strengths but it is not yet implemented.

That's fine, it will come a bit later but we should keep this in mind.

For CombinedFunctionMagnet() array you will get a strengths attribute with a non constant length.

Yes if you return a MagnetArray then you will have the magnet methods but your arrays is not supposed to contains only magnets.

What I mean is that we should detect at the pyAML software level what type of array should be returned (or what type of property/method should be exposed if you wish to keep always the same array type) depending of the result of the filtering.

Is the result of the filtering is only Magnet, then you get an MagnetArray with all its possibilites. If not then have a regular ElementArray with only the different kind of names as common properties.

OK so the commutativity depends on the way the user configure its arrays. In a & b, you cannot say a is the master array or if you do so then it is equivalent to say that it is non commutative.

The results only depends on the pyAML configuration, not on the way the arrays are configured. The idea is to do all operations with respect to the "master array".

Here is a simplified code which does this with int

import numpy as np

master_array = np.array([2, 4, 5, 6, 11])

a = np.array([6, 4, 5])

b = np.array([11, 4, 5, 2])

a_master, a_ind, master_ind_wa = np.intersect1d(a, master_array, return_indices=True)

b_master, b_ind, master_ind_wb = np.intersect1d(b, master_array, return_indices=True)

bool_array_a = np.isin(np.arange(len(master_array)), master_ind_wa)

bool_array_b = np.isin(np.arange(len(master_array)), master_ind_wb)

idx_intersection = bool_array_a & bool_array_b

idx_intersection_b = bool_array_b & bool_array_a

print(master_array[idx_intersection]) # return [4 5]

print(master_array[idx_intersection_b]) # return [4 5]

[JeanLucPons]

What I mean is that we should detect at the pyAML software level what type of array should be returned (or what type of property/method should be exposed if you wish to keep always the same array type) depending of the result of the filtering.

Is the result of the filtering is only Magnet, then you get an MagnetArray with all its possibilites. If not then have a regular ElementArray with only the different kind of names as common properties.

I will see how to implement this. The idea is that I want something fully typed to allow completion in editors (not necessary in a console having constructed objects). For subset selection in already typed array no problem. For subset of ElementArray this might be tricky.

OK so the commutativity depends on the way the user configure its arrays. In a & b, you cannot say a is the master array or if you do so then it is equivalent to say that it is non commutative.

The results only depends on the pyAML configuration, not on the way the arrays are configured. The idea is to do all operations with respect to the "master array".

Here is a simplified code which does this with int

import numpy as np

master_array = np.array([2, 4, 5, 6, 11])

a = np.array([6, 4, 5])

b = np.array([11, 4, 5, 2])

a_master, a_ind, master_ind_wa = np.intersect1d(a, master_array, return_indices=True)

b_master, b_ind, master_ind_wb = np.intersect1d(b, master_array, return_indices=True)

bool_array_a = np.isin(np.arange(len(master_array)), master_ind_wa)

bool_array_b = np.isin(np.arange(len(master_array)), master_ind_wb)

idx_intersection = bool_array_a & bool_array_b

idx_intersection_b = bool_array_b & bool_array_a

print(master_array[idx_intersection]) # return [4 5]

print(master_array[idx_intersection_b]) # return [4 5]

OK I see, that means that a master array has to be defined somewhere.
Should we build a master array containing all elements in the order found in the config file ?

[GamelinAl]

I think the idea of master array will also resolve a few other issues of things we wanted in the specification like the device_list access like in MML (cell, number) but I will make a proposal on how to implement this in a future issue if we agree on this.

Should we build a master array containing all elements in the order found in the config file ?

I think so, this delegate to the user the responsability to have a consistent configuration file if they want to keep the numbering of the index (which we all agree should not be used anyway).
Dynamically defined pyAML objects can be append at the end of the master array.

[simoneliuzzo]

Subsets (e.g., qcorr[0:5]) should return a MagnetArray, not a list.

* **Bug:** `qcorr[0:5].strengths` → `AttributeError: 'list' object has no attribute 'strengths'`

Dear @GamelinAl ,

I do not see this feature as strongly needed. I would rather create at configuration level an other array of magnets.
It may be used for scripting, but I think that also something like

str = qcorr.strengths[0:5]
newstrengths = qcorr.strengths 
newstrengths[0:5] = str * 2
qcorr. strengths = newstrength

would work, and it could be less difficult to implement.

[simoneliuzzo]

• name or list of name (with wildcard type regex possible)

Dear @GamelinAl,

To select by name, python index function could also help

names = qcorr.names  # list of names
specialquads = names.index('OCT.1')

Always with the idea to keep a simple implementation for the time being.
Also, if this selection is very often used, I would simply create a dedicated family in the configuration.

[simoneliuzzo]

Dear @GamelinAl ,

• bpm_cell8 = sr.cell8 & sr.bpm # return an ElementArray with the common element between the two arrays sr.cell8 and sr.bpm.

Also here I think that if we ask for the cell-element# indexing option (as in MML), then we do not need to have this boolean indexing. It could be useful, but I have the feeling that we are asking for two different solution of the same problem.