yaml loader and factory error management: cycle detection, line mapping, reporting

[JeanLucPons]

Handle error management in factory and config file loader.

pyaml/pyaml/configuration/fileloader.py

Line 9 in 0e306fb

#Implement cycle detection in case of wrong yaml/json link that creates a cycle

Use

model_config = ConfigDict(arbitrary_types_allowed=True,extra="forbid")

in all pydatic schema and remove BaseModel from abstract classes

[JeanLucPons]

The location is lost when a model validation error happens and it does not contains the filename.
I try to add it (factory.py:98) but the location points to the root object and not to the error itself.
The error stack produced is also difficult to read.
I would expect something simple such as: "test/config/sr.yaml: 'array' unexpected keyword at line 11"

        except Exception as e:            
            raise PyAMLException(f'{type_str}.{cls_name}: {location_str}') from e

type: pyaml.pyaml
instruments:
  - type: pyaml.instrument
    name: sr
    energy: 6e9
    simulators:
      - type: pyaml.lattice.simulator
        lattice: sr/lattices/ebs.mat
        name: design
    data_folder: /data/store
    array: # Error here (array instead of arrays)
      - type: pyaml.arrays.hcorrector
        name: HCORR
        elements:
          - SH1A-C01-H
          - SH1A-C02-H
      - type: pyaml.arrays.vcorrector
        name: VCORR
        elements:
          - SH1A-C01-V
          - SH1A-C02-V
    devices:
      - sr/quadrupoles/QF1AC01.yaml
      - sr/correctors/SH1AC01.yaml
      - sr/correctors/SH1AC02.yaml

Traceback (most recent call last):
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 87, in build_object
    cfg = config_cls.model_validate(d)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pydantic/main.py", line 705, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ConfigModel
array
  Extra inputs are not permitted [type=extra_forbidden, input_value=[<pyaml.arrays.hcorrector...ject at 0x7f44ecd407d0>], input_type=list]
    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 135, in depth_first_build
    obj = self.build_object(d)
          ^^^^^^^^^^^^^^^^^^^^
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 99, in build_object
    raise PyAMLException(f'{type_str}.{cls_name}: {location_str}') from e
pyaml.exception.PyAMLException: pyaml.instrument.Instrument:  at line 3, column 5.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 111, in depth_first_build
    obj = self.depth_first_build(e)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 137, in depth_first_build
    raise PyAMLConfigException(None, pyaml_ex) from pyaml_ex
pyaml.configuration.config_exception.PyAMLConfigException: An exception occurred while building object: pyaml.instrument.Instrument:  at line 3, column 5.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 125, in depth_first_build
    obj = self.depth_first_build(value)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 114, in depth_first_build
    raise PyAMLConfigException(f"[{index}]", pyaml_ex) from pyaml_ex
pyaml.configuration.config_exception.PyAMLConfigException: An exception occurred while building key '[0]': An exception occurred while building object: pyaml.instrument.Instrument:  at line 3, column 5.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/esrf/pons/pyaml/tests/test_jlp.py", line 8, in <module>
    ml:PyAML = pyaml("tests/config/sr.yaml")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/pyaml.py", line 48, in pyaml
    aml:PyAML = depthFirstBuild(aml_cfg)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/operation/common/miniconda/envs/jlp-py312/lib/python3.12/site-packages/pyaml/configuration/factory.py", line 129, in depth_first_build
    raise PyAMLConfigException(key, pyaml_ex) from pyaml_ex
pyaml.configuration.config_exception.PyAMLConfigException: An exception occurred while building key 'instruments.[0]': pyaml.instrument.Instrument:  at line 3, column 5.

[JeanLucPons]

I investigated using PyYAML instead ruamel.yaml and the result is the same. No way to tag fields 👎
However the way to handle line mapping is much more simple using PyYAML.

class SafeLineLoader(SafeLoader):
    def construct_mapping(self, node, deep=False):
        mapping = super(SafeLineLoader, self).construct_mapping(node, deep=deep)
        # Add 1 so line numbering starts at 1
        mapping['__location__'] = (node.start_mark.line + 1 , node.start_mark.column + 1) 
        return mapping
    
# YAML loader
class YAMLLoader(Loader):
    
    def load(self,fileName:str) -> Union[dict,list]:
        self.path:Path = get_root_folder() / fileName
        self.suffixes = [".yaml"]
        with open(self.path) as file:
            try:
                return self.expand(yaml.load(file,Loader=SafeLineLoader))
            except yaml.YAMLError as e:
                raise Exception(self.path + ": " + str(e))

[JeanLucPons]

I finally found a way to get a complete mapping using PyYAML.
Just need to create a field list with locations, extract fields on model validation error and attach error location. 😨

    def construct_object(self, node, deep=False):
        obj = super(SafeLineLoader, self).construct_object(node, deep=deep)
        print(f"{obj}: {(node.start_mark.line + 1 , node.start_mark.column + 1)}")
        return obj

[JeanLucPons]

I finally used the mapping bellow which seems rather good.
However, mapping for lists is still not implemented and, using list, location still point to the root object in case of error.

    def construct_mapping(self, node, deep=False):
        mapping = {}
        field_mapping = {}

        for key_node, value_node in node.value:
            key = self.construct_object(key_node, deep=deep)
            if not isinstance(key, collections.abc.Hashable):
                raise ConstructorError("while constructing a mapping", node.start_mark,
                        "found unhashable key", key_node.start_mark)
            value = self.construct_object(value_node, deep=deep)
            mapping[key] = value
            field_mapping[key] = (self.filename, value_node.start_mark.line + 1 , value_node.start_mark.column + 1)

        # Add location information inside the dict
        mapping['__location__'] = (self.filename, node.start_mark.line + 1 , node.start_mark.column + 1) 
        mapping['__fieldlocations__'] = field_mapping
        return mapping