CIM XML (IEC 61970-552) as Lang based on current RDF/XML parser (read-only at first)

[arne-bdt]

The ENTSO-E RDF-SYNTAX USER GUIDE describes some key differences between CIM XML (IEC 61970-552) and RDF XML (W3C).

Currently, CIM XML is the dominant format for IT processes and standards defined by ENTSO-E for the European transport system operators. The most important is the CGMES (Common Grid Model Exchange Standard), but there are many additional application profiles with published RDF Schemas and SHACL constraints.

The European Network of Transmission System Operators for Electricity (ENTSO-E) and its member TSOs are among the most important drivers of the energy transition in the European electricity grid. A working toolchain for CIM XML would greatly support these projects. It could also lead to more software architectures that are truly graph-based and oriented on RDF, SPARQL, and SHACL.

In my opinion, it would help a lot if there was a Lang/parser in Jena that uses a given RDF Schema to determine the datatypes for parsing.

That parser could also provide the missing xml:base from the corresponding RDF Schema as a default.

I already have the necessary code snippets and sample data to build such a parser.

My current implementations all work as an implementation of StreamRDF. But that causes most of the object literal nodes and their triples to be created twice, once as a string literal and once parsed with the correct datatype.
An efficient implementation could modify the determination of the datatype in org.apache.jena.riot.lang.rdfxml.rrx.ParserRDFXML_SAX#startPropertyElement. I would like to avoid a second parser implementation and duplicated tests when the difference is so small. But I read the comment "parsing is sensitive to the JIT optimizer" by @afs on the mailing list yesterday, so I am not sure about the best approach here.

What do you think? Would the project benefit from CIM XML as Lang? What else should I consider?

Note: A corresponding CIM XML writer could be implemented later. But that would be another discussion.

[afs]

To meet the objective of precisely following CIM XML then there should be a separate Lang with a derived RDF/XML parser.

Adding a Lang+parser is not so bad. Only differences need new tests.

There are items in CIM that are not correct RDF/XML.

While PR #2477 can be justified on the grounds of "compatibility", if the claim is "Jena parses CIM", there needs to be detailed compliance. The way to do that is parser adjusted for CIM, derived (subclass) from RRX. That gives the freedom for precise control.

What do other CIM parsers do?

Do they accept arbitrary (non-CIM) RDF/XML or enforce CIM features?

I would like to avoid a second parser implementation and duplicated tests when the difference is so small.

What I want to avoid is CIM features/profile of RDF/XML leaking back into general use. The Postel's law argument of "be conservative in what you send, be liberal in what you accept" only goes so far when people look to Jena as an accurate implementation of the W3C standards.

[afs]

This feels like it would fit in nicely as a new "extras" module jena-extras/jena-cim-xml.

[arne-bdt]

I have been working with CIMXML and CGMES since 2015, and with Apache Jena since 2017. Only the RDF-Syntax User Guide from 2024 made me realize that I have not been too stupid to use Apache Jena properly all that time.
We discovered a workaround for every problem but never doubted the specifications, which stated, for example:

However, the CIMXML format (described in IEC 61970-552:2016) is based on RDF technology
and RDF-compliant software natively can validate CIMXML files against profiles,
or rather an ontology generated from the profile. (source)

The IEC standard claims they use a simplified RDF syntax, which is a proper subset of the RDF standard and can be deserialized by RDF software like SirPAC.

What do other CIM parsers do?
Do they accept arbitrary (non-CIM) RDF/XML or enforce CIM features?

--> So I am not sure if there are many real CIM parsers or rather a lot of pre- and post-processing around RDF parsers.

What I want to avoid is CIM features/profile of RDF/XML leaking back into general use.

I had absolutely no intention to make the RDF/XML parser more general or mix implementations. I wanted to know if a copy of the RDF parser was necessary or if there are any pain points when I inherit the parser, as I would need to split some methods and make them protected.

Implementation Levels

There are different levels at which CIMXML could be supported:

1. Datatype Parsing Only
   • Reading all predicates and the corresponding datatypes from one or more RDFS files (there is an RDFS for the file header and one for the content).
   • For each triple, trying to parse with the matching datatype.
   • There could be warnings if a property was not part of the RDFS.
   • There should be errors when a property could not be properly parsed.
2. Semantic Matching of CIMXML File Header model.profile Against the RDFS owl:versionIRI
   • The given RDFS file must match the profile of the CIMXML file.
   • Maybe a schema registry would be nice, where the corresponding profile is looked up by the header data.
3. Variant of 2, Where the File Header Data is Read into a Different Graph
   • This makes sense since a separation of the file header is already planned when switching to JSON-LD.

In my opinion, level 1 would be best suited for Jena. It would pave the way and remove the most disturbing hurdles. That would suffice to load CIMXML with parsed datatypes into Jena. With a proper xml:base, users would stop removing the source path from every subject's IRI with code like BIND (strafter(str(?ConductingEquipment), "#") AS ?eqMRID) to join data from two different CIMXML files.

xml:base

In QUALITY OF CGMES DATASETS AND CALCULATIONS, some specialties around rdf:ID/rdf:about are explained, and they interestingly define xml:base=“urn:uuid:”. --> I have not tried using this as a base for the parser yet.

[VladimirAlexiev]

@arne-bdt

QUALITY OF CGMES DATASETS AND CALCULATIONS they interestingly define xml:base="urn:uuid:"

Interesting indeed, because you cannot use "urn:uuid:" as base:
Sveino/Inst4CIM-KG#159

[afs]

Yes - that one has strange effects.

This:

PREFIX uuid: <urn:uuid:>
uuid:df89cc9d-84f3-11f1-bd6c-7486e2254909 :property "object" .

is legal as URI - it has a scheme violation at <urn:uuid:> (legal basic URI, bad by URN uuid rules).

However, the trick involved is going to cause all sorts of problems!

[arne-bdt]

Based on ParserRRX_StAX_SR.java, I started a separate module "jena-iec61970-552" in a branch .
I failed to derive from the original parser as the necessary changes went too deep. So I copied and adopted it.

For several reasons I don't think it should be a Jena module for now:

1. There may be frequent changes and/or fixes needed in the coming months, that do not match the Apache Jena release cycle.
2. I am not sure, if the CGMES profiles are 100% IEC compliant or just a European variation of the original CIM. CIMXML parser itself should be IEC compliant, but it needs these profiles.
3. The parser does something that Jena would not: It fixes incoming UUIDs (with uppercase or missing dashes) to always get valid "urn:uuid:..." IRIs. Otherwise the CGMES 2.4.15 "Conformity Assessment Scheme Test Configurations" and data from https://energy.referencedata.eu/ could not be used.
4. It might help the OpenCGMES project to get attraction.

So for now, I derived the OpenCGMES IEC 61970-552 CIMXML Parser from the "jena-iec61970-552" module in my branch.

[afs]

2.I am not sure, if the CGMES profiles are 100% IEC compliant or just a European variation of the original CIM. CIMXML parser itself should be IEC compliant, but it needs these profiles.

Profiles means a subset of full RDF/XML? i.e. there is legal RDF/XML that is not legal as CIMXML.

3. The parser does something that Jena would not: It fixes incoming UUIDs (with uppercase or missing dashes) to always get valid "urn:uuid:..." IRIs. Otherwise the CGMES 2.4.15 "Conformity Assessment Scheme Test Configurations" and data from https://energy.referencedata.eu/ could not be used.

That can be done in the StreamRDF that is the compiler output.

[arne-bdt]

The IEC CIM "profiles" are regular RDF Schema using regular RDF/XML.
They describe classes, inheritance, properties, datatypes, relations etc.
The datatypes are mostly special datatypes like "ActivePower" which typically can be mapped to a XSDDatatype like XSDfloat.

CIMXML was planned to be a subset of RDF/XML. The idea was, that it needed to be readable as regular XML. For older CGMES releases, they provided XSD-Schema definitions to use, e.g. to build parsers.
CIMXML requires the syntax to be flat. Only resource + property is allowed, no nested resources like in RDF/XML.
--> so there is legal RDF/XML, which is not legal as CIMXML.

There is more:

• CIMXML has a special syntax for URN forms where the prefix "urn:uuid:" is replaxed by "_".
• For DifferenceModels (basically the additon and deletion of Delta in Jena) they introduced rdf:parseType="Statements" which should create a sub graph with the contained statements.
  --> so at least these DifferenceModels cannot be expressed as one graph but as multiple graphs.
• CIMXML does not contain any rdf:datatype. The parser is supposed to enhance the data using the datatypes from the matching profile definitions. This requires to look into the "md:Model" header data and find the referenced "md:Model.profile"s.

The old format UCTE, that is currently used to exchange Transport System Operator (TSO) grid data is basically an ASCII format for punch cards. One TSO grid today is between 600-900 kB ASCII. Roughly the the same data as CIM/CGMES is 40-60 MB in CIMXML.
CIM/CGMES is able to describe the grid in much more detail, which is needed for e.g. redispatch and topology optimization. At this detail level, I expect the data for one grid to grow up to 140-250 MB of CIMXML.
There are 35+ interconnected TSOs in Europe. For many processes like the coordinated security analysis all these grids are needed at once and not only for one timestamp but typically for 24h.
So we have roughly 35 TSOs * 160 MB * 24h of CIMXML data, that we need to parse and then work with. These coordination process typically have limited time windows between 15-30 minutes for reading all the data over calculation to grid operator decisions.

While we used StreamRDF in the past to fix each resource node and each typed literal node, this is slow and requires many new object creations. The cimxml parser fixes this and is optimized for performance.
For DifferenceModels the problem was that the nested graphs were first parsed as XML literals and then needed to be parsed again as RDF/XML.
The cimxml parser uses aalto-xml instead of xerces, which showed to be bit faster for most of the relevant larger graphs.
On my machine, the Jena RDFParser takes about 40s to read bsbm-25m.xml into GraphMem2Roaring where the new parser takes only 30s. For CIMXML the speed up compared to our previous implementations is >2x.
The average speed on my machine (13th Gen Intel(R) Core(TM) i9-13950HX, 2200 MHz, 24 Cores) now is about 0.7-0.83 million triples per second for parsing large RDF/XML or CIMXML files into GraphMem2Roaring.

Many software projects around CIM/CGMES had bad experiences when they tried to work with CIMXML as RDF. I hope the cimxml parser might ease the pain for future projects in that area.
This week I spoke about CGMES and CIMXML at the LF Energy Summit 2025 in Aachen: Breaking Down CGMES Barrieres

[VladimirAlexiev]

@arne-bdt: I made similar fixes towards in the fall of 2024.
They use external tools and thus are very short:

• https://github.com/Sveino/Inst4CIM-KG/blob/develop/rdf-improved/cim-urn-uuid.pl
  perl script that replaces the defective CIM XML URLs with "urn:uuid:"
• https://github.com/Sveino/Inst4CIM-KG/blob/develop/rdf-improved/cim-trig.pl
  perl script that converts CIM XML to Trig (turlte with graphs) by splitting a DifferenceModel into forward/backwardDifferences graphs
• https://github.com/Sveino/Inst4CIM-KG/blob/develop/rdf-improved/fix-datatypes-new.ru:
  SPARQL update that adds datatypes based on a long list of properties.
  Can be used on the command line (eg with jena's update) or in a semantic repository

using the datatypes from the matching profile definitions. This requires to look into ... "md:Model.profile"

• the values are like md:Model.profile <http://iec.ch/TC57/ns/CIM/Topology-EU/3.0>
• Actual ontology URLs are like <http://iec.ch/TC57/ns/CIM/Topology-EU#Ontology>
• whereas owl:versionIRI <http://iec.ch/TC57/ns/CIM/Topology-EU/3.0> so this can be matched to md:Model.profile

Many props are redundantly defined in multiple profiles. So I use a global list of props.
One needs to be vey careful about the cim (and nc) namespaces that have been changing in the past.

datatypes are mostly special datatypes like "ActivePower" which typically can be mapped to a XSDDatatype like XSDfloat

CIM datatypes are modeled wrong, see https://github.com/Sveino/Inst4CIM-KG/tree/develop/rdfs-improved#quantitykinds-and-units-of-measure

https://github.com/Sveino/Inst4CIM-KG/blob/develop/rdfs-improved/fix-all.ru fixes this (and a number of other problems in the ontologies)

Cheers!

[arne-bdt]

Thank you @VladimirAlexiev.
Especially for the Link to your fixes for the datatypes. I will check If we need to update https://github.com/SOPTIM/OpenCGMES/tree/main/cimxml.