DBEntry.get() fails after transferring IMAS HDF5 data through HSDS

[satelite2517]

Hi, thanks for developing and maintaining IMAS-Python.

Summary

We are currently working on replacing an OMAS HDF5–based remote HSDS database with an IMAS-Python HDF5 backend in the database platform used for the VEST device.
HSDS (Highly Scalable Data Service) provides REST-based access to HDF5 data stored on distributed or cloud storage systems. Our goal is to store IMAS IDS data locally using the IMAS HDF5 backend, upload the resulting HDF5 files to an HSDS server, and later download and read them again using imas.DBEntry.get().
However, after transferring the file through HSDS, reading the IDS fails with an ImasCoreBackendException.

What we did

We create and store an IDS using the IMAS HDF5 backend:

import imas
import os

factory = imas.IDSFactory()
pf_active = factory.pf_active()

pf_active.ids_properties.homogeneous_time = imas.ids_defs.IDS_TIME_MODE_HOMOGENEOUS
pf_active.time = [0.01]

dir_path = "/tmp/tet"
os.makedirs(dir_path, exist_ok=True)

with imas.DBEntry("imas:hdf5?path=" + dir_path, "w") as dbentry:
    dbentry.put(pf_active)

The file appears correct when inspected using h5py:

import h5py

f = h5py.File("/tmp/tet/pf_active.h5")
print(list(f.keys()))

Output:

['pf_active']

Problem

When trying to read the data again:

with imas.DBEntry("imas:hdf5?path=/tmp/tet", "r") as dbentry:
    ids = dbentry.get("pf_active")

we get the following error:

ImasCoreBackendException:

Question

Even though we upload and download the exact same file, it becomes unreadable by imas.DBEntry.get().

Is there any additional requirement (e.g., metadata, directory structure, or backend-specific information) that must be preserved when transferring IMAS HDF5 data between systems (for example through HSDS)?

[olivhoenen]

Hi @satelite2517,

Before digging in, please note that you don't need to call the os.makedirs prior to creating your DBEntry, it will create needed folders automatically.

Do you have any exception message in your traceback?
Do you have a master.h5 file in your folder /tmp/tet when reading back the data? A DBEntry with HDF5 backend always expect a master.h5 file that has external links to <idsname>.h5 files containing the data for each IDS type.

[satelite2517]

Hi @olivhoenen
Thanks for the reply, and sorry for not providing enough details earlier.

Yes, the master.h5 file is also uploaded and downloaded together with <idsname>.h5.

Our workflow is roughly the following:

1. Create and store an IDS using the IMAS HDF5 backend:

factory = imas.IDSFactory()
equilibrium = factory.equilibrium()

equilibrium.ids_properties.homogeneous_time = imas.ids_defs.IDS_TIME_MODE_HOMOGENEOUS
equilibrium.ids_properties.comment = "testing"
equilibrium.time = [0.01]

dir_path = "./test"
with imas.DBEntry("imas:hdf5?path=" + dir_path, "w") as dbentry:
    dbentry.put(equilibrium)

2.	Upload both equilibrium.h5 and master.h5 to HSDS using hsload. / Download them again using hsget into another directory .

import subprocess
def _ids_top_level_name(ids_obj):
    name = getattr(getattr(ids_obj, "metadata", None), "name", None)
    if name:
        return name
    cls = type(ids_obj).__name__
    return cls

ids_name = _ids_top_level_name(equilibrium)
filename = f"{ids_name}.h5"
remote_uri = f"hdf5://admin/{filename}"
command = ["hsload","./test/"+filename, remote_uri]
subprocess.run(command, capture_output=False, text=True, check=True)

#master.h5
remote_uri = f"hdf5://admin/master.h5"
command = ["hsload","./test/master.h5", remote_uri]
subprocess.run(command, capture_output=False, text=True, check=True)

command = ["hsget" ,"hdf5://admin/equilibrium.h5", "./test_r/equilibrium.h5"]
subprocess.run(command, capture_output=False, text=True, check=True)

command = ["hsget" ,"hdf5://admin/master.h5", "./test_r/master.h5"]
subprocess.run(command, capture_output=False, text=True, check=True)

3.	Try to read the data again:

uri = "imas:hdf5?path=./test_r/"
with imas.DBEntry(uri, "r") as dbentry:
    eq = dbentry.get("equilibrium")

However, this fails when opening the DBEntry with the following error:

---------------------------------------------------------------------------
ImasCoreBackendException                  Traceback (most recent call last)
Cell In[22], [line 3](vscode-notebook-cell:?execution_count=22&line=3)
      1 uri = "imas:hdf5?path=./test_r/"
----> [3](vscode-notebook-cell:?execution_count=22&line=3) with imas.DBEntry(uri, "r") as dbentry:
      4     eq = dbentry.get("equilibrium")

File ~/.local/lib/python3.10/site-packages/imas/db_entry.py:187, in DBEntry.__init__(self, dd_version, xml_path, *args, **kwargs)
    185 self.mode = mode
    186 cls = self._select_implementation(self.uri)
--> [187](https://vscode-remote+ssh-002dremote-002buser1.vscode-resource.vscode-cdn.net/home/user1/h5pyd/vaft/notebooks/~/.local/lib/python3.10/site-packages/imas/db_entry.py:187) self._dbe_impl = cls.from_uri(self.uri, mode, self._ids_factory)

File ~/.local/lib/python3.10/site-packages/imas/backends/imas_core/db_entry_al.py:88, in ALDBEntryImpl.from_uri(cls, uri, mode, factory)
     86     modes = list(_OPEN_MODES)
     87     raise ValueError(f"Unknown mode {mode!r}, was expecting any of {modes}")
---> [88](https://vscode-remote+ssh-002dremote-002buser1.vscode-resource.vscode-cdn.net/home/user1/h5pyd/vaft/notebooks/~/.local/lib/python3.10/site-packages/imas/backends/imas_core/db_entry_al.py:88) return cls._from_uri(uri, _OPEN_MODES[mode], factory)

File ~/.local/lib/python3.10/site-packages/imas/backends/imas_core/db_entry_al.py:144, in ALDBEntryImpl._from_uri(cls, uri, mode, factory)
    141 backend = urlparse(uri).path.lower().lstrip("/")
    142 cls._setup_backend(backend, mode, factory)
--> [144](https://vscode-remote+ssh-002dremote-002buser1.vscode-resource.vscode-cdn.net/home/user1/h5pyd/vaft/notebooks/~/.local/lib/python3.10/site-packages/imas/backends/imas_core/db_entry_al.py:144) status, ctx = ll_interface.begin_dataentry_action(uri, mode)
    145 if status != 0:
    146     raise LowlevelError("opening/creating data entry", status)

File _al_lowlevel.pyx:318, in imas_core._al_lowlevel.al_begin_dataentry_action()

ImasCoreBackendException: b'al_begin_dataentry_action: [ALBackendException = Unable to read attribute: HDF5_BACKEND_VERSION\n]'

Both master.h5 and equilibrium.h5 are present in the directory I’ve tried to read.

Could this indicate that some attribute or metadata required by the IMAS HDF5 backend (such as HDF5_BACKEND_VERSION) is not preserved when transferring the files through HSDS?

[olivhoenen]

Thanks for the information, could you also please run h5dump --superblock --header on the original master.h5 and compare it with the file you have after hsget?

[satelite2517]

This log is for the original file

$ h5dump --superblock --header master.h5 
HDF5 "master.h5" {
SUPER_BLOCK {
   SUPERBLOCK_VERSION 0
   FREELIST_VERSION 0
   SYMBOLTABLE_VERSION 0
   OBJECTHEADER_VERSION 0
   OFFSET_SIZE 8
   LENGTH_SIZE 8
   BTREE_RANK 16
   BTREE_LEAF 4
   ISTORE_K 32
   FILE_SPACE_STRATEGY H5F_FSPACE_STRATEGY_FSM_AGGR
   FREE_SPACE_PERSIST FALSE
   FREE_SPACE_SECTION_THRESHOLD 1
   FILE_SPACE_PAGE_SIZE 4096
   USER_BLOCK {
      USERBLOCK_SIZE 1024
   }
}
GROUP "/" {
   ATTRIBUTE "HDF5_BACKEND_VERSION" {
      DATATYPE  H5T_STRING {
         STRSIZE 20;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
   }
   EXTERNAL_LINK "equilibrium" {
      TARGETFILE "./equilibrium.h5"
      TARGETPATH "equilibrium"
         GROUP "equilibrium" {
            DATASET "ids_properties&comment" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&homogeneous_time" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&version_put&access_layer" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&version_put&access_layer_language" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&version_put&data_dictionary" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "time" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SIMPLE { ( 1 ) / ( H5S_UNLIMITED ) }
            }
         }
   }
}
}

And this part below is with the new file, which was made with hsget:

$ h5dump --superblock --header master.h5 
HDF5 "master.h5" {
SUPER_BLOCK {
   SUPERBLOCK_VERSION 0
   FREELIST_VERSION 0
   SYMBOLTABLE_VERSION 0
   OBJECTHEADER_VERSION 0
   OFFSET_SIZE 8
   LENGTH_SIZE 8
   BTREE_RANK 16
   BTREE_LEAF 4
   ISTORE_K 32
   FILE_SPACE_STRATEGY H5F_FSPACE_STRATEGY_FSM_AGGR
   FREE_SPACE_PERSIST FALSE
   FREE_SPACE_SECTION_THRESHOLD 1
   FILE_SPACE_PAGE_SIZE 4096
   USER_BLOCK {
      USERBLOCK_SIZE 0
   }
}
GROUP "/" {
   ATTRIBUTE "HDF5_BACKEND_VERSION" {
      DATATYPE  H5T_STRING {
         STRSIZE 3;
         STRPAD H5T_STR_NULLPAD;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
   }
   EXTERNAL_LINK "equilibrium" {
      TARGETFILE "./equilibrium.h5"
      TARGETPATH "equilibrium"
         GROUP "equilibrium" {
            DATASET "ids_properties&comment" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&homogeneous_time" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&version_put&access_layer" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&version_put&access_layer_language" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "ids_properties&version_put&data_dictionary" {
               DATATYPE  H5T_STRING {
                  STRSIZE H5T_VARIABLE;
                  STRPAD H5T_STR_NULLTERM;
                  CSET H5T_CSET_UTF8;
                  CTYPE H5T_C_S1;
               }
               DATASPACE  SCALAR
            }
            DATASET "time" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SIMPLE { ( 1 ) / ( H5S_UNLIMITED ) }
            }
         }
   }
}
}

[olivhoenen]

Thanks, indeed the user block seems affected by this transfer to and from HSDS.
I moved this issue to IMAS-Core, as this related to the HDF5 backend and not to IMAS-Python.

[satelite2517]

Thanks!

[prasad-sawantdesai]

Thank you, @satelite2517. Is it possible to attach the HDF5 files received from the HSDS service? I’m also wondering if there are any changes in pf_active.h5

[satelite2517]

Hi @prasad-sawantdesai, Thanks for the help.
Here are the files. I couldn’t attach the .h5 files directly, so I compressed them into a .zip archive.

HSDS-imas.zip

The “(hsds)” part at the end of the filenames was added by me for clarity and is not part of the original filenames.
The pf_active.h5 file also appears to have a similar issue.

[prasad-sawantdesai]

Thank you. Could you please try using appropriate wheel from this page and test. Download zip according to the Python version and Operating system.

https://github.com/iterorganization/IMAS-Core/actions/runs/23001945704

In the environment you were testing

pip uninstall imas_core
pip install imas_core-5.6.1.dev3-cp313-cp313-macosx_14_0_arm64.whl

>>python
import imas_core
imas_core.__version.__

[satelite2517]

I tried testing with the wheel you suggested, but now put() fails entirely on my side.
I tested with both Python 3.10 and 3.12(on linux). The versions I am using are:
• imas_core = 5.6.1.dev3
However, when I try a simple dbentry.put(...) with a fresh directory, it fails with:

---------------------------------------------------------------------------
ImasCoreBackendException                  Traceback (most recent call last)
Cell In[9], line 17
     15 dir_path = "./test2"
     16 with imas.DBEntry("imas:hdf5?path=" + dir_path, "w") as dbentry:
---> 17     dbentry.put(equilibrium)
     18     dbentry.put(pf_active)
     19 print("Saved to HDF5:", dir_path)

File ~/miniconda3/lib/python3.12/site-packages/imas/db_entry.py:651, in DBEntry.put(self, ids, occurrence)
    631 def put(self, ids: IDSToplevel, occurrence: int = 0) -> None:
    632     """Write the contents of an IDS into this Database Entry.
    633 
    634     The IDS is written entirely, with all time slices it may contain.
   (...)
    649             imas_entry.put(ids)
    650     """
--> 651     self._put(ids, occurrence, False)

File ~/miniconda3/lib/python3.12/site-packages/imas/db_entry.py:739, in DBEntry._put(self, ids, occurrence, is_slice)
    736     version_put.access_layer = self._dbe_impl.access_layer_version()
    737     version_put.access_layer_language = f"IMAS-Python {imas.__version__}"
--> 739 self._dbe_impl.put(ids, occurrence, is_slice)

File ~/miniconda3/lib/python3.12/site-packages/imas/backends/imas_core/db_entry_al.py:322, in ALDBEntryImpl.put(self, ids, occurrence, is_slice)
    316         raise DataEntryException(
    317             f"Cannot change homogeneous_time from {db_time_mode} to {time_mode}"
    318         )
    320 if not is_slice:
    321     # put() must first delete any existing data
--> 322     with self._db_ctx.global_action(ll_path, WRITE_OP) as write_ctx:
    323         # New IDS to ensure all fields in "our" DD version are deleted
    324         # If ids is in another version, we might not erase all fields
    325         delete_children(self._ids_factory.new(ids_name).metadata, write_ctx)
    327 if is_slice:

File ~/miniconda3/lib/python3.12/site-packages/imas/backends/imas_core/al_context.py:74, in ALContext.global_action(self, path, rwmode, datapath)
     63 def global_action(self, path: str, rwmode: int, datapath: str = "") -> "ALContext":
     64     """Begin a new global action for use in a ``with`` context.
     65 
     66     Args:
   (...)
     72         The created context.
     73     """
---> 74     status, ctx = ll_interface.begin_global_action(self.ctx, path, rwmode, datapath)
     75     if status != 0:
     76         raise LowlevelError("global_action", status)

File imas_core/_al_lowlevel.pyx:382, in imas_core._al_lowlevel.al_begin_global_action()

ImasCoreBackendException: b'al_plugin_begin_global_action: [ALBackendException = Unable to read attribute: HDF5_BACKEND_VERSION\n]'
Error status=-3

[prasad-sawantdesai]

Thank you for testing. Indeed, that was the issue. I have fixed it

You will find wheels here -> https://github.com/iterorganization/IMAS-Core/actions/runs/23007164059

Please let me know if the issue still persists.

Regards,

[satelite2517]

Hi, @prasad-sawantdesai
I tested the new wheel, and it works perfectly now. Thanks a lot for fixing this so quickly!! 👍
Do you plan to include this fix in a future release? Also, just out of curiosity, what exactly was causing the issue?

Best,