Incorrect Kinetic Energy Calculations

[enjyashraf18]

There is an issue with the calculation of kinetic_energy properties in run module.

During data migration, the computed values for kinetic energy do not match the expected values stored in the msgpack files. This mismatch causes all related tests in atomdb/test/test_wfn_slater to fail for multiple atomic species.

From the errors, it seems the problem arises from the kinetic energy density calculations or the integration logic within run module

Example failure

atomdb/test/test_wfn_slater.py

args = (0.14730019881254125, 37.708843614), kwds = {'decimal': 5}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
                   ^^^^^^^^^^^^^^^^^^^
E           AssertionError: 
E           Arrays are not almost equal to 5 decimals
E            ACTUAL: 0.14730019881254125
E            DESIRED: 37.708843614

All related failing tests

FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[He-0-1-2.861679997-5]
FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[Be-0-1-14.57302313-5]
FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[B-0-2-24.529060725-5]
FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[Cl-0-2-459.482072308-3]
FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[Ag-0-2-5197.698468984-2]
FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[C-1-2-37.292223758-5]
FAILED atomdb/test/test_wfn_slater.py::test_slater_positive_definite_kinetic_energy[C--1-4-37.708843614-5]

@gabrielasd @marco-2023 Could you please check this?

[PaulWAyers]

I remember there was some issue once before with this. Perhaps @gabrielasd remembers what it was. I'd also search the repo to see if there is another issue/discussion on the k.e.d..

[gabrielasd]

This is the old issue about a problem with the positive kinetic energy density in the Slater dataset:
#128

And here the PR that solved it:
#139

@enjyashraf18 is the problem you find when comparing to files in the test module? Could you compare against files in the AtomDBDataset repo?

[enjyashraf18]

This is the old issue about a problem with the positive kinetic energy density in the Slater dataset: #128

And here the PR that solved it: #139

@enjyashraf18 is the problem you find when comparing to files in the test module? Could you compare against files in the AtomDBDataset repo?

Yes, the problem appeared in the test modules.
But sure, I can provide a comparison between the files as well to check if there are any other mismatches not showing up in the test modules.

[gabrielasd]

How are you setting up the tests after the update to the database? By which I mean, what data are you validating (your new dataset ported to HDF5 format?), and against what are you comparing (the reference values set up in the tests or the .msg files in the test/data folder?) ?

For example, it can be that the problem are the reference values we took as the truth for the positive kinetic energy density.

This would be most likely my fault as I don't recall if I updated the tests after PR #139, and the reference data in the Slater test was derived based on the old code for the Slater dataset (here

One way to check if this is the problem is to compare against the data in the AtomDBData repo which was updated after the fix.
There is also Ali's reply where he gives link to his updated code for the Slater dataset (and also tests, I think):
#128 (comment)

[enjyashraf18]

How are you setting up the tests after the update to the database? By which I mean, what data are you validating (your new dataset ported to HDF5 format?), and against what are you comparing (the reference values set up in the tests or the .msg files in the test/data folder?) ?

For example, it can be that the problem are the reference values we took as the truth for the positive kinetic energy density.

This would be most likely my fault as I don't recall if I updated the tests after PR #139, and the reference data in the Slater test was derived based on the old code for the Slater dataset (here

One way to check if this is the problem is to compare against the data in the AtomDBData repo which was updated after the fix. There is also Ali's reply where he gives link to his updated code for the Slater dataset (and also tests, I think): #128 (comment)

I made a comparison between the ported data in the HDF5 files and the latest data files in the AtomDBData repo. The result is that all properties match exactly, except for ked_tot property.

Let’s take an example like Be_000_001_000

From datasets_data.h5

ked_tot: [848954.98337262  28677.91563367 848954.98337262 ...      0.
      0.              0.        ]

From the latest version of Be_000_001_000.msg

ked_tot: [2.83147907e+02 2.83147873e+02 2.83147839e+02 ... 2.70972040e-11
 2.62252138e-11 2.53800514e-11]

Was the data in the AtomDBData repo updated after the fix using the latest calculations from the run module?
If not, I think there might still be an issue in run

[gabrielasd]

Yes, the current data for the Slater set in AtomDBData was updated after that PR.

I am not sure yet if the problem is with the Slater run script. Isn't the run script being also ran as part of the data migration?
@enjyashraf18 could you compare your migrated data (say for that Be species) against the ked_tot data from the gaussian and/or HCI datasets? For example create a plot similar to the second one in Marco's PR #139 (comment): KED (log scale) vs radial distance.

@marco-2023 could you take a look into this? thanks

[enjyashraf18]

Yes, run script was included in the data migration process. the calculations run first, and then I ported the data into the HDF5 file.

I already checked ked_tot for the gaussian dataset, and everything matched fine there (cant do the same for hci yet, since its migration hasn’t been done as the raw files aren’t accessible)

Taking Be_001_002_000 from gaussian as an example
The comparison of ked_tot vs radial distance between the msgpack and HDF5 data shows they’re identical. So I think the main issue might be coming from the Slater run module

From HDF5 file

From msgpack file