X axis doesn't work for ScSm (multiple spectra, e.g. Time-Resolved Step Scan)

[ChrisHodgesUK]

I've got time resolved Stepscan data in an ScSm data block. This is a 2d dataset with a time dimension as well as a spectral one. get_range returns far too many X values, quite possibly by a factor of the number of time steps. I might be able to fix it, if there's enough metadata in there to extract the number of timeslices.

[tdyku]

Hi @ChrisHodgesUK !
Do you have an example of the opus file that is not working for you?

[ChrisHodgesUK]

Hi Tomasz,

Part of the issue is that PIP gave me an old version, and the fix to trust NPT wasn't included. That gets me a little further but then the data can't be reshaped into a 2d array because the number of values in ScSm isn't a multiple of NPT (it's just over NPT * 2 * number_of_timeslices). It doesn't look like simply a bunch of excess points after each NPT points, though I can't yet rule it out ; at the end I have values like 10^19 and nan, and this is single channel data so there should be no divide-by-zero issues. Unfortunately I can't export the same data as DPT (ASCII) because OPUS throws an error, though it displays fine.

A sample file is at https://www.dropbox.com/s/7ff8qh2xw4x9jq9/ScSm_demo.0?dl=0 but it's 40MB - it should have 12444 datapoints in each of 340 timeslices, from 600 to 3000 cm-1. I'm going to get something smaller tomorrow when I'm in front of the instrument again - I need it for debugging and there's a better chance of DPT export so I an check for consistency.

From looking at the source and viewing the file parameters in OPUS, I think we'd need to read the "Trace Data Parameters" data block to extract the time information, but that still doesn't help if it reshape can't map onto an NPT*n array

[ChrisHodgesUK]

I've generated a more sensible file for debugging: https://www.dropbox.com/s/enjgkztnkj3vxec/100cm-1_test.0?dl=0 along with a .dpt ascii file of the same data: https://www.dropbox.com/s/5y78nvslucmaleh/100cm1_test.dpt?dl=0

With this I hope I'll be able to figure out where the junk is and maybe propose a fix

[ChrisHodgesUK]

Progress so far: In the data linked in my previous comment ScSm contains 8 junk elements at the start, 195 good ones (matching NPT), 39 junk, 195 good etc. I haven't checked the whole file, just the top few spectra and the last data value expected from this pattern; with 100 time slices, 8+(195+39)*100=23408=len(opus_data["ScSm"]) which is good. Now I'd like to figure out where the 39 nonsense values come from, and whether it's always 39. They range from 1e-43 to 4e19 and include 0.0... and nan. Unfortunately they includes plausible data values so we can't filter them out. I'll grab another few datasets today while I've got the instrument, varying the resolution and the number of slices

Also, the number of timeslices is stored in opus_data["Acquisition"]["WAS"] so we don't need to parse the "Trace" block to get this info; indeed it isn't always present

[ChrisHodgesUK]

I've taken a load of debug data for various resolutions and times, with ascii (.dpt) files exported as well for comparison. I'm hoping to have a good look at it over the next few days.
https://www.dropbox.com/s/kqhpg4r3sp2hata/OPUS_reader_debug_data.zip?dl=0

[ChrisHodgesUK]

I'm nearly there with a new parse_sm function for ScSm and IgSm blocks. Testing/tweaking will hopefully be finished tomorrow and when it is I'll submit PR with the function and test code. My earlier comment had an off-by-one error - it's 38 lines of junk, not 39 between timeslices, but it's consistent across all my tests including for IgSm. A 1-slice TR spectrum is a special case, that I think can be treated the same as Sc data though it's in an ScSm block