Best practice for handling unevenly sampled (jagged) observations in Black-Box models? (SimpleBbAsciiFile length constraint & Documentation inquiry)

[hurys20]

Hello OpenDA developers/community,

I am currently integrating a 2D hydrodynamic and water quality model (CE-QUAL-W2) into OpenDA using the BlackBoxWrapper. Since the model is not natively supported, I am using Python scripts to bridge the model I/O with OpenDA.

Currently, I am using org.openda.blackbox.io.SimpleBbAsciiFile for model results and noosObserver for observations. However, I have hit a severe architectural limitation regarding real-world, unevenly sampled data.

The Physical Scenario (The "Jagged Data" Problem):
In our real-world reservoir monitoring (multiple sites and multiple depths), the sampling frequency is naturally uneven.
For example, within a 60-day simulation window:

Site_A_Depth_0.5m might have 15 observations.

Site_B_Depth_10.0m might have 8 observations.

Site_C_Depth_90.0m might only have 2 observations.

The Technical Bottleneck:
My Python bridge successfully extracts a full, continuous 60-day time series for all simulation points and writes them into model_results.output. However, SimpleBbAsciiFile seems to enforce a strict length contract based on the .noo files.

If the .noo file for Site_B has 8 records, OpenDA throws an error when parsing the 60-day model output:

“Error preparing algorithm.
Error message: expecting vector of length 8 for time, but length was 60
...
at org.openda.blackbox.io.SimpleBbAsciiFile.initialize”

Workarounds we considered (but are not ideal):

Intersection Method: Only keep the exact dates where all sites were simultaneously monitored. (This forces all .noo files to be the exact same length, but we lose a massive amount of valuable field data).

Scalarization Method: Abandon timeSeries entirely and treat every single observation point at every specific timestamp as an independent, length-1 scalar variable (e.g., SiteA_Day1, SiteA_Day2). This explodes the XML configuration and loses the semantic meaning of a time series.

My Questions:

Advanced I/O Handling: Is there a more advanced generic IO class (e.g., a generic NetCDF wrapper) or a specific Observation Operator configuration in OpenDA that allows us to feed a full, continuous model time series (e.g., length 60) and let OpenDA automatically interpolate or pick the matching timestamps based on the varying lengths of individual .noo files?

Best Practices: How do advanced models integrated via the Black-Box approach usually handle varying observation frequencies across different measurement vectors without triggering the length mismatch error?

Documentation & Manuals: Is there a continuously updated reference manual or comprehensive documentation for OpenDA's built-in wrapper classes and algorithms? We often find ourselves unsure about what newer classes might exist, what their underlying constraints/limitations are, how to correctly format their XML paradigms, and how they compare to one another. A detailed guide would greatly help us fully utilize the framework's potential.

Any guidance, documentation links, or examples pointing to a more advanced I/O handler for Black-Box models would be greatly appreciated. Thank you!

[nilsvanvelzen]

Hi hurts20,

Over 10 years ago I have made a CE-QUAL-W2 coupling with OpenDA. Unfortunately the client dit not put this code into the public domain. Advantage is that I do know some things about CE-QUAL-W2. I assume that a lot of CE-QUAL-W2 changes have been made over the years hence the old wrapper will not be of much use in 2026.

Other issue you might run into... If you want to do Kalman filtering you run into tilmestep challenges as a result of the adaptive time steps of the integration inside CE-QUAL-W2 :-). The output of CE-QUAL-W2 is interpolated in time but restarts do not always exactly correspond to output times. But that is a side note and only an challange for Kalman filtering (not calibration)

About your question. The problem is that you have time series for some locations but you have missing values in these series (or at least not all observations are always available at the same time.

My suggestion is to use for handling your observations. See e.g. course/exercise_black_box_calibration_polution. You just provide a time series file (time-value) for each observation location. Missing observations should be handled automatically. (otherwise we have an even more fancy NOOS observer that can definitely deal with it).

When the model provides the corresponding predictions at all times (which should not be a problem) then OpenDA (in the algorithm) should do the matching and ignore model results for which there is no observation available.

Most important tip is to use the models and configuration of course/exercise_black_box_calibration_polution and course/exercise_black_box_enkf_polution as a starting point/main example to try to get your model to run. (use the latest greatest version (main-branch) from GitHub)

For black box coupling there is no fancy interpolation/transformation. The trick is that your (wrapped) model should be able to produce the corresponding values in some file corresponding to the observations (but the model should be agnostic to missing values in the observation series some of the model output (predictions) will not be used when no corresponding observation is available.

I hope this kind of answers you question and helps you tackle this hurdle.

Best regards,
Nils

[theavuik]

Does this documentation website help? https://docs.openda.org/en/latest/
We are currently working on a general page about wrappers, but it is not yet ready (see issue #467)

Furthermore, the XSD schemas are posted on https://schemas.openda.org/