From 9ba5d8c2a303e3d8743a8d6d227a95182f1dbe1e Mon Sep 17 00:00:00 2001 From: Thea Vuik Date: Fri, 27 Feb 2026 10:29:28 +0100 Subject: [PATCH 1/2] Add page on wrappers, data objects, exchange items, aliases and compute actions --- docs/source/DFlowFMWrapper.rst | 15 +-- docs/source/index.rst | 15 +-- docs/source/wrappers.rst | 162 +++++++++++++++++++++++++++++++++ 3 files changed, 175 insertions(+), 17 deletions(-) create mode 100644 docs/source/wrappers.rst diff --git a/docs/source/DFlowFMWrapper.rst b/docs/source/DFlowFMWrapper.rst index c68f631ba..bc762cfa2 100644 --- a/docs/source/DFlowFMWrapper.rst +++ b/docs/source/DFlowFMWrapper.rst @@ -85,6 +85,9 @@ OpenDA will overwrite the original ``.bc`` file. As an example, we provide:: NetCDF concatenater ------------------- + +.. _NetCDF concatenater: + OpenDA splits longer runs into smaller parts (as we already explained in the section on :ref:`restarts of a model`). Therefore, lots of smaller NetCDF output files are written instead of one single NetCDF output file that @@ -93,7 +96,8 @@ in order to merge them together, OpenDA has a NetcdfFileConcatenater. This is general functionality available in OpenDA and can be used for many different models that create NetCDF output files. This concatenater (formally spelled as concatenator) should be configured as compute action after the compute action -of the model run. [#action]_ It takes two arguments: the first argument is the name of +of the model run (see the section on :ref:`compute actions` for more information). +It takes two arguments: the first argument is the name of the file with all concatenated data and the second argument is the file that should be concatenated to it. Since the concatenator will run after each small model run, only 1 NetCDF file has to be concatenated to the main file @@ -293,12 +297,3 @@ exception because files for the higher partitioning do not exist. When the number of partitions is set too small, then only part of the grid will be taken into account, leading to inconsistent changes which makes model runs unpredictable. - -.. rubric:: Footnotes - -.. [#action] - Actions are configurations that specify which (external) executables - OpenDA should run. Typically, this would be the model itself. Other actions can - include pre- or post-processing steps. For example, the NetCDF concatenator is - a post-processing step that merges smaller NetCDF result files from multiple - runs into a single NetCDF file. diff --git a/docs/source/index.rst b/docs/source/index.rst index afd44cde8..0e3210208 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -26,19 +26,20 @@ Index :caption: Wrappers :maxdepth: 2 + wrappers DFlowFMWrapper .. toctree:: :caption: Additional information :maxdepth: 2 - data_objects.rst - localization.rst - continuing.rst - estimate_missing_observations.rst - DFlowFM_info.rst - calibration_twin.rst - Quadratic_cost_function.rst + data_objects + localization + continuing + estimate_missing_observations + DFlowFM_info + calibration_twin + Quadratic_cost_function .. toctree:: :caption: Contributing to the source diff --git a/docs/source/wrappers.rst b/docs/source/wrappers.rst new file mode 100644 index 000000000..2e130cdaa --- /dev/null +++ b/docs/source/wrappers.rst @@ -0,0 +1,162 @@ +======================== +Introduction to wrappers +======================== +OpenDA model wrappers are used to connect existing models (such as D-Flow FM) +to OpenDA, and call these external models from within an OpenDA run. +In OpenDA, models are used in *black-box* form, which means that data-assimilation +and model calibration techniques are applied to a model without changing the +existing model code. OpenDA and the black-box wrapper have no knowledge of the +model internals (black box) and only use the input and output files of the +model. In order to connect such +models to OpenDA, specific functionality needs to be created to supply OpenDA +with information about the model runs. Examples include run periods, model input +values or files that OpenDA can adjust, and model output values OpenDA can +read. Note that the file containing the model run time must be rewritten as +OpenDA cuts the whole run period into smaller pieces. +The existing wrappers can be +found in the ``model_*`` directories in the OpenDA root. + +Data objects +------------ + +For each input or output file, OpenDA will read or write a *data object*, +which needs to be configured in a wrapper configuration XML file. The wrapper +configuration files are based on the ``blackboxWrapperConfig.xsd`` schema. +In such a wrapper configuration file, we have:: + + + + + + +The ``dataObject`` contains the following information: + + - ``className="[package].[className]"``: refers to the main Java code that handles (wraps) the file; + - ````: refers to the file that needs to be read / written; + - ````: specifies the identifier of the object so it can be referred to at other places in the configuration. + +For example, we can construct the following data object:: + + + rstfilename + rstfile + + +Exchange items +-------------- + +For each data object, OpenDA will create virtual objects in memory containing +the needed values. These virtual objects are called `exchange items`, because +OpenDA has the possibility to change these values according to its algorithms. +An exchange item can be seen as a container of values with optionally some +meta information. In the most simple form an exchange item contains a single value or array +and has an identifier. This identifier can be used to refer to from other +parts of the OpenDA configuration. The identifiers of the exchange items are +determined by the specific data objects and ideally resemble the meaning of +the values they hold in memory. For instance, a data object created for a +file that contains information about the start and end time of a run will +typically create two exchange items with ``id`` ``start_time`` and +``end_time``. But a data object for model results would typically create +exchange items with identifiers resembling station and parameter identifiers, like +``waterlevel_delft``, ``temperature_delft``, ``waterlevel_rotterdam`` +etc. Since each data object is model and file specific and there are +no strict rules about or automatic functionality for the exact identifiers, +they will differ a lot between models. + +In the easiest case, all exchange items of a data object are used with the +identifiers the data object has created for them:: + + + +Here, the ``dataObjectId`` refers to the ``rstfile`` identifier that was chosen +in the example above. + +It is also possible to explicitly configure which exchange items are used and +which identifiers they have throughout the code. For example:: + + + + + +Here, + + - ``id`` determines the identifier that can be referenced elsewhere in the configuration; + - ``dataObjectId`` refers to the identifier of the data object as specified in the wrapper configuration; + - ``elementId`` has to match the exchange item identifier that the data object has created for it. + +When ``elementId`` is not specified, the ``id`` has to match the exchange item +identifier that the data object has created for it. + +Within the model configuration XML file, there is a specific element to +determine which exchange items are used for the model run period:: + + + +These ``start_time`` and ``end_time`` should be the ``elementId`` identifiers of the data object. + +Aliases +------- + +In the OpenDA model configuration (``blackBoxModelConfig``) aliases can be used, +for instance for file references:: + + + + + + + +These aliases can be used as file references in other parts of the OpenDA +configuration. Benefit of this is the longer paths do not need to be repeated and +if a file path or name changes, it only needs to be changed in one place. It +also enables configurators to use names which make more sense to them. +In the wrapper configuration, the following lines are needed to make sure the +aliases can be used:: + + + + + + + +The surrounding ``%`` make sure we refer to tags: its content is +defined elsewhere. +Below we will see an example where aliases are used in practice. + +Compute actions +--------------- + +.. _compute actions: + +Actions are configurations that specify which (external) executables +OpenDA should run. Typically, this would be the model itself. +This will be configured as ````:: + + + + %optional_argument_1% + %optional_argument_2% + + + + + +OpenDA will run the model executable matching the operating system. +This would be the main and most important execute action. Here, ``checkOutput`` checks whether a certain output file has been created. + +Other actions can include pre- or postprocessing steps. For example, the +:ref:`NetCDF concatenater` is a postprocessing step that merges smaller NetCDF result +files from multiple runs into a single NetCDF file. +Other examples include the copy or rename of certain files. +This can be the case when a model uses time stamps in files that OpenDA needs to read or change. +Then a postprocessing step can find the most recent file and locate (and rename) it such that OpenDA can find it or use it for a restart for the next model run. +An example of this is used in ``model_dflowfm_blackbox/dcsmv5_kalman_rst/stochModel/dflowfmWrapper.xml``:: + + + runId=%runid% + sourceRestartFileSubDir=%outputDir% + targetRestartFileNamePostFix=00000000_000000_rst.nc + deleteOlderRstFiles=true + + +This action does not refer to a Windows or Linux executable but to an executable Java class which is part of the OpenDA source code and in this case also part of the D-Flow FM specific wrapper code. More explanation on the specific contents of this action can be found at the :ref:`DFlow-FM wrapper` page. From 485fdb7e995bc3e3b2d0b6e3071533dfa98f04cc Mon Sep 17 00:00:00 2001 From: Thea Vuik Date: Fri, 27 Mar 2026 15:40:02 +0100 Subject: [PATCH 2/2] Fix typo --- docs/source/estimate_missing_observations.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/estimate_missing_observations.rst b/docs/source/estimate_missing_observations.rst index 6d884d1cf..bb9ba0bd2 100644 --- a/docs/source/estimate_missing_observations.rst +++ b/docs/source/estimate_missing_observations.rst @@ -83,7 +83,7 @@ In order to use :math:`H*K` in the steady-state filter, the :ref:`Kalman gain` is used, :math:`H*K` needs to be element-wise multiplied with the weight factor between observations called ``rho``. ``rho`` will be retrieved via a call to ``IModelInstance.getRhoForLocalization()``. The weight factor ``rho`` is currently only calculated for the black box model in ``BBStochModelInstance.java``. For all other models, a default implementation (``IModelInstance.getDefaultRhoForLocalization()``) is available such that the interface contracts will not be broken. This default implementation will return a rho matrix consisting of only the value 1, so it does not affect :math:`H*K`. +- **Hamill localization**: When :ref:`Hamill localization` is used, :math:`H*K` needs to be element-wise multiplied with the weight factor between observations called ``rho``. ``rho`` will be retrieved via a call to ``IModelInstance.getRhoForLocalization()``. The weight factor ``rho`` is currently only calculated for the black-box model in ``BBStochModelInstance.java``. For all other models, a default implementation (``IModelInstance.getDefaultRhoForLocalization()``) is available such that the interface contracts will not be broken. This default implementation will return a rho matrix consisting of only the value 1, so it does not affect :math:`H*K`. - For all **other localizations**, :math:`H*K` is not computed and the user is informed by a message. :math:`H*K` will be added to ``KalmanGainStorage`` in NetCDF CF