Decouple ecflow from prod_util

[RussellManser-NCO]

Description
Currently, this package is coupled to NCO-HPC/prod_util such that when prod_util is upgraded, this package also needs to be upgraded. This is because prod_util is listed in versions/run.ver and loaded in include/head.h. This introduces more work for an upgrade to prod_util because an upgrade to our ecflow package requires an ecFlow server switch.

This makes introducing small, manageable changes into prod_util inefficient due to the extra work of needing to perform an ecFlow server switch. The current approach is to collect multiple changes to prod_util and then upgrade the package with all of these changes. However, if one aspect of the upgrade turns out to need modification, then (a) all of the changes need to be rolled back, and (b) another ecFlow server switch needs to be performed.

This package uses the following prod_util utilities:

• ush/err_exit - called once in include/envir-p1.h
• ush/mail.py - called four times in include/switch_fun.h

Aside from loading prod_util in include/head.h and the calls above, I do not see another place where ecflow uses prod_util.

Proposed Solution
To maintain the module load for prod_util in include/head.h while introducing flexibility for upgrades to prod_util, I suggest we set prod_util_ver as an ecFlow variable.

CC: @CarlosMDiaz-NCO

[RussellManser-NCO]

Here are some of my observations and results from testing so far:

• Setting prod_util_ver at the server level / is possible without performing an ecFlow server switch, allowing for the desired flexibility
• Setting prod_util_ver at the suite level, e.g., /prod, is also possible and still provides the desired flexibility.
• However, it isn't clear to me how setting prod_util_ver at the suite level would affect mail.py calls in include/switch_fun.h.
  • Does a WCOSS switch have access to ecFlow variables at the suite level?
  • Would it be possible to remove the dependency on mail.py from include/switch_fun.h?

Additionally, I searched for packages with dependencies on switch_fun.h by running

grep -r "switch_fun" /lfs/h1/ops/prod/packages/*/ecf/* | less

and found that the following WCOSS2 packages are dependent:

• paratest.v1.1.0
• recover_dcom.v1.0.2
• wcoss_switch.v1.1.6

More extensive testing should be performed to assess the impacts of setting prod_util_ver as an ecFlow variable on include/switch_fun.h and its dependencies. Personally, I also think it's worth investigating if mail.py can be replaced with mailx to remove the dependency on prod_util. However, I expect that there is a reason that mail.py is used instead of mailx, and that removing mail.py may not be reasonable.

What I've described above includes one or two additional projects that are outside the scope of moving a prod_util upgrade along at this time. I suggest we move forward with completing NCO-HPC/prod_util#13 without attempting to fix this issue first.