Process manager integration tests

[PawelPlesniak]

Description

Fixes issue #728

Also fixes issue #819

Depends on DUNE-DAQ/integrationtest#151

Adds some test for the Process Manager based on the DUNE DAQ integ test framework.

The relevant changes in the user workflow have been documented in the doc changes thing see the file change its an md

TODO:

• Document changes in the user workflow (update the line above)
• Push for the new PR in integrationtest Added capture of the run control console output so that it can be used for validations integrationtest#151 (review)
• Discuss with Kurt to remove dependency on terminal width. When terminal window is too small, the ps table concatenates. Since the current test is incredibly dependent on obtaining the stdout of the drunc process, it would be nice if the terminal window dependency can be removed
  • Turns out we can do it in drunc natively and much cleaner too :)
• Miruna has done a bit of development on this front: https://github.com/DUNE-DAQ/drunc/tree/miruuna%2Ffeature-base-sequence-tests-for-ssh-shell-pm---integration-testing-framework. See if we can integrate any of the changes here.
  • Looking at this branch, a lot of it has been focused on the controllers. While this will be useful to test the controllers, these are beyond the purview of the Process Manager tests.

Type of change

• New feature / enhancement
• Bug fix

List of required branches from other repositories

A new PR in integrationtest repo. Currently, the test requires the branch kbiery/temp_branch_drunc_output_capture, of which a PR has not been made yet.

Change log

An integtest-like set of tests have been written for the SSH Process Manager. It also includes several helper functions which may come in handy for other sort of tests, which is done in the integ_test_utils.py file.

The current testing framework is a bit janky. Its entirely dependent on reading the terminal output of the drunc process, which is saved into an stdout object that can be read by DUNE DAQ's integtesting framework.

With the echo command (see below), this PR also adds a bunch of testing utility that allows the integtest to process the stdout of the drunc session. This lets us parse the logs directly, checking if specific processes exist in the ps command, or if specific log lines are found with specific numbers. This lets us test if, for example, the wait command waits for the correct amount of time, or if the mlt process exists or not depending on if we've killed it.

Echo

To help with this, an echo function is added in drunc, which lets you 'echo' stuff in the terminal like in bash

drunc-unified-shell > echo "hello world"
[2026/03/17 11:19:07 UTC] INFO       commands.py:266                          drunc.echo                                         hello world

This echo command is liberally used when setting up the testing framework. For now, there are discrete blocks of code that tests a particular command, and the echo command is used by the test to identify which line the tests happen in the log file. Further processing is done to perform the required tests, which should be documented in code and easy to follows.

Comment

A comment function is also added in drunc. It lets you write 'comments' in the terminal if required, and nothing will come out

drunc-unified-shell > comment "hello, world"
drunc-unified-shell > 

It could come in handy. Note that we cannot bind # or // or similar as a function for comments, because the click parsing relies on the characters being ascii and alphanumeric.

pythoncode vs sourcecode

A slight change in the drunc integtest bundle also lets the drunc repo exist in either the pythoncode folder or the sourcecode folder. The sourcecode folder was not supported before

Fix width dependnecy on logs

A slight change in the logs unified shell command,k in which we add a soft_wrap. The logs command uses Console.print() (from the Rich library) to get the logs to terminal, but that command is a 'print it once' output, so with this there is no mechanism to re-render those lines at a new width. This effectively single log lines can be split into two, which breaks the current testing framework. soft-wrap fixes this.

Fix width dependency on ps tables (and other tables from the PM)

We now introduce an optional parameter for eg. ps -w [int]. If parmater is not used, follow usual procedure.

If parameter is included, it will expand to the desired width and correctly wraps around. Very useful for the integ tests :)

Suggested manual testing checklist

From the drunc folder, run ./scripts/drunc_integtest_bundle.sh

Follow up

Two issues were discovered when writing the tests for these (as intended!!)

#819 <- also being fixed in this PR
#821

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: np04-srv-028 from release NFD_DEV_260313_A9.

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed
  • Only concern yourself if failures related to drunc are in the log files
  • If non-drunc failure appears:
    • Validate failure in fresh working area
    • Contact Pawel if unsure

Once the features are validated and both the unit and integration tests pass, the PRs is ready to be merged.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #daq-sw-librarians Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for #daq-sw-librarians Slack channel

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[PawelPlesniak]

Markled as ready for review to see which PRs are being actively worked on for now. Please ignore that update

[emmuhamm]

Latest changes are up, I think they are janky but thats the downside of using logs. With more time, I can get it better but for now they work in most circumstances. I would advice against putting this in any default integtests though, at least until monday when I reverify everything.

It also depends on Kurt's branch, kbiery/temp_branch_drunc_output_capture, on integtests

TODO for monday

• go through again to make sure everything works
• Update readme + the PR desc
• Go through personal notes to see if I forgotten anythign
• The terminal window bug for the table needs to be fixed on the daqsystemtest front

[PawelPlesniak]

Thank you @emmuhamm, I am about to start reviewing. Before I start, some questions

A slight change in the drunc integtest bundle also lets the drunc repo exist in either the pythoncode folder or the sourcecode folder. The sourcecode folder was not supported before

What is the motivation for needing this? drunc is a python-only package, I would not have expected us to need to put it into sourcecode

[emmuhamm]

Thanks Pawel. Regarding

What is the motivation for needing this? drunc is a python-only package, I would not have expected us to need to put it into sourcecode

Honestly there's not much motivation, it stems from me not fully understanding how strict the pythoncode v sourcecode distinction is. I've understood the need for it but not 'how strict do we want it to be' kinda deal.

Anyways, this was a minor change and whipped it up in no time. Can undo it by reverting this one commit for that specific file 7149dbf.

I'm not too attached to the idea so if you want me to get rid of it I'd be happy to

[PawelPlesniak]

Integration test passed, nice work :)

[emmuhamm]

Reverting back to draft so it doesnt accidentally get merged. It now depends on #824

[emmuhamm]

Only major change I can think of is this commit

73188d1

This is built off Aurash's PR with the flush crash thing, so that should work as expected

[emmuhamm]

Since technically you are the author of this PR you may not be able to approve it. In which case I'll approve it now but keep it mark as draft so you can 'approve' by undrafting and merging ;)