[Test] Add end2end test to cover cluster patching.

[gmarciani]

Description of changes

Add end2end test to cover cluster patching.

Minor changes made in this PR:

1. move the gpu workload script to a common folder so that it can be reused by this new test.

Test flow

1. Retrieve the AMI the cluster will use
2. Run EC2 instance with the source AMI, run patching commands and create a new patched AMI.
3. Create a cluster that uses the source AMI
4. Validate that the cluster is functional (GPU workload submitted from login node)
5. Stop the login nodes.
6. Update the cluster so that compute and login nodes use the patched AMI
7. Patch the head node in place and reboot it.
8. Start login nodes
9. Validate that the cluster is functional (GPU workload submitted from login node)

Tests

ONGOING:

test-suites:
  patching:
    test_patching.py::test_patching_cluster:
      dimensions:
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_3_INSTANCES_2_HOURS_NOPG_rhel8 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["rhel8"]
          schedulers: ["slurm"]
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_3_INSTANCES_2_HOURS_NOPG_alinux2023 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["alinux2023"]
          schedulers: ["slurm"]
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_3_INSTANCES_2_HOURS_NOPG_ubuntu2204 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["ubuntu2204"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.92%. Comparing base (bdf724a) to head (29db146).
⚠️ Report is 12 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #7456   +/-   ##
========================================
  Coverage    89.91%   89.92%           
========================================
  Files          180      180           
  Lines        16200    16226   +26     
========================================
+ Hits         14567    14592   +25     
- Misses        1633     1634    +1     

Flag	Coverage Δ
unittests	89.92% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hanwen-cluster]

Does this respect the "AMI_OWNER" specification in the testing infrastructure? If not, how about retrieving AMIs from the CloudFormation template of the first cluster creation?

[gmarciani]

Godd catch, it may not. Fixing

[hanwen-cluster]

Are you sure the number of kernel modules don't change before and after the update?

[gmarciani]

Good point, what we want to enforce is that all the kernel modules loaded before the patching are still loaded after the patching (will fix the assertion).

That said, it may be that this assertion will be anyways too hard because a kernel bump could legitimately invalidate some kernel modules. My proposal is to start with a hard assertion as long as it succeeds and we will relax if we face legitimate removals.