Microsoft Slurm apt repo does not reference a "signed-by" key which seems to break CycleCloud VM initialisation with the latest Ubuntu release?

[garymansellricardo]

Hi,

I think this is a CycleCloud team issue (as CycleCloud installs the slurm packages ontop of the image during VM initialisation), rather than an AzHPC image team issue, or an Ubuntu issue, but feel free to correct me, as I may well be wrong...

Using the latest AzHPC Ubuntu 22.04 Marketplace image works fine, but if I "roll-my-own" image from the latest Ubuntu 22.04 LTS Marketplace image and apply the AzHPC repo build scripts to it (so I have all the latest security fixes), CycleCloud fails to initialise the Scheduler node, giving this error in the UI:

STDOUT:
(...64 lines trimmed...)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.microsoft.com/repos/amlfs-jammy jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EB3E94ADBE1229CF
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.microsoft.com/repos/slurm-ubuntu-jammy insiders InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EB3E94ADBE1229CF
W: Failed to fetch https://packages.microsoft.com/repos/amlfs-jammy/dists/jammy/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EB3E94ADBE1229CF
W: Failed to fetch https://packages.microsoft.com/repos/slurm-ubuntu-jammy/dists/insiders/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EB3E94ADBE1229CF
W: Some index files failed to download. They have been ignored, or old ones used instead.

• apt install -y python3.11 python3.11-venv python3.11-dev

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.What is the issue?

It looks to me (I may be wrong?) as if the latest version released of Ubuntu has tightened up on not allowing repo's that don't use the new "signed-by" key mechanism which they have been switching to for several years now. But, this is a guess, am no expert, I may be wrong.

This is the Slurm repo config file that CycleCloud installs (which gives the key error above with apt update):

root@test-scheduler:/etc/apt/sources.list.d# cat slurm.list
deb [arch=amd64] https://packages.microsoft.com/repos/slurm-ubuntu-jammy/ stable main

if I change the slurm.list file to this:
deb [arch=amd64 signed-by=/usr/share/keyrings/microsoft-prod.gpg] https://packages.microsoft.com/repos/slurm-ubuntu-jammy/ stable main

Then apt update works fine and the scheduler node goes green in the CycleCloud UI.

So - I think the Slurm repo config or CycleCloud initialisation steps need fixing so that the /etc/apt/sources.list.d/slurm.list file is of a format that the very latest Ubuntu 22.04 release will allow:

deb [arch=amd64 signed-by=/usr/share/keyrings/microsoft-prod.gpg] https://packages.microsoft.com/repos/slurm-ubuntu-jammy/ stable main

Note - I saw this issue with with the latest Marketplace image:

grma@emospot-scheduler:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

grma@emospot-scheduler:~$ uname -a
Linux emospot-scheduler 5.15.0-1111-azure #120-Ubuntu SMP Wed Apr 15 15:09:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

[garymansellricardo]

I just checked my Custom Ubuntu 22.04 AzHPC image (my own built one, not the marketplace one), and it seems to have the slurm sources list file in the correct format with the signing key, so this must get over-written by CycleCloud at VM initialisation:

root@ubuntu2204-test:/etc/apt/sources.list.d# ls -la
total 24
drwxr-xr-x 2 root root 4096 Jun 9 09:40 .
drwxr-xr-x 8 root root 4096 May 3 07:55 ..
-rw-r--r-- 1 root root 139 Apr 20 2023 cuda-ubuntu2204-x86_64.list
-rw-r--r-- 1 root root 310 Dec 11 10:38 doca.list
-rw-r--r-- 1 root root 89 Jun 9 08:50 microsoft-prod.list
-rw-r--r-- 1 root root 140 Jun 9 08:47 slurm.list

root@ubuntu2204-test:/etc/apt/sources.list.d# more slurm.list
deb [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/microsoft-prod.gpg] https://packages.microsoft.com/repos/slurm-ubuntu-jammy/ insiders main

root@ubuntu2204-test:/etc/apt/sources.list.d# apt list --installed | grep slurm

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

root@ubuntu2204-test:/etc/apt/sources.list.d#

[garymansellricardo]

I think I have found the issue in ubuntu.sh in the azure-slurm-install-pkg-4.0.6.tar.gz archive that is downloaded by CycleCloud to the Scheduler node at node initialisation time, as it only sets the signed-by option for Ubuntu 24.x and not 22.x versions:

So, changing that section to this should fix things - but when I do this and update the tar file in the storage account, it keeps getting over-written by the old one from Microsoft repo:

# Always use signed-by (modern APT requirement)
echo "deb [arch=$arch signed-by=/usr/share/keyrings/microsoft-prod.gpg] https://packages.microsoft.com/repos/$REPO/ $REPO_GROUP main" \
  > /etc/apt/sources.list.d/slurm.list

if [ "$arch" == "arm64" ] && [[ ! $UBUNTU_VERSION =~ ^24\.* ]]; then
    echo "Slurm is not supported on arm64 architecture for Ubuntu versions < 24.04"
    exit 1
fi

So - it needs updating server / repo side as a proper change, I suspect?

Is there someway I can temporarily fix my version of the tar file so that it does not get over-written by cyclecloud in the short term, so I can keep working on my new custom image until the proper fix is in?

BTW - I am running latest CycleCloud 8.9.0-3754

[garymansellricardo]

@aditigaur4 - hi, I hope you don't mind me reaching out, but you have been really helpful in the past...

This is quite an important issue for us - as it prevents us deploying our "roll-our-own" version of the AzHPC Ubuntu image using the latest version of Ubuntu 22.04 LTS, which addresses a bunch of serious security issues in the current Ubuntu 22.04 Marketplace AzHPC image. Some of our CycleCloud environments are "sensitive" - so we can't run with High/Critical security vulnerabilities and would have to rip out CycleCloud :-(

Until you can address the backend issue with the azure-slurm-install-pkg-4.0.6.tar.gz in the CycleCloud Storage Account (which I appreciate is gonna take time), is there some way to work-around this issue perhaps that you can advise on?

I have tried fixing the ubuntu.sh file and recreating the azure-slurm-install-pkg-4.0.6.tar.gz file myself and replacing the version in the CycleCloud Storage account with my updated one, but every time I start the cluster, it seems to get over-written with the default/problem version again.

Not sure what else to try - maybe a cluster image script to replace the ubuntu.sh file after it has been deployed from the tar file (or is the tar file downloaded again each jetpack retry)?

Do you have any advice/suggestions?

[garymansellricardo]

Is there anyone monitoring this that can respond?

[aditigaur4]

Hi Gary-- We can include this fix in the latest release. Apologies for the late response, I was out of office.

re this:

I have tried fixing the ubuntu.sh file and recreating the azure-slurm-install-pkg-4.0.6.tar.gz file myself and replacing the version in the CycleCloud Storage account with my updated one, but every time I start the cluster, it seems to get over-written with the default/problem version again.

One way to achieve this is to make a change in your template in the cluster init section (for all nodearrays)--

replace cyclecloud/slurm-4.0.6.tar.gz with
slurm-4.0.6.tar.gz.

This will make sure it reads from location in your storage account. Make sure you do cyclecloud project upload <your storage account> (this needs to be done from the root of cyclecloud-slurm directory where you have the custom blobs) too to make sure it gets the right blob. But that should work.

Once you make the change to your template:

cyclecloud export_parameters -c <cluster> -o params.json
cyclelcoud import_cluster -f <template.txt> -p <params.json> -c slurm

This should update your cluster to use your staged blobs.

cyclecloud/ makes it read from the cyclecloud folder in the storage account. Also-- that blob actually gets staged in CC VM the first time you start a cluster and from that point on it uses the cached copy in /opt/cycle-server/work/staging/projects directory. You could try replacing there too but try the template approach first-- because thats the method for having clusters reference your blobs versus the one's shipped with CC

[garymansellricardo]

@aditigaur4 - hi, I see that you have made and committed the change, please can you tell me which final Ubuntu CycleCloud package version this will land in?