ssh-keyscan failing intermittently

[eshattow]

Hello, I'm observing what appears to be intermittent failure depending on which particular runner gets the job. This is in parallel with runs for the standard aarch64 and amd64 GitHub Actions runners which do not exhibit the failures, so... I think it's something ssh adjacent.

Broad strokes description is that the workflow builds python wheels and uploads them (ssh rsync) to a web server which will later serve as the python wheels index for some other workflows. The private key is in a secret and the print of a pubkey is simply proof that the key entered is valid before continuing; however one of those validation commands fails sometimes on the RISE RISC-V runners, but not always:

  if [[ "false" =~ false|False ]] && [ -z "" ]; then
    umask g-rwx,o-rwx
    chmod -f g-rwx,o-rwx .ssh .ssh/id_rsa .ssh/known_hosts .ssh/wheels-key || true
    mkdir -p .ssh
  
    # Write valid wheels-key or wrapped with PEM header-footer
    echo "***
...snip...
  ***" > .ssh/wheels-key
    ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null || \
      sed -e '1i-----BEGIN RSA PRIVATE KEY-----' \
          -e '$a-----END RSA PRIVATE KEY-----' -i .ssh/wheels-key
    ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null && \
      cat .ssh/wheels-key >> .ssh/id_rsa && rm -f .ssh/wheels-key
  
    # Validate & update known_hosts
    ssh-keygen -y -e -f .ssh/id_rsa
    ssh-keyscan -H "ai6fs.net" >> .ssh/known_hosts
  fi
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "3072-bit RSA, converted by runner@riscv-runner-12 from OpenS"
AAAAB3NzaC1yc2EAAAADAQABAAABgQDAFGj/8WmGysWZwN9aXWtwfjVhyy821k7Fg1QQ1k
l4qdQ0GJNyh64oWBK7dNsJLd53gSeijK/Yvz0fFZybp7ZGaR12N/0/baV8IQ0Ga9b+rR2m
+eKI/YinsWJej+iP7yWdZ/VataLDMUtbg2TpufawFbBmeX5m3h/Ffjc7ArD5LAjBqB0NQc
CSldMboNcoESi7bcqAXvUako+69gl3QKryHdEDR1tLykKRPE7ItTt479OkVRUPRdv986l0
dUwXK8Kjf6cqukCAf5eWgoiocyJRm3WdHdm+LfK8AUsWQjk/4yDCqHiWbYxryVW+5ck7Ss
Zem7ennP4VBPxLWFtpd6FYlmLPh7aPNUECDCdY6mKWwwzqFdzaOLj2FmVdF0uFLAxuwnWy
iI13onMyYTHqyq9JOm6uK+pDYXDMETG4ZgfCaQ3XW/xvmy+ev0FPa4sZhDu+dbW+nAdnsb
9hdokl7UdP3ZTlFdnIbj5IVr8XbV1kuZ/a+oGn8UmAFeiW2Q96Ox8=
---- END SSH2 PUBLIC KEY ----
Error: Process completed with exit code 1.

and then, the same exact job re-run is successful:

  if [[ "false" =~ false|False ]] && [ -z "" ]; then
    umask g-rwx,o-rwx
    chmod -f g-rwx,o-rwx .ssh .ssh/id_rsa .ssh/known_hosts .ssh/wheels-key || true
    mkdir -p .ssh
  
    # Write valid wheels-key or wrapped with PEM header-footer
    echo "***
...snip...
  ***" > .ssh/wheels-key
    ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null || \
      sed -e '1i-----BEGIN RSA PRIVATE KEY-----' \
          -e '$a-----END RSA PRIVATE KEY-----' -i .ssh/wheels-key
    ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null && \
      cat .ssh/wheels-key >> .ssh/id_rsa && rm -f .ssh/wheels-key
  
    # Validate & update known_hosts
    ssh-keygen -y -e -f .ssh/id_rsa
    ssh-keyscan -H "ai6fs.net" >> .ssh/known_hosts
  fi
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "3072-bit RSA, converted by runner@riscv-runner-28 from OpenS"
AAAAB3NzaC1yc2EAAAADAQABAAABgQDAFGj/8WmGysWZwN9aXWtwfjVhyy821k7Fg1QQ1k
l4qdQ0GJNyh64oWBK7dNsJLd53gSeijK/Yvz0fFZybp7ZGaR12N/0/baV8IQ0Ga9b+rR2m
+eKI/YinsWJej+iP7yWdZ/VataLDMUtbg2TpufawFbBmeX5m3h/Ffjc7ArD5LAjBqB0NQc
CSldMboNcoESi7bcqAXvUako+69gl3QKryHdEDR1tLykKRPE7ItTt479OkVRUPRdv986l0
dUwXK8Kjf6cqukCAf5eWgoiocyJRm3WdHdm+LfK8AUsWQjk/4yDCqHiWbYxryVW+5ck7Ss
Zem7ennP4VBPxLWFtpd6FYlmLPh7aPNUECDCdY6mKWwwzqFdzaOLj2FmVdF0uFLAxuwnWy
iI13onMyYTHqyq9JOm6uK+pDYXDMETG4ZgfCaQ3XW/xvmy+ev0FPa4sZhDu+dbW+nAdnsb
9hdokl7UdP3ZTlFdnIbj5IVr8XbV1kuZ/a+oGn8UmAFeiW2Q96Ox8=
---- END SSH2 PUBLIC KEY ----
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1

[luhenry]

Hi @eshattow thank you for your report. I'm not sure I understand exactly what's going wrong. Do you know which command exactly fails? The TH1520 chip (which underlines the EM-RV1) has a known hardware bug which triggers with atomic operations where an atomic operation returns success when it hasn't. Could it be that?

[eshattow]

Hi @eshattow thank you for your report. I'm not sure I understand exactly what's going wrong. Do you know which command exactly fails? The TH1520 chip (which underlines the EM-RV1) has a known hardware bug which triggers with atomic operations where an atomic operation returns success when it hasn't. Could it be that?

After the second attempt today where it fails I have the sense to set GitHub Actions Secret 'ACTIONS_STEP_DEBUG'=true but now it does not fail.... of course, that is how these "fun" intermittent problems are so anyhow the "good" output with debug:

2026-05-12-0538-debug-success-job-logs.txt

And just previous to this the two attempts (no debug, unfortunately) that fail:

2026-05-12-0448-job-logs.txt

and in the host that is the target of ssh keyscan I read through the journal but it is not clear to me which information corresponds to the github actions activity.

then the other github attempt that failed:

2026-05-12-0502-job-logs.txt

Looking in the log for "SSH2" pubkey validation are the list of commands above it, framed for our understanding as funneling the GitHub Actions Secret 'WHEELS_KEY' (containing ssh id_rsa private key) into file '.ssh/wheels-key' and then follow along from there; this is preparation for rsync the produced python wheels to a webhost with id_rsa privkey as the authentication mechanism. The invocation of 'ssh-keygen' in multiple possible tests is to validate the user input is a valid private key (and to print the corresponding pubkey what we see in the log) prior to appending it to the target file. So... we do see the pubkey is output but then somewhere between that and the assumed completion of the very next command performing the keyscan amending the known_hosts file it fails the workflow. My guesses are one of:

• hardware bug (is this at all possible?)
• slight variation in software between GitHub official runners and RISE RISC-V runners, where some action of the ssh tools differs
• network disruption on the RISE RISC-V runner in some way that it fails to complete the keyscan
• something along the idea of "user error" in how I have the ssh host configured at the virtual private server (Possible? I think not as likely for it is always successful with default official GitHub actions runners)

Would it be worthwhile to craft a workflow specifically to try to loop and get a failure from these ssh commands in sequence, could it be expected to re-try this sequence of commands until failure or would that be flawed in some way? - I want to ask before any such thing as it is not my intent to be wasteful of any community resource offering. Also if in a side channel you want the privkey to set up your own such reproducer for this issue and you think it's more than just some user error on my part I'd be sure glad to share if it helps make troubleshooting easier.

[eshattow]

P.S. I should mention what it is, this all, is. I'm readying Home Assistant for RISC-V and the superset work of that is making Home Assistant CI/CD workflows able to be forked and built outside of the home-assistant organization. There's layers of docker images and reusable actions up to the "wheels" builder image, and then after the "docker" image does use the former to build some python wheels and rsync those to a webhost as it becomes the PEP503 index for the build of "core" image (what we think of as the canonical Home Assistant docker image) which also does more of the same building a long list of python wheels and rsync to the webhost. I've proven it's possible to get there and now backtracking to make the whole thing repeatable by anyone that wants to, at: Special Interest Group for Home Assistant on RISC-V