[release-1.26] Fix CVE-2025-47913 with x/crypto + minimum go bump by cevich · Pull Request #6813 · containers/buildah

cevich · 2026-04-27T13:45:18Z

What type of PR is this?

/kind other

What this PR does / why we need it:

Resolves https://issues.redhat.com/browse/RHEL-134784
Resolves https://issues.redhat.com/browse/RHEL-168611

Also bump CI image to gain access to (required) golang 1.24.

How to verify it

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix CVE-2025-47913 with x/crypto + minimum go bump

nalind

#6678 may help with one of the errors we're seeing in the unit tests.
#6271 and #6674 should help with one of the errors we're seeing in integration tests.
Some of the changes from https://github.com/containers/buildah/pull/4610/changes#diff-dcdc10b43f652991a881c6d816518d2f1b52fe22f34abb9103f90528dff35a68 appear to be indicated for errors we're seeing in conformance tests.

nalind · 2026-04-27T17:33:30Z

+go 1.24.0

-toolchain go1.22.8
+toolchain go1.24.10


Given that we try to avoid having this line pop up in later branches, I'm surprised to see it here.

...may help...

Thanks so much for all the pointers! I'll get started tracking them down and applying.

edit: Fixed- All backported, and w/ selections from fa145b0

...avoid having this line...

I won't pretend to understand why, but I tried removing it then make vendor just adds it back in 😞

IIRC, there might be an environment variable that controls it? If so, I'd hazard to guess I'm not setting it 😁 Sounds like I should?

Edit: Fixed - manually removed line and now it's not re-appearing, so 🤷

FWIW, results from running system tests on RHEL (I haven't looked at them yet for comparison).

86147cc-gating:local_root_system_runc:root.txt

86147cc-gating:local_root_system_crun:root.txt

Try setting "go 1.24.10" in go.mod ("go mod edit -go=1.24.10" should also work). make vendor-in-container shouldn't reintroduce a "toolchain" directive then.

There seem to be a few missing test helper binaries in the installed buildah-tests package (buildah-inet, buildah-dumpspec, buildah-tutorial) and a missing "DUMPSPEC_BINARY=/usr/bin/buildah-dumpspec" in the gating environment. I don't see the corresponding .spec file in the source tree, though, so I'm not sure what to suggest.

Does the gating test environment have "crun" installed? Or a version that doesn't recognize a "--debug" flag?

The SELinux test in the gating environment may need #2125.

Try setting

I hand edited it then ran make vendor-in-container and it did not re-add the toolchain directive, so I'm not sure why I was struggling with it before.

There seem to be a few missing test helper binaries

Yes (and good catch). This unfortunately has been a problem across multiple releases. We've got issues open on it, and I've already got the specfile fixes in place for this version.

Does the gating test environment have "crun" installed?

Ahh! Another good catch, indeed it's unspecified so may not have been installed. I've fixed this so it's explicit now.

The SELinux test in the gating environment

Thanks for the pointer, we see this for other releases as well but have been ignoring it. I wasn't aware it had a fix. I'll add a reference to our notes for the other releases, and cherry-pick the fix here.

The cherry-pick from 595af50 appears to have pulled in unit tests that weren't in the original commit.

Yeah I generally default to --theirs to make future backports easier but it may have caused more trouble here than it resolved. I'll come back and strip out the tag-a-longs if this continues to be the case.

The "extra" tests seem to be passing fine, though the copier TestGetSingleChroot still seems to be flaking on broken pipes. I'm fine with re-running or ignoring this as it seems to be a test bug not a buildah bug(?).

Resolves https://issues.redhat.com/browse/RHEL-134784 Resolves https://issues.redhat.com/browse/RHEL-168611 Also bump CI image to gain access to (required) golang 1.24. Drop updates to the build-push image as it hasn't been used in this repo for years and causes CI jobs to fail. [NO NEW TESTS NEEDED] Signed-off-by: Chris Evich <cevich@redhat.com>

cevich · 2026-04-28T14:49:35Z

Ugh now that the conformance tests are running, they're throwing:

Error response from daemon: client version 1.43 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version.

This is surprising given I've updated the environment to Debian 14 😕 Or is this some client version buildah is reporting? I'm confused.

nalind · 2026-04-28T15:10:21Z

Ugh now that the conformance tests are running, they're throwing:

Error response from daemon: client version 1.43 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version.

This is surprising given I've updated the environment to Debian 14 😕 Or is this some client version buildah is reporting? I'm confused.

That's the conformance test client's half of the API version negotiation not matching up with what the docker daemon is offering. 8d6f3fc is what we used on main.

Ensure sockets are also disabled, and guarantee that /etc/resolved.conf symlink is removed/replaced by the file. Signed-off-by: Chris Evich <cevich@redhat.com>

cevich · 2026-04-28T15:40:37Z

That's the conformance test client's half of the API version negotiation not matching up with what the docker daemon is offering. 8d6f3fc is what we used on main.

Cherry-picking that didn't seem to result in any conformance test change. Could it be that the vendored code is "too old"? go.mod shows:

	github.com/docker/distribution v2.8.3+incompatible
	github.com/docker/docker v24.0.7+incompatible
	github.com/docker/go-units v0.5.0

cevich · 2026-04-28T15:55:27Z

Answering my own question: "Yes", it seems an upgrade of github.com/docker/docker to v25 or later is required to get 1.44. However, doing this snowballs into also requiring a golang bump up to 1.25 which isn't available in the CI images.

I'm informed another way around this is to set DOCKER_API_VERSION=1.44, which could break things later. However, by the time "later" comes there's a good chance RHEL won't care about this branch any longer.

cevich · 2026-04-28T18:39:52Z

Ugh, I tried, but no matter where I export DOCKER_API_VERSION, it's not getting picked up or is being masked going into the conformance tests.

I tried 8d6f3fc but as stated above, this won't work without a bump to the docker module version.

Just now I've made an attempt at updating the ci vm images to support golang 1.25, bumped the docker module etc, but gave up. It really just wants to snowball into bigger and bigger changes, so this seems like a dead-end.

The goal of this PR is just to update the x/crypto module version. The targeted version of RHEL goes EOL at the end of May, and the CI system shuts down completely in June. So I really don't care to waste more time grinding away at this CI-only problem.

I'm going to simply disable the conformance tests, and then we 🤞 nothing in the next month or so would benefit from them...

nalind · 2026-04-28T19:08:14Z

Ugh, I tried, but no matter where I export DOCKER_API_VERSION, it's not getting picked up or is being masked going into the conformance tests.

The conformance tests also use containers/image to read images in the docker daemon, and that would negotiate with, at the latest, the version that the corresponding docker client libraries listed as the most recent, which is 1.43 in the current tree, so that's not going to work so well. As you've noted, you'd have to update to 25.x to get something that claimed to be 1.44 or later.

cevich · 2026-04-28T19:12:55Z

also use containers/image to read images in the docker daemon

Ahh, I vaguely remember seeing other erors about an even older unsupported client, so that's probably the explanation there.

Yeah, I don't see any practical way around this. Though I'll be open to suggestions tomorrow once my blood pressure goes back down 😁

cevich · 2026-04-29T13:17:57Z

@nalind @TomSweeneyRedHat PTAL when you get a chance.

Note: Validate is likely broken due to use of EOL OS.

nalind · 2026-04-29T13:54:38Z

Note: Validate is likely broken due to use of EOL OS.

Indeed I don't see it listed at https://github.com/actions/runner-images. FWIW the version of .github/workflows/pr.yml on main looks more or less the same give or take versions.

nalind · 2026-04-29T14:40:32Z

Yeah, I don't see any practical way around this. Though I'll be open to suggestions tomorrow once my blood pressure goes back down 😁

Later versions of dockerd relaxed this minimum down to 1.40, but if that's not an option, a

echo '{"min-api-version":"1.40"}' > /etc/docker/daemon.json

before the daemon starts might override the requirement on its end.

nalind · 2026-04-29T14:40:47Z

+	sort.Strings(matches)
+	return matches, nil
+}
+


If we update, this should be dropped from the backport.

I'm assuming you're meaning drop this entire function along with TestSortedExtendedGlob() in copier_test.go 👍

Ref: Old commit was 08dfb82

Fix race condition where tar.Reader returns EOF after reading the standard EOF marker (two 512-byte null blocks) but before consuming trailing null bytes that many tar implementations add. This caused the copier subprocess to exit early, resulting in EPIPE errors when the sender tried to write remaining data. Fixes: containers#6573 Signed-off-by: Jan Rodák <hony.com@seznam.cz> (cherry picked from commit 595af50) Signed-off-by: Chris Evich <cevich@redhat.com>

Update "the bud with --cpu-shares" test to expect the a cgroupsv2 value computed using either the older formula or the newer one introduced in github.com/opencontainers/cgroups v0.0.3, and give it a unique name so that it can be selected more easily with bats's "--filter" flag. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (cherry picked from commit 16c0bda) Signed-off-by: Chris Evich <cevich@redhat.com>

Update the weights expected in the "from cpu-shares test" test to account for both the old way that runtimes would convert from cgroupsv1 cpu shares to cgroupsv2 cpu weights, and the new way, catching it up with the "bud with --cpu-shares, checked" test, which will now also check with a few different values. Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (cherry picked from commit 17e30d5) Signed-off-by: Chris Evich <cevich@redhat.com>

See issue containers#4639 (cherry picked from commit fa145b0) Signed-off-by: Chris Evich <cevich@redhat.com>

Depending on the test execution environment, there are two acceptable values for role. Match both of them to tollerate both environments. Ref: containers#2125 Signed-off-by: Chris Evich <cevich@redhat.com>

Signed-off-by: Chris Evich <cevich@redhat.com>

cevich · 2026-04-29T19:59:00Z

Force push:

Stripped out copier/copier.go extendedGlob() function + test as requested.
Added back the conformance tests.
Added conformance test setup step to force acceptance of client versions >= 1.40

nalind · 2026-04-29T20:18:12Z

Looks like the DOCKER_API_VERSION environment variable is still going to be needed for the go-dockerclient bits in the conformance tests.

dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Apr 27, 2026

cevich force-pushed the fix_crypto_cve branch 2 times, most recently from 2ddbdb2 to e5f3af4 Compare April 27, 2026 14:46

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Apr 27, 2026

cevich force-pushed the fix_crypto_cve branch from e5f3af4 to 86147cc Compare April 27, 2026 14:51

nalind reviewed Apr 27, 2026

View reviewed changes

cevich marked this pull request as draft April 27, 2026 18:17

cevich force-pushed the fix_crypto_cve branch 2 times, most recently from 8c75f09 to 7107680 Compare April 27, 2026 20:06

nalind reviewed Apr 27, 2026

View reviewed changes

Comment thread contrib/cirrus/setup.sh Outdated

cevich force-pushed the fix_crypto_cve branch 2 times, most recently from fc1623e to 87251b5 Compare April 28, 2026 14:15

CI: Fix disablement of systemd-resolved

f64ea48

Ensure sockets are also disabled, and guarantee that /etc/resolved.conf symlink is removed/replaced by the file. Signed-off-by: Chris Evich <cevich@redhat.com>

cevich force-pushed the fix_crypto_cve branch from 87251b5 to 06857d3 Compare April 28, 2026 15:18

cevich force-pushed the fix_crypto_cve branch 2 times, most recently from 4e5a315 to 9108f9a Compare April 28, 2026 18:02

cevich force-pushed the fix_crypto_cve branch from 9108f9a to 0049aab Compare April 28, 2026 18:49

cevich marked this pull request as ready for review April 28, 2026 19:45

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Apr 28, 2026

nalind reviewed Apr 29, 2026

View reviewed changes

Honny1 and others added 6 commits April 29, 2026 15:54

Skip conformance test failures with Docker 23.0.1

7422309

See issue containers#4639 (cherry picked from commit fa145b0) Signed-off-by: Chris Evich <cevich@redhat.com>

Fix selinux spc test

c200ddc

Depending on the test execution environment, there are two acceptable values for role. Match both of them to tollerate both environments. Ref: containers#2125 Signed-off-by: Chris Evich <cevich@redhat.com>

CI: Workaround conformance testing bugaboo

171adb9

Signed-off-by: Chris Evich <cevich@redhat.com>

cevich force-pushed the fix_crypto_cve branch from 0049aab to 171adb9 Compare April 29, 2026 19:55

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Apr 29, 2026

Conversation

cevich commented Apr 27, 2026

What type of PR is this?

What this PR does / why we need it:

How to verify it

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

nalind left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cevich Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cevich commented Apr 28, 2026

Uh oh!

nalind commented Apr 28, 2026

Uh oh!

cevich commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cevich commented Apr 28, 2026

Uh oh!

cevich commented Apr 28, 2026

Uh oh!

nalind commented Apr 28, 2026

Uh oh!

cevich commented Apr 28, 2026

Uh oh!

cevich commented Apr 29, 2026

Uh oh!

nalind commented Apr 29, 2026

Uh oh!

nalind commented Apr 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cevich commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nalind commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cevich Apr 27, 2026 •

edited

Loading

cevich commented Apr 28, 2026 •

edited

Loading

cevich commented Apr 29, 2026 •

edited

Loading