qvm-template-upgrade: add orchestration workflow and in-VM agent#213
qvm-template-upgrade: add orchestration workflow and in-VM agent#213nihalxkumar wants to merge 3 commits into
Conversation
ben-grande
left a comment
ben-grande
left a comment
There was a problem hiding this comment.
Yay, started. As I've done a review now, I will receive a notification every time you commit. Let me know when you need another look or have doubts by mentioning me.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #213 +/- ##
==========================================
- Coverage 71.72% 69.39% -2.34%
==========================================
Files 12 28 +16
Lines 1337 2470 +1133
==========================================
+ Hits 959 1714 +755
- Misses 378 756 +378 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
PipelineRetryFailed |
|
It's showing a successful run 
|
|
Some tests are not enabled on this repo, such as mypy, black and pylint. See this as an example: https://github.com/QubesOS/qubes-core-admin/blob/main/.gitlab-ci.yml. Can you create a separate PR to enable those checks? |
|
Sure, will do |
|
We can squash here if it's looks fine. Will also have to rebase as this is 24 commits behind. |
|
Looks fine to squash. |
nihalxkumar
force-pushed
the
qvm-template-upgrade
branch
from
71dee9e to
6c3d431
Compare
|
Oh no, commit message in the other PR closed it ... |
|
I see a conflict here - CI will not run until it's resolved. |
nihalxkumar
force-pushed
the
qvm-template-upgrade
branch
6 times, most recently
from
16b99f5 to
f1f8fc1
Compare
nihalxkumar
changed the title
ben-grande
left a comment
ben-grande
left a comment
There was a problem hiding this comment.
I have taken a look through the new code. Thanks for the progress, I like to see it evolving. Just minor review though
nihalxkumar
force-pushed
the
qvm-template-upgrade
branch
from
ad97dca to
a0289a7
Compare
|
Can you share more details of what fails on cleanup? |
|
When I tried upgrading fedora 41 -> 42 there was a cleanup failure which happened after the successful version upgrade. logs as shared privately on tuesday: https://gist.github.com/nihalxkumar/01fb990deab3960cb28d680773bb1089#file-upgrade42-log-L48-L55 we can see https://gist.github.com/nihalxkumar/0e4edccb4f4409b1a4daf5c44b6f576f After
|
|
But we can't ignore all cleanup failures. We don't know what is causing it, so this needs to be investigated. |
nihalxkumar
force-pushed
the
qvm-template-upgrade
branch
from
a90eee1 to
f66d1e7
Compare
|
a90eee1 wasn't ignoring all cleanup failures 😅 Anyways, we finally have a full end to end successful fedora upgrade pipeline More logs are at https://gist.github.com/nihalxkumar/e02475c11b6dc8a2e855ceec66599d49 For the fix had to extend the try block in qubes-core-admin-linux/vmupdate/update_manager.py Lines 362 to 366 in 497d467 I have mentioned details in the f66d1e7's description. |
Add qvm-template-upgrade as an initial safe upgrade workflow for TemplateVMs and StandaloneVMs. The command validates the source qube, derives the next distro-version clone name, clones the source, updates template metadata, and cleans up failed clones unless explicitly asked to keep them. The version-upgrade agent hook remains a stub for now, so the command can land the orchestration, rollback behavior, and tests without pretending to perform in-VM distro upgrades yet. Fixes: QubesOS/qubes-issues#8605
This adds the in-qube side of the distro version upgrade and connects it to the dom0 orchestrator. The package manager grows a version_upgrade entry point whose default fails loud, so families without a real path return an error. The dnf path re-reads the distribution from inside the qube, refuses anything that isn't a single-step move to a RedHat-family release, and runs distro-sync onto the target releasever. A --version-upgrade flag selects it, and value-bearing agent args are skipped when unset so a normal update never injects a bare "None". On dom0, run_agent drives the clone through the existing vmupdate qrexec transport, forwarding the agent's streamed output to the log so the user sees progress. On failure the clone is shut down before deletion. StandaloneVM clones keep their inherited template-* features untouched.
nihalxkumar
force-pushed
the
qvm-template-upgrade
branch
from
f66d1e7 to
bd9a723
Compare
|
Unsure about this one also. If shutdown failed, the upgrade might have failed despite the package manager not returning an error during upgrade. Which exception were you receiving? |
By "all" I thought we are including the regular-updates cleanup behavior as well, which were untouched. On the version-upgrade path, though, you're right: once the upgrade returned I've since dropped that tolerance change. As in every run |
The exception was an It's raised in the pre-existing To your concern that a failed shutdown could mean a failed upgrade: in this case it didn't. The agent finishes cleanly: My dom0 is on
We could also harden the source instead, |
Can you upgrade to R4.3? It would make testing much easier, cause currently you are testing an EOL release.
The exception should be raised so the command fails. It won't be raised when you have R4.3. |
nihalxkumar
force-pushed
the
qvm-template-upgrade
branch
from
bd9a723 to
d3ab852
Compare
|
I upgraded my system to R4.3 and the upgrade ran successfully end to end upgrade with no errors. I have reset the branch to d3ab852 as of june15. The shutdown exception propagates, and failure detail it was trying to surface is already in Next thing is disk handling for temporary cache volume |
ben-grande
left a comment
ben-grande
left a comment
There was a problem hiding this comment.
Hi. Could you also take a look at:
- Debian integration
- API usage for better progress reporting
|
|
||
| os_data = get_os_data(self.log) | ||
| family = os_data.get("os_family") | ||
| if family != "RedHat": |
There was a problem hiding this comment.
Did you find a case where this happens? We are in the Dnf module, Dnf was already chosen, and here we are checking again the os_family for which specific reason?
| if not supported: | ||
| raise ValidationError( | ||
| f"Unsupported distro {distro!r}; only Fedora- and " | ||
| f"Debian-based qubes are supported for now." |
There was a problem hiding this comment.
Instead of writing the supported distros as strings, use something like this untested code: [s.capitalize() for s in SUPPORTED_DISTROS]
f"Unsupported distro {distro!r}; Supported distros are ...| # still be running; Qubes refuses to delete a running VM, so force | ||
| # it down and wait before removing it. | ||
| if self.cloned_qube.is_running(): | ||
| shutdown_domains([self.cloned_qube], self.log) |
There was a problem hiding this comment.
I think kill() is appropriate here as we don't care about this qube anymore.
There was a problem hiding this comment.
Makes sense. While doing this I dropped the preceding is_running() guard too. kill() already raises QubesVMNotStartedError when the qube is halted, so the extra check was a redundant round-trip.
| if upgrader.cloned_qube.klass == "TemplateVM" | ||
| else "standalone" | ||
| ) | ||
| print(f"Upgrade complete. New {label}: {upgrader.cloned_qube.name}") |
There was a problem hiding this comment.
label value should be upgrader.cloned_qube.klass.lower(), so even if ther eare other klass supported in the future, this would handle it.
There was a problem hiding this comment.
Sure. I went with klass.lower().removesuffix("vm") so that output to user is "New template:" / "New standalone:" instead of "New templatevm:" / "New standalonevm:"
You can take the final call
|
In the future, this tool will be tested by openqa and when it is good enough, it should serve this use case: In other words, it should be able to do, what this command from the link, does. After you add API support, check if upgrading simultaneously with CLI and API breaks things or it is safely handled. |
|
@ben-grande Should debian integration be also included in this PR? |
|
Yes please. |
4 participants
This PR introduces the
qvm-template-upgradedom0 command-line utility, that performs an in-place N -> N+1 distribution upgrade of Debian and Fedora TemplateVM or StandaloneVMfixes: QubesOS/qubes-issues#8605
GSoC 2026 project: Automate Template Version Upgrade