-
Notifications
You must be signed in to change notification settings - Fork 5
Restructure os version layout #49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
AnHeuermann
wants to merge
11
commits into
OpenModelica:main
Choose a base branch
from
AnHeuermann:restructure-os-version-layout
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
1ed45ab
Restructure to OS/version layout (Option B); implement Ubuntu
AnHeuermann 476537d
Fixup
AnHeuermann b02a14a
Fix review comments
AnHeuermann ea496b1
Adding GBD add-on
AnHeuermann 7d91b49
Automate rolling release
AnHeuermann fbcb397
Add first draft of Rust addon
sjoelund d021c00
Also install Qt Quick3D and the Qt env for MSVC
sjoelund b74445d
Will also need LLVM for llvm-lib, etc
sjoelund df672fb
Install Quick3D also for the base image, and qt5compat for aqt
sjoelund ef7ad70
We will need wine to run qtdeploy
sjoelund 37080e3
Add qtwebengine and qhttpserver
sjoelund File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| #!/usr/bin/env python3 | ||
| """Helpers for the CI. | ||
|
|
||
| Reads ``.ci/matrix.yml`` (the single source of truth for which images exist) | ||
| and answers questions for the GitHub Actions workflows: | ||
|
|
||
| matrix.py all | ||
| Print, on one line, a JSON array of every image. Used as the | ||
| ``strategy.matrix`` for the build workflow. Each element looks like:: | ||
|
|
||
| {"os": "ubuntu", "version": "24.04", "dir": "ubuntu/24.04", | ||
| "base_tag": "ubuntu-24.04", "context": "ubuntu", | ||
| "dockerfile": "ubuntu/Dockerfile", "target": "full", | ||
| "build_args": "UBUNTU_VERSION=24.04", "addons": "cmake-4"} | ||
|
|
||
| ``target`` is the build stage for the base image ("" = final stage). | ||
| Each add-on in ``addons`` is itself a build stage (--target). Both | ||
| ``build_args`` and ``addons`` are space-separated strings so they can be | ||
| looped over in shell directly. | ||
|
|
||
| matrix.py image <tag> | ||
| Resolve a release tag such as ``ubuntu-24.04-2.1.0`` to the image it | ||
| refers to and print shell ``key='value'`` assignments to stdout:: | ||
|
|
||
| dir='ubuntu/24.04' | ||
| base_tag='ubuntu-24.04' | ||
| semver='2.1.0' | ||
| context='ubuntu' | ||
| dockerfile='ubuntu/Dockerfile' | ||
| target='full' | ||
| build_args='UBUNTU_VERSION=24.04' | ||
| addons='cmake-4' | ||
|
Comment on lines
+22
to
+32
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Simply make the tags v2.1.0 instead and rebuild the images based on that. Needing to tag multiple things is annoying. |
||
|
|
||
| Intended to be consumed with ``eval "$(python .ci/matrix.py image …)"``. | ||
|
|
||
| matrix.py publish-matrix <date> | ||
| Print, on one line, a JSON array of ``{"tag": "<base_tag>-<date>"}`` | ||
| objects for every image in the matrix. Used by the scheduled workflow | ||
| to build the publish job matrix with date-stamped immutable tags:: | ||
|
|
||
| [{"tag":"ubuntu-24.04-2026.06.23"},{"tag":"ubuntu-22.04-2026.06.23"}] | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import os | ||
| import re | ||
| import shlex | ||
| import sys | ||
|
|
||
| import yaml | ||
|
|
||
| HERE = os.path.dirname(os.path.abspath(__file__)) | ||
| MATRIX_FILE = os.path.join(HERE, "matrix.yml") | ||
|
|
||
| SEMVER_RE = re.compile(r"^(?P<prefix>.+)-(?P<semver>\d+\.\d+\.\d+)$") | ||
|
|
||
|
|
||
| def load_images(): | ||
| with open(MATRIX_FILE, encoding="utf-8") as handle: | ||
| data = yaml.safe_load(handle) | ||
| images = data.get("images") or [] | ||
| result = [] | ||
| for img in images: | ||
| os_name = str(img["os"]) | ||
| version = str(img["version"]) | ||
| directory = f"{os_name}/{version}" | ||
| context = str(img.get("context") or f"{directory}/base") | ||
| dockerfile = str(img.get("dockerfile") or f"{context}/Dockerfile") | ||
| target = str(img.get("target") or "") | ||
| build_args = " ".join( | ||
| f"{k}={v}" for k, v in (img.get("build_args") or {}).items() | ||
| ) | ||
| addons = [str(a) for a in (img.get("addons") or [])] | ||
| result.append( | ||
| { | ||
| "os": os_name, | ||
| "version": version, | ||
| "dir": directory, | ||
| "base_tag": f"{os_name}-{version}", | ||
| "context": context, | ||
| "dockerfile": dockerfile, | ||
| "target": target, | ||
| "build_args": build_args, | ||
| "addons": " ".join(addons), | ||
| } | ||
| ) | ||
| return result | ||
|
|
||
|
|
||
| def cmd_all(): | ||
| print(json.dumps(load_images(), separators=(",", ":"))) | ||
|
|
||
|
|
||
| def cmd_publish_matrix(date: str): | ||
| images = load_images() | ||
| result = [{"tag": f"{img['base_tag']}-{date}"} for img in images] | ||
| print(json.dumps(result, separators=(",", ":"))) | ||
|
|
||
|
|
||
| def cmd_image(tag: str): | ||
| match = SEMVER_RE.match(tag) | ||
| if not match: | ||
| sys.exit( | ||
| f"error: tag '{tag}' is not of the form <os>-<version>-<semver> " | ||
| f"(e.g. ubuntu-24.04-2.1.0)" | ||
| ) | ||
| prefix = match.group("prefix") | ||
| semver = match.group("semver") | ||
|
|
||
| for img in load_images(): | ||
| if img["base_tag"] == prefix: | ||
| print(f"dir={shlex.quote(img['dir'])}") | ||
| print(f"base_tag={shlex.quote(img['base_tag'])}") | ||
| print(f"semver={shlex.quote(semver)}") | ||
| print(f"context={shlex.quote(img['context'])}") | ||
| print(f"dockerfile={shlex.quote(img['dockerfile'])}") | ||
| print(f"target={shlex.quote(img['target'])}") | ||
| print(f"build_args={shlex.quote(img['build_args'])}") | ||
| print(f"addons={shlex.quote(img['addons'])}") | ||
| return | ||
|
|
||
| valid = ", ".join(img["base_tag"] for img in load_images()) | ||
| sys.exit( | ||
| f"error: no image matches tag prefix '{prefix}'. " | ||
| f"Known images: {valid}" | ||
| ) | ||
|
|
||
|
|
||
| def main(argv): | ||
| if len(argv) >= 2 and argv[1] == "all": | ||
| cmd_all() | ||
| elif len(argv) >= 3 and argv[1] == "image": | ||
| cmd_image(argv[2]) | ||
| elif len(argv) >= 3 and argv[1] == "publish-matrix": | ||
| cmd_publish_matrix(argv[2]) | ||
| else: | ||
| sys.exit(f"usage: {argv[0]} all | image <tag> | publish-matrix <date>") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main(sys.argv) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # Declares every image this repository builds. | ||
| # | ||
| # Each entry is one OS / OS-version. Every OS uses a single multi-stage | ||
| # Dockerfile at <os>/Dockerfile (e.g. ubuntu/Dockerfile) covering all of its | ||
| # versions: the base image is the `full` stage and add-ons are extra stages. | ||
| # | ||
| # Per-image fields: | ||
| # context build context directory (default: <os>/<version>/base) | ||
| # dockerfile path to the Dockerfile (default: <context>/Dockerfile) | ||
| # build_args map of --build-arg values, e.g. the version (default: none) | ||
| # target build stage for the BASE image (default: final stage) | ||
| # addons list of add-ons. Each add-on is a build STAGE (--target) in the | ||
| # image's own Dockerfile (default: none) | ||
| # The shared-Dockerfile images set context/dockerfile/target explicitly (e.g. | ||
| # all Ubuntu versions use ubuntu/Dockerfile and differ only by UBUNTU_VERSION; | ||
| # the base is the `full` stage and add-ons such as `cmake-4` are extra stages). | ||
| # | ||
| # Tag grammar (see README.md): | ||
| # base moving : <os>-<version> e.g. ubuntu-24.04 | ||
| # base immutable : <os>-<version>-<semver> e.g. ubuntu-24.04-2.1.0 | ||
| # addon moving : <os>-<version>-<addon> e.g. ubuntu-24.04-cmake-4 | ||
| # addon immutable : <os>-<version>-<addon>-<semver> e.g. ubuntu-24.04-cmake-4-2.1.0 | ||
| # | ||
| # <semver> is THIS repository's version (the recipe), independent of the | ||
| # OpenModelica version. Pushing the git tag `<os>-<version>-<semver>` releases | ||
| # the base image and all of its add-ons (see RELEASING.md). | ||
|
|
||
| images: | ||
| # All Ubuntu versions share the multi-stage ubuntu/Dockerfile; only | ||
| # UBUNTU_VERSION differs. The base image is the `full` stage; add-ons are | ||
| # further stages (e.g. `cmake-4`). | ||
| - os: ubuntu | ||
| version: "26.04" | ||
| context: ubuntu | ||
| dockerfile: ubuntu/Dockerfile | ||
| target: full | ||
| build_args: | ||
| UBUNTU_VERSION: "26.04" | ||
| addons: | ||
| - rust | ||
|
|
||
| - os: ubuntu | ||
| version: "24.04" | ||
| context: ubuntu | ||
| dockerfile: ubuntu/Dockerfile | ||
| target: full | ||
| build_args: | ||
| UBUNTU_VERSION: "24.04" | ||
| addons: | ||
| - cmake-4 | ||
|
|
||
| - os: ubuntu | ||
| version: "22.04" | ||
| context: ubuntu | ||
| dockerfile: ubuntu/Dockerfile | ||
| target: full | ||
| build_args: | ||
| UBUNTU_VERSION: "22.04" | ||
| addons: [] | ||
|
|
||
| # Placeholders — not implemented yet (see <os>/Dockerfile). Each OS uses the | ||
| # same single multi-stage Dockerfile layout as Ubuntu. Uncomment and flesh out | ||
| # once the corresponding Dockerfile is real, so CI starts building them. | ||
| # | ||
| # - os: debian | ||
| # version: "13" | ||
| # context: debian | ||
| # dockerfile: debian/Dockerfile | ||
| # target: full | ||
| # build_args: | ||
| # DEBIAN_VERSION: "13" | ||
| # addons: [] | ||
| # | ||
| # - os: almalinux | ||
| # version: "9" | ||
| # context: almalinux | ||
| # dockerfile: almalinux/Dockerfile | ||
| # target: full | ||
| # build_args: | ||
| # ALMALINUX_VERSION: "9" | ||
| # addons: [] | ||
| # | ||
| # - os: arch | ||
| # version: "rolling" | ||
| # context: arch | ||
| # dockerfile: arch/Dockerfile | ||
| # target: full | ||
| # addons: [] | ||
| # | ||
| # - os: opensuse-leap | ||
| # version: "16.0" | ||
| # context: opensuse-leap | ||
| # dockerfile: opensuse-leap/Dockerfile | ||
| # target: full | ||
| # build_args: | ||
| # LEAP_VERSION: "16.0" | ||
| # addons: [] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # Build and push one image (base + all its add-ons) to a registry. | ||
| # | ||
| # Used by both publish.yml (GHCR) and publish-nexus.yml (Nexus). The base image | ||
| # is pushed under a moving tag (<os>-<version>) and an immutable tag | ||
| # (<os>-<version>-<semver>). Each add-on is a build STAGE (--target) in the same | ||
| # Dockerfile and is pushed under its own moving + immutable tags; shared layers | ||
| # come from the build cache, so the base is effectively built only once. | ||
| # | ||
| # Required environment variables: | ||
| # REGISTRY Image repository, e.g. ghcr.io/openmodelica/build-deps | ||
| # TAG Release tag, e.g. ubuntu-24.04-2.1.0 | ||
| # SIGN "true" to cosign-sign every pushed tag (keyless OIDC), else "false" | ||
| set -euo pipefail | ||
|
|
||
| : "${REGISTRY:?REGISTRY is required}" | ||
| : "${TAG:?TAG is required}" | ||
| SIGN="${SIGN:-false}" | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
|
|
||
| # Resolve the tag to base_tag / semver / context / dockerfile / target / | ||
| # build_args / addons. | ||
| eval "$(python3 "${SCRIPT_DIR}/matrix.py" image "${TAG}")" | ||
|
|
||
| PUSHED_TAGS=() | ||
|
|
||
| build_and_push() { | ||
| # $1 = moving tag, $2 = immutable tag, remaining args go to `docker buildx`. | ||
| local moving="$1" immutable="$2" | ||
| shift 2 | ||
| echo "::group::Building ${moving}" | ||
| docker buildx build \ | ||
| --pull \ | ||
| --file "${dockerfile}" \ | ||
| --tag "${REGISTRY}:${moving}" \ | ||
| --tag "${REGISTRY}:${immutable}" \ | ||
| --push \ | ||
| "$@" \ | ||
| "${context}" | ||
| echo "::endgroup::" | ||
| PUSHED_TAGS+=("${REGISTRY}:${moving}" "${REGISTRY}:${immutable}") | ||
| } | ||
|
|
||
| # Common build-args (e.g. UBUNTU_VERSION) apply to every stage. | ||
| common_args=() | ||
| for kv in ${build_args}; do | ||
| common_args+=(--build-arg "${kv}") | ||
| done | ||
|
|
||
| # 1. Base image: the image's `target` stage (or the final stage). | ||
| base_target_arg=() | ||
| [ -n "${target}" ] && base_target_arg=(--target "${target}") | ||
| build_and_push "${base_tag}" "${base_tag}-${semver}" \ | ||
| "${common_args[@]}" "${base_target_arg[@]}" | ||
|
|
||
| # 2. Add-ons: each is a --target stage in the same Dockerfile. | ||
| for addon in ${addons}; do | ||
| build_and_push "${base_tag}-${addon}" "${base_tag}-${addon}-${semver}" \ | ||
| "${common_args[@]}" --target "${addon}" | ||
| done | ||
|
|
||
| # 3. Optionally sign every pushed tag (GHCR / cosign keyless). | ||
| if [ "${SIGN}" = "true" ]; then | ||
| for image in "${PUSHED_TAGS[@]}"; do | ||
| echo "Signing ${image}" | ||
| cosign sign --yes "${image}" | ||
| done | ||
| fi | ||
|
|
||
| printf '%s\n' "${PUSHED_TAGS[@]}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| pyyaml==6.0.3 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that important? A list would be much more natural...