diff --git a/.gitignore b/.gitignore index 771ac766e..f7b76ad54 100644 --- a/.gitignore +++ b/.gitignore @@ -4,7 +4,6 @@ public/ themes/flatcar/node_modules themes/flatcar/yarn.lock themes/flatcar/static/css/.sass-cache -external-docs/ resources/ # Temporary lock file while building diff --git a/README.md b/README.md index f27dd2e69..6c4d916b5 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,33 @@ categories: *TODO: Add more content sections (events, job postings, etc.)* +## Documentation Versioning + +Documentation versions are stored in `content/docs/` and sorted by a `weight` parameter in each version's `_index.md` frontmatter. Higher weight = newer version. + +### Directory structure + +- Current version: `content/docs/latest/` (highest weight) +- Snapshots: `content/docs/DD-MM-YYYY/` (date when they became outdated, lower weights) + +### Creating a new version snapshot + +When you need to snapshot the current documentation: + +1. **Rename `latest` to today's date and lower its weight**: + ```bash + mv content/docs/latest content/docs/$(date +%d-%m-%Y) + # Edit content/docs/DD-MM-YYYY/_index.md and set weight to previous value - 1 + ``` + +2. **Copy the snapshot back to `latest` with a higher weight**: + ```bash + cp -r content/docs/DD-MM-YYYY content/docs/latest + # Edit content/docs/latest/_index.md and set weight to highest value (e.g., 40) + ``` + +3. **Make your changes** in `content/docs/latest/` + ### Presentations Create HTML presentations from screenshots using Marp (requires Docker): @@ -104,51 +131,3 @@ The above command will run a server with the the site available at `http://local ### Testing Pull requests Each pull request will run some checks and create a new preview of the changes that can be access by clicking on the Github pull request status section. - -### Testing documentation locally - -If you are working on documentation and would like to see the changes -be reflected in a local run of the website, then you need to generate a -module to import the docs and run the website locally with it. - -You can use the `tools/preview_docs.sh` script for conveniently generating -an import module, e.g.: - -`./tools/preview_docs.sh ../flatcar-docs/docs new_latest` - -(this creates a `tmp_modules.yaml`) - -and then start the local server again with `make run`. - -## Changing the published documentation - -The documentation is set under `params.docs` in [config.yaml](./config.yaml) and -should look similar to: - -``` -github_edit_url: https://github.com/flatcar/flatcar-docs/edit/main/docs/ -issues_url: https://github.com/kinvolk/flatcar/issues/new?labels=kind/docs -external_docs: -- repo: https://github.com/flatcar/flatcar-docs.git - name: "latest" - branch: "main" - dir: "docs" -``` - -If you want to add a new version of the documentation, this can be done by adding a new entry to external_docs: -``` -external_docs: -- repo: https://github.com/flatcar/flatcar-docs.git - name: "latest" - branch: "main" - dir: "docs" -- repo: https://github.com/flatcar/flatcar-docs.git - name: "old" - branch: "tag-1.2.3" - dir: "docs" -``` - -This will pull the docs that were versioned by the `tag-1.2.3` and place them under a version called `old`. - -By default, the first version in the list of `external_docs` is considered to be -the latest version and so it is the one linked to automatically in the site. diff --git a/config.yaml b/config.yaml index 139dbe8a3..79382117f 100644 --- a/config.yaml +++ b/config.yaml @@ -43,13 +43,8 @@ params: url: "https://app.element.io/#/room/#flatcar:matrix.org" favicon: "/media/brand-logo.svg" docs: - github_edit_url: https://github.com/flatcar/flatcar-website/edit/main/content/docs/latest + base_github_edit_url: https://github.com/flatcar/flatcar-website/edit/main/content/ issues_url: https://github.com/flatcar/flatcar/issues/new?labels=kind/docs - external_docs: - - repo: https://github.com/flatcar/flatcar-website.git - name: "latest" - branch: "master" - dir: "content/docs" pygmentsCodefences: true pygmentsCodefencesGuessSyntax: true diff --git a/content/docs/.gitignore b/content/docs/.gitignore index f20385a1a..51527e2bf 100644 --- a/content/docs/.gitignore +++ b/content/docs/.gitignore @@ -1 +1 @@ -_index.md +/_index.md diff --git a/content/docs/23-10-2025/_index.md b/content/docs/23-10-2025/_index.md new file mode 100644 index 000000000..d65550d18 --- /dev/null +++ b/content/docs/23-10-2025/_index.md @@ -0,0 +1,273 @@ +--- +content_type: flatcar +title: Flatcar Container Linux +main_menu: true +weight: 39 +--- + +Flatcar Container Linux is a container optimized OS that ships a minimal OS +image, which includes only the tools needed to run containers. The OS is +shipped through an immutable filesystem and includes automatic atomic +updates. + + +### Getting Started + +If you're new to Flatcar and if you're looking for a brief introduction on getting Flatcar up and running, please have a look at our [quickstart guide][quick-start]. + +### Installing Flatcar + +Flatcar Container Linux runs on most cloud providers, virtualization +platforms and bare metal servers. + +#### Cloud Providers + * [Amazon EC2][ec2] + * [Microsoft Azure][azure] + * [Google Compute Engine][gce] + * [Equinix Metal][equinix-metal] + * [VMware][vmware] + * [DigitalOcean][digital-ocean] + * [Hetzner][hetzner] + * [OpenStack][openstack] + * [Brightbox][brightbox] + * [Scaleway][scaleway] (community support) + * [OVHcloud][ovhcloud] (community support) + * [Akamai/Linode][akamai] + * [STACKIT][stackit] + +#### Virtualization options +It's easy to run a local Flatcar VM on your laptop for testing and debugging +purposes. You can use any of the following options. + + * [QEMU][qemu] + * [libVirt][libvirt] + * [VirtualBox][virtualbox] (community support) + * [Vagrant][vagrant] (community support) + * [Hyper-V][hyper-v] (community support) + * [KubeVirt][kubevirt] (community support) + * [Proxmox VE][proxmoxve] (community support) + +#### Bare Metal +You can install Flatcar on bare metal machines in different ways: using ISO +images, booting from PXE or iPXE, and even by running an installation +script on an existing Linux system. + + * [Installing from ISO images][boot-iso] + * [Booting with PXE][pxe] + * [Booting with iPXE][ipxe] + * [Installing with flatcar-install][install-to-disk] + +If you want to provide metadata to your baremetal machines, we recommend +using [Matchbox][matchbox]. + +#### Upgrading from CoreOS Container Linux + +Flatcar Container Linux is a drop-in replacement of CoreOS Container Linux. +If you are a CoreOS Container Linux user looking for a replacement, +checkout our guides to [migrate from CoreOS Container +Linux][migrate-from-container-linux], or you can [update from CoreOS +Container Linux][update-from-container-linux] directly. + +### Provisioning Tools + +[Ignition][ignition-what] is the recommended way to provision Flatcar +Container Linux at first boot. Ignition uses a JSON configuration file, +and it is recommended to generate it from the [Container Linux +Config][container-linux-config] YAML format, which has additional features. +The [Container Linux Config Transpiler][config-transpiler] converts a +Container Linux Config to an Ignition config. + + * [Understanding the Boot Process][ignition-boot] + * [Configuring the Network with Ignition][ignition-network] + * [Using metadata during provisioning][ignition-metadata] + * [Getting started with Butane][config-intro] + * [Examples of using Butane][config-examples] + * [Using Terraform to provision Flatcar Container Linux][terraform] + * [Extending the base OS with systemd-sysext images][sysext] + +### Setting Flatcar Up and Common Operations + +Follow these guides to connect your machines together as a cluster, +configure machine parameters, create users, inject multiple SSH keys, and +more. + +#### Customizing Flatcar + * [Using networkd to customize networking][networkd-customize] + * [Using systemd drop-in units][systemd-drop-in] + * [Using environment variables in systemd units][environment-variables-systemd] + * [Using systemd and udev rules][udev-rules] + * [Using NVIDIA GPUs on Flatcar][using-nvidia] + * [Scheduling tasks with systemd timers][tasks-with-systemd] + * [Configuring DNS][dns] + * [Configuring date & timezone][date-timezone] + * [Adding users][users] + * [Kernel modules / sysctl parameters][parameters] + * [Adding swap][swap] + * [Power management][power-management] + * [ACPI][acpi] + +#### Managing Releases and Updates + * [Switching release channels][release-channels] + * [Configuring the update strategy][update-strategies] + * [Flatcar update configuration specification][update-conf] + * [Verifying Flatcar Images with GPG][verify-container-linux] + * [Nebraska][nebraska] + +#### Creating Clusters + * [Cluster architectures][cluster-architectures] + * [Clustering machines][clustering-machines] + * [Using Amazon EC2 Container Service][ec2-container-service] + +#### Managing Storage + * [Using RAID for the root filesystem][filesystem-placement] + * [Adding disk space][disk-space] + * [Mounting storage][mounting-storage] + * [iSCSI configuration][iscsi] + * [ZFS Extension][zfsextension] + +#### Additional security options + * [Setting up LUKS disk encryption][luks-encryption] + * [Customizing the SSH daemon][ssh-daemon] + * [Configuring SSSD on Flatcar Container Linux][sssd-container-linux] + * [Hardening a Flatcar Container Linux machine][hardening-container-linux] + * [Trusted Computing Hardware Requirements][hardware-requirements] + * [Adding Cert Authorities][cert-authorities] + * [Using SELinux][selinux] + * [Disabling SMT][disabling-smt] + * [Enabling FIPS][enabling-fips] + * [Using the audit subsystem][audit-system] + +#### Debugging Flatcar + * [Install debugging tools][debugging-tools] + * [Working with btrfs][btrfs] + * [Reading the system log][system-log] + * [Collecting crash logs][crash-log] + * [Manual Flatcar Container Linux rollbacks][container-linux-rollbacks] + +### Container Runtimes +Flatcar Container Linux supports all of the popular methods for running +containers, and you can choose to interact with the containers at a +low-level, or use a higher level orchestration framework. Listed below are +some guides to help you choose and make use of the different runtimes. + + * [Getting started with Docker][docker] + * [Customizing Docker][customizing-docker] + * [Using systemd to manage Docker containers][manage-docker-containers] + * [Use a custom Docker or containerd version][use-a-custom-docker-or-containerd-version] + * [Authenticating to Container registries][registry-authentication] + * [Getting started with Kubernetes][kubernetes] + * [High availability Kubernetes][ha-kubernetes] + +### Developer guides and Reference +APIs and troubleshooting guides for working with Flatcar Container Linux. + +* [Developer guides][developer-guides]: Comprehensive guides on developing for Flatcar, working with the SDK, and on building and extending OS images. +* [Integrations][integrations] +* [Migrating from cloud-config to Container Linux Config][migrating-from-cloud-config] +* [Flatcar Supply Chain Security (SLSA and SPDX SBOM)][supply-chain-security] detailing security mechanisms employed at build / release time as well as at run-time to ensure validity of inputs processed and outputs shipped. + +### Tutorial +Flatcar tutorial to deep dive into some Flatcar fundamental concepts. +* [Introduction][tutorial-introduction] +* [Hands-on 1: Discovering][tutorial-hands-on-1] +* [Hands-on 2: Provisioning][tutorial-hands-on-2] +* [Hands-on 3: Deploying][tutorial-hands-on-3] +* [Hands-on 4: Updating][tutorial-hands-on-4] + +[quick-start]: installing +[supply-chain-security]: reference/supply-chain +[ignition-what]: provisioning/ignition/ +[ignition-boot]: provisioning/ignition/boot-process +[ignition-network]: provisioning/ignition/network-configuration +[ignition-metadata]: provisioning/ignition/dynamic-data +[container-linux-config]: provisioning/cl-config/ +[config-transpiler]: provisioning/config-transpiler/ +[config-intro]: provisioning/config-transpiler/getting-started +[config-dynamic-data]: provisioning/config-transpiler/dynamic-data +[config-examples]: provisioning/config-transpiler/examples +[matchbox]: https://matchbox.psdn.io/ +[ipxe]: installing/bare-metal/booting-with-ipxe +[pxe]: installing/bare-metal/booting-with-pxe +[install-to-disk]: installing/bare-metal/installing-to-disk +[boot-iso]: installing/bare-metal/booting-with-iso +[filesystem-placement]: setup/storage/raid +[migrate-from-container-linux]: migrating-from-coreos/ +[update-from-container-linux]: migrating-from-coreos/update-from-container-linux +[ec2]: installing/cloud/aws-ec2 +[digital-ocean]: installing/cloud/digitalocean +[gce]: installing/cloud/gcp +[azure]: installing/cloud/azure +[qemu]: installing/vms/qemu +[equinix-metal]: installing/cloud/equinix-metal +[libvirt]: installing/vms/libvirt +[virtualbox]: installing/vms/virtualbox +[vagrant]: installing/vms/vagrant +[hyper-v]: installing/vms/hyper-v +[kubevirt]: installing/vms/kubevirt +[proxmoxve]: installing/community-platforms/proxmoxve +[vmware]: installing/cloud/vmware +[cluster-architectures]: setup/clusters/architectures +[update-strategies]: setup/releases/update-strategies +[clustering-machines]: setup/clusters/discovery +[verify-container-linux]: setup/releases/verify-images +[networkd-customize]: setup/customization/network-config-with-networkd +[systemd-drop-in]: setup/systemd/drop-in-units +[environment-variables-systemd]: setup/systemd/environment-variables +[dns]: setup/customization/configuring-dns +[date-timezone]: setup/customization/configuring-date-and-timezone +[users]: setup/customization/adding-users +[parameters]: setup/customization/other-settings +[disk-space]: setup/storage/adding-disk-space +[mounting-storage]: setup/storage/mounting-storage +[power-management]: setup/customization/power-management +[registry-authentication]: container-runtimes/registry-authentication +[iscsi]: setup/storage/iscsi +[swap]: setup/storage/adding-swap +[zfsextension]: setup/storage/zfs +[ec2-container-service]: setup/clusters/booting-on-ecs/ +[manage-docker-containers]: setup/systemd/getting-started +[udev-rules]: setup/systemd/udev-rules +[update-conf]: setup/releases/update-conf +[release-channels]: setup/releases/switching-channels +[tasks-with-systemd]: setup/systemd/timers +[luks-encryption]: setup/security/luks +[ssh-daemon]: setup/security/customizing-sshd +[sssd-container-linux]: setup/security/sssd +[hardening-container-linux]: setup/security/hardening-guide +[hardware-requirements]: setup/security/trusted-computing-hardware-requirements +[cert-authorities]: setup/security/adding-certificate-authorities +[selinux]: setup/security/selinux +[disabling-smt]: setup/security/disabling-smt +[enabling-fips]: setup/security/fips +[audit-system]: setup/security/audit +[debugging-tools]: setup/debug/install-debugging-tools +[btrfs]: setup/debug/btrfs-troubleshooting +[system-log]: setup/debug/reading-the-system-log +[crash-log]: setup/debug/collecting-crash-logs +[container-linux-rollbacks]: setup/debug/manual-rollbacks +[docker]: container-runtimes/getting-started-with-docker +[customizing-docker]: container-runtimes/customizing-docker +[use-a-custom-docker-or-containerd-version]: container-runtimes/use-a-custom-docker-or-containerd-version +[developer-guides]: reference/developer-guides/ +[integrations]: reference/integrations/ +[migrating-from-cloud-config]: provisioning/cl-config/from-cloud-config +[containerd-for-kubernetes]: container-runtimes/switching-from-docker-to-containerd-for-kubernetes +[terraform]: provisioning/terraform/ +[hetzner]: installing/cloud/hetzner +[sysext]: provisioning/sysext/ +[acpi]: setup/customization/ACPI +[openstack]: installing/cloud/openstack +[brightbox]: installing/cloud/brightbox +[kubernetes]: container-runtimes/getting-started-with-kubernetes +[ha-kubernetes]: container-runtimes/high-availability-kubernetes +[using-nvidia]: setup/customization/using-nvidia +[tutorial-introduction]: tutorial/ +[tutorial-hands-on-1]: tutorial/hands-on-1 +[tutorial-hands-on-2]: tutorial/hands-on-2 +[tutorial-hands-on-3]: tutorial/hands-on-3 +[tutorial-hands-on-4]: tutorial/hands-on-4 +[scaleway]: installing/community-platforms/scaleway +[ovhcloud]: installing/community-platforms/ovhcloud +[akamai]: installing/cloud/akamai +[nebraska]: nebraska +[stackit]: installing/cloud/stackit diff --git a/content/docs/23-10-2025/container-runtimes/_index.md b/content/docs/23-10-2025/container-runtimes/_index.md new file mode 100644 index 000000000..09837b7e2 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/_index.md @@ -0,0 +1,10 @@ +--- +title: Container Runtimes +description: > + Flatcar Container Linux supports all of the popular methods for running + containers, and you can choose to interact with the containers at a + low-level, or use a higher level orchestration framework. These guides + can help you choose and use the different container runtimes supported. +weight: 60 +--- + diff --git a/content/docs/23-10-2025/container-runtimes/customizing-docker.md b/content/docs/23-10-2025/container-runtimes/customizing-docker.md new file mode 100644 index 000000000..ea4185288 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/customizing-docker.md @@ -0,0 +1,383 @@ +--- +title: Customizing Docker +description: > + How to select which runtime to use, make docker available on a + TCP socket, enable TLS, and other customizations. +weight: 30 +aliases: + - ../os/customizing-docker +--- + +The Docker systemd unit can be customized by overriding the unit that ships with the default Flatcar Container Linux settings or through a drop-in unit. Common use-cases for doing this are covered below. + +For switching to using containerd with Kubernetes, there is an [extra guide](../switching-from-docker-to-containerd-for-kubernetes/). + +## Use a custom containerd configuration + +The default configuration under `/run/torcx/unpack/docker/usr/share/containerd/config.toml` can't be changed but you can copy it to `/etc/containerd/config.toml` and modify it. +**NOTE** that newer Flatcar major releases (above major release version 3760) ship the default configuration under `/usr/share/containerd/config.toml`. + +Create a `/etc/systemd/system/containerd.service.d/10-use-custom-config.conf` unit drop-in file to select the new configuration: + +```ini +[Service] +ExecStart= +ExecStart=/usr/bin/containerd +``` + +On a running system, execute `systemctl daemon-reload ; systemctl restart containerd` for it to take effect. + +## Enable the remote API on a new socket + +Create a file called `/etc/systemd/system/docker-tcp.socket` to make Docker available on a TCP socket on port 2375. + +```ini +[Unit] +Description=Docker Socket for the API + +[Socket] +ListenStream=2375 +BindIPv6Only=both +Service=docker.service + +[Install] +WantedBy=sockets.target +``` + +Then enable this new socket: + +```shell +systemctl enable docker-tcp.socket +systemctl stop docker +systemctl start docker-tcp.socket +systemctl start docker +``` + +Test that it's working: + +```shell +docker -H tcp://127.0.0.1:2375 ps +``` + +### Butane Config + +To enable the remote API on every Flatcar Container Linux machine in a cluster, use a [Butane Config][butane-configs]. We need to provide the new socket file and Docker's socket activation support will automatically start using the socket: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker-tcp.socket + enabled: true + contents: | + [Unit] + Description=Docker Socket for the API + + [Socket] + ListenStream=2375 + BindIPv6Only=both + Service=docker.service + + [Install] + WantedBy=sockets.target +``` + +To keep access to the port local, replace the `ListenStream` configuration above with: + +```yaml + [Socket] + ListenStream=127.0.0.1:2375 +``` + +## Enable the remote API with TLS authentication + +Docker TLS configuration consists of three parts: keys creation, configuring new [systemd socket][systemd-socket] unit and systemd [drop-in][drop-in] configuration. + +### TLS keys creation + +Please follow the [instruction][self-signed-certs] to know how to create self-signed certificates and private keys. Then copy the following files into `/etc/docker` Flatcar Container Linux's directory and fix their permissions: + +```shell +scp ~/cfssl/{server.pem,server-key.pem,ca.pem} flatcar.example.com: +ssh core@flatcar.example.com +sudo mv {server.pem,server-key.pem,ca.pem} /etc/docker/ +sudo chown root:root /etc/docker/{server-key.pem,server.pem,ca.pem} +sudo chmod 0600 /etc/docker/server-key.pem +``` + +On your local host copy certificates into `~/.docker`: + +```shell +mkdir ~/.docker +chmod 700 ~/.docker +cd ~/.docker +cp -p ~/cfssl/ca.pem ca.pem +cp -p ~/cfssl/client.pem cert.pem +cp -p ~/cfssl/client-key.pem key.pem +``` + +### Enable the secure remote API on a new socket + +Create a file called `/etc/systemd/system/docker-tls-tcp.socket` to make Docker available on a secured TCP socket on port 2376. + +```ini +[Unit] +Description=Docker Secured Socket for the API + +[Socket] +ListenStream=2376 +BindIPv6Only=both +Service=docker.service + +[Install] +WantedBy=sockets.target +``` + +Then enable this new socket: + +```shell +systemctl enable docker-tls-tcp.socket +systemctl stop docker +systemctl start docker-tls-tcp.socket +``` + +### Drop-in configuration + +Create `/etc/systemd/system/docker.service.d/10-tls-verify.conf` [drop-in][drop-in] for systemd Docker service: + +```ini +[Service] +Environment="DOCKER_OPTS=--tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server.pem --tlskey=/etc/docker/server-key.pem" +``` + +Reload systemd config files and restart docker service: + +```shell +sudo systemctl daemon-reload +sudo systemctl restart docker.service +``` + +Now you can access your Docker's API through TLS secured connection: + +```shell +docker --tlsverify -H tcp://server:2376 images +# or +docker --tlsverify -H tcp://server.example.com:2376 images +``` + +If you've experienceed problems connection to remote Docker API using TLS connection, you can debug it with `curl`: + +```shell +curl -v --cacert ~/.docker/ca.pem --cert ~/.docker/cert.pem --key ~/.docker/key.pem https://server:2376 +``` + +Or on your Flatcar Container Linux host: + +```shell +journalctl -f -u docker.service +``` + +In addition you can export environment variables and use docker client without additional options: + +```shell +export DOCKER_HOST=tcp://server.example.com:2376 DOCKER_TLS_VERIFY=1 +docker images +``` + +### Butane Config (TLS) + +A Butane Config for Docker TLS authentication will look like: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + files: + - path: /etc/docker/ca.pem + mode: 0644 + contents: + inline: | + -----BEGIN CERTIFICATE----- + MIIFNDCCAx6gAwIBAgIBATALBgkqhkiG9w0BAQswLTEMMAoGA1UEBhMDVVNBMRAw + DgYDVQQKEwdldGNkLWNhMQswCQYDVQQLEwJDQTAeFw0xNTA5MDIxMDExMDhaFw0y + NTA5MDIxMDExMThaMC0xDDAKBgNVBAYTA1VTQTEQMA4GA1UEChMHZXRjZC1jYTEL + ... ... ... + - path: /etc/docker/server.pem + mode: 0644 + contents: + inline: | + -----BEGIN CERTIFICATE----- + MIIFajCCA1SgAwIBAgIBBTALBgkqhkiG9w0BAQswLTEMMAoGA1UEBhMDVVNBMRAw + DgYDVQQKEwdldGNkLWNhMQswCQYDVQQLEwJDQTAeFw0xNTA5MDIxMDM3MDFaFw0y + NTA5MDIxMDM3MDNaMEQxDDAKBgNVBAYTA1VTQTEQMA4GA1UEChMHZXRjZC1jYTEQ + ... ... ... + - path: /etc/docker/server-key.pem + mode: 0644 + contents: + inline: | + -----BEGIN RSA PRIVATE KEY----- + MIIJKAIBAAKCAgEA23Q4yELhNEywScrHl6+MUtbonCu59LIjpxDMAGxAHvWhWpEY + P5vfas8KgxxNyR+U8VpIjEXvwnhwCx/CSCJc3/VtU9v011Ir0WtTrNDocb90fIr3 + YeRWq744UJpBeDHPV9opf8xFE7F74zWeTVMwtiMPKcQDzZ7XoNyJMxg1wmiMbdCj + ... ... ... +systemd: + units: + - name: docker-tls-tcp.socket + enabled: true + contents: | + [Unit] + Description=Docker Secured Socket for the API + + [Socket] + ListenStream=2376 + BindIPv6Only=both + Service=docker.service + + [Install] + WantedBy=sockets.target + - name: docker.service + dropins: + - name: flags.conf + contents: | + [Service] + Environment="DOCKER_OPTS=--tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server.pem --tlskey=/etc/docker/server-key.pem" +``` + +## Use attached storage for Docker images + +Docker containers can be very large and debugging a build process makes it easy to accumulate hundreds of containers. It's advantageous to use attached storage to expand your capacity for container images. Check out the guide to [mounting storage to your Flatcar Container Linux machine][mounting-storage] for an example of how to bind mount storage into `/var/lib/docker`. + +## Enabling the Docker debug flag + +Set the `--debug` (`-D`) flag in the `DOCKER_OPTS` environment variable by using a drop-in file. For example, the following could be written to `/etc/systemd/system/docker.service.d/10-debug.conf`: + +```ini +[Service] +Environment=DOCKER_OPTS=--debug +``` + +Now tell systemd about the new configuration and restart Docker: + +```shell +systemctl daemon-reload +systemctl restart docker +``` + +To test our debugging stream, run a Docker command and then read the systemd journal, which should contain the output: + +```shell +docker ps +journalctl -u docker +``` + +### Butane Config (flags) + +If you need to modify a flag across many machines, you can add the flag with a Butane Config: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker.service + dropins: + - name: flags.conf + contents: | + [Service] + Environment="DOCKER_OPTS=--debug" +``` + +## Use an HTTP proxy + +If you're operating in a locked down networking environment, you can specify an HTTP proxy for Docker to use via an environment variable. First, create a directory for drop-in configuration for Docker: + +```shell +mkdir /etc/systemd/system/docker.service.d +``` + +Now, create a file called `/etc/systemd/system/docker.service.d/http-proxy.conf` that adds the environment variable: + +```ini +[Service] +Environment="HTTP_PROXY=http://proxy.example.com:8080" +``` + +To apply the change, reload the unit and restart Docker: + +```shell +systemctl daemon-reload +systemctl restart docker +``` + +Proxy environment variables can also be set [system-wide][systemd-env-vars]. + +### Butane Config (proxy) + +The easiest way to use this proxy on all of your machines is via a Butane Config: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker.service + enabled: true + dropins: + - name: 20-http-proxy.conf + contents: | + [Service] + Environment="HTTP_PROXY=http://proxy.example.com:8080" +``` + +## Increase ulimits + +If you need to increase certain ulimits that are too low for your application by default, like memlock, you will need to modify the Docker service to increase the limit. First, create a directory for drop-in configuration for Docker: + +```shell +mkdir /etc/systemd/system/docker.service.d +``` + +Now, create a file called `/etc/systemd/system/docker.service.d/increase-ulimit.conf` that adds increased limit: + +```ini +[Service] +LimitMEMLOCK=infinity +``` + +To apply the change, reload the unit and restart Docker: + +```shell +systemctl daemon-reload +systemctl restart docker +``` + +### Butane Config (ulimits) + +The easiest way to use these new ulimits on all of your machines is via a Butane Config: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + - name: docker.service + enabled: true + dropins: + - name: 30-increase-ulimit.conf + contents: | + [Service] + LimitMEMLOCK=infinity +``` + +## Using a dockercfg file for authentication + +A json file `.dockercfg` can be created in your home directory that holds authentication information for a public or private Docker registry. + +[docker-socket-systemd]: https://github.com/docker/docker/pull/17211 +[drop-in]: ../setup/systemd/drop-in-units +[mounting-storage]: ../setup/storage/mounting-storage +[self-signed-certs]: ../setup/security/generate-self-signed-certificates +[systemd-socket]: https://www.freedesktop.org/software/systemd/man/systemd.socket.html +[systemd-env-vars]: ../setup/systemd/environment-variables/#system-wide-environment-variables +[butane-configs]: ../../provisioning/config-transpiler diff --git a/content/docs/23-10-2025/container-runtimes/getting-started-with-docker.md b/content/docs/23-10-2025/container-runtimes/getting-started-with-docker.md new file mode 100644 index 000000000..275dd1d79 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/getting-started-with-docker.md @@ -0,0 +1,175 @@ +--- +title: Getting started with Docker +description: Basic Docker operations on Flatcar +weight: 10 +aliases: + - ../os/getting-started-with-docker +--- + +Docker is an open-source project that makes creating and managing Linux containers really easy. Containers are like extremely lightweight VMs – they allow code to run in isolation from other containers but safely share the machine’s resources, all without the overhead of a hypervisor. + +Docker containers can boot extremely fast (in milliseconds!) which gives you unprecedented flexibility in managing load across your cluster. For example, instead of running chef on each of your VMs, it’s faster and more reliable to have your build system create a container and launch it on the appropriate number of Flatcar Container Linux hosts. This guide will show you how to launch a container, install some software on it, commit that container, and optionally launch it on another Flatcar Container Linux machine. Before starting, make sure you've got at least one Flatcar Container Linux machine up and running — try it on [Amazon EC2][aws-ec2] or locally with [Vagrant][vagrant]. + +## Docker CLI basics + +Docker has a [straightforward CLI][docker-cli] that allows you to do almost everything you could want to a container. All of these commands use the image id (ex. be29975e0098), the image name (ex. myusername/webapp) and the container id (ex. 72d468f455ea) interchangeably depending on the operation you are trying to do. This is confusing at first, so pay special attention to what you're using. + +## Launching a container + +Launching a container is simple as `docker run` + the image name you would like to run + the command to run within the container. If the image doesn't exist on your local machine, Docker will attempt to fetch it from the public image registry. Later we'll explore how to use Docker with a private registry. It's important to note that containers are designed to stop once the command executed within them has exited. For example, if you ran `/bin/echo hello world` as your command, the container will start, print hello world and then stop: + +```shell +docker run ubuntu /bin/echo hello world +``` + +Let's launch an Ubuntu container and install Apache inside of it using the bash prompt: + +```shell +docker run -t -i ubuntu /bin/bash +``` + +The `-t` and `-i` flags allocate a pseudo-tty and keep stdin open even if not attached. This will allow you to use the container like a traditional VM as long as the bash prompt is running. Install Apache with `apt-get update && apt-get install apache2`. You're probably wondering what address you can connect to in order to test that Apache was correctly installed...we'll get to that after we commit the container. + +## Committing a container + +After that completes, we need to `commit` these changes to our container with the container ID and the image name. + +To find the container ID, open another shell (so the container is still running) and read the ID using `docker ps`. + +The image name is in the format of `username/name`. We're going to use `flatcar` as our username in this example but you should [sign up for a Docker.IO user account][docker-signup] and use that instead. + +It's important to note that you can commit using any username and image name locally, but to push an image to the public registry, the username must be a valid [Docker.IO user account][docker-signup]. + +Commit the container with the container ID, your username, and the name `apache`: + +```shell +docker commit 72d468f455ea myname/myapache +``` + +The overlay filesystem works similar to git: our image now builds off of the `ubuntu` base and adds another layer with Apache on top. These layers get cached separately so that you won't have to pull down the ubuntu base more than once. + +## Keeping the Apache container running + +Now we have our Ubuntu container with Apache running in one shell and an image of that container sitting on disk. Let's launch a new container based on that image but set it up to keep running indefinitely. The basic syntax looks like this, but we need to configure a few additional options that we'll fill in as we go: + +```shell +docker run [options] [image] [process] +``` + +The first step is to tell Docker that we want to run our `myname/myapache` image: + +```shell +docker run [options] myname/myapache [process] +``` + +### Run container detached + +When running Docker containers manually, the most important option is to run the container in detached mode with the `-d` flag. This will output the container ID to show that the command was successful, but nothing else. At any time you can run `docker ps` in the other shell to view a list of the running containers. Our command now looks like: + +```shell +docker run -d myname/myapache [process] +``` + +After you are comfortable with the mechanics of running containers by hand, it's recommended to use [systemd units][systemd-getting-started] to run your containers on a cluster of Flatcar Container Linux machines. + +Do not run containers with detached mode inside of systemd unit files. Detached mode prevents your init system, in our case systemd, from monitoring the process that owns the container because detached mode forks it into the background. To prevent this issue, just omit the `-d` flag if you aren't running something manually. + +### Run Apache in foreground + +We need to run the apache process in the foreground, since our container will stop when the process specified in the `docker run` command stops. We can do this with a flag `-D` when starting the apache2 process: + +```shell +/usr/sbin/apache2ctl -D FOREGROUND +``` + +Let's add that to our command: + +```shell +docker run -d myname/myapache /usr/sbin/apache2ctl -D FOREGROUND +``` + +### Permanently running a container + +While the sections above explained how to run a container when configuring it, for a production setup, you should not manually start and babysit containers. + +Instead, create a systemd unit file to make systemd keep that container running. See [Getting Started with systemd][systemd-getting-started] for details. + +Alternatively, Docker also has a feature to start existing containers on boot, when the container has the `restart` attribute set to `always`. +This requires the Docker service to get started on boot instead of using the default socket activation that starts on-demand. + +Here is a Butane Config to enable the Docker service while disabling socket activation: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + # Ensure docker starts automatically instead of being only socket-activated + - name: docker.service + enabled: true +storage: + links: + - path: /etc/systemd/system/multi-user.target.wants/docker.service + target: /usr/lib/systemd/system/docker.service + hard: false + overwrite: true +``` + +**NOTE** for Flatcar versions prior to (older than) the 3761 major release the soft link is unnecessary. The following configuration suffices: + +```yaml +variant: flatcar +version: 1.0.0 +systemd: + units: + # Ensure docker starts automatically instead of being only socket-activated + - name: docker.service + enabled: true +``` + +### Network access to 80 + +The default apache install will be running on port 80. To give our container access to traffic over port 80, we use the `-p` flag and specify the port on the host that maps to the port inside the container. In our case we want 80 for each, so we include `-p 80:80` in our command: + +```shell +docker run -d -p 80:80 myname/myapache /usr/sbin/apache2ctl -D FOREGROUND +``` + +You can now run this command on your Flatcar Container Linux host to create the container. You should see the default apache webpage when you load either `localhost:80` or the IP of your remote server. Be sure that any firewall or EC2 Security Group allows traffic to port 80. + +## Using the Docker registry + +Earlier we downloaded the ubuntu image remotely from the Docker public registry because it didn't exist on our local machine. We can also push local images to the public registry (or a private registry) very easily with the `push` command: + +```shell +docker push myname/myapache +``` + +To push to a private repository the syntax is very similar. First, we must prefix our image with the host running our private registry instead of our username. List images by running `docker images` and insert the correct ID into the `tag` command: + +```shell +docker tag f455ea72d468 registry.example.com:5000/myname/myapache +``` + +After tagging, the image needs to be pushed to the registry: + +```shell +docker push registry.example.com:5000/myname/myapache +``` + +Once the image is done uploading, you should be able to start the exact same container on a different Flatcar Container Linux host by running: + +```shell +docker run -d -p 80:80 registry.example.com:5000/myname/myapache /usr/sbin/apache2ctl -D FOREGROUND +``` + +## More information + + * [Docker Website](http://www.docker.com/) + * [docker's Getting Started Guide](https://docs.docker.com/mac/started/) + +[aws-ec2]: ../installing/cloud/aws-ec2 +[vagrant]: ../installing/vms/vagrant +[docker-cli]: https://docs.docker.com/engine/reference/commandline/cli/ +[docker-signup]: https://hub.docker.com/account/signup/ +[systemd-getting-started]: ../setup/systemd/getting-started diff --git a/content/docs/23-10-2025/container-runtimes/getting-started-with-kubernetes.md b/content/docs/23-10-2025/container-runtimes/getting-started-with-kubernetes.md new file mode 100644 index 000000000..8f838934e --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/getting-started-with-kubernetes.md @@ -0,0 +1,410 @@ +--- +title: Getting started with Kubernetes +description: Operate Kubernetes from Flatcar +aliases: + - ../os/switching-from-docker-to-containerd-for-kubernetes + - ./switching-from-docker-to-containerd-for-kubernetes +weight: 11 +--- + +One of the Flatcar purposes is to run container workloads, this term is quite generic: it goes from running a single Docker container to operate a Kubernetes cluster. + +This documentation will cover preliminary aspects of operating Kubernetes cluster based on Flatcar. + +# Supported Kubernetes version + +A Kubernetes basic scenario (deploy a simple Nginx) is being tested on Flatcar accross the channels and various CNIs, it mainly ensures that Kubernetes can be correctly installed and can operate in a simple way. + +One way to contribute to Flatcar would be to extend the covered CNIs (example: [kubenet][kubenet]) or to provide more complex scenarios (example: [cilium extension][cilium]). + +This is a compatibility matrix between Flatcar and Kubernetes deployed using vanilla components and Flatcar provided software: +| :arrow_down: Flatcar channel \ Kubernetes Version :arrow_right: | 1.29 | 1.30 | 1.31 | 1.32 | 1.33 | 1.34 | +|--------------------------------------|--------------------|--------------------|--------------------|---------------------------------|------|------| +| Alpha | :large_orange_diamond: | :large_orange_diamond: |:large_orange_diamond: |:white_check_mark: |:white_check_mark: |:white_check_mark: | +| Beta | :large_orange_diamond: | :large_orange_diamond: |:large_orange_diamond: |:white_check_mark: |:white_check_mark: |:white_check_mark: | +| Stable | :large_orange_diamond: | :large_orange_diamond: |:large_orange_diamond: |:white_check_mark: |:white_check_mark: |:white_check_mark: | +| LTS (2024) | :large_orange_diamond: | :large_orange_diamond: |:large_orange_diamond: |:white_check_mark: |:white_check_mark: |:white_check_mark: | + +:large_orange_diamond:: The version is not tested anymore before a release but was known for working. + +Tested CNIs: +- Cilium +- Flannel +- Calico + +_Known issues_: +* Flannel > 0.17.0 does not work with enforced SELinux ([flatcar#779][flatcar-779]) + +# Deploy a Kubernetes cluster with Flatcar + +## Using Kubeadm + +`kubeadm` remains one standard way to quickly deploy and operate a Kubernetes cluster. It's possible to install the tools (`kubeadm`, `kubelet`, etc.) using Ignition or directly with the Kubernetes sysext image distributed from the [flatcar/sysext-bakery][sysext-bakery] release page. + +### Setup the control plane + +Here are two examples to setup a control plane with [Butane][butane]. The first example is using the systemd-sysext approach to bring in the binaries and update them through systemd-sysupdate. The second approach fetches the binaries but has no way of updating them in-place. + +
+ +
+
+
+ This is an example using systemd-sysext and systemd-sysupdate. NOTE: We are using Kured to coordinate nodes reboot when there is a new Kubernetes sysext image available (or if Flatcar has been updated), hence the /run/reboot-required file. + **Note:** In these examples, the hostname is manually set via initial provisioning (`/etc/hostname`). + In cloud environments, the hostname can be set dynamically from metadata exposed at `/run/metadata/flatcar` + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  links:
+    - target: /opt/extensions/kubernetes/kubernetes-v1.33.2-x86-64.raw
+      path: /etc/extensions/kubernetes.raw
+      hard: false
+  files:
+    - path: /etc/sysupdate.kubernetes.d/kubernetes-v1.33.conf
+      contents:
+        source: https://extensions.flatcar.org/extensions/kubernetes/kubernetes-v1.33.conf
+    - path: /etc/sysupdate.d/noop.conf
+      contents:
+        source: https://extensions.flatcar.org/extensions/noop.conf
+    - path: /opt/extensions/kubernetes/kubernetes-v1.33.2-x86-64.raw
+      contents:
+        source: https://extensions.flatcar.org/extensions/kubernetes-v1.33.2-x86-64.raw
+    - path: /etc/hostname
+      contents:
+        inline: "flatcar-node1"
+systemd:
+  units:
+    - name: systemd-sysupdate.timer
+      enabled: true
+    - name: systemd-sysupdate.service
+      dropins:
+        - name: kubernetes.conf
+          contents: |
+            [Service]
+            ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
+            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
+            ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
+            ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi"
+    - name: locksmithd.service
+      # NOTE: To coordinate the node reboot in this context, we recommend to use Kured.
+      mask: true
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        After=network-online.target
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        ExecStartPre=/usr/bin/kubeadm init
+        ExecStartPre=/usr/bin/mkdir /home/core/.kube
+        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
+        ExecStart=/usr/bin/chown -R core:core /home/core/.kube
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ :warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  files:
+    - path: /opt/bin/kubectl
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.33.2/bin/linux/amd64/kubectl
+    - path: /opt/bin/kubeadm
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.33.2/bin/linux/amd64/kubeadm
+    - path: /opt/bin/kubelet
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.33.2/bin/linux/amd64/kubelet
+    - path: /etc/systemd/system/kubelet.service
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/refs/tags/v0.18.0/cmd/krel/templates/latest/kubelet/kubelet.service
+    - path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/refs/tags/v0.18.0/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf
+    - path: /etc/kubeadm.yml
+      contents:
+        inline: |
+          apiVersion: kubeadm.k8s.io/v1beta3
+          kind: InitConfiguration
+          nodeRegistration:
+            kubeletExtraArgs:
+              volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
+          ---
+          apiVersion: kubeadm.k8s.io/v1beta3
+          kind: ClusterConfiguration
+          controllerManager:
+            extraArgs:
+              flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
+    - path: /etc/hostname
+      contents:
+        inline: "flatcar-node1"
+systemd:
+  units:
+    - name: kubelet.service
+      enabled: true
+      dropins:
+        - name: 20-kubelet.conf
+          contents: |
+            [Service]
+            ExecStart=
+            ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        After=network-online.target
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
+        ExecStartPre=/opt/bin/kubeadm config images pull
+        ExecStartPre=/opt/bin/kubeadm init --config /etc/kubeadm.yml
+        ExecStartPre=/usr/bin/mkdir /home/core/.kube
+        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
+        ExecStart=/usr/bin/chown -R core:core /home/core/.kube
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ + +This minimal configuration can be used with Flatcar on QEMU (:warning: be sure that the instance has enough memory: 4096mb is good). + +```bash +butane < config.yaml > config.json +./flatcar_production_qemu.sh -i config.json -- -display curses +kubectl get nodes +NAME STATUS ROLES AGE VERSION +flatcar-node1 NotReady control-plane 2m10s v1.33.2 +``` + +The control plane will appear has non-ready until a CNI is deployed, here's an example with calico: +```bash +kubectl \ + apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml +kubectl get nodes +NAME STATUS ROLES AGE VERSION +flatcar-node1 Ready control-plane 5m30s v1.33.2 +``` + +If you want to coordinate the nodes reboot when there is a new Kubernetes sysext image or a Flatcar update, you can deploy [`Kured`][kured]: +```bash +latest=$(curl -s https://api.github.com/repos/kubereboot/kured/releases | jq -r '.[0].tag_name') +kubectl apply -f "https://github.com/kubereboot/kured/releases/download/$latest/kured-$latest-dockerhub.yaml" +``` + +We can now prepare the nodes to join the cluster. + +### Setup the nodes + +Here's are two examples for a [butane][butane] configuration to setup the nodes. The first example is using the systemd-sysext approach to bring in the binaries and update them through systemd-sysupdate. The second approach fetches the binaries but has no way of updating them in-place. + +
+ +
+
+
+ This is an example using systemd-sysext and systemd-sysupdate. NOTE: We are using Kured to coordinate nodes reboot when there is a new Kubernetes sysext image available (or if Flatcar has been updated), hence the /run/reboot-required file. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  links:
+    - target: /opt/extensions/kubernetes/kubernetes-v1.33.2-x86-64.raw
+      path: /etc/extensions/kubernetes.raw
+      hard: false
+  files:
+    - path: /etc/sysupdate.kubernetes.d/kubernetes-v1.33.conf
+      contents:
+        source: https://extensions.flatcar.org/extensions/kubernetes/kubernetes-v1.33.conf
+    - path: /etc/sysupdate.d/noop.conf
+      contents:
+        source: https://extensions.flatcar.org/extensions/noop.conf
+    - path: /opt/extensions/kubernetes/kubernetes-v1.33.2-x86-64.raw
+      contents:
+        source: https://extensions.flatcar.org/extensions/kubernetes-v1.33.2-x86-64.raw
+    - path: /etc/hostname
+      contents:
+        inline: "flatcar-node2"
+systemd:
+  units:
+    - name: systemd-sysupdate.timer
+      enabled: true
+    - name: systemd-sysupdate.service
+      dropins:
+        - name: kubernetes.conf
+          contents: |
+            [Service]
+            ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
+            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
+            ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
+            ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi"
+    - name: locksmithd.service
+      # NOTE: To coordinate the node reboot in this context, we recommend to use Kured.
+      mask: true
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        After=network-online.target
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        ExecStart=/usr/bin/kubeadm join $(output from 'kubeadm token create --print-join-command')
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ :warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts. + 
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  files:
+    - path: /opt/bin/kubeadm
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.33.2/bin/linux/amd64/kubeadm
+    - path: /opt/bin/kubelet
+      mode: 0755
+      contents:
+        source: https://dl.k8s.io/v1.33.2/bin/linux/amd64/kubelet
+    - path: /etc/systemd/system/kubelet.service
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/refs/tags/v0.18.0/cmd/krel/templates/latest/kubelet/kubelet.service
+    - path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
+      contents:
+        source: https://raw.githubusercontent.com/kubernetes/release/refs/tags/v0.18.0/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf
+    - path: /etc/hostname
+      contents:
+        inline: "flatcar-node2"
+systemd:
+  units:
+    - name: kubelet.service
+      enabled: true
+      dropins:
+        - name: 20-kubelet.conf
+          contents: |
+            [Service]
+            ExecStart=
+            ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        After=network-online.target
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
+        ExecStart=/opt/bin/kubeadm join $(output from 'kubeadm token create --print-join-command')
+        [Install]
+        WantedBy=multi-user.target
+
+
+
+
+
+ +This method is far from being ideal in terms of infrastructure as code as it requires a two steps manipulation: create the control plane to generate the join configuration then pass that configuration to the nodes. Other solutions exist to make things easier, like Cluster API or [Typhoon][typhoon]. + +### Switching from Docker to containerd for Kubernetes + +In Kubernetes v1.20, `dockershim` was deprecated and it has been [officially](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#dockershim-removed-from-kubelet) removed in Kubernetes v1.24. + +The `containerd` CRI plugin is enabled by default and you can use containerd for Kubernetes while still allowing Docker to function. +Recent Kubernetes versions will prefer containerd over Docker automatically on recent Flatcar versions. + +If you run `kubelet` in a Docker container, make sure it has access +to the following directories on the host file system: +- `/run/docker/libcontainerd/` +- `/run/containerd/` +- `/var/lib/containerd/` + +And that it has access to the following binaries on the host file system and that they are included in `PATH`: +- For Flatcar releases until major version 3760: + - `/run/torcx/unpack/docker/bin/containerd-shim-runc-v1` + - `/run/torcx/unpack/docker/bin/containerd-shim-runc-v2` +- For Flatcar releases above major version 3760: + - `/usr/bin/containerd-shim-runc-v1` + - `/usr/bin/containerd-shim-runc-v2` + +Finally, tell `kubelet` to use containerd by adding to it the following flags: +- `--container-runtime=remote` +- `--container-runtime-endpoint=unix:///run/containerd/containerd.sock` + +## Cluster API + +From the official [documentation][capi-documentation]: +> Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. + +As it requires to have some tools already installed on the OS to work correcly with CAPI, Flatcar images can be built using the [image-builder][image-builder] project. + +While CAPI is an evolving project and Flatcar support is in-progress regarding the various providers, here's the current list of supported providers: +* [AWS][capi-aws] +* [Akamai / Linode][capi-linode] +* [Azure][capi-azure] +* [Kubevirt][capi-kubevirt] +* [OpenStack][openstack] +* [Proxmox][capi-proxmox] +* [vSphere][capi-vsphere] + +## Kubespray + +Kubespray is an open-source project used to deploy production ready Kubernetes cluster, learn more about it on the [documentation][kubespray-documentation]. + +Based on users feedback, Flatcar is known to work with Kubespray - you can read more about it in this section: . + +[butane]: https://coreos.github.io/butane/ +[capi-documentation]: https://cluster-api.sigs.k8s.io/ +[capi-aws]: https://cluster-api-aws.sigs.k8s.io/ +[capi-azure]: https://capz.sigs.k8s.io/ +[capi-kubevirt]: https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt +[capi-linode]: https://linode.github.io/cluster-api-provider-linode/topics/flavors/flatcar.html +[capi-proxmox]: https://github.com/ionos-cloud/cluster-api-provider-proxmox +[capi-vsphere]: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/docs/ignition.md +[cilium]: https://github.com/flatcar/mantle/pull/292 +[flatcar-779]: https://github.com/flatcar/Flatcar/issues/779 +[image-builder]: https://github.com/kubernetes-sigs/image-builder +[kubenet]: https://github.com/flatcar/Flatcar/issues/579 +[kubespray-documentation]: https://kubespray.io +[kured]: https://kured.dev/docs/ +[openstack]: https://cluster-api-openstack.sigs.k8s.io/clusteropenstack/configuration.html#ignition-based-images +[sysext-bakery]: https://flatcar.github.io/sysext-bakery/ +[typhoon]: https://typhoon.psdn.io/ diff --git a/content/docs/23-10-2025/container-runtimes/high-availability-kubernetes.md b/content/docs/23-10-2025/container-runtimes/high-availability-kubernetes.md new file mode 100644 index 000000000..f55810e87 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/high-availability-kubernetes.md @@ -0,0 +1,1097 @@ +--- +title: "High Availability Kubernetes" +date: 2025-02-13T16:30:38-05:00 +draft: false +weight: 12 +--- + +After you have created a kubernetes cluster using the [getting started +guide](./getting-started-with-kubernetes.md), we can take a look at a more +complex example that involves a highly available control plane nodes and +dedicated worker nodes. The result will be similar to a [Typhoon][Typhoon] +cluster, but this version will be a little more "vanilla" and will run on +libvirt VMs, which is not mentioned in their documentation as of time of +writing. + +## Architecture + +This documentation will walk you through creating 5 VMs with the following +properties: + +| VM Name | Hostname | IP | Role | Runtime | +| --------- | ---------- | ---- | ------ | ------- | +| virt-node1-ha | node1-cp | 192.168.122.31 | Control Plane | CRIO | +| virt-node2-ha | node2-cp | 192.168.122.32 | Control Plane | CRIO | +| virt-node3-ha | node3-cp | 192.168.122.33 | Control Plane | CRIO | +| virt-node4-ha | node4-worker | 192.168.122.34 | Worker | CRIO | +| virt-node5-ha | node5-worker | 192.168.122.35 | Worker | CRIO | + +### Control Plane + +Each control plane node will be responsible for running the regular Kubernetes +control plane components, as well as HAProxy and Keepalived. The api-server +with have a VIP of `192.168.122.30`. HAProxy and Keepalived can be installed +through various methods, but this article will show you how to install them +using [Quadlet][Quadlet] through Ignition. + +### Bootstrap Sequence + +We will install the necessary systemd files to use kubeadm to bootstrap a +cluster. This means essentially that `node1-cp` will run `kubeadm init ...`, +`node2-cp` and `node3-cp` will run `kubeadm join --control-plane ...`, and the +worker nodes will run `kubeadm join ...`. This is a [very common pattern][HA K8S] +for bootstrapping a kubernetes cluster. + +Typically, one node will call `kubeadm init ...`, then `kubeadm print +join-token`, which then can be copied between nodes. The problem with this +approach is that it requires manual intervention at install-time which sort-of +defeats the purpose of Ignition. Luckily, we can provide all the information +ahead of time so nothing needs to be copied between nodes and the cluster +installation can still be secure and hands-off. + +In order to do this, there are a few things that we need to pre-generate so +that we can pass them to the `kubeadm` commands via ignition. These are: + +- /etc/kubernetes/pki/ca.crt +- /etc/kubernetes/pki/ca.key +- /etc/kubernetes/pki/sa.key +- /etc/kubernetes/pki/sa.pub +- /etc/kubernetes/pki/front-proxy-ca.crt +- /etc/kubernetes/pki/front-proxy-ca.key +- /etc/kubernetes/pki/etcd/ca.crt +- /etc/kubernetes/pki/etcd/ca.key + +We also will generate an initial join token and hash of the CA and provide that +in an EnvironmentFile called `/etc/kubernetes/certs.conf`. + +Fortunately, this can all be created via script, called +`generate-k8s-certs.sh`. Before we talk about the script, we should +talk about the overall file structure. + +## Files + +Building the configuration files and VMs is not a small feat, so we should +create a dedicated workspace for this. + +```bash +mkdir -p butane certs ign scripts +``` + +Now we can get started looking at our butane configs. + +### Butane Structure + +All of our butane files will be in the `butane` directory. Each VM will have +its own butane file that will have the same name as the VM and it will depend +on one or more other butane file. In order to get dependencies correct, the +base butane files will be prefixed by 2 numbers and they will be built in +numerical order. This way, our butane files can depend on any file sorted +before it, but not after. Files named `00_base*` should not depend on any +other butane config. The final butane configs for the VMs should only depend +on these base configs. + +``` +> tree butane/ +butane/ +├── 00_base-k8s-token.yaml +├── 00_base-k8s.yaml +├── 10_base-ha.yaml +├── 20_base-ha-cp.yaml +├── 30_base-ha-init.yaml +├── 31_base-ha-join-cp.yaml +├── 32_base-ha-join.yaml +├── node1-ha.yaml +├── node2-ha.yaml +├── node3-ha.yaml +├── node4-ha.yaml +└── node5-ha.yaml +``` + +This already seems a bit unwieldy, so let's create a `Makefile` to help build everything. + +### Makefile + +
Makefile + +```Makefile +# Options are stable | alpha | beta +CHANNEL := stable + +# Make sure to https://www.flatcar.org/docs/latest/installing/vms/qemu/#gentoo + +TOKEN_FILE = butane/00_base-k8s-token.yaml +BUTANE = $(TOKEN_FILE) $(shell find butane -type f -not -name 'node*' | sort -h) $(shell find butane -type f -name "node*.yaml") +IGN = $(patsubst butane/%.yaml,ign/%.ign,$(BUTANE)) + +ifeq ($(shell command -v podman 2> /dev/null),) + CR=docker +else + CR=podman +endif + +# Remove files that we can easily recreate +clean: + rm -f flatcar_production_qemu_image.img.sig config.json ign/* /var/lib/libvirt/images/flatcar/*.ign + +# Remove files that take longer to recreate/download +cleanclean: clean + rm -f flatcar_production_qemu_image.img.fresh Flatcar_Image_Signing_Key.asc $(TOKEN_FILE) + +.PHONY: cleanclean clean + +Flatcar_Image_Signing_Key.asc: + curl -L -O https://www.flatcar.org/security/image-signing-key/Flatcar_Image_Signing_Key.asc + +verify-gpg: Flatcar_Image_Signing_Key.asc + gpg --import --keyid-format LONG Flatcar_Image_Signing_Key.asc + +.PHONY: verify-gpg + +# Download and verify the image that we'll use as a base +flatcar_production_qemu_image.img.fresh: + wget "https://${CHANNEL}.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img" + wget "https://${CHANNEL}.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.sig" + gpg --verify flatcar_production_qemu_image.img.sig + cp -f flatcar_production_qemu_image.img flatcar_production_qemu_image.img.fresh + +# Create vm disks from the base image +%.qcow2: flatcar_production_qemu_image.img.fresh + mkdir -p /var/lib/libvirt/images/flatcar + qemu-img create -f qcow2 -F qcow2 -b $(shell pwd)/flatcar_production_qemu_image.img.fresh /var/lib/libvirt/images/flatcar/$@ + +# Convert butane configs to ignition configs +ign/%.ign: + mkdir -p ign + $(CR) run --rm -i -v $(shell pwd)/butane:/config/butane:ro -v $(shell pwd)/ign:/config/ign quay.io/coreos/butane:latest --pretty -d /config < butane/$*.yaml > $@ + +# Create VMs +virt-node%: flatcar_production_qemu_image-%.qcow2 $(IGN) + cp -fv ign/*.ign /var/lib/libvirt/images/flatcar + virt-install --connect qemu:///system --import --name $@ --ram 4096 --vcpus 4 --os-variant=generic --network network=default,model=virtio --disk path=/var/lib/libvirt/images/flatcar/flatcar_production_qemu_image-$*.qcow2,format=qcow2,bus=virtio --vnc --qemu-commandline="-fw_cfg name=opt/org.flatcar-linux/config,file=/var/lib/libvirt/images/flatcar/node$*.ign" --noautoconsole + +# Don't delete intermediate files +.PRECIOUS: %.qcow2 ign/%.ign + +# Stop and Remove VMs +rm-virt-node%: + virsh destroy virt-node$* || : + virsh undefine virt-node$* + +# Create token butane file +butane/00_base-k8s-token.yaml: + ./scripts/generate-k8s-certs.sh + +# Create all ignition configs +new-cluster: clean butane/00_base-k8s-token.yaml $(IGN) + +# Create all VMs +ha: virt-node1-ha virt-node2-ha virt-node3-ha virt-node4-ha virt-node5-ha + +# Stop and Remove all VMs +rm-ha: rm-virt-node1-ha rm-virt-node2-ha rm-virt-node3-ha rm-virt-node4-ha rm-virt-node5-ha + +.PHONY: new-cluster ha rm-ha +``` +
+
+ +This makefile is maybe a bit long, but we'll go through each piece in due time. +First, we should look at these lines: + +```Makefile +TOKEN_FILE = butane/00_base-k8s-token.yaml +BUTANE = $(TOKEN_FILE) $(shell find butane -type f -not -name 'node*' | sort -h) $(shell find butane -type f -name "node*.yaml") +IGN = $(patsubst butane/%.yaml,ign/%.ign,$(BUTANE)) +``` + +This is specifying our special token file that will contain all of our +certificates and such, as well as specifying the order in which our butane +files should be built. Then, it uses path substitution to generate the name of +the resulting ignition files. + +```Makefile +# Convert butane configs to ignition configs +ign/%.ign: + mkdir -p ign + $(CR) run --rm -i -v $(shell pwd)/butane:/config/butane:ro -v $(shell pwd)/ign:/config/ign quay.io/coreos/butane:latest --pretty -d /config < butane/$*.yaml > $@ + +# ... + +# Create token butane file +butane/00_base-k8s-token.yaml: + ./scripts/generate-k8s-certs.sh +``` + +These lines show us how to to build each ignition file using a container. We +also have to specify how to build our token file using a shell script. + +### Token File + +Generating the token file is a fun exercise using `openssl`, yaml, and bash +heredocs. In case you actually value your time, the script is provided here in +full. As always, please make sure you understand what commands you are running +before running a script you found on the internet. + +
scripts/generate-k8s-certs.sh + +```bash +#!/usr/bin/env bash + +set -e + +# Set file paths +cert_dir="./certs" +output_yaml="butane/00_base-k8s-token.yaml" + +# Create cert directory +mkdir -p "$cert_dir" + +# Generate the token +token=$(echo "$(tr -dc 'a-z0-9' < /dev/urandom | head -c 6).$(tr -dc 'a-z0-9' < /dev/urandom | head -c 16)") +encoded_token=$(echo -n "$token" | base64) + +# Generate Kubernetes CA (used for cluster signing) +openssl req -x509 -newkey rsa:2048 -keyout "$cert_dir/ca.key" -out "$cert_dir/ca.crt" -days 365 -nodes -subj "/CN=k8s-ca" + +# Generate Front Proxy CA (used for API server aggregation) +openssl req -x509 -newkey rsa:2048 -keyout "$cert_dir/front-proxy-ca.key" -out "$cert_dir/front-proxy-ca.crt" -days 365 -nodes -subj "/CN=front-proxy-ca" + +# Generate Service Account Signing Key +openssl genrsa -out "$cert_dir/sa.key" 2048 +openssl rsa -in "$cert_dir/sa.key" -pubout -out "$cert_dir/sa.pub" + +# Generate API server certificate (signed by Kubernetes CA) +openssl req -new -newkey rsa:2048 -keyout "$cert_dir/apiserver.key" -out "$cert_dir/apiserver.csr" -nodes -subj "/CN=kube-apiserver" +openssl x509 -req -in "$cert_dir/apiserver.csr" -CA "$cert_dir/ca.crt" -CAkey "$cert_dir/ca.key" -CAcreateserial -out "$cert_dir/apiserver.crt" -days 365 + +# Generate etcd CA (if using external etcd) +openssl req -x509 -newkey rsa:2048 -keyout "$cert_dir/etcd-ca.key" -out "$cert_dir/etcd-ca.crt" -days 365 -nodes -subj "/CN=etcd-ca" + +indent=" " + +# Encode certificates for YAML +ca_crt=$(sed "s/^/${indent}/" "$cert_dir/ca.crt") +ca_key=$(sed "s/^/${indent}/" "$cert_dir/ca.key") +front_proxy_ca_crt=$(sed "s/^/${indent}/" "$cert_dir/front-proxy-ca.crt") +front_proxy_ca_key=$(sed "s/^/${indent}/" "$cert_dir/front-proxy-ca.key") +sa_key=$(sed "s/^/${indent}/" "$cert_dir/sa.key") +sa_pub=$(sed "s/^/${indent}/" "$cert_dir/sa.pub") +etcd_ca_crt=$(sed "s/^/${indent}/" "$cert_dir/etcd-ca.crt") +etcd_ca_key=$(sed "s/^/${indent}/" "$cert_dir/etcd-ca.key") + +# Compute CA hash +ca_hash="sha256:$(openssl x509 -pubkey -in "$cert_dir/ca.crt" | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')" +encoded_base64_ca_hash=$(echo -n "$ca_hash" | base64 -w 0) + +# Write the header to the output YAML file +cat > "$output_yaml" <<-EOF +--- +# This is generated using generate-k8s-certs.sh +variant: flatcar +version: 1.1.0 +storage: + files: + - path: /etc/kubernetes/pki/ca.crt + contents: + inline: | +$ca_crt + - path: /etc/kubernetes/pki/ca.key + contents: + inline: | +$ca_key + - path: /etc/kubernetes/pki/front-proxy-ca.crt + contents: + inline: | +$front_proxy_ca_crt + - path: /etc/kubernetes/pki/front-proxy-ca.key + contents: + inline: | +$front_proxy_ca_key + - path: /etc/kubernetes/pki/sa.key + contents: + inline: | +$sa_key + - path: /etc/kubernetes/pki/sa.pub + contents: + inline: | +$sa_pub + - path: /etc/kubernetes/pki/etcd/ca.crt + contents: + inline: | +$etcd_ca_crt + - path: /etc/kubernetes/pki/etcd/ca.key + contents: + inline: | +$etcd_ca_key + - path: /etc/kubernetes/certs.conf + contents: + inline: | + K8S_TOKEN='$token' + K8S_HASH='$ca_hash' +EOF + +echo "Kubernetes certificates have been generated successfully!" +echo "YAML file '$output_yaml' has been successfully overwritten!" +``` +
+
+ +Now we we can save this script in `scripts/generate-k8s-certs.sh` run `make +butane/00_base-k8s-token.yaml` and check out our SSL goodies. Treat this file +like a password. If it gets made public (for anything other than test +clusters) make sure you recreate them. It's usually recommended to not provide +sensitive information via Ignition configuration files. This is only for demo +purposes. + +### Butane Content + +Now that we have the basic structure in place, we can get started filling out +the content of our butane files. We will first look at the "base" configs, +then look at node specific configs. + +#### Base + +
+ +
butane/00_base-k8s.yaml + +```yaml +variant: flatcar +version: 1.1.0 +storage: + links: + - path: /etc/extensions/docker-flatcar.raw + target: /dev/null + overwrite: true + - path: /etc/extensions/containerd-flatcar.raw + target: /dev/null + overwrite: true + - target: /opt/extensions/crio/crio-v1.31.3-x86-64.raw + path: /etc/extensions/crio.raw + hard: false + - target: /opt/extensions/kubernetes/kubernetes-v1.31.3-x86-64.raw + path: /etc/extensions/kubernetes.raw + hard: false + files: + - path: /etc/flatcar/enabled-sysext.conf + contents: + inline: | + zfs + podman + mode: 0644 + - path: /etc/sysupdate.d/noop.conf + contents: + source: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf + - path: /opt/extensions/crio/crio-v1.31.3-x86-64.raw + mode: 0644 + contents: + source: https://github.com/flatcar/sysext-bakery/releases/download/latest/crio-v1.31.3-x86-64.raw + - path: /etc/sysupdate.crio.d/crio.conf + contents: + source: https://github.com/flatcar/sysext-bakery/releases/download/latest/crio.conf + - path: /etc/sysupdate.kubernetes.d/kubernetes-v1.31.conf + contents: + source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-v1.31.conf + - path: /opt/extensions/kubernetes/kubernetes-v1.31.3-x86-64.raw + contents: + source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-v1.31.3-x86-64.raw + - path: /opt/bin/cilium.tar + mode: 0755 + contents: + source: https://github.com/cilium/cilium-cli/releases/download/v0.16.24/cilium-linux-amd64.tar.gz + compression: gzip +systemd: + units: + - name: systemd-sysupdate.timer + enabled: true + - name: locksmithd.service + # NOTE: To coordinate the node reboot in this context, we recommend to use Kured. + mask: true + - name: systemd-sysupdate.service + dropins: + - name: kubernetes.conf + contents: | + [Service] + ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes" + ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update + ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new" + ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi" + - name: crio.conf + contents: | + [Service] + ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/crio.raw > /tmp/crio" + ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C crio update + ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/crio.raw > /tmp/crio-new" + ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/crio /tmp/crio-new; then touch /run/reboot-required; fi" + - name: cilium-setup.service + enabled: true + contents: | + [Unit] + Description=Download and install Cilium + After=network-online.target + Wants=network-online.target + + [Service] + Type=oneshot + ExecStart=/bin/sh -c 'tar -C /opt/bin -xf /opt/bin/cilium.tar' + RemainAfterExit=true + + [Install] + WantedBy=multi-user.target + - name: zfs-setup.service + enabled: true + contents: | + [Unit] + Description=Load zfs kernel modules + After=local-fs-pre.target + Wants=local-fs-pre.target + + [Service] + Type=oneshot + ExecStart=/usr/bin/modprobe zfs + RemainAfterExit=true + + [Install] + WantedBy=multi-user.target +``` +
+
+ +This base config does a couple of things. It disables the `docker` and +`containerd` sysexts, installs `kubernetes` and `crio` sysexts, and enables +the `zfs` and `podman` sysexts. It also creates a systemd unit to download the +cilium cli. This cli may only be needed on the first node, but we install it +on all hosts to make day 2 operations easier. We also disable `locksmithd` to +prepare for using [Kured][Kured]. + +
butane/10_base-ha.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/00_base-k8s.ign + - local: ign/00_base-k8s-token.ign +storage: + files: + - path: /etc/kubernetes/kubeadm-env.conf + mode: 0644 + contents: + inline: | + K8S_APISERVER_URL="192.168.122.30" + - path: /etc/containers/policy.json + contents: + inline: | + { + "default": [ + { + "type": "insecureAcceptAnything" + } + ] + } +``` +
+
+ +We extend the first config by including it in this config, along with the token +file. We also set any environment variales that are shared across hosts, in +this case `K8S_APISERVER_URL`, as well as set up some default podman/crio +policies. + +
butane/20_base-ha-cp.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/10_base-ha.ign +storage: + files: + - path: /etc/keepalived/check_apiserver.sh + mode: 0755 + contents: + inline: | + #!/bin/bash + curl --silent --max-time 2 --insecure https://localhost:6443/healthz -o /dev/null || exit 1 + - path: /etc/containers/systemd/keepalived.container + contents: + inline: | + [Unit] + Description=Keepalived Container + Requires=crio.service + After=crio.service + + [Container] + Image=docker.io/osixia/keepalived:latest + AutoUpdate=registry + AddCapability=NET_ADMIN + AddCapability=NET_BROADCAST + AddCapability=NET_RAW + PodmanArgs=--privileged + Exec=--copy-service + Volume=/etc/keepalived/keepalived.conf:/container/service/keepalived/assets/keepalived.conf:ro + Volume=/etc/keepalived/check_apiserver.sh:/etc/keepalived/check_apiserver.sh:ro + Network=host + + [Service] + Restart=always + RestartSec=30 + + [Install] + WantedBy=multi-user.target default.target + - path: /etc/containers/systemd/haproxy.container + contents: + inline: | + [Unit] + Description=HAProxy Container + Requires=keepalived.service + After=keepalived.service + + [Container] + Image=docker.io/library/haproxy:alpine + AutoUpdate=registry + Network=host + Volume=/etc/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro + + [Service] + Restart=always + RestartSec=30 + + [Install] + WantedBy=multi-user.target default.target + - path: /etc/haproxy/haproxy.cfg + contents: + inline: | + # /etc/haproxy/haproxy.cfg + #--------------------------------------------------------------------- + # Global settings + #--------------------------------------------------------------------- + global + log stdout format raw local0 + daemon + + #--------------------------------------------------------------------- + # common defaults that all the 'listen' and 'backend' sections will + # use if not designated in their block + #--------------------------------------------------------------------- + defaults + mode http + log global + option httplog + option dontlognull + option http-server-close + option forwardfor except 127.0.0.0/8 + option redispatch + retries 1 + timeout http-request 10s + timeout queue 20s + timeout connect 5s + timeout client 35s + timeout server 35s + timeout http-keep-alive 10s + timeout check 10s + + #--------------------------------------------------------------------- + # apiserver frontend which proxys to the control plane nodes + #--------------------------------------------------------------------- + frontend apiserver + bind 192.168.122.30:6444 + mode tcp + option tcplog + default_backend apiserverbackend + + #--------------------------------------------------------------------- + # round robin balancing for apiserver + #--------------------------------------------------------------------- + backend apiserverbackend + mode tcp + balance roundrobin + + server node1-cp 192.168.122.31:6443 check verify none + server node2-cp 192.168.122.32:6443 check verify none + server node3-cp 192.168.122.33:6443 check verify none +systemd: + units: + - name: ip_vs-setup.service + enabled: true + contents: | + [Unit] + Description=Load ip_vs kernel modules + After=local-fs-pre.target + Wants=local-fs-pre.target + + [Service] + Type=oneshot + ExecStart=/usr/bin/modprobe ip_vs + RemainAfterExit=true + + [Install] + WantedBy=multi-user.target +``` + +
+
+ +Here we are defining services that should be running on each of the control +plane nodes. We specify keepalived and haproxy containers that should be +running via Quadlet. These need to be running before kubernetes so its easier +to define them here. We also specify the haproxy config since it will be the +same across all nodes, but we don't specify the keepalived config. That will +be different per node because it will include the nodes ip address and +rankings. Finally we make sure to run `modprobe ip_vs` which is required when +running keepalived. + +
butane/30_base-ha-init.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/20_base-ha-cp.ign +storage: + files: +systemd: + units: + - name: kubeadm.service + enabled: false + contents: | + [Unit] + StartLimitInterval=200 + StartLimitBurst=5 + Description=Kubeadm service + Requires=haproxy.service + After=haproxy.service + ConditionPathExists=!/etc/kubernetes/kubelet.conf + [Service] + EnvironmentFile=/etc/kubernetes/kubeadm-env.conf + EnvironmentFile=/etc/kubernetes/certs.conf + # We skip kube-proxy because we install cilium + ExecStartPre=/usr/bin/kubeadm init --token ${K8S_TOKEN} --cri-socket=unix:///var/run/crio/crio.sock --skip-phases=addon/kube-proxy --control-plane-endpoint ${K8S_APISERVER_URL}:6444 --upload-certs + ExecStartPre=/usr/bin/mkdir /home/core/.kube + ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config + ExecStart=/usr/bin/chown -R core:core /home/core/.kube + [Install] + WantedBy=multi-user.target + - name: cilium-install-cluster.service + enabled: true + contents: | + [Unit] + Description=Install Cilium as Kubernetes CNI + Requires=kubeadm.service + After=kubeadm.service + [Service] + Environment=KUBECONFIG='/home/core/.kube/config' + ExecStart=/opt/bin/cilium install --set kubeProxyReplacement=true --namespace=kube-system + [Install] + WantedBy=multi-user.target +``` + +
+
+ +This service defines what to do when intializing the kubernetes cluster. This +only needs to be run on one node. Essentially we are calling `kubeadm init +...` and `cilium install ...`, but its worth pointing out the dependencies +here. We specify that kubeadm should only start AFTER the haproxy service +(which is generated after we specify the haproxy container) because that is +load-balancing our api-server VIP that's defined by keepalived. We also don't +install Cilium to the cluster until there is a cluster to install to. So +cilium can't start until kubeadm starts, kubeadm can't start until haproxy +starts, and haproxy can't start until keepalived starts. This ordering is +important for an HA control plane. + +In the `kubeadm init ...` command, we also specify variables that are defined +in `EnvironmentFile=/etc/kubernetes/kubeadm-env.conf` and +`EnvironmentFile=/etc/kubernetes/certs.conf`. This ensures we are using the +same secrets across all nodes, and since we pre-generated them, no information +needs to be copied from node to node. + +
butane/31_base-ha-join-cp.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/20_base-ha-cp.ign +storage: + files: +systemd: + units: + - name: kubeadm.service + enabled: true + contents: | + [Unit] + StartLimitInterval=200 + StartLimitBurst=5 + Description=Kubeadm service + Requires=crio.service + After=crio.service + ConditionPathExists=!/etc/kubernetes/kubelet.conf + [Service] + Restart=always + RestartSec=30 + EnvironmentFile=/etc/kubernetes/certs.conf + EnvironmentFile=/etc/kubernetes/kubeadm-env.conf + ExecStartPre=/usr/bin/kubeadm config images pull + # Wait until first control plane is up + ExecStartPre=/usr/bin/sh -c "while ! nc -z ${K8S_APISERVER_URL} 6444; do sleep 5; done" + ExecStart=/usr/bin/kubeadm join ${K8S_APISERVER_URL}:6444 --token ${K8S_TOKEN} --discovery-token-ca-cert-hash ${K8S_HASH} --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt --cri-socket=unix:///var/run/crio/crio.sock --control-plane --certificate-key ${K8S_CERT_KEY} + [Install] + WantedBy=multi-user.target +``` + +
+
+ +This config specifies how to join the cluster as a control plane node using the +information in the same `EnvironmentFile`s. We also specify a `ExecStartPre` +which essentially just sleeps until the first nodes api-server is available. + +
butane/32_base-ha-join.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/10_base-ha.ign +systemd: + units: + - name: kubeadm.service + enabled: true + contents: | + [Unit] + StartLimitInterval=200 + StartLimitBurst=5 + Description=Kubeadm service + Requires=crio.service + After=crio.service + ConditionPathExists=!/etc/kubernetes/kubelet.conf + [Service] + Restart=always + RestartSec=30 + EnvironmentFile=/etc/kubernetes/certs.conf + EnvironmentFile=/etc/kubernetes/kubeadm-env.conf + ExecStartPre=/usr/bin/kubeadm config images pull + ExecStartPre=/usr/bin/sh -c "while ! nc -z ${K8S_APISERVER_URL} 6444; do sleep 5; done" + ExecStart=/usr/bin/kubeadm join ${K8S_APISERVER_URL}:6444 --token ${K8S_TOKEN} --discovery-token-ca-cert-hash ${K8S_HASH} --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt --cri-socket=unix:///var/run/crio/crio.sock + [Install] + WantedBy=multi-user.target +``` + +
+
+ +This is similar to the previous command, but we join as a worker node, not a +control plane. + +#### Node + +
+ +
butane/node1-ha.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/30_base-ha-init.ign +storage: + files: + - path: /etc/hostname + overwrite: true + contents: + inline: node1-cp + - path: /etc/systemd/network/00-eth0.network + contents: + inline: | + [Match] + Name=eth0 + + [Network] + Address=192.168.122.31/24 + Gateway=192.168.122.1 + DNS=192.168.122.1 + - path: /etc/keepalived/keepalived.conf + mode: 0644 + contents: + inline: | + ! /etc/keepalived/keepalived.conf + ! Configuration File for keepalived + global_defs { + router_id LVS_DEVEL + } + vrrp_script check_apiserver { + script "/etc/keepalived/check_apiserver.sh" + interval 3 + weight -2 + fall 10 + rise 2 + } + vrrp_instance VI_1 { + state MASTER + interface eth0 + virtual_router_id 51 + priority 101 + authentication { + auth_type PASS + auth_pass kubevip + } + virtual_ipaddress { + 192.168.122.30/24 + } + track_script { + check_apiserver + } + } +``` + +
+
+ +Here we define the first node. We are including everything from the +`ign/30_base-ha-init.ign` config, as well as defining our hostname and static +IP. We also provide the keepalived config that specifies this node as the +`MASTER`. + +
butane/node2-ha.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/31_base-ha-join-cp.ign +storage: + files: + - path: /etc/hostname + overwrite: true + contents: + inline: node2-cp + - path: /etc/systemd/network/00-eth0.network + contents: + inline: | + [Match] + Name=eth0 + + [Network] + Address=192.168.122.32/24 + Gateway=192.168.122.1 + DNS=192.168.122.1 + - path: /etc/keepalived/keepalived.conf + mode: 0644 + contents: + inline: | + ! /etc/keepalived/keepalived.conf + ! Configuration File for keepalived + global_defs { + router_id LVS_DEVEL + } + vrrp_script check_apiserver { + script "/etc/keepalived/check_apiserver.sh" + interval 3 + weight -2 + fall 10 + rise 2 + } + vrrp_instance VI_1 { + state MASTER + interface eth0 + virtual_router_id 51 + priority 100 + authentication { + auth_type PASS + auth_pass kubevip + } + virtual_ipaddress { + 192.168.122.30/24 + } + track_script { + check_apiserver + } + } +``` + +
+
+ +
butane/node3-ha.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/31_base-ha-join-cp.ign +storage: + files: + - path: /etc/hostname + overwrite: true + contents: + inline: node3-cp + - path: /etc/systemd/network/00-eth0.network + contents: + inline: | + [Match] + Name=eth0 + + [Network] + Address=192.168.122.33/24 + Gateway=192.168.122.1 + DNS=192.168.122.1 + - path: /etc/keepalived/keepalived.conf + mode: 0644 + contents: + inline: | + ! /etc/keepalived/keepalived.conf + ! Configuration File for keepalived + global_defs { + router_id LVS_DEVEL + } + vrrp_script check_apiserver { + script "/etc/keepalived/check_apiserver.sh" + interval 3 + weight -2 + fall 10 + rise 2 + } + vrrp_instance VI_1 { + state MASTER + interface eth0 + virtual_router_id 51 + priority 99 + authentication { + auth_type PASS + auth_pass kubevip + } + virtual_ipaddress { + 192.168.122.30/24 + } + track_script { + check_apiserver + } + } +``` + +
+
+ +These two files are similar in that they define the other two control plane +nodes. The main differences are the hostnames, IPs, and the priority in the +keepalived config. + +
butane/node4-ha.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/32_base-ha-join.ign +storage: + files: + - path: /etc/hostname + overwrite: true + contents: + inline: node4-worker + - path: /etc/systemd/network/00-eth0.network + contents: + inline: | + [Match] + Name=eth0 + + [Network] + Address=192.168.122.34/24 + Gateway=192.168.122.1 + DNS=192.168.122.1 +``` + +
+
+ +
butane/node5-ha.yaml + +```yaml +variant: flatcar +version: 1.1.0 +ignition: + config: + merge: + - local: ign/32_base-ha-join.ign +storage: + files: + - path: /etc/hostname + overwrite: true + contents: + inline: node5-worker + - path: /etc/systemd/network/00-eth0.network + contents: + inline: | + [Match] + Name=eth0 + + [Network] + Address=192.168.122.35/24 + Gateway=192.168.122.1 + DNS=192.168.122.1 +``` + +
+
+ +These final two configs specify the worker nodes. Each inherits +`ign/32_base-ha-join.ign`, so we only need to specify the hostname and IP +addresses. + +## Running + +Now that we have all of our files defined, we can actually spin up the cluster. +We start by downloading the VM image, generating the configs, and running the +VMs. + +NOTE: You may want to run these make commands as root, depending on how +your permissions are setup with libvirt, but again, make sure you understand +what the commands are doing before you run them. There is no lifeguard at the +pool. + +Running `make verify-gpg` should download flatcars GPG key and import it. +Then, running `make new-cluster` should generate our token file and generate +all the Ignition files from the butane configs. Finally, running `make ha` +will create and run each VM. This involves downloading a base VM image, +verifying it with GPG, cloning the base image to create a working image, and +copying the image (and Igntion files) to `/var/lib/libvirt/images/flatcar`. + +After that, we should have a running kubernetes cluster. We can shut +everything down by running `make rm-ha` or connect to a node via `virsh connect +virt-node1-ha`. Once we are in the first node we can run commands like +`kubectl get nodes` or `cilium status` to see the status of the cluster. Do +note, however, that this may take some time for everything to settle. Each +node will be downloading things from the internet and automatically registering +itself so you may not see all the nodes or see cilium in an error state for a +few minutes. + +Once everything has settled, you can create an example deployment by running +`kubectl apply -f +https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/controllers/nginx-deployment.yaml`. + +That's it! You now have a fully functional, highly available kubernetes +cluster running on your VMs! + +[Typhoon]: https://typhoon.psdn.io +[Quadlet]: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html +[HA K8S]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/ +[Kured]: https://kured.dev/ diff --git a/content/docs/23-10-2025/container-runtimes/incus.md b/content/docs/23-10-2025/container-runtimes/incus.md new file mode 100644 index 000000000..1925e2b41 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/incus.md @@ -0,0 +1,263 @@ +--- +title: Incus +description: Operate Incus from Flatcar +weight: 11 +--- + +While Flatcar proposes Containerd and Docker by default, [Incus][incus] can be used to run containers. The goal of this guide is not to re-write the Incus documentation but to give key aspects of Incus usage on Flatcar. + +# Installing Incus + +Incus is provided as an [official][official-sysext] Systemd sysext Flatcar extension. To install it automatically at boot: +```yaml +--- +# config.yaml +# butane < config.yaml > config.json +variant: flatcar +version: 1.1.0 +storage: + files: + - path: /etc/subuid + append: + - inline: | + root:1065536:65536 + - path: /etc/subgid + append: + - inline: | + root:1065536:65536 + - path: /etc/flatcar/enabled-sysext.conf + contents: + inline: | + incus +``` + +Once the instance booted, `incus` command is available and `incus.{socket,service}` are started. Note: the `core` user is added by default to the `incus-admin` group. +```bash +core@localhost ~ $ systemd-sysext status +HIERARCHY EXTENSIONS SINCE +/opt none - +/usr flatcar-incus Fri 2025-05-16 08:43:36 UTC +core@localhost ~ $ incus --version +6.0.3 +core@localhost ~ $ userdbctl groups-of-user core +USER GROUP +core docker +core docker +core incus-admin +core portage +core systemd-journal +core systemd-journal +core wheel +core wheel + +8 memberships listed. +``` + +# Initialize Incus + +Before using Incus, it must be initialized using one of those two ways: +* Via CLI - Ideal for testing and quick setups. +* Using a pre-seed file - Recommended for automation and repeatable deployments. + +The CLI can output the pre-seed file at the end of the manual initialization. Once done, it is possible to use pre-seed file from Butane: +```yaml +--- +# config.yaml +# butane < config.yaml > config.json +variant: flatcar +version: 1.1.0 +storage: + files: + - path: /etc/subuid + append: + - inline: | + root:1065536:65536 + - path: /etc/subgid + append: + - inline: | + root:1065536:65536 + - path: /etc/flatcar/enabled-sysext.conf + contents: + inline: | + incus + - path: /etc/incus/pre-seed.yaml + contents: + inline: | + config: {} + networks: + - config: + ipv4.address: auto + ipv6.address: auto + description: "" + name: incusbr0 + type: "" + project: default + storage_pools: + - config: + size: 5GiB + description: "" + name: default + driver: btrfs + profiles: + - config: {} + description: "Flatcar profile" + devices: + eth0: + name: eth0 + network: incusbr0 + type: nic + root: + path: / + pool: default + type: disk + name: default + projects: [] + cluster: null +systemd: + units: + - name: incus-admin-init.service + enabled: true + contents: | + [Unit] + Description=incus admin init + After=systemd-sysext.service + Requires=systemd-sysext.service + [Service] + StandardInput=file:/etc/incus/pre-seed.yaml + Type=oneshot + RemainAfterExit=yes + ExecStart=/usr/bin/incus admin init --preseed + [Install] + WantedBy=multi-user.target +``` + +# Using Incus + +Once Incus is installed and initialized, Incus [documentation][documentation] can be consulted to learn how to operate containers. + +```bash +core@localhost ~ $ incus launch images:ubuntu/22.04 ubuntu-01 +Launching ubuntu-01 +core@localhost ~ $ incus list ++-----------+---------+----------------------+------+-----------+-----------+ +| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | ++-----------+---------+----------------------+------+-----------+-----------+ +| ubuntu-01 | RUNNING | 10.126.29.162 (eth0) | | CONTAINER | 0 | ++-----------+---------+----------------------+------+-----------+-----------+ +core@localhost ~ $ incus exec ubuntu-01 cat /etc/os-release +PRETTY_NAME="Ubuntu 22.04.5 LTS" +NAME="Ubuntu" +VERSION_ID="22.04" +VERSION="22.04.5 LTS (Jammy Jellyfish)" +VERSION_CODENAME=jammy +ID=ubuntu +ID_LIKE=debian +HOME_URL="https://www.ubuntu.com/" +SUPPORT_URL="https://help.ubuntu.com/" +BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" +PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" +UBUNTU_CODENAME=jammy +``` + +# Using Incus and ZFS + +[ZFS][zfs] is provided as a Systemd sysext Flatcar extension as well, it fits perfectly with Incus as a storage driver. This configuration can be used to deactivate Docker and Containerd and enable Incus runtime using ZFS storage. + +```yaml +--- +# config.yaml +# butane < config.yaml > config.json +variant: flatcar +version: 1.1.0 +storage: + files: + - path: /etc/subuid + append: + - inline: | + root:1065536:65536 + - path: /etc/subgid + append: + - inline: | + root:1065536:65536 + - path: /etc/flatcar/enabled-sysext.conf + contents: + inline: | + incus + zfs + - path: /etc/incus/pre-seed.yaml + contents: + inline: | + config: {} + networks: + - config: + ipv4.address: auto + ipv6.address: auto + description: "" + name: incusbr0 + type: "" + project: default + storage_pools: + - config: + size: 5GiB + description: "Incus ZFS pool" + name: default + driver: zfs + profiles: + - config: {} + description: "" + devices: + eth0: + name: eth0 + network: incusbr0 + type: nic + root: + path: / + pool: default + type: disk + name: default + projects: [] + cluster: null + links: + - path: /etc/extensions/docker-flatcar.raw + target: /dev/null + overwrite: true + - path: /etc/extensions/containerd-flatcar.raw + target: /dev/null + overwrite: true +systemd: + units: + - name: incus-admin-init.service + enabled: true + contents: | + [Unit] + Description=incus admin init + After=systemd-sysext.service + Requires=systemd-sysext.service + [Service] + StandardInput=file:/etc/incus/pre-seed.yaml + Type=oneshot + RemainAfterExit=yes + ExecStart=/usr/bin/incus admin init --preseed + [Install] + WantedBy=multi-user.target +``` + +Incus will now use ZFS pool to store images and containers: + +```bash +core@localhost ~ $ incus storage list ++---------+--------+----------------+---------+---------+ +| NAME | DRIVER | DESCRIPTION | USED BY | STATE | ++---------+--------+----------------+---------+---------+ +| default | zfs | Incus ZFS pool | 1 | CREATED | ++---------+--------+----------------+---------+---------+ +core@localhost ~ $ zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +default 4.50G 592K 4.50G - - 1% 0% 1.00x ONLINE - +``` + + +[incus]: https://linuxcontainers.org/incus/ +[documentation]: https://linuxcontainers.org/incus/docs/main/ +[official-sysext]: https://www.flatcar.org/docs/latest/provisioning/sysext/#flatcar-release-extensions-official +[zfs]: ../setup/storage/zfs diff --git a/content/docs/23-10-2025/container-runtimes/registry-authentication.md b/content/docs/23-10-2025/container-runtimes/registry-authentication.md new file mode 100644 index 000000000..c442bcc28 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/registry-authentication.md @@ -0,0 +1,179 @@ +--- +title: Authenticating to Container Registries +description: Configuration examples for authenticating to different container registries. +weight: 50 +aliases: + - ../os/registry-authentication + - ../clusters/management/registry-authentication +--- + +Many container image registries require authentication. This document explains how to configure container management software like Docker, Kubernetes, rkt, and Mesos to authenticate with and pull containers from registries like [Quay][quay-site] and [Docker Hub][docker-hub-site]. + +## Using a Quay robot for registry auth + +The recommended way to authenticate container manager software with [quay.io][quay-site] is via a [Quay Robot][quay-robot]. The robot account acts as an authentication token with some nice features, including: + +* Readymade repository authentication configuration files +* Credentials are limited to specific repositories +* Choose from read, write, or admin privileges +* Token regeneration + +![Quay Robot settings][quay-bot-img] + +Quay robots provide config files for Kubernetes, Docker, Mesos, and rkt, along with instructions for using each. Find this information in the **Robot Accounts** tab under your Quay user settings. For more information, see the [Quay robot documentation][quay-robot]. + +## Manual registry auth setup + +If you are using a registry other than Quay (e.g., Docker Hub, Docker Store, etc) you will need to manually configure your credentials with your container-runtime or orchestration tool. + +### Docker + +The Docker client uses an interactive command to authenticate with a centralized service. + +```shell +docker login -u -p https://registry.example.io +``` + +This command creates the file `$HOME/.docker/config.json`, formatted like the following example: + +**/home/core/.docker/config.json:** + +```json +{ + "auths": { + "https://index.docker.io/v1/": { + "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" + }, + "quay.io": { + "xxxx": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" + }, + "https://registry.example.io/v0/": { + "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" + } + } +} +``` + +On Flatcar Container Linux, this process can be automated by writing out the config file during system provisioning [with a Butane Config][butane-configs]. Since the config is written to the `core` user's home directory, ensure that your systemd units run as that user, by adding, e.g., `User=core`. + +Docker also offers the ability to configure a credentials store, such as your operating system's keychain. This is outlined in the [Docker login documentation][docker-login]. + +### Kubernetes + +Kubernetes uses [*Secrets*][k8s-secrets] to store registry credentials. + +When manually configuring authentication with *any* registry in Kubernetes (including Quay and Docker Hub) the following command is used to generate the Kubernetes registry-auth secret: + +```shell +$ kubectl create secret docker-registry my-favorite-registry-secret --docker-username=giffee_lover_93 --docker-password='passphrases are great!' --docker-email='giffee.lover.93@example.com' --docker-server=registry.example.io +secret "my-favorite-registry-secret" created +``` + +If you prefer you can store this in a YAML file by adding the `--dry-run` and `-o yaml` flag to the end of your command and copying or redirecting the output to a file: + +```shell +kubectl create secret docker-registry my-favorite-registry [...] --dry-run -o yaml | tee credentials.yaml +``` + +```yaml +apiVersion: v1 +data: + .dockercfg: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx== +kind: Secret +metadata: + creationTimestamp: null + name: my-favorite-registry-secret +type: kubernetes.io/dockercfg +``` + +```shell +$ kubectl create -f credentials.yaml +secret "my-favorite-registry-secret" created +``` + +You can check that this secret is loaded with with the `kubectl get` command: + +```shell +$ kubectl get my-favorite-registry-secret +NAME TYPE DATA AGE +my-favorite-registry-secret kubernetes.io/dockercfg 1 30m +``` + +The secret can be used in a Pod spec with the `imagePullSecrets` variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: somepod + namespace: all +spec: + containers: + - name: web + image: registry.example.io/v0/giffee_lover_93/somerepo + + imagePullSecrets: + - name: my-favorite-registry-secret +``` + +For more information, check the [docker-registry Kubernetes secret][k8s-docker-registry] and [Kubernetes imagePullSecrets][k8s-image-pull] documentation. + +## Copying the config file with a Butane Config + +[Butane Configs][butane-configs] can be used to provision a Flatcar Container Linux node on first boot. Here we will use it to copy registry authentication config files to their appropriate destination on disk. This provides immediate access to your private Docker Hub and Quay image repositories without the need for manual intervention. The same Butane Config file can be used to copy registry auth configs onto an entire cluster of Flatcar Container Linux nodes. + +Here is an example of using a Butane Config to write the `.docker/config.json` registry auth configuration file mentioned above to the appropriate path on the Flatcar Container Linux node: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + files: + - path: /home/core/.docker/config.json + mode: 0644 + contents: + inline: | + { + "auths": { + "quay.io": { + "auth": "AbCdEfGhIj", + "email": "your.email@example.com" + } + } + } +``` + +Butane Configs can also download a file from a remote location and verify its integrity with a SHA512 hash: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + files: + - path: /home/core/.docker/config.json + mode: 0644 + contents: + source: http://internal.infra.example.com/cluster-docker-config.json + verification: + hash: sha512-0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +``` + +For details, check out the [Butane Config examples][butane-examples]. + +[config-valid]: https://coreos.com/validate/ +[docker-hub-site]: https://hub.docker.com/ +[docker-instructions]: #docker +[docker-login]: https://docs.docker.com/engine/reference/commandline/login/ +[docker-reg-v2]: https://docs.docker.com/registry/spec/auth/jwt/ +[k8s-docker-registry]: https://kubernetes.io/docs/user-guide/kubectl/kubectl_create_secret_docker-registry/ +[k8s-docker-registry]: https://kubernetes.io/docs/user-guide/kubectl/kubectl_create_secret_docker-registry/ +[k8s-image-pull]: https://kubernetes.io/docs/user-guide/images/ +[k8s-secrets]: https://kubernetes.io/docs/user-guide/secrets/ +[mesos-registry]: https://mesosphere.github.io/marathon/docs/native-docker-private-registry.html +[quay-bot-img]: ../img/quay-robot-screen.png +[quay-robot]: https://docs.quay.io/glossary/robot-accounts.html +[quay-site]: https://quay.io/ +[rfc-2397]: https://tools.ietf.org/html/rfc2397 +[rkt-config]: registry-authentication/#rkt +[butane-configs]: ../provisioning/config-transpiler +[butane-examples]: ../provisioning/config-transpiler/examples diff --git a/content/docs/23-10-2025/container-runtimes/switching-to-unified-cgroups.md b/content/docs/23-10-2025/container-runtimes/switching-to-unified-cgroups.md new file mode 100644 index 000000000..c68a7c981 --- /dev/null +++ b/content/docs/23-10-2025/container-runtimes/switching-to-unified-cgroups.md @@ -0,0 +1,153 @@ +--- +title: Switching to Unified Cgroups +linktitle: Switching to unified cgroups +description: Overview of changes necessary to use unified cgroups with Kubernetes +weight: 20 +aliases: +--- + +Beginning with Flatcar version 2969.0.0, Flatcar Linux has migrated to the unified +cgroup hierarchy (aka cgroup v2). Much of the container ecosystem has already +moved to default to cgroup v2. Cgroup v2 brings exciting new features in +areas such as eBPF and rootless containers. + +Flatcar nodes deployed prior to this change will be kept on cgroups v1 (legacy +hierarchy) and will require manual migration. During an update from an older +Flatcar version, a post update script does two things: + +* adds the kernel command line parameters `systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller` + to `/usr/share/oem/grub.cfg` +* creates a systemd drop-in unit at `/etc/systemd/system/containerd.service.d/10-use-cgroupfs.conf` that + configures `containerd` to keep using cgroupfs for cgroups. + +# Migrating old nodes to unified cgroups + +To undo the changes performed by the post update script, execute the following commands as root (or using `sudo`): + +```bash +rm /etc/systemd/system/containerd.service.d/10-use-cgroupfs.conf +sed -i -e '/systemd.unified_cgroup_hierarchy=0/d' /usr/share/oem/grub.cfg +sed -i -e '/systemd.legacy_systemd_cgroup_controller/d' /usr/share/oem/grub.cfg +reboot +``` + +# Starting new nodes with legacy cgroups + +Nodes deployed with the release incorporating the described changes use cgroups v2 by default. To revert to cgroups v1 on new +nodes during provisioning, use the following Ignition snippet (here as Butane YAML to be transpiled to Ignition JSON): + +```yaml +variant: flatcar +version: 1.0.0 +storage: + filesystems: + - name: "OEM" + mount: + device: "/dev/disk/by-label/OEM" + format: "btrfs" +kernel_arguments: + should_exist: + - systemd.unified_cgroup_hierarchy=0 + - systemd.legacy_systemd_cgroup_controller +systemd: + units: + - name: containerd.service + dropins: + - name: 10-use-cgroupfs.conf + contents: | + [Service] + Environment=CONTAINERD_CONFIG=/usr/share/containerd/config-cgroupfs.toml +``` + +However, the kernel commandline setting doesn't take effect on the first boot, and a reboot is required before the snippet becomes active. + +If your deployment can't tolerate the required reboot, consider using the following snippet to switch to legacy cgroups without a reboot. This is supported by Flatcar 3033.2.4 or newer: + +```yaml +variant: flatcar +version: 1.0.0 +storage: + filesystems: + - device: "/dev/disk/by-label/OEM" + format: "btrfs" + files: + - path: /etc/flatcar-cgroupv1 + mode: 0444 +kernel_arguments: + should_exist: + - systemd.unified_cgroup_hierarchy=0 + - systemd.legacy_systemd_cgroup_controller +systemd: + units: + - name: containerd.service + dropins: + - name: 10-use-cgroupfs.conf + contents: | + [Service] + Environment=CONTAINERD_CONFIG=/usr/share/containerd/config-cgroupfs.toml +``` + +Beware that over time it is expected that upstream projects will drop support for cgroups v1. + +**Known issues:** Unprivileged containers with user namespaces may lack permissions to access the bind mount ([Flatcar#722](https://github.com/flatcar/Flatcar/issues/722)) and unmounting the bind mount before starting the container is needed. + +## Generate AWS EC2 cgroups v1 AMIs + +The [`create_cgroupv1_ami.sh` script](https://raw.githubusercontent.com/flatcar/flatcar-docs/main/create_cgroupv1_ami.sh) performs the image modification as outlined above for you to upload your own Flatcar AMI that directly boots into cgroup v1 without needing an additonal reboot. + +# Kubernetes + +The unified cgroup hierarchy is supported starting with Docker v20.10 and +Kubernetes v1.19. Users that need to run older version will need to revert to +cgroups v1, but are urged to find a migration path. Flatcar now ships with Docker +v20.10, older versions can be deployed following the instructions on [running custom docker versions](use-a-custom-docker-or-containerd-version). + +Flatcar nodes that had Kubernetes deployed on them before the introduction of +cgroups v2 should be careful when migrating. Depending on the deployment method, +the `cgroupfs` cgroup driver may be hardcoded in the `kubelet` configuration. +Cgroups v2 are only supported with the `systemd` cgroup driver. See [configuring a cgroup driver][kube-cgroup-docs] in the Kubernetes documentation for a discussion of cgroup drivers and how to migrate nodes. We recommend redeploying Kubernetes on fresh nodes instead of migrating inplace. + +The cgroup driver used by `kubelet` should be the same as the one used by `docker` daemon. `docker` defaults to `systemd` cgroup driver when started on a system running cgroup v2 and `cgroupfs` when running on a system with cgroup v1. The cgroup driver can be explicitly configured for `docker` by extending `/etc/docker/daemon.json`: +```json +{ + "exec-opts": ["native.cgroupdriver=systemd"] +} +``` +or adding a `docker.service` drop-in at `/etc/systemd/system/docker.service.d/10-cgroup-v2.conf`: +```ini +[Service] +Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver=systemd" +``` + +## Container Runtimes + +When deploying Kubernetes through `kubeadm`, the default container runtime on Flatcar is currently `dockershim`. In this setup, `kubelet` talks to `dockershim`, which talks to `docker`, which interfaces with `containerd`. The `SystemdCgroup` setting in `containerd`'s `config.toml` is ignored. `docker`'s cgroup driver and `kubelet` cgroup driver settings must match. Starting with Kubernetes v1.22, `kubeadm` will default to the `systemd` `cgroupDriver` setting if no setting is provided explicitly. Out of the box, Flatcar defaults are compatible with Docker and Kubernetes defaults - everything will use `systemd` cgroup driver. + +When using `kubeadm`, add the snippet to your `kubeadm-config.yaml` to configure the `kubelet` cgroup driver: + +```yaml +--- +kind: KubeletConfiguration +apiVersion: kubelet.config.k8s.io/v1beta1 +cgroupDriver: systemd +``` + +## Containerd + +If users choose the `containerd` runtime, they must ensure that `containerd`'s setting for `SystemdCgroup` is consistent with `kubelet` and `docker` settings. Flatcar enables `SystemdCgroup` by default for `containerd`. Users may change the setting to suit their deployment. +If you maintain your own containerd configuration or did follow the instructions on +[how to customize containerd configuration](customizing-docker), you should add the relevant lines to your `config.toml`: +```toml +version = 2 + +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] + # setting runc.options unsets parent settings + runtime_type = "io.containerd.runc.v2" + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] + SystemdCgroup = true + ``` + +For a more detailed discussion of container runtimes, see the [Kubernetes documentation][kube-runtime-docs]. + +[kube-cgroup-docs]: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#migrating-to-the-systemd-driver +[kube-runtime-docs]: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ diff --git a/content/docs/23-10-2025/contribute/_index.md b/content/docs/23-10-2025/contribute/_index.md new file mode 100644 index 000000000..03c4f5c2d --- /dev/null +++ b/content/docs/23-10-2025/contribute/_index.md @@ -0,0 +1,58 @@ +--- +title: How to Contribute +content_type: contribute +weight: 130 +--- + +Flatcar documentation is released under the [Apache 2.0 License][asl], and we welcome contributions. Check out the [help-wanted tag][help-wanted] in this project's Issues list for good places to start participating. + +Submit fixes and additions in the form of [GitHub *Pull Requests* (PRs)][pull-requests]. The general process is the typical git fork-branch-PR-review-merge cycle: + +1. Fork this repository into your GitHub account +2. Make changes in a topic branch or your fork's `master` +3. Send a Pull Request from that topic branch to flatcar/flatcar-website +4. Maintainers will review the PR and either merge it or make comments + +Cognizance of the tribal customs described and linked to below will help get your contributions incorporated with the greatest of ease. + +## Clear commit messages + +Commit messages follow a format that makes clear **what** changed and **why** it changed. The first line of each commit message should clearly state what module or file changed, summarize the change very briefly, and should end, without a period, somewhere short of 70 characters. After a blank line, the body of the commit message should then explain why the change was needed, with lines wrapped at 72 characters wide and sentences normally punctuated. Cite related issues or previous revisions as appropriate. For example: + +``` +ignition: Update etcd example to use %m + +Make the etcd configuration example use ignition's %m instead of the +ETCD_NAME environment variable. Fixes #123. +``` + +This format can be described somewhat more formally as: + +``` +: + + + +[