NodeOpUpgrade
The NodeOpUpgrade custom resource is a Kairos-specific resource for upgrading Kairos nodes. Under the hood, it creates a NodeOp with the appropriate upgrade script and configuration, so you only need to specify the target image and a few options.
One-off operations and reusing manifestsβ
A NodeOpUpgrade represents a single upgrade run on the target nodes. The operator drives one upgrade flow per object; changing spec on an existing resource is not a supported way to βstart overβ or switch to a different upgrade plan. The API allows spec updates, but behavior after an update is undefined from a product perspectiveβcreate a new NodeOpUpgrade for a new run.
Reusing the same manifest with generateNameβ
To run the same upgrade configuration repeatedly, use metadata.generateName instead of metadata.name and kubectl create (not apply) so each run creates a new NodeOpUpgrade. See NodeOp: One-off operations and reusing manifests for the same pattern and rationale.
GitOps alternative: static-name bump with a versioned suffixβ
generateName + kubectl create is an imperative workflow β the CR name is unknown ahead of time and the run doesn't live in git. In a GitOps setup (ArgoCD / Flux / etc.) both properties are inverted: every resource has a known static name that git owns, and drift-detection expects the same name to describe the same object across reconciliations.
The GitOps equivalent of "new run = new object" is to make the version part of the name, and bump the name and the image reference in the same commit. Each merged commit produces a new named resource, which the operator treats as a new one-shot run. Same semantic guarantee as generateName, but declarative and reviewable.
apiVersion: operator.kairos.io/v1alpha1
kind: NodeOpUpgrade
# The version fragment in the name is REQUIRED. Same name across two commits
# would look like a spec update to the operator (behavior undefined per the
# warning above). Bumping the name makes the new commit a new object.
metadata:
name: hadron-mgmt-v0-4-0 # <-- bumped alongside spec.image
namespace: default
spec:
image: quay.io/kairos/hadron:v0.4.0-standard-amd64-generic-v1.31.0-k3s-v1.31.0-k3s1
concurrency: 1
stopOnFailure: true
Bumping the name by hand at every release defeats the point of GitOps. A dependency-update bot (Renovate, dependabot, etc.) can watch the image tag and open a PR that rewrites both the image: line and the version fragment in metadata.name:.
For Renovate, a custom regex manager targeting a composite tag (Kairos + k8s version) looks like:
{
"customManagers": [
{
"customType": "regex",
"fileMatch": ["^upgrades/.*\\.ya?ml$"],
"matchStrings": [
"image:\\s+quay\\.io/kairos/hadron:(?<currentValue>v\\d+\\.\\d+\\.\\d+-standard-(?:amd64|arm64)-generic-v\\d+\\.\\d+\\.\\d+-k3s-v\\d+\\.\\d+\\.\\d+-k3s1)"
],
"datasourceTemplate": "docker",
"depNameTemplate": "quay.io/kairos/hadron",
"versioningTemplate": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-standard-(?:amd64|arm64)-generic-v\\d+\\.\\d+\\.\\d+-k3s-v\\d+\\.\\d+\\.\\d+-k3s1$"
},
{
"customType": "regex",
"fileMatch": ["^upgrades/.*\\.ya?ml$"],
"matchStrings": [
"name:\\s+hadron-(?<cluster>[a-z]+)-v(?<major>\\d+)-(?<minor>\\d+)-(?<patch>\\d+)"
],
"currentValueTemplate": "v{{{major}}}.{{{minor}}}.{{{patch}}}",
"autoReplaceStringTemplate": "name: hadron-{{{cluster}}}-v{{{newMajor}}}-{{{newMinor}}}-{{{newPatch}}}",
"datasourceTemplate": "docker",
"depNameTemplate": "quay.io/kairos/hadron",
"versioningTemplate": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-standard-(?:amd64|arm64)-generic-v\\d+\\.\\d+\\.\\d+-k3s-v\\d+\\.\\d+\\.\\d+-k3s1$"
}
],
"packageRules": [
{
"matchPackageNames": ["quay.io/kairos/hadron"],
"allowedVersions": "/.*-k3s-v1\\.35\\.\\d+-k3s1$/",
"minimumReleaseAge": "24 hours",
"automerge": false
}
]
}
Three things to note:
allowedVersionspins updates to the current k3s minor line (1.35.x). This prevents Renovate from silently bundling a k3s minor upgrade with a Kairos patch bump. To adopt a new k3s minor, update the regex explicitly.- The second custom manager handles
metadata.name.currentValueTemplateconverts the dash-format slug (v0-3-0βv0.3.0) so Renovate can compare it against the docker datasource.autoReplaceStringTemplatewrites the new version back in dash-format. Both managers share the samedepNameTemplate(quay.io/kairos/hadron) anddatasourceTemplate(docker), so Renovate coalesces them into a single PR by default β nogroupNamerequired. Note this is emergent behaviour from matching the same package, not a transactional guarantee: if one manager's regex fails to match (e.g. a filename not covered byfileMatch), only the other field is updated. - Do not use
extractVersionTemplateto parse the dash-format inmetadata.nameβ it is not a valid field for Renovate custom managers and is silently ignored. The result is thatspec.imagegets bumped butmetadata.namestays at the old version, so the operator sees no new CR and does nothing. - The
versioningTemplateregex hardcodes(?:amd64|arm64)in both managers. If your CR uses a different arch (e.g.riscv64) neither regex matches and Renovate silently skips the CR. Adapt the alternation to match the arch segment in your actual image tags.
generateName and static-name bump are equivalent in semantics β both produce a new one-shot CR. The difference is operational: generateName suits imperative workflows (CLI, CI job) where the run doesn't need to live in git. Static-name bump suits GitOps where every change is a reviewed commit.
Basic Exampleβ
The following is an example of a "canary upgrade", which upgrades Kairos nodes one-by-one (master nodes first). It will stop upgrading if one of the nodes doesn't complete the upgrade and reboot successfully.
The nodeSelector matches every Kairos node. To upgrade a subset, select on
the other labels the operator writes, such as the architecture or the boot
state. See Node labels and annotations.
The image references below show a valid tag format, but these non-Hadron flavor repositories are not actively updated by the Kairos release pipeline anymore. Build and publish your own upgrade image with BYOI and Kairos Factory.
apiVersion: operator.kairos.io/v1alpha1
kind: NodeOpUpgrade
metadata:
name: kairos-upgrade
namespace: default
spec:
# The container image containing the new Kairos version
image: quay.io/kairos/opensuse:leap-15.6-standard-amd64-generic-v3.4.2-k3sv1.30.11-k3s1
# NodeSelector to target specific nodes (optional)
nodeSelector:
matchLabels:
kairos.io/managed: "true"
# Maximum number of nodes that can run the upgrade simultaneously
# 0 means run on all nodes at once
concurrency: 1
# Whether to stop creating new jobs when a job fails
# Useful for canary deployments
stopOnFailure: true
Only 4 fields is all it takes to safely upgrade the whole cluster.
Spec Referenceβ
| Field | Type | Default | Description |
|---|---|---|---|
image | string | (required) | Container image containing the new Kairos version |
imagePullSecrets | []LocalObjectReference | (none) | Secrets for pulling from private registries (details) |
nodeSelector | LabelSelector | (none) | Standard Kubernetes label selector to target specific nodes |
concurrency | int | 0 | Max nodes running the upgrade simultaneously (0 = all at once) |
stopOnFailure | bool | false | Stop creating new jobs when a job fails (canary mode) |
upgradeActive | bool | true | Whether to upgrade the active partition |
upgradeRecovery | bool | false | Whether to upgrade the recovery partition |
force | bool | false | When true, run the upgrade on every targeted node regardless of whether it is already at spec.image. Disables the preflight skip β see Skipping no-op upgrades. |
debug | bool | false | Run kairos-agent with the global --debug flag for verbose upgrade output. See Debugging upgrades. |
uncordonOnFailure | bool | false | Uncordon a node if its upgrade fails, instead of leaving it unschedulable. Passed through to the underlying NodeOp. See Recovering nodes after a failed upgrade and How cordoning works for the full lifecycle. |
excludePaths | []string | (none) | Paths of spec.image to leave out of the new system, passed to kairos-agent upgrade as --exclude-path. Requires kairos-agent v3.6.0+ in spec.image. See Which files the upgrade installs. |
resources | ResourceRequirements | (none) | Resource requests/limits for the main nodeop container of the generated NodeOp. See Resource specification. |
preflightResources | ResourceRequirements | (built-in) | Resource requests/limits for the preflight Pod of the generated NodeOp. See Resource specification. |
rebootResources | ResourceRequirements | (built-in) | Resource requests/limits for the reboot Pod of the generated NodeOp. See Resource specification. |
Additional Optionsβ
apiVersion: operator.kairos.io/v1alpha1
kind: NodeOpUpgrade
metadata:
name: kairos-upgrade
namespace: default
spec:
image: quay.io/kairos/opensuse:leap-15.6-standard-amd64-generic-v3.4.2-k3sv1.30.11-k3s1
# ImagePullSecrets for private registries (optional)
imagePullSecrets:
- name: private-registry-secret
nodeSelector:
matchLabels:
kairos.io/managed: "true"
concurrency: 1
stopOnFailure: true
# Whether to upgrade the active partition (defaults to true)
upgradeActive: true
# Whether to upgrade the recovery partition (defaults to false)
upgradeRecovery: false
# Whether to force the upgrade. When true, the controller skips the
# preflight version check and runs the upgrade on every targeted node
# even if it is already at spec.image. See "Skipping no-op upgrades" below.
force: false
# Run kairos-agent with --debug for verbose upgrade output (defaults to false)
debug: false
# Uncordon a node if its upgrade fails (defaults to false)
uncordonOnFailure: false
# Paths of spec.image to leave out of the new system. Requires
# kairos-agent v3.6.0+ in spec.image.
excludePaths:
- /var/lib/mystuff
- /opt/keep
To upgrade the "recovery" partition instead of the active one, set upgradeRecovery: true and upgradeActive: false:
spec:
# ... other fields ...
upgradeActive: false
upgradeRecovery: true
Skipping no-op upgradesβ
By default, NodeOpUpgrade avoids cordoning, draining, and rebooting nodes that are already running the requested spec.image. This is useful for staggered rollouts β for example, upgrading just the control plane first and then a cluster-wide NodeOpUpgrade with the same image: control-plane nodes are detected as already up-to-date and left alone, while worker nodes go through the normal flow.
This works by leveraging the generic NodeOp preflight mechanism. When the NodeOpUpgrade controller creates the underlying NodeOp, it populates spec.preflight with a short script that:
- Runs in the upgrade image (the one you set in
spec.image), so the script can read the target/etc/kairos-releasedirectly from inside that image without pulling anything else. - Mounts the host's
/etcread-only at/host/etc, so the same script can also read the currently installed/etc/kairos-releaseon the node. - Computes the version triple β
${KAIROS_VERSION}-${KAIROS_SOFTWARE_VERSION_PREFIX}${KAIROS_SOFTWARE_VERSION}β from each side and compares them. - If both versions are known and equal, writes the skip reason to
/dev/termination-log(e.g.node is already at v4.0.3-k3sv1.32.4-k3s1). - If versions differ, can't be determined, or anything else, exits 0 silently β the controller proceeds with the normal cordon β drain β upgrade Job β reboot flow on that node.
The controller honors the preflight verdict by not creating a Job, not cordoning, and not rebooting any node the preflight skipped. The node's entry in status.nodeStatuses is marked Completed with the skip reason from /dev/termination-log, and the per-node concurrency slot is freed immediately for the next node.
When the skip kicks in (and when it doesn't)β
The preflight comparison runs the script against the actual /etc/kairos-release contents on both sides, so it's reliable across:
- Image mirrors / re-tags. It doesn't matter that you mirror
quay.io/kairos/fedora:0.7.1to your own registry; the script reads the file contents, not the image reference. - Nodes that were bootstrapped from a particular image (you didn't have to install via the operator first).
It won't fire when:
- The image has been rebuilt with the same
KAIROS_VERSIONvalues but different content (e.g. a CI re-run with the same tag but a different commit). Version-triple equality is the only signal the script uses; if the metadata is the same, the script will mark the node as already up-to-date. Usespec.force: trueto override. - The host's
/etc/kairos-releaseis missing or doesn't carryKAIROS_VERSION. The script treats either side as "unknown" and falls through to "proceed" rather than wrongly skip β the in-Pod upgrade flow then runs and the user will see whatever it reports.
Forcing the upgradeβ
Set spec.force: true to disable the preflight entirely. The controller creates the NodeOp without spec.preflight, so every targeted node goes straight through cordon β drain β upgrade Job β reboot, regardless of what version is already installed. Use this when you want to re-run an upgrade with the same image but different flags, or to recover from a previous run that ended in a weird state.
Which files the upgrade installsβ
A NodeOpUpgrade runs kairos-agent in a Pod that uses spec.image, and kairos-agent builds the node's new system image from that Pod's root filesystem. Kubernetes mounts Pod-specific copies of /etc/resolv.conf, /etc/hosts and /etc/hostname into every Pod. They hold the cluster DNS settings and the Pod's own name and IP, and they must not end up on the node.
The upgrade script therefore copies from a plain bind mount of the Pod's root, which shows the image's own files without anything Kubernetes mounted on top.
When the image does not ship one of the three files, the container runtime creates an empty file at that path so it has something to mount the Pod's copy onto, and the bind mount shows that empty file. The script leaves any of the three that is missing, or is an empty regular file, out of the new system with --exclude-path. A symlink is always copied as is, whatever its target holds inside the Pod. This matters for /etc/resolv.conf on images that ship none and rely on systemd to create the /etc/resolv.conf symlink at boot, which it only does when nothing is at that path.
For an image without /etc/resolv.conf, the generated script looks like this:
UPGRADE_ROOTFS=$(mktemp -d)
mount --bind / "${UPGRADE_ROOTFS}"
# adds --exclude-path for each of /etc/resolv.conf, /etc/hosts and
# /etc/hostname that is missing or an empty regular file under ${UPGRADE_ROOTFS}
kairos-agent upgrade --source "dir:${UPGRADE_ROOTFS}" --exclude-path /etc/resolv.conf
The node gets the same /etc/resolv.conf, /etc/hosts and /etc/hostname that a fresh install of spec.image would have. On Kairos images, the boot stages then set the hostname and hosts file from your cloud-config, or from the defaults if it sets nothing.
Leaving paths out of the new systemβ
Use spec.excludePaths to leave paths of spec.image out of the new system:
spec:
# ... other fields ...
excludePaths:
- /var/lib/mystuff
- /opt/keep
Each path is passed to kairos-agent upgrade as --exclude-path:
kairos-agent upgrade --source "dir:${UPGRADE_ROOTFS}" --exclude-path '/var/lib/mystuff' --exclude-path '/opt/keep'
They come after any exclusions the script adds for /etc/resolv.conf, /etc/hosts and /etc/hostname.
kairos-agent builds the new system image from scratch, so an excluded path is missing from the new system. The node's current copy of that path is not carried over. To keep data across upgrades, store it on a persistent path such as /usr/local or one of the paths Kairos bind-mounts from the persistent partition.
Requirementsβ
--exclude-path was added in kairos-agent v3.6.0. Older versions of kairos-agent do not recognize the flag and the upgrade Job will fail with unknown flag: --exclude-path. The flag is used when spec.excludePaths is set, and also when spec.image has no /etc/resolv.conf, /etc/hosts or /etc/hostname, or ships one of them as an empty regular file. In practice this only affects users who pin spec.image to a pre-v3.6.0 Kairos release. The kairos-agent that runs during the upgrade is the one shipped in spec.image, not the one on the current node, so upgrading a very old node to a v3.6.0 or newer image works fine.
Debugging upgradesβ
When an upgrade fails and the Job logs don't explain why, set spec.debug: true. The controller then runs kairos-agent with its global --debug flag, so the upgrade Job produces verbose output. Because --debug is a global flag, it is placed before the upgrade subcommand in the generated script (kairos-agent --debug upgrade ...).
spec:
# ... other fields ...
debug: true
Recovering nodes after a failed upgradeβ
Every NodeOpUpgrade cordons each targeted node before running the upgrade and uncordons it once the upgrade has completed and the node has rebooted. If the upgrade fails, the affected node stays cordoned (unschedulable) by default so you can inspect it. On a cluster with a single control plane node this can leave the control plane unschedulable until you intervene manually.
Set spec.uncordonOnFailure: true to have the operator uncordon a node whose upgrade failed, returning it to a schedulable state automatically. The value is passed through to the underlying NodeOp, and the operator only uncordons nodes it cordoned itself, so any pre-existing cordon (from a human or a different NodeOp) is left alone. Kairos applies upgrades to a separate partition, so a failed upgrade generally leaves the running system intact and safe to schedule again.
For the full cordon lifecycle (how cordon ownership is claimed, when the uncordon happens after a successful upgrade, and what the operator does with cordons it did not set), see How cordoning works in the NodeOp docs.
spec:
# ... other fields ...
uncordonOnFailure: true
How Upgrade Is Performedβ
Before you attempt an upgrade, it's good to know what to expect. Here is how the process works:
- The operator is notified about the NodeOpUpgrade resource and creates a NodeOp with the appropriate script, options, and (unless
spec.forceistrue) aspec.preflightthat compares the image's/etc/kairos-releaseagainst the host's. - The NodeOp controller lists matching Nodes using the provided label selector. If no selector is provided, all Nodes will match.
- The list is sorted with master nodes first, and based on the
concurrencyvalue, the first batch of Nodes will be processed (could be just 1 Node). - For each targeted node, the controller first runs a preflight Pod on that node β a short-lived, non-disruptive Pod (no cordon, no drain) using the upgrade image. The preflight script writes a skip reason to
/dev/termination-logwhen the node is already at the target version, or stays silent otherwise. See Skipping no-op upgrades. - If preflight says skip, the node is recorded as
Completedwith the skip reason and the controller moves on to the next node. No cordon, no drain, no reboot for that node. - If preflight says proceed, the controller starts a reboot Pod on the node. Once the reboot Pod is ready, the controller cordons and drains the node and starts the upgrade Job.
- The Job upgrades the node and finishes. The reboot Pod then reboots the node, and the controller confirms the reboot before it uncordons the node. See How the Reboot Works.
- If everything worked successfully, the operator advances to the next batch of nodes, respecting
concurrencyandstopOnFailure.
The result of the above process is that each upgrade Job finishes successfully, with no unnecessary restarts. The upgrade logs can be found in the Job's Pod logs.
How the Reboot Worksβ
Every time Linux starts, it picks a new random boot ID. Kubernetes shows it on each Node (status.nodeInfo.bootID). If the boot ID changed, the machine rebooted. The operator uses this to confirm the reboot after an upgrade.
The same flow applies to any NodeOp with rebootOnSuccess: true. In that case, the upgrade Job below is the NodeOp's main Job.
For each node:
- The operator starts a reboot Pod on the node and tells it the name of the upgrade Job it is about to create.
- Once the reboot Pod is ready, the operator cordons and drains the node.
- The operator starts the upgrade Job on the node.
- The Job upgrades the node. As its last step it reads the node's current boot ID (say
A) and reports it in its final status. - The operator sees the Job finish and stores
Afor this node. - The reboot Pod sees the Job finish and reads
Afrom the Job's final status. The node is still onA, so it reboots the node. - The node boots into the new version with a new boot ID (say
B). The reboot Pod starts again, readsA, seesB, and exits. - The operator sees the Node report
Band Ready.Bis notA, so the reboot is confirmed. The operator uncordons the node and moves on to the next one.
Steps 5 and 6 do not depend on each other: the node reboots even if the operator is unavailable at that moment (for example, because its own node is rebooting in the same batch). The operator stores A when it is back.
The reboot Pod only reads from the Kubernetes API: its own upgrade Job and that Job's Pods, in its own namespace.
Edge casesβ
These are handled automatically:
- The reboot command fails. The reboot Pod exits with an error and Kubernetes restarts it. The node is still on
A, so it tries again. A node that refuses to reboot shows up as a reboot Pod that keeps restarting, and the node stays cordoned. - The reboot command reports success but nothing happens. The reboot Pod waits 5 minutes, exits with an error, and tries again.
- The node reboots for an unrelated reason during the upgrade. The upgrade attempt is lost and the Job retries it. Only the attempt that succeeds reports a boot ID, so that reboot is not mistaken for the upgrade reboot.
- The node reboots for an unrelated reason after the upgrade finished, before the reboot Pod acts. The reboot Pod sees a new boot ID and does not reboot again. The operator counts it as the reboot, which is correct: the upgrade is already in place.
- The Kubernetes API is unreachable when the upgrade finishes (for example, the control plane node is rebooting in the same batch). The node waits to reboot until the API is back. Nothing is lost.
- The operator is down when the upgrade finishes. The Job's final status stays in the cluster. The reboot happens without the operator, and the operator confirms it when it is back.
- The reboot Pod cannot start again after the reboot (for example, the upgraded kubelet cannot start containers yet). Nothing depends on it after the reboot. The operator already has the old boot ID and reads the new one from the Node.
These need a human:
- The reboot Pod cannot start at all (for example, its image cannot be pulled). The operator does not cordon the node or start the upgrade until the reboot Pod is ready. The node keeps running workloads, but it holds one of the batch's slots, and the NodeOp status says it is waiting for the reboot Pod.
- The Job finishes without reporting a valid boot ID (for example, a custom
sentinelImagewithout a shell). The reboot Pod does not reboot without it, and the operator cannot confirm a reboot. The node stays cordoned, with the upgrade on disk but not active. - The operator is upgraded from v0.2.x while a node is mid-upgrade. Jobs created by the older version do not report a boot ID, so that node stays cordoned. Upgrade the operator while no upgrade is running.
Sandboxed container runtimes (for example, Kata Containers or gVisor) are not supported for the upgrade Job. Inside them the Job reads the sandbox's boot ID instead of the node's.
The NodeOpUpgrade stores the statuses of the various Jobs it creates so it can be used to monitor the summary of the operation.
Monitoringβ
You can monitor the progress of an upgrade:
$ kubectl get jobs -A
NAMESPACE NAME STATUS COMPLETIONS DURATION AGE
default kairos-upgrade-localhost-wr26f Running 0/1 24s 24s
$ kubectl get nodeopupgrades
NAME AGE
kairos-upgrade 5s
Resource specificationβ
The three resource fields of NodeOpUpgrade are passed through to the NodeOp the operator generates, so they have exactly the same semantics as their NodeOp counterparts:
| Field | Applies to | Description |
|---|---|---|
resources | Main nodeop container | The upgrade Job container (its init container in reboot mode, alongside the small boot-id-reporter container, which uses fixed resources). |
preflightResources | Preflight Pod container | The preflight Pod that compares the target version against the running one (always created, unless force: true skips preflight). |
rebootResources | Reboot Pod container | The reboot Pod that waits for the upgrade Job to finish and reboots the node. |
Semantics differ slightly per field:
resources- unset means no resource constraints on the main container; set means the given requests and limits are used.preflightResources/rebootResources- tri-state:- unset (nil): a built-in default of 200m CPU / 128Mi memory is applied to both requests and limits (Guaranteed QoS).
- explicit empty (
preflightResources: {}): opt-out - no resources are set at all. - set: the given requests and limits are used (requests and limits are independent).
Example: cap the upgrade Job container and tune the auxiliary Pods:
apiVersion: operator.kairos.io/v1alpha1
kind: NodeOpUpgrade
metadata:
name: kairos-upgrade
namespace: default
spec:
image: quay.io/kairos-io/kairos:v4.2.0
concurrency: 1
stopOnFailure: true
resources:
requests:
memory: "2Gi"
cpu: "1"
limits:
memory: "4Gi"
preflightResources:
requests:
cpu: "50m"
limits:
memory: "64Mi"
rebootResources: {}
In this example:
- The upgrade
nodeopcontainer requests 2Gi memory and is capped at 4Gi - useful on shared clusters where a large rsync-based upgrade should not starve other workloads. - The preflight Pod (a small version-comparison script) gets a smaller custom allocation than the built-in default.
- The reboot Pod opts out of any resource constraints entirely (the built-in 200m/128Mi default is skipped).
See NodeOp: Resource specification for the general description of the fields and the tri-state behavior.
What's next?β
- Upgrading from Kubernetes β full upgrade workflow guide
- Trusted Boot upgrades β upgrades with Trusted Boot enabled
- NodeOp β for custom upgrade logic or other operations
- Bandwidth Optimized Upgrades β optimize bandwidth during upgrades