#Upgrading Clusters on VMware vSphere

#Create the target machine templates

Before you start the rolling upgrade, create new VSphereMachineTemplate resources for the control plane and workers.

1. Export the existing control plane template

   kubectl get vspheremachinetemplate <cluster_name>-control-plane -n <namespace> -o yaml > new-cp-template.yaml

2. Modify the control plane template

   Edit new-cp-template.yaml:

   • Set metadata.name to a new unique name (for example, <cluster_name>-control-plane-v2)
   • Update spec.template.spec.template to the target VM template name
   • Update CPU, memory, or disk settings if needed
   • Remove server-generated fields: metadata.resourceVersion, metadata.uid, metadata.generation, metadata.creationTimestamp, metadata.managedFields, metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], and status
   • Leave spec.template.spec.providerID unset. The vSphere provider sets providerID to the VM's BIOS UUID once the VM is created; pre-filling it in the template breaks the controller's identity binding.

3. Export and modify the worker template

   kubectl get vspheremachinetemplate <cluster_name>-worker -n <namespace> -o yaml > new-worker-template.yaml

   Edit new-worker-template.yaml:

   • Set metadata.name to a new unique name (for example, <cluster_name>-worker-v2)
   • Update spec.template.spec.template to the target VM template name
   • Update CPU, memory, or disk settings if needed
   • Remove the same server-generated fields listed above

4. Apply both new templates

   kubectl apply -f new-cp-template.yaml
   kubectl apply -f new-worker-template.yaml

#Upgrade the control plane

Before you start, collect every required value from the target ACP row in the OS Support Matrix as described in Required Values From the OS Support Matrix.

1. Patch the KubeadmControlPlane with the target Kubernetes values

   Update the KubeadmControlPlane resource in a single edit to keep spec.version, the CoreDNS image tag, the etcd image tag, and the infrastructure template reference consistent with the same VM template:

   • spec.version ← Kubernetes Version from the OS Support Matrix row

   • spec.kubeadmConfigSpec.clusterConfiguration.dns.imageTag ← coredns column from the same row

   • spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.imageTag ← etcd column from the same row

   • spec.machineTemplate.infrastructureRef.name ← the new VSphereMachineTemplate name created above

     kubectl edit kubeadmcontrolplane <cluster_name> -n <namespace>

   Updating only spec.version is not sufficient. The CoreDNS and etcd image tags must move together with the Kubernetes version because they are built from the same release; leaving them at the previous values can result in CoreDNS and etcd pods that do not match the new Kubernetes minor version.

2. Update the Kube-OVN version annotation on the Cluster resource

   If the target ACP row in the OS Support Matrix shows a different kube-ovn (chart) value than the current cluster, patch the annotation on the Cluster resource so the vSphere provider reconciles the new Kube-OVN AppRelease.

   INFO

   Prerequisite — Kube-OVN reconcile gating annotation (vSphere only)

   On vSphere, the provider reconciles the Kube-OVN AppRelease only when the Cluster resource carries the annotation cpaas.io/network-type: kube-ovn. This annotation is normally set at cluster creation. If it is missing, the steps below will succeed at writing the version annotation but the Kube-OVN AppRelease will not be reconciled. Verify before proceeding:

   kubectl get cluster <cluster_name> -n <namespace> \
     -o jsonpath='{.metadata.annotations.cpaas\.io/network-type}{"\n"}'
   # Expected output: kube-ovn

   This precondition is vSphere-specific. The DCS and Huawei Cloud Stack providers use the DCSCluster / HCSCluster spec.networkType field instead and do not require this annotation.

   kubectl annotate cluster <cluster_name> -n <namespace> \
     cpaas.io/kube-ovn-version=<kube-ovn-version-from-matrix> --overwrite

   Kube-OVN is a Core lifecycle component, but on immutable OS the vSphere provider drives its delivery from this annotation; the annotation does not update automatically when spec.version changes.

   The vSphere provider reconciles a single Kube-OVN AppRelease named cni-kube-ovn in the cpaas-system namespace of the workload cluster. Run the following on the workload cluster (not the bootstrap KIND or the global cluster) to follow the reconciliation:

   # Overall AppRelease state — Sync and Health columns must reach a Success-equivalent reason
   kubectl get apprelease cni-kube-ovn -n cpaas-system
   
   # Installed revision and chart phase
   kubectl get apprelease cni-kube-ovn -n cpaas-system \
     -o jsonpath='Installed: {.status.charts.*.installedRevision}{"\n"}Phase: {.status.charts.*.phase}{"\n"}'

   The normal sequence is Upgrading → HealthChecking → Success. On small clusters the full transition typically completes within about one minute. Read the phases as follows:

   Phase	Meaning	installedRevision
   Upgrading	Helm release upgrade in progress. Sync condition is Unknown(Syncing).	Still the previous version
   HealthChecking	Helm release applied; controller is verifying Kube-OVN pods. Sync condition is True(Synced).	Already the target version
   Success	All three conditions (Validate, Sync, Health) are True.	Target version

   WARNING

   Do not declare the upgrade complete on installedRevision alone. The field flips to the target value during HealthChecking, before pods have been verified Ready. The chart is only considered upgraded when phase is Success and installedRevision matches the target.

   The AppRelease API also defines Downloading, Installing, Syncing, DownloadFailed, DeployFailed, and NotReady. The first three are transient and the upgrade should converge on its own. The last three indicate a failure that needs manual investigation; start with kubectl describe apprelease cni-kube-ovn -n cpaas-system to read the per-condition message field.

3. Monitor the control plane rollout

   kubectl -n <namespace> get kubeadmcontrolplane <cluster_name> -w
   kubectl -n <namespace> get machine -l cluster.x-k8s.io/control-plane

#Upgrade the worker nodes

After the control plane upgrade completes, update the MachineDeployment to reference the new worker template and the target Kubernetes version.

Typical changes include:

• spec.template.spec.version — the target Kubernetes version
• spec.template.spec.infrastructureRef.name — the new VSphereMachineTemplate name
• spec.template.spec.bootstrap.configRef.name — the new KubeadmConfigTemplate name, if bootstrap settings must change (see Updating Bootstrap Templates)

Apply the changes:

kubectl patch machinedeployment <cluster_name>-md-0 -n <namespace> \
  --type='merge' -p='{
    "spec": {
      "template": {
        "spec": {
          "version": "<target_kubernetes_version>",
          "infrastructureRef": {
            "name": "<new-worker-template-name>"
          }
        }
      }
    }
  }'

Monitor the worker rollout:

kubectl -n <namespace> get machinedeployment <cluster_name>-md-0 -w
kubectl -n <namespace> get machine
kubectl --kubeconfig=/tmp/<cluster_name>.kubeconfig get nodes -o wide