ZeroDownTime
/
kubezero
3
0
Fork 0
Code Issues 1 Pull Requests 24 Releases Wiki Activity
kubezero/docs/v1.24.md

2.6 KiB
Raw Permalink Blame History

KubeZero 1.24

TODO

What's new - Major themes

  • Cilium is now the default CNI, calico gets removed
  • cluster-autoscaler is enabled by default on AWS
  • worker nodes are now fully automatically updated to latest AMI and config in a rolling fashion
  • integrated Bitnami Sealed Secrets controller
  • reduced avg. CPU load on controller nodes, well below the 20% threshold to prevent extra costs from CPU credits in most cases

Version upgrades

  • cilium
  • metallb
  • nvidia-device-plugin
  • aws-node-termination-handler
  • aws-ebs-csi-driver
  • aws-efs-csi-driver
  • istio 1.16
  • argocd 2.5.5 + tweaks
  • all things prometheus incl. automated muting of certain alarms, eg. CPUOverCommit when cluster-autoscaler is available

FeatureGates

  • PodAndContainerStatsFromCRI
  • DelegateFSGroupToCSIDriver

Upgrade

(No, really, you MUST read this before you upgrade)

Ensure your Kube context points to the correct cluster !

  1. Review CFN config for controller and workers, no mandatory changes during this release though

  2. Upgrade CFN stacks for the control plane ONLY ! Updating the workers CFN stacks would trigger rolling updates right away !

  3. Trigger cluster upgrade:
    ./admin/upgrade_cluster.sh <path to the argocd app kubezero yaml for THIS cluster>

  4. Review the kubezero-config and if all looks good commit the ArgoApp resouce for Kubezero via regular git
    git add / commit / push <cluster/env/kubezero/application.yaml>
    DO NOT yet re-enable ArgoCD before all pre v1.24 workers have been replaced !!!

  5. Reboot controller(s) one by one
    Wait each time for controller to join and all pods running. Might take a while ...

  6. Upgrade CFN stacks for the workers.
    This in turn will trigger automated worker updates by evicting pods and launching new workers in a rolling fashion. Grab a coffee and keep an eye on the cluster to be safe ... Depending on your cluster size it might take a while to roll over all workers!

  7. Re-enable ArgoCD by hitting on the still waiting upgrade script

  8. Quickly head over to ArgoCD and sync the KubeZero main module as soon as possible to reduce potential back and forth in case ArgoCD has legacy state

Known issues

existing EFS volumes

If pods are getting stuck in Pending during the worker upgrade, check the status of any EFS PVC. In case any PVC is in status Lost, edit the PVC and remove the following annotation: pv.kubernetes.io/bind-completed: "yes" This will instantly rebind the PVC to its PV and allow the pods to migrate.
Going to be fixed during the v1.25 cycle by a planned rework of the EFS storage module.