Update EKS cluster

Explains how to update the EKS cluster

For general repository structure, authentication, and execution instructions, see our terraform docs.

Update terraform variables

EKS update

The update process must be always incremental. Update only one kubernetes minor version at a time.

To update the EKS cluster, go to igaawi-infra-aws.

Open the environment-specific file envs/<env>/aws/eks/<env>.tfvars and update the following variables to the next minor version:

kubernetes_version = "1.36"
kubectl_version    = "1.36.0"

For example, do not update directly from v1.33 to v1.36 in one jump.

First update to v1.34, complete the full update process, make sure everything is stable, and only then repeat the same steps and update to v1.35, and so on.

Example EKS update PR: https://github.com/saritasa-nest/igaawi-infra-aws/pull/635/changes

EKS nodegroup version bump

Version bumping

Bumping EKS nodegroup versions during an EKS cluster update is considered deprecated and should be avoided unless required.

Nodegroup version bump is not needed

There is no need to bump the nodegroup version if you are only updating the EKS cluster version.

In this case, the terraform plan will show that the nodegroup will be updated in-place. Terraform will handle the process of creating the new nodes and removing the old ones automatically:

In this case, follow the regular update steps from these docs. The update should be completed successfully with a single terraform apply.

Nodegroup version bump is needed

If you also update nodegroup parameters that change the launch template (i.e. instance_types, ami_type, capacity_type), and similar settings, the terraform plan will show that the nodegroup must be replaced:

In this case, you cannot perform a single terraform apply, because terraform will fail with the ResourceInUseException: NodeGroup already exists with name <…> error:

To work around this error, you need to bump the version of the affected nodegroup.

Create a copy of the affected nodegroup in <env>.tfvars inside the eks_managed_node_groups map, next to the existing nodegroup.

For example, if the affected nodegroup is ops-v1, copy and paste the entire ops-v1 configuration and rename the copy to ops-v2.

Then run a targeted terraform apply to create the new nodegroup:

terragrunt apply \
  -var-file=prod.tfvars \
  -target='module.eks.module.eks_managed_node_group["ops-v2"]'

The new ops-v2 nodegroup will use the updated EKS cluster version and the updated nodegroup parameters.

After the targeted apply completes, monitor the cluster and confirm that the new nodes were created successfully.

Once the new nodegroup is healthy, remove ops-v1 from the eks_managed_node_groups map and run a full terraform apply.

If multiple nodegroups are affected, bump all of them and create them in the same targeted apply.

Notify the team

Before starting the update, notify the team in the project-igaawi Slack channel.

Example messages

Notify the team ahead of time so they know when the update is planned:

Hi, team!
I am planning to update the `igaawi-prod-eks` cluster on MM/DD at 10AM UTC+7.
The update should take around 2 hours. I do not expect any downtime.

Also send a reminder on the planned date before starting the update:

Hi, team!
In one hour, I will start updating the `igaawi-prod-eks` cluster.
I will post updates in this channel.

Prepare the cluster

Before fully running the EKS update, prepare the cluster to make the update faster and reduce possible downtime.

When EKS nodegroups are updated, new nodes are created with the updated kubernetes version (i.e. v1.36).

After that, old nodes (i.e. v1.35) are cordoned and drained.

This means that pods from the old nodes must be evicted and rescheduled to the new nodes.

The update will get stuck if some pods cannot be evicted.

The most common reason for this are PodDisruptionBudget resources having a minAvailable: 1 option, which often results in AllowedDisruptions: 0.

In this case kubernetes won’t allow the pod to be evicted. Terraform will keep waiting for the EKS nodegroup update to finish, but eventually it will fail with a timeout.

EKS nodegroup update timeout

Terraform waits around 40 minutes before failing the nodegroup update, and exiting with an error.

Disable ArgoCD auto-sync

First, you must start by disabling auto-sync in all root apps. Otherwise, ArgoCD will recreate the deleted PDBs during the update.

To disable auto-sync only in root apps you can use this helper (or do it in ArgoCD GUI):

for app in addons apps apps-dev apps-staging; do
  kubectl patch applications.argoproj.io "$app" -n argo-cd --type=merge \
    -p='{"spec":{"syncPolicy":null}}'
done

Then, you should disable auto-sync for child apps that can recreate PDBs during the update.

You can update the helper used for root apps to include child ArgoCD apps (i.e. teleport karpenter coredns etc) that you need to disable auto-sync for.

Or you can use this helper to disable auto-sync for all apps:

kubectl get applications.argoproj.io -n argo-cd -o name | while read app; do
  kubectl patch "$app" -n argo-cd --type=merge \
    -p='{"spec":{"syncPolicy":null}}'
done

You can check that the applications were updated with the:

kubectl get applications.argoproj.io -n argo-cd

Delete PDBs

After auto-sync is disabled, delete all PDBs in all namespaces with:

kubectl delete pdb --all -A

To monitor if any PDBs are being recreated, run this command in another shell:

kubectl get pdb -w -A

Disable auto-sync

If you see that a PDB was recreated, check which ArgoCD application manages it and make sure again that auto-sync/self-heal is disabled for that application.

Some PDBs can keep getting recreated even after they were removed from an unsynced ArgoCD application.

This can happen when they are not managed by gitops and are created by a kubernetes controller instead.

For convenience, you can run this command during the update in another shell, to constantly remove PDBs:

while true; do kubectl delete pdb -A --all; sleep 10; done

Don’t forget to stop this command after the EKS update is done.

Apply terraform code

At the agreed time, run:

cd envs/<env>/aws/eks
terragrunt apply

Terraform will start by updating the EKS control plane first: module.eks.aws_eks_cluster.this[0]. This usually takes around 10 minutes.

After that, terraform will update EKS managed nodegroups: module.eks.module.eks_managed_node_group (i.e. ops-v3, apps-v5 etc).

This usually takes around 20 minutes, but it can take longer if pod eviction is being blocked.

So it’s wise to keep an eye on nodes and PDBs during the update.

Watch nodes in another shell:

kubectl get nodes -o wide -L workload,karpenter.sh/nodepool -w

Watch PDBs in another shell:

kubectl get pdb -A -w

After the update

After terraform has successfully finished applying the changes and the EKS update is done, there are some steps to take to ensure that the update was successful.

Check kubernetes cluster

First thing to check after the update is the cluster itself.

You can go to the AWS EKS GUI and check cluster info.

The status should be Active, and Kubernetes version must show the one you were updating into:

Then check that all nodes are in the Ready status, and use the updated version (i.e. v1.35.3-eks):

Check pods health

Pending pods

Check that there are no pods in Pending state:

kubectl get pods -A | grep Pending

CrashLoopBackOff or Error pods

Check that there are no pods in CrashLoopBackOff or Error state:

kubectl get pods -A | grep -E 'CrashLoopBackOff|Error'

EBS CSI controller logs

Check logs of the EBS CSI controller for errors:

kubectl logs -n kube-system -l app=ebs-csi-controller --all-containers --tail=200 | grep -iE 'error|failed'

EFS CSI controller logs

Check logs of the EFS CSI controller for errors:

kubectl logs -n kube-system -l app=efs-csi-controller --all-containers --tail=200 | grep -iE 'error|failed'

If any of these commands return errors, you should investigate them before considering the upgrade complete.

Check and sync ArgoCD apps

After the update is complete, check that there are no apps in Unknown state, and that no app sync statuses are in Error state.

Then you can restore ArgoCD auto-sync only for the applications where it was enabled before the upgrade.

This can be done either through the ArgoCD GUI or by using this helper:

kubectl get applications.argoproj.io -n argo-cd -o json \
  | jq -r '.items[]
  | select(.metadata.name as $n | ["addons","apps","apps-dev","apps-staging"] | index($n) | not)
  | "applications.argoproj.io/\(.metadata.name)"' \
  | xargs -I{} kubectl patch {} -n argo-cd --type=merge \
  -p='{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'

This will sync back all apps, except for the root apps, and in result recreate all of the previously removed PDBs.

Don't sync root apps

Do not enable auto-sync for root apps if it was disabled before. Enabling sync for a root app can overwrite the current branch of child apps and move them back to the root app's branch.

For example:

addons app points to main branch.
prometheus app points to feature/xxx branch.

If you enable sync for addons, it can sync prometheus back to main. As a result, changes from the currently deployed child app branch can disappear.

Check alerts

Check that there are no Firing or Pending alerts in Grafana, and no alarms in project-igaawi-alarms Slack channel.