Troubleshooting GuideAKSKubernetesReal Solutionskubectl

Azure Kubernetes Service (AKS) Troubleshooting Guide:
Real Solutions to Common Problems

CrashLoopBackOff at 2am. Pods stuck Pending with no obvious cause. Nodes going NotReady mid-deployment. DNS resolution silently failing in production. Every AKS engineer encounters these — the difference between engineers who panic and engineers who stay calm is knowing the exact sequence of diagnostic commands to run. This guide gives you that sequence, the root cause analysis for each failure mode, and the fix.
3 commands
90% of AKS problems are diagnosed with the same three kubectl commands: describe pod, logs --previous, and get events — in that order, every time
Exit 137
The exit code that tells you everything: container killed by SIGKILL — either the Linux OOM killer (memory limit exceeded) or kubelet after grace period expired
5 min
The CrashLoopBackOff ceiling: Kubernetes applies exponential backoff (10s → 20s → 40s → 80s → 160s → 300s) capped at 5 minutes between restarts
9 problems
This guide covers 9 of the most common AKS failure modes — from pod crashes and image pull failures to node pressure, DNS, storage, networking, and ingress issues
Table of Contents
01The Golden Three: Diagnostic Commands for Every AKS Problem
02Problem 1 — CrashLoopBackOff: Pod Starts, Crashes, Repeats
03Problem 2 — ImagePullBackOff / ErrImagePull: Cannot Pull Container Image
04Problem 3 — OOMKilled: Container Killed by Memory Limit
05Problem 4 — Pending Pods: Stuck in Scheduling Limbo
06Problem 5 — Node NotReady: Node Lost Contact with Control Plane
07Problem 6 — DNS Resolution Failures Inside Pods
08Problem 7 — Persistent Volume / PVC Stuck in Pending
09Problem 8 — Service 503 / Ingress Not Routing Traffic
10Problem 9 — AKS Auto-Upgrade Not Running / Maintenance Window Skipped
11Prevention: Production Health Baseline Every AKS Cluster Should Have

The Golden Three: Diagnostic Commands for Every AKS Problem

Before diving into specific problems, burn these three commands into muscle memory. They diagnose 90% of AKS failures. Run them in this exact order every time something breaks — not because there is anything magic about the sequence, but because each command's output guides what you look for in the next one.
Figure 1 — Universal AKS diagnostic flow: the sequence that identifies root cause for any pod or node failure
These three commands, in this order, resolve 90% of AKS problems. The exit code in describe pod tells you the failure category. The --previous logs tell you the exact error message. Everything else is narrowing down the root cause within that category.
The essential AKS diagnostic sequence — bookmark this# STEP 1: Get the lay of the land
kubectl get pods -A -o wide # All namespaces, with node assignment
kubectl get nodes -o wide # Node status and versions
kubectl get events -A --sort-by='.lastTimestamp' | tail -30 # Recent events, newest last

# STEP 2: Describe the problem resource
kubectl describe pod <pod-name> -n <namespace> # Read Events section carefully
kubectl describe node <node-name> # If node-level issue suspected

# STEP 3: Read the crash logs
kubectl logs <pod-name> -n <namespace> --previous # ← last crashed container (KEY)
kubectl logs <pod-name> -n <namespace> --previous --tail=100 # Last 100 lines
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous # Multi-container pod

# BONUS: Resource consumption snapshot
kubectl top pods -n <namespace> # Actual CPU/memory usage
kubectl top nodes # Node-level resource pressure
The 9 Problem Runbooks
Problem 1CrashLoopBackOff: Pod Starts, Crashes, Repeats
Most CommonPod Layer
What you see:
kubectl get pods outputNAME READY STATUS RESTARTS AGE
myapp-7d8f6b9c-xk4mp 0/1 CrashLoopBackOff 8 14m
What CrashLoopBackOff actually means: The container image pulled successfully. The container started. Then it crashed with a non-zero exit code. Kubernetes restarted it. It crashed again. After enough consecutive failures, Kubernetes applies exponential backoff between restarts: 10s → 20s → 40s → 80s → 160s → 300s (5-minute cap). The pod is not "stuck" — it is actively being restarted on that schedule. The RESTARTS counter tells you how many cycles have completed.
Exit Code in describe Root Cause First Fix to Try
Exit 1 Application error — exception on startup, missing env var, cannot connect to DB/API at boot time Read kubectl logs --previous for the actual error message
Exit 0 Entrypoint command completed and exited. Container is not a daemon/service, it is a one-shot job. Fix the Dockerfile CMD or Deployment command to run a long-running process
Exit 137 OOMKilled — container exceeded memory limit and was killed by Linux kernel See Problem 3 (OOMKilled runbook)
Exit 143 Liveness probe timed out and kubelet sent SIGTERM to restart the container Increase initialDelaySeconds or add a startupProbe
Exit 127 Command not found — binary in CMD/entrypoint does not exist in the container image Verify the binary exists: kubectl exec into a running copy or check the image locally
1

Identify the exit code

kubectl describe pod <name> -n <ns> → scroll to "Containers" → "Last State: Terminated" → read the "Reason" and "Exit Code". This single line tells you which root cause category you are in.
2

Read the previous container's logs

kubectl logs <pod> --previous -n <ns>. Current container logs are often empty because the container crashes immediately on startup. The --previous flag retrieves logs from the last terminated container instance — the one that actually printed the error before dying.
3

Use sleep infinity to inspect a crashing container

If the container crashes before printing anything useful, temporarily override the command to keep it alive: patch the Deployment with command: ["sleep", "infinity"], then kubectl exec -it <pod> -- sh inside it. Run the actual entrypoint command by hand and read the real error output.
4

Check for missing ConfigMaps, Secrets, or environment variables

kubectl describe pod Events will show CreateContainerConfigError if a referenced ConfigMap or Secret does not exist. Check: kubectl get configmap -n <ns> and kubectl get secret -n <ns>. Secret and ConfigMap names are namespace-scoped — a missing namespace prefix is the most common cause.
5

Fix liveness probe timing if exit code is 143

A liveness probe that fires before the application has finished starting will kill the container on every cycle. Add a startupProbe that blocks liveness checks until the application is ready, or increase initialDelaySeconds on the liveness probe to give the app time to boot.
YAML — startupProbe to prevent premature liveness killscontainers:
- name: myapp
  startupProbe: # Blocks liveness until app is ready
    httpGet:
      path: /health
      port: 8080
    failureThreshold: 30 # Allow up to 3010s = 5 minutes to start
    periodSeconds: 10
  livenessProbe: # Only runs AFTER startupProbe succeeds
    httpGet:
      path: /health
      port: 8080
    periodSeconds: 15
    failureThreshold: 3
  readinessProbe: # Controls traffic, does NOT restart
    httpGet:
      path: /ready
      port: 8080
    periodSeconds: 5
Problem 2ImagePullBackOff / ErrImagePull: Cannot Pull Container Image
AKS+ACR SpecificPre-start Failure
What you see:
kubectl get pods and describe eventsmyapp-xyz   0/1   ImagePullBackOff   0   3m

Events:
  Warning  Failed  kubelet  Failed to pull image "acrprod.azurecr.io/myapp:v2.0":
    rpc error: code = Unknown desc = failed to pull and unpack image:
    unauthorized: authentication required
Key distinction: ImagePullBackOff happens before* the container starts — the image cannot be pulled. CrashLoopBackOff happens after the image pulls successfully. Confusing the two wastes significant debugging time. ErrImagePull is the first failure; ImagePullBackOff is the same error after Kubernetes begins applying exponential backoff between retry attempts.
1

Check AKS ↔ ACR authorisation (most common AKS-specific cause)

AKS clusters that pull from Azure Container Registry use the kubelet's managed identity with the acrpull role. Run: az aks check-acr --resource-group <rg> --name <aks> --acr <acr-name>.azurecr.io. If it reports authorization failure: az aks update --resource-group <rg> --name <aks> --attach-acr <acr-name> — this assigns the acrpull role to the AKS kubelet managed identity.
2

Verify the image name and tag exist

az acr repository show-tags --name <acr-name> --repository myapp -o table. A typo in the tag name (including case sensitivity — Docker tags are case-sensitive) returns a 404 that manifests as ImagePullBackOff. Confirm the exact tag matches what is in the Deployment spec.
3

Check imagePullSecrets for private registries (non-ACR)

For third-party registries: kubectl create secret docker-registry regcred --docker-server=registry.example.com --docker-username=user --docker-password=pass -n <ns>. Reference the secret in the pod spec: imagePullSecrets: [{name: regcred}]. Verify the secret exists in the same namespace as the pod — imagePullSecrets are namespace-scoped.
4

Check node network access to the registry

On AKS with private endpoints or firewall rules: verify the AKS node subnet has outbound access to <acr-name>.azurecr.io. Private ACR requires a private endpoint on the AKS VNet and correct DNS resolution — test from a debug pod: kubectl run -it debug --image=alpine --restart=Never -- nslookup <acr-name>.azurecr.io.
Azure CLI — Check and fix AKS-to-ACR authorization# Diagnose: check if AKS can pull from ACR
az aks check-acr \
  --resource-group rg-prod \
  --name aks-prod \
  --acr acrprod001.azurecr.io

# Fix: attach ACR to AKS (assigns acrpull role to kubelet MI)
az aks update \
  --resource-group rg-prod \
  --name aks-prod \
  --attach-acr acrprod001

# Verify image exists in ACR
az acr repository show-tags \
  --name acrprod001 \
  --repository myapp -o table

# Debug pull access from inside the cluster
kubectl run pull-test --image=acrprod001.azurecr.io/myapp:v2.0 \
  --restart=Never -n default -- echo "pulled successfully"
kubectl describe pod pull-test -n default
Figure 2 — Pod status decision tree: navigate from symptom to root cause in under 60 seconds
Start with the STATUS column in kubectl get pods. Each status maps to a specific investigation path. Never skip directly to restarting pods — the status tells you which of the 9 problem runbooks in this guide to follow.
Problem 3OOMKilled: Container Killed by Memory Limit
Exit Code 137Resource Layer
What you see:
kubectl describe pod — Last State sectionLast State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Thu, 10 Jul 2026 14:32:10 +0000
  Finished:     Thu, 10 Jul 2026 14:34:51 +0000
Root cause: The Linux kernel's OOM killer terminated the container because it exceeded the memory limit set in the container spec. OOMKilled is a deterministic signal — the container used more memory than allowed, so the kernel killed it. The fix is either to increase the limit (if the application legitimately needs more memory) or to fix a memory leak (if the application is consuming memory it should not be).
1

Confirm OOMKilled and read the current limit

kubectl describe pod <name> → "Last State: Terminated Reason: OOMKilled". Then: kubectl get pod <name> -o yaml | grep -A 5 resources to see the current memory limit. Also run kubectl top pod <name> to see actual peak consumption (requires metrics-server, which AKS installs by default).
2

Immediate fix: increase the memory limit

Edit the Deployment: kubectl edit deployment <name> -n <ns>. Increase limits.memory. Start with 2× the current limit as a diagnostic baseline. The application will restart with the new limit and you can observe whether memory usage stabilises or continues growing (memory leak indicator).
3

Distinguish memory limit from node memory pressure

kubectl describe node <node-name> → check "Conditions" for MemoryPressure: True. If the node has memory pressure, pods are being evicted by kubelet regardless of their individual limits — the entire node is out of memory. Fix: scale the node pool or move memory-heavy pods to a dedicated node pool.
4

Use kubectl top and Vertical Pod Autoscaler for right-sizing

Run kubectl top pods -n <ns> during normal and peak load. The "MEMORY" column shows actual usage — set your limit to 120–150% of peak observed usage to leave headroom for spikes. For production clusters, deploy VPA in recommendation mode to get automatic right-sizing suggestions without automatic pod eviction.
YAML — Correctly set resource requests and limitsresources:
  requests:
    memory: "256Mi" # Minimum guaranteed allocation (affects scheduling)
    cpu: "250m" # 0.25 CPU cores minimum
  limits:
    memory: "512Mi" # Maximum before OOMKill — set to 1.5-2x observed peak
    cpu: "1000m" # CPU is throttled (not killed) when exceeded

# Diagnose: observe actual usage over time
kubectl top pods -n <ns> --sort-by=memory
# Watch mode (updates every 2s)
watch -n 2 kubectl top pods -n <ns>
Problem 4Pending Pods: Stuck in Scheduling Limbo
Scheduler LayerMultiple Causes
What you see:
kubectl get pods and describe eventsmyapp-xyz   0/1   Pending   0   15m

Events:
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available:
    3 Insufficient memory. preemption: 0/3 nodes are available:
    3 No preemption victims found for incoming pod.
Critical diagnosis point: Pending means the Kubernetes scheduler cannot find a node that satisfies all the pod's requirements. The scheduler's reason is always in the Events section of kubectl describe pod. Read that message carefully — it tells you exactly which constraint is failing: insufficient memory, insufficient CPU, taint/toleration mismatch, node affinity rule not met, or PVC not bound.
Event Message Root Cause Fix
Insufficient memory No node has enough free memory for the pod's requests.memory Scale node pool: az aks scale --node-count N or reduce memory request
Insufficient CPU No node has enough free CPU for requests.cpu Scale node pool or reduce CPU request
node(s) had untolerated taint Pod needs a toleration for a node taint (e.g., dedicated GPU nodes) Add tolerations to pod spec or remove node taint
node(s) didn't match node affinity nodeSelector or nodeAffinity rules don't match any node's labels Fix nodeSelector labels or add label to node: kubectl label node
PersistentVolumeClaim is not bound A PVC referenced by the pod has not been provisioned yet See Problem 7 (PVC runbook)
pod has unbound immediate PersistentVolumeClaims PVC using WaitForFirstConsumer binding — pod must schedule first Check StorageClass and PVC status
kubectl — Diagnose why a pod is Pending# Read the scheduler failure reason
kubectl describe pod <pending-pod> -n <ns> | grep -A 10 "Events:"

# Check total available resources across all nodes
kubectl describe nodes | grep -A 5 "Allocated resources"
kubectl top nodes # Real-time CPU and memory

# Check node taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Scale AKS node pool to add capacity
az aks scale -g rg-prod -n aks-prod \
  --node-count 5 --nodepool-name nodepool1

# Add a toleration to a pod spec for a tainted node
# tolerations:
# - key: "sku"
# operator: "Equal"
# value: "gpu"
# effect: "NoSchedule"
Problem 5Node NotReady: Node Lost Contact with Control Plane
Node LayerHigh Impact
What you see:
kubectl get nodesNAME                     STATUS     ROLES   AGE   VERSION
aks-nodepool1-27392819-0   NotReady   agent   3d    v1.32.0
Root cause: The kubelet on the node has stopped sending heartbeats to the AKS control plane. The node's status transitions to NotReady after the heartbeat timeout (default 40 seconds). After 5 minutes in NotReady, Kubernetes moves pods to Unknown status and begins rescheduling them to healthy nodes. Common AKS-specific causes: the underlying Azure VM is experiencing hardware issues, the node has run out of disk space, or the kubelet process crashed due to memory pressure.
1

Read the node conditions

kubectl describe node <node-name> → "Conditions" section. Look for: MemoryPressure: True (node out of memory), DiskPressure: True (node out of disk — most common cause in AKS), PIDPressure: True (too many processes), Ready: False with reason KubeletNotReady.
2

Drain the node before attempting any repair

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data. This safely moves all pods off the node and prevents new pods from being scheduled to it. Never attempt to repair a node while workloads are running on it — the repair may cause sudden pod termination.
3

Check the Azure VM status in the portal

In Azure Portal → the AKS resource → Node pools → the specific node pool → Nodes. If the underlying VM shows "Unhealthy" or "Failed", the Azure infrastructure is the issue. Use az aks nodepool upgrade --node-image-only to reimage the node, or delete the node (AKS will automatically provision a replacement).
4

Delete the NotReady node and let AKS replace it

kubectl delete node <node-name>. AKS automatically provisions a new node in the node pool to replace the deleted one. This is almost always faster than attempting to repair a broken node in production. The new node joins the pool within 3–5 minutes.
kubectl — Diagnose and remediate a NotReady node# Identify NotReady nodes
kubectl get nodes | grep NotReady

# Read conditions — what is wrong with the node
kubectl describe node aks-nodepool1-xxx | grep -A 20 "Conditions:"

# Check node resource allocation
kubectl describe node aks-nodepool1-xxx | grep -A 10 "Allocated resources:"

# Drain: safely evict all workloads from the node
kubectl drain aks-nodepool1-xxx \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=120

# Delete the node — AKS auto-provisions a replacement
kubectl delete node aks-nodepool1-xxx

# Alternative: reimage the node pool (all nodes)
az aks nodepool upgrade \
  --resource-group rg-prod \
  --cluster-name aks-prod \
  --name nodepool1 \
  --node-image-only
Problem 6DNS Resolution Failures Inside Pods
Silent FailureNetworking
What you see: Your pods are Running. The application logs show connection refused, timeout, or "Name or service not known" errors when trying to reach other services within the cluster or external endpoints. DNS failures are the most insidious AKS problem because the pod appears healthy while silently failing to communicate.
1

Test DNS resolution from a debug pod

Deploy a debug pod into the affected namespace and run DNS lookups: kubectl run dns-debug --image=nicolaka/netshoot -it --rm -n <namespace> -- bash. Inside: nslookup kubernetes.default (tests cluster DNS), nslookup <service-name>.<namespace>.svc.cluster.local (tests service discovery).
2

Check CoreDNS pod status

kubectl get pods -n kube-system | grep coredns. CoreDNS pods in CrashLoopBackOff or fewer than expected replicas will cause cluster-wide DNS failures. Check: kubectl logs -n kube-system -l k8s-app=kube-dns --previous. Restart CoreDNS: kubectl rollout restart deployment coredns -n kube-system.
3

Verify the service name format for cross-namespace DNS

In Kubernetes, a service in a different namespace is accessed as <service-name>.<namespace>.svc.cluster.local. The shortform <service-name> only works within the same namespace. Verify the full DNS name resolves: nslookup <service>.<ns>.svc.cluster.local in the debug pod.
4

Check node DNS configuration for external resolution

If internal DNS works but external DNS (e.g., api.github.com) fails, the node-level DNS configuration or NSG/firewall may be blocking UDP 53 outbound. Check: cat /etc/resolv.conf inside the debug pod. For AKS private clusters, the VNet must have DNS resolution configured to reach upstream resolvers.
kubectl — Full DNS diagnostic sequence# Deploy a network debugging pod
kubectl run dns-debug --image=nicolaka/netshoot \
-it --rm -n myapp -- bash

# Inside the debug pod:
nslookup kubernetes.default # Test cluster DNS works
nslookup myservice.myapp.svc.cluster.local # Test service discovery
nslookup api.github.com # Test external DNS
dig @10.0.0.10 myservice.myapp.svc.cluster.local # Query CoreDNS directly (10.0.0.10 = default CoreDNS ClusterIP)

# On the cluster: check CoreDNS health
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Restart CoreDNS if it's unhealthy
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout status deployment coredns -n kube-system
Problem 7Persistent Volume / PVC Stuck in Pending
Storage LayerAKS Azure Disks
What you see: A pod is Pending and kubectl describe pod shows "persistentvolumeclaim <name> not found" or "pod has unbound immediate PersistentVolumeClaims". The PVC itself shows STATUS: Pending.
1

Check PVC status and events

kubectl get pvc -n <ns> → note STATUS (Pending/Bound/Lost). kubectl describe pvc <pvc-name> -n <ns> → Events section. The most common AKS cause: the StorageClass is set to WaitForFirstConsumer binding mode — the PVC won't bind until a pod is scheduled that claims it, creating a chicken-and-egg problem with topology constraints.
2

Verify the StorageClass exists and is correct

kubectl get storageclass. AKS provides default storage classes: default (Azure managed disk, LRS), managed-csi (CSI driver), and azurefile. If the PVC references a StorageClass that does not exist, it stays Pending indefinitely with no useful error message.
3

Check if the PVC is in a different AZ than the node

Azure managed disks are zone-specific. If a PVC is provisioned in zone 1 and the pod is scheduled to a node in zone 2, the disk cannot attach. Use the WaitForFirstConsumer binding mode on the StorageClass — this delays disk provisioning until the pod is scheduled, ensuring the disk is created in the same zone as the pod.
4

Check the CSI driver pods

kubectl get pods -n kube-system | grep csi. The Azure Disk CSI driver pods (csi-azuredisk-node-*) must be Running on the nodes where the PVC needs to be attached. If CSI driver pods are crashing, disk attachment will fail silently.
kubectl + YAML — PVC diagnostic and StorageClass with WaitForFirstConsumer# Diagnose PVC state
kubectl get pvc -n myapp -o wide
kubectl describe pvc my-data-pvc -n myapp
kubectl get storageclass # List available storage classes

# Check CSI drivers are healthy
kubectl get pods -n kube-system | grep -E "csi|disk|file"

# YAML: StorageClass with WaitForFirstConsumer (zone-aware)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-csi-zrs
provisioner: disk.csi.azure.com
parameters:
skuName: Premium_LRS
volumeBindingMode: WaitForFirstConsumer # ← Zone-aware disk creation
reclaimPolicy: Retain # Keep disk if PVC is deleted
allowVolumeExpansion: true
Problem 8Service 503 / Ingress Not Routing Traffic
NetworkingIngress / Service
What you see: Pods are Running and appear healthy. External requests return HTTP 503 Service Unavailable or the connection is refused. The Ingress controller is reachable but traffic is not being routed to the application pods.
1

Verify Service selector matches pod labels

The most common cause: the Service selector does not match the pod labels. kubectl get svc <svc-name> -n <ns> -o yaml | grep selector then kubectl get pod -n <ns> --show-labels. Every key-value pair in the Service selector must exactly match a pod's labels.
2

Check that Endpoints are populated

kubectl get endpoints <svc-name> -n <ns>. If ENDPOINTS shows <none>, the Service has no healthy pods to route to. This means either the selector doesn't match any pod, or all matched pods are failing their readiness probe (not Ready).
3

Verify pod readiness probes are passing

kubectl describe pod <name> -n <ns> → "Conditions" → check Ready: True and ContainersReady: True. A pod that is Running but not Ready (readiness probe failing) is excluded from Service endpoints — traffic is not sent to it. Fix the readiness probe path or increase failureThreshold.
4

Check Ingress backend service name and port

kubectl describe ingress <ingress-name> -n <ns>. Verify the backend service name and port match an actual Service in the same namespace. The Ingress controller logs show routing errors: kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=100.
5

Test internal connectivity from a debug pod

Isolate whether the problem is at the Ingress or Service layer: kubectl run test --image=curlimages/curl -it --rm -- curl http://<svc-name>.<ns>.svc.cluster.local:<port>/health. If this returns 200, the Service and pods are healthy and the problem is in the Ingress configuration. If it fails, the Service → pod connectivity is broken.
kubectl — Full Service and Ingress diagnostic sequence# Check Service selector and Endpoints
kubectl get svc myapp-svc -n myapp -o yaml | grep -A 5 selector
kubectl get endpoints myapp-svc -n myapp # Should show pod IPs, not <none>

# Check pod labels match the service selector
kubectl get pods -n myapp --show-labels

# Check pod readiness
kubectl get pods -n myapp -o wide # READY column: 1/1 means ready
kubectl describe pod <pod-name> -n myapp | grep -A 5 "Conditions:"

# Describe the Ingress resource
kubectl describe ingress myapp-ingress -n myapp

# Test internal routing from a debug pod
kubectl run curl-test --image=curlimages/curl -it --rm -n myapp \
  -- curl -v http://myapp-svc.myapp.svc.cluster.local:8080/health

# Read Ingress controller logs
kubectl logs -n ingress-nginx \
  -l app.kubernetes.io/name=ingress-nginx \
  --tail=100 | grep -E "error|warning|upstream"
Problem 9AKS Auto-Upgrade Not Running / Maintenance Window Skipped
AKS-SpecificControl Plane
What you see: The AKS cluster has an auto-upgrade channel configured (patch, stable, or rapid) with a scheduled maintenance window, but upgrades are not executing. The Kubernetes version is falling behind and security patches are not being applied.
Root cause patterns: The most common cause is a startDate in the maintenance configuration set to a future date — the window is defined but not yet active. Other causes: the maintenance window duration is under 4 hours (AKS minimum), the cluster has PodDisruptionBudgets that block node draining, or the autoUpgradeProfile.upgradeChannel is not set despite maintenance window configuration.
1

Check the maintenance configuration for startDate issues

az aks maintenanceconfiguration show -g <rg> --cluster-name <aks> -n default -o yaml. Examine the startDate field. If it is set to a future date, the maintenance window has not yet become active and no upgrades will run until that date arrives. Fix: remove the startDate or set it to today.
2

Verify the auto-upgrade channel is set

az aks show -g <rg> -n <aks> --query "autoUpgradeProfile.upgradeChannel" -o tsv. If the result is null or none, the upgrade channel is not configured — the maintenance window has no upgrades to apply. Set the channel: az aks update -g <rg> -n <aks> --auto-upgrade-channel patch.
3

Verify the maintenance window duration is at least 4 hours

AKS requires a minimum 4-hour maintenance window duration. Windows shorter than 4 hours are silently skipped. Re-configure: az aks maintenanceconfiguration add -g <rg> --cluster-name <aks> --name default --weekday Sunday --start-hour 2 --duration 6. The --duration parameter sets the window length in hours.
4

Check for PodDisruptionBudgets blocking node drain

kubectl get pdb -A. A PDB with maxUnavailable: 0 or minAvailable equal to the total replica count prevents AKS from draining nodes during the upgrade. The upgrade will time out and be marked as failed. Temporarily relax PDB constraints during scheduled maintenance windows.
Azure CLI — Diagnose and fix AKS auto-upgrade configuration# Check current upgrade channel
az aks show -g rg-prod -n aks-prod \
  --query "autoUpgradeProfile.upgradeChannel" -o tsv

# Check maintenance configuration
az aks maintenanceconfiguration list -g rg-prod \
  --cluster-name aks-prod -o table
az aks maintenanceconfiguration show -g rg-prod \
  --cluster-name aks-prod -n default -o yaml

# Set auto-upgrade channel
az aks update -g rg-prod -n aks-prod \
  --auto-upgrade-channel patch # patch | stable | rapid | node-image

# Add maintenance window: every Sunday at 2am, 6 hours, starting immediately
az aks maintenanceconfiguration add \
  -g rg-prod --cluster-name aks-prod \
  --name default \
  --weekday Sunday \
  --start-hour 2 \
  --duration 6
# DO NOT set startDate unless you want a future activation

# Check available upgrades
az aks get-upgrades -g rg-prod -n aks-prod -o table

# Check for blocking PodDisruptionBudgets
kubectl get pdb -A -o wide

Prevention: Production Health Baseline Every AKS Cluster Should Have

These are the controls that catch problems before they become incidents. Configure them at cluster creation, not after the first outage.
kubectl — Quick cluster health audit (run this first on any cluster you inherit)# Overall cluster health snapshot
kubectl get nodes -o wide
kubectl top nodes
kubectl get pods -A | grep -v Running | grep -v Completed # Non-healthy pods
kubectl get events -A --field-selector type=Warning --sort-by='.lastTimestamp' | tail -20

# Resource requests/limits coverage (pods without limits are high risk)
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[].resources.limits == null) | .metadata.name'

# Pods with high restart counts
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' | tail -10

# PVCs not bound
kubectl get pvc -A | grep -v Bound

# AKS cluster version vs available upgrades
kubectl version --short
az aks get-upgrades -g rg-prod -n aks-prod -o table
Production Baseline Checklist
Resource requests and limits on every container. Pods without limits are OOMKill candidates and disrespect node capacity. Without requests, the scheduler cannot make informed placement decisions.
Readiness probes on every web-facing container. Without a readiness probe, a pod is added to Service endpoints immediately on start — before the application is ready to accept traffic — causing 503 errors during rolling deployments.
PodDisruptionBudgets with maxUnavailable ≥ 1. Allows AKS to drain nodes during upgrades and maintenance without blocking. A PDB with maxUnavailable: 0 will block all node operations.
Azure Monitor container insights enabled. az aks enable-addons -a monitoring -g <rg> -n <aks>. Provides memory/CPU metrics, OOMKill event alerts, and container log forwarding to Log Analytics without any application code changes.

Quick Reference: Match Your Symptom to the Fix

→CrashLoopBackOff → kubectl describe pod for exit code + kubectl logs --previous for error message. Exit 1 = app error. Exit 137 = OOMKilled. Exit 143 = liveness probe killed it.
→ImagePullBackOff on AKS+ACR → run az aks check-acr. If unauthorized: az aks update --attach-acr. For private ACR, also verify DNS resolution and private endpoint connectivity from the node subnet.
→OOMKilled (Exit 137) → increase limits.memory to 1.5-2× observed peak from kubectl top pods. Check kubectl describe node for node-level MemoryPressure — that requires adding nodes, not just raising limits.
→Pending pod → always read Events in kubectl describe pod first. Insufficient memory/CPU = scale the node pool. Taint issue = add tolerations. PVC issue = check StorageClass and CSI driver health.
→Node NotReady → drain first (kubectl drain --ignore-daemonsets), then delete the node (kubectl delete node). AKS auto-provisions a replacement within 5 minutes. Do not attempt to repair a broken production node in-place.
→503 / no traffic → check kubectl get endpoints. If <none>, the Service selector doesn't match pod labels or all pods are failing their readiness probe. Test internal connectivity with a debug pod before touching the Ingress.
→Auto-upgrade not running → check az aks maintenanceconfiguration show for a future startDate. Verify autoUpgradeProfile.upgradeChannel is not null. Ensure the maintenance window is at least 4 hours. Check for blocking PodDisruptionBudgets.
→The universal first step for any AKS problem: kubectl describe pod <name> then kubectl logs --previous <name>. These two commands, in this order, resolve 90% of AKS failures before reaching for any other diagnostic tool.

Frequently Asked Questions

A pod shows CrashLoopBackOff but kubectl logs returns nothing. How do I get the error?
Use kubectl logs --previous <pod-name> — note the --previous flag. Without it, you are reading logs from the current (new) container instance, which may have just started and hasn't printed anything before crashing. The --previous flag reads logs from the last terminated container instance — the one that actually crashed and printed the error. If even --previous returns nothing, the container crashes too quickly to emit any output. In this case, temporarily patch the Deployment with command: ["sleep", "infinity"] to prevent the crash, then kubectl exec -it <pod> -- sh inside it and run your entrypoint command manually.
How do I safely restart all pods in a Deployment without downtime?
kubectl rollout restart deployment <name> -n <ns>. This performs a rolling restart — it gradually replaces pods one by one according to your Deployment's strategy.rollingUpdate settings. Monitor the rollout: kubectl rollout status deployment <name> -n <ns>. If the new pods are unhealthy, roll back: kubectl rollout undo deployment <name> -n <ns>. Never use kubectl delete pods in production — it creates an immediate outage for the number of pods you delete simultaneously.
Pods are randomly restarting every few hours but logs show nothing wrong. What is causing this?
Random restarts with no obvious log error typically have two causes. First, check if the liveness probe is timing out: kubectl describe pod → "Last State: Terminated Reason: Error Exit Code: 143" indicates a SIGTERM from the liveness probe. The application is taking longer than timeoutSeconds to respond to the probe. Fix: increase timeoutSeconds or add a startupProbe. Second, check for intermittent OOMKills: kubectl describe pod → "Last State: Terminated Reason: OOMKilled" with Exit Code 137. The application has periodic memory spikes (common in JVM applications during GC or in ML model inference). Use kubectl top pods under load to profile actual peak memory consumption.
How do I run a one-off debug command in a production cluster without installing anything permanently?
kubectl run debug-pod --image=nicolaka/netshoot -it --rm -n <namespace> -- bash. The --rm flag automatically deletes the pod when you exit the shell. The nicolaka/netshoot image contains nslookup, curl, dig, nmap, tcpdump, iperf, and dozens of other network diagnostic tools. For a minimal debugging session: kubectl run debug --image=alpine -it --rm -- sh. For running a command inside an existing running container (no new pod needed): kubectl exec -it <existing-pod> -n <ns> -- sh.

Related FAVRITE Articles

How to Fix AKS Cluster Auto-Upgrade Not Executing During Scheduled Maintenance Window
Top 100 Azure CLI Commands Every Cloud Engineer Should Know
Stop Using Connection Strings: A Step-by-Step Guide to Azure Managed Identities in 2026
Troubleshooting Azure DevOps Pipeline Timeouts: 5 Real-World Fixes That Work