AKS CrashLoopBackOff, Pending Pods, and
NotReady Nodes — The Real Fixes Engineers Use
Every AKS engineer eventually faces the same nightmare: CrashLoopBackOff at 2am, pods stuck Pending for no clear reason, or nodes flipping to NotReady mid-deployment. The difference between panic and control is knowing the exact diagnostic sequence — and the real fixes that work in production. This guide gives you both.
The engineers who stay calm during an AKS incident are not smarter than everyone else — they just run the same diagnostic sequence every time, in the same order, and let the cluster tell them what is wrong. Kubernetes is remarkably honest about its own failures: almost every problem announces itself in a pod's status, its events, or its logs. The skill is knowing where to look and what each signal means. This guide walks the failures you will actually hit in production — CrashLoopBackOff, Pending, NotReady, DNS, OOM, and ImagePullBackOff — and gives the exact commands and fixes for each.
Before learning any specific fix, internalise the diagnostic sequence. Nearly every AKS incident is solved by three commands run in order. Skipping ahead — guessing at the fix before reading the events — is what turns a ten-minute incident into an hour.
1. kubectl get pods — find the target
Start wide. This shows you which pods are unhealthy, their exact status (CrashLoopBackOff, Pending, ImagePullBackOff, OOMKilled), how many times they have restarted, and — with -o wide — which node they landed on. The restart count and status column alone often tell you which failure class you are dealing with.
2. kubectl describe pod — read the events
This is the most important command in Kubernetes troubleshooting, and the Events section at the bottom is where the real cause lives. FailedScheduling, Failed to pull image, Back-off restarting failed container, Liveness probe failed, OOMKilled — they all surface here in plain language, usually with the exact reason attached.
3. kubectl logs — hear the app's last words
When a container is crashing, its own logs tell you why — a missing environment variable, a failed database connection, an unhandled exception on startup. The critical flag is --previous (or -p): for a container that already crashed and restarted, this shows the logs from the failed instance, not the fresh one that is about to fail again.
For a pod in CrashLoopBackOff, plain kubectl logs shows the container that just started and hasn't crashed yet — often empty or misleading. kubectl logs --previous shows the instance that actually failed. This one flag is the difference between seeing the real error and staring at a blank log.
CrashLoopBackOff is not an error in itself — it is Kubernetes telling you that a container keeps starting, crashing, and being restarted, and that the restarts are now spaced out with an exponential back-off (10s, 20s, 40s, up to 5 minutes). The status is a symptom. The cause is always why the container exits, and that is found in the logs and the container's exit code.
| Exit Code | Meaning | Typical Cause |
|---|---|---|
| 0 | Clean exit — but container wasn't meant to stop | Wrong entrypoint / command; the main process finished and exited instead of staying up |
| 1 | General application error | Unhandled exception on startup, bad config, missing env var or file |
| 126 / 127 | Command not runnable / not found | Bad command in the manifest, missing binary in the image, wrong path |
| 137 | SIGKILL (128 + 9) — OOMKilled or forced kill | Container exceeded its memory limit, or a failed liveness probe forced a kill |
| 143 | SIGTERM (128 + 15) — graceful termination | Probe failure or eviction asked the container to stop; often a slow-starting app |
The most common real causes, in order of frequency:
- Application config error — a missing environment variable, an unreachable database, a bad connection string. The app throws on startup and exits 1. Fix: read logs --previous, correct the config/secret, redeploy.
- Failing liveness probe — the app is fine but slow to start, and an aggressive liveness probe kills it before it is ready. Fix: raise initialDelaySeconds / failureThreshold, or use a startupProbe for slow starters.
- OOMKilled (exit 137) — the container exceeds its memory limit and is killed. Fix: raise the memory limit or fix the leak (see Section 6).
- Wrong command / entrypoint (exit 0 or 127) — the container runs the wrong thing and exits immediately. Fix: correct command/args in the manifest.
- Missing dependency at boot — the app needs a service, volume, or secret that isn't ready. Fix: add an init container or readiness gate so it waits.