// HACKER NEWS — CYBERSECURITY
How Kubernetes Probes Work
I’m going to show you, really show you, how probes work in Kubernetes. How they can make your application more resilient, and how they can help you prevent avoidable mistakes. Like restart loops that take hours to recover from, and dropping requests during rollouts.
Every interactive demo in this post uses webernetes, my partial port of Kubernetes to TypeScript. It contains more than 100,000 lines of ported Kubernetes Go code to run a simulated cluster right here in your browser. I verified the behaviour of these demos against k3s and managed to find a bug in Kubernetes! More on that later.
I want to run a pod with a single container. Here’s its manifest, pod-a.yaml:
This image, my-app:latest, spends a few seconds initialising before listening on port 8080. You will see this below when you click restart to send the container a signal, causing it to crash and get started back up by Kubernetes. You can pause or reset any demo at any time.
After the first crash, the container restarts straight away. After the second, Kubernetes imposes a CrashLoopBackOff on it before starting it again. By default this delay is 10 seconds, doubling with each crash up to a maximum wait of 5 minutes. I shortened it to 3 seconds for this demo.
In both cases, Kubernetes considers the container Ready as soon as it starts, even though we know it’s not. It’s still doing startup work and not listening on port 8080.
Next I’ll add pod-b, which sends a request to pod-a every 2 seconds. Throughout the post, you can think of pod-b as any source of client traffic: an ingress controller, a load balancer, inter-service requests, etc.
If you restart pod-a in the demo below while a request is on its way, that request will fail.
From the moment you restart the container until its startup work finishes, requests will fail, even though the container is considered Ready! This is not what I want. I need Kubernetes to know when pod-a is ready to receive traffic.
For this, Kubernetes gives us probes. Probes are periodic checks sent to containers to determine their health. They come in three flavours: