Tomas Vik

Simulate kubernetes pod resisting shutdown

This is a short post about kubernetes pod lifecycle, namely terminating a pod.

When kubernetes shuts down a pod it will:

  1. Send SIGTERM signal to the container process
  2. wait for the grace period (default is 30s)
  3. Send SIGKILL signal to the process

When a Pod is being deleted, it is shown as Terminating by some kubectl commands. This Terminating status is not one of the Pod phases. A Pod is granted a term to terminate gracefully, which defaults to 30 seconds. You can use the flag --force to terminate a Pod by force.

If needed to simulate simulate a container process that ignores the “softer” SIGTERM and has to be shut down using SIGKILL. This is how I did it:

Create pod that ignores SIGTERM

I created bash.yml file with the following content

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: bash
   labels:
     app: bash
 spec:
   selector:
     matchLabels:
       app: bash
   template:
     metadata:
       labels:
         app: bash
     spec:
       containers:
         - image: ubuntu
           name: ubuntu
           command: ["bash", "-c", "trap echo sigterm SIGTERM;while :;do echo hello; sleep 10;done"]
  • This is the key part of the template: trap echo sigterm SIGTERM;while :;do echo hello; sleep 10;done.
  • The bash script that “traps” the SIGTERM signal and only prints sigterm instead.
  • And to keep the container alive, we print hello every 10 seconds.

I created deployment with kubectl apply -f bash.yml.

I deleted that deployment with kubectl delete deployment bash.

I observed the pod hanging in terminating state.

kubectl get pod
NAME                    READY   STATUS        RESTARTS   AGE
bash-7cf45b6b47-8km59   1/1     Terminating   0          22s

Additional resources

Define a Command and Arguments for a Container | Kubernetes

Kubernetes best practices: terminating with grace | Google Cloud Blog