<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Kubernetes Production Troubleshoot]]></title><description><![CDATA[Why Kubernetes DNS failures look like app errors — and the exact step-by-step commands to diagnose and fix CoreDNS issues in production.]]></description><link>https://skj02.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6a00a1fde3eebc2e2095024e/f11acf78-bb6e-40f0-8429-b89c7054977d.jpg</url><title>Kubernetes Production Troubleshoot</title><link>https://skj02.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Fri, 23 Oct 2026 11:31:55 GMT</lastBuildDate><atom:link href="https://skj02.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Kubernetes Services Explained Simply: Why Your Application Doesn't Break When Pod IPs Change]]></title><description><![CDATA[Learn how Kubernetes Services provide stable networking, route traffic to healthy Pods, and prevent applications from breaking during deployments.]]></description><link>https://skj02.hashnode.dev/kubernetes-services-explained-simply-why-your-application-doesn-t-break-when-pod-ips-change</link><guid isPermaLink="true">https://skj02.hashnode.dev/kubernetes-services-explained-simply-why-your-application-doesn-t-break-when-pod-ips-change</guid><dc:creator><![CDATA[Sumankant Jha]]></dc:creator><pubDate>Sat, 20 Jun 2026 13:23:51 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a00a1fde3eebc2e2095024e/9ffc1e5f-a999-469a-97bc-ea55c1304ca7.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>One of the most common Kubernetes questions I hear is:</p>
<blockquote>
<p>"If Pods keep getting recreated and their IP addresses change, how does my application continue to work?"</p>
</blockquote>
<p>It's a great question because understanding the answer unlocks one of the most important concepts in Kubernetes networking.</p>
<p>The short answer is:</p>
<p><strong>Applications should never communicate directly with Pods. They should communicate with Services.</strong></p>
<p>Let's see why.</p>
<hr />
<h1>The Problem with Pod IPs</h1>
<p>Every Pod in Kubernetes gets its own IP address.</p>
<p>For example:</p>
<pre><code class="language-text">Pod-1 → 10.244.1.10
Pod-2 → 10.244.1.11
Pod-3 → 10.244.1.12
</code></pre>
<p>Everything works fine until a deployment happens.</p>
<p>A Pod crashes or gets replaced.</p>
<pre><code class="language-text">Pod-1 → 10.244.1.10
Pod-4 → 10.244.3.55
Pod-3 → 10.244.1.12
</code></pre>
<p>Notice what happened?</p>
<p>Pod-2 disappeared and Pod-4 got a completely different IP address.</p>
<p>If applications were configured to connect directly to Pod IPs, every deployment would risk breaking connectivity.</p>
<p>Clearly, we need something more stable.</p>
<hr />
<h1>Enter Kubernetes Services</h1>
<img src="https://cdn.hashnode.com/uploads/covers/6a00a1fde3eebc2e2095024e/380faadd-24ca-498e-9f89-a61067c678bb.png" alt="" style="display:block;margin:0 auto" />

<p>A Service provides a stable way to access a group of Pods.</p>
<p>Instead of talking directly to Pod IPs, applications talk to a Service.</p>
<pre><code class="language-text">      Application
         │
        ▼
 Kubernetes Service
        │
 ┌───┼───┐
 ▼    ▼    ▼
Pod    Pod   Pod
</code></pre>
<p>The Service remains constant even when Pods are created, deleted, scaled, or replaced.</p>
<p>Think of it like a company's customer support number.</p>
<p>Employees may come and go, but customers continue calling the same number.</p>
<p>The company routes the call internally.</p>
<p>A Kubernetes Service works similarly.</p>
<hr />
<h1>How Traffic Flows</h1>
<p>Suppose you have a backend application running three Pods.</p>
<pre><code class="language-text">backend-1
backend-2
backend-3
</code></pre>
<p>A Service selects those Pods using labels.</p>
<p>Example Service:</p>
<pre><code class="language-yaml">apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 8080
</code></pre>
<p>Any Pod with:</p>
<pre><code class="language-yaml">labels:
  app: backend
</code></pre>
<p>becomes part of the Service.</p>
<p>When a request reaches the Service:</p>
<ol>
<li><p>Kubernetes identifies healthy Pods matching the selector.</p>
</li>
<li><p>One Pod is chosen.</p>
</li>
<li><p>Traffic is forwarded to that Pod.</p>
</li>
<li><p>If a Pod disappears, it is removed automatically.</p>
</li>
</ol>
<p>The client never notices the change.</p>
<hr />
<h1>What Happens During a Deployment?</h1>
<p>Let's say we start with:</p>
<pre><code class="language-text">Service
   │
   ├── Pod-1
   ├── Pod-2
   └── Pod-3
</code></pre>
<p>Now a rolling update begins.</p>
<pre><code class="language-text">Pod-2 terminated
Pod-4 created
</code></pre>
<p>Kubernetes updates the Service backend list automatically.</p>
<pre><code class="language-text">Service
   │
   ├── Pod-1
   ├── Pod-4
   └── Pod-3
</code></pre>
<p>Traffic continues flowing.</p>
<p>No application configuration changes.</p>
<p>No DNS updates.</p>
<p>No manual intervention.</p>
<p>This is one of the reasons Kubernetes can perform rolling deployments with minimal disruption.</p>
<hr />
<h1>A Real Production Incident</h1>
<p>I once investigated an incident where:</p>
<ul>
<li><p>All Pods were Running</p>
</li>
<li><p>The application was unreachable</p>
</li>
<li><p>Container logs showed no errors</p>
</li>
</ul>
<p>The team spent hours checking application code.</p>
<p>The actual problem was much simpler.</p>
<p>The Service selector didn't match the Pod labels.</p>
<p>Service:</p>
<pre><code class="language-yaml">selector:
  app: payment
</code></pre>
<p>Pods:</p>
<pre><code class="language-yaml">labels:
  app: payments
</code></pre>
<p>Notice the difference?</p>
<pre><code class="language-text">payment
payments
</code></pre>
<p>Because the labels didn't match, the Service had no backend Pods.</p>
<p>The application was effectively disconnected.</p>
<p>A small typo caused a major outage.</p>
<hr />
<h1>The First Three Things I Check</h1>
<p>Whenever a Kubernetes application is unreachable, I usually check these first:</p>
<h2>1. Verify the Service</h2>
<pre><code class="language-bash">kubectl get svc
</code></pre>
<p>Ensure the Service exists and exposes the expected port.</p>
<hr />
<h2>2. Check Endpoints</h2>
<pre><code class="language-bash">kubectl get endpoints
</code></pre>
<p>or</p>
<pre><code class="language-bash">kubectl get endpointslices
</code></pre>
<p>If there are no endpoints, the Service has no healthy backend Pods.</p>
<hr />
<h2>3. Verify Labels</h2>
<pre><code class="language-bash">kubectl get pods --show-labels
</code></pre>
<p>Compare Pod labels with the Service selector.</p>
<p>Most Service-related issues can be identified within minutes using these commands.</p>
<hr />
<h1>The Mental Model That Helps</h1>
<p>Whenever you're learning Kubernetes networking, remember:</p>
<pre><code class="language-text">Pods are temporary.

Services are permanent.
</code></pre>
<p>Pods can be:</p>
<ul>
<li><p>Created</p>
</li>
<li><p>Deleted</p>
</li>
<li><p>Restarted</p>
</li>
<li><p>Rescheduled</p>
</li>
</ul>
<p>Services remain stable.</p>
<p>Applications should trust Services, not Pod IPs.</p>
<p>Once this concept clicks, topics like:</p>
<ul>
<li><p>CoreDNS</p>
</li>
<li><p>Ingress</p>
</li>
<li><p>Load Balancers</p>
</li>
<li><p>API Gateways</p>
</li>
<li><p>Service Meshes</p>
</li>
</ul>
<p>become much easier to understand because they all build on top of the Service abstraction.</p>
<hr />
<h1>Key Takeaways</h1>
<ul>
<li><p>Pod IPs are temporary and can change at any time.</p>
</li>
<li><p>Services provide stable networking for applications.</p>
</li>
<li><p>Services use label selectors to identify backend Pods.</p>
</li>
<li><p>Kubernetes automatically updates Service backends during deployments.</p>
</li>
<li><p>Many production outages occur because Service selectors don't match Pod labels.</p>
</li>
<li><p>Always check Services, Endpoints, and Labels during troubleshooting.</p>
</li>
</ul>
<p>The next time an application becomes unreachable in Kubernetes, don't start with container logs.</p>
<p>Start by asking:</p>
<blockquote>
<p>"Does the Service actually know where my Pods are?"</p>
</blockquote>
<p>That simple question can save hours of debugging.  </p>
<p>#kubernetes #devops #sre #cloud-native #networking #platform-engineering #containers #docker</p>
]]></content:encoded></item><item><title><![CDATA[Kubernetes DNS Failures: Why They're Invisible — and How to Fix Them]]></title><description><![CDATA[Why Kubernetes DNS failures look like app errors — and the exact step-by-step commands to diagnose and fix CoreDNS issues in production.]]></description><link>https://skj02.hashnode.dev/kubernetes-dns-failures-why-they-re-invisible-and-how-to-fix-them</link><guid isPermaLink="true">https://skj02.hashnode.dev/kubernetes-dns-failures-why-they-re-invisible-and-how-to-fix-them</guid><category><![CDATA[Kubernetes]]></category><category><![CDATA[SRE]]></category><category><![CDATA[Devops]]></category><category><![CDATA[k8s]]></category><dc:creator><![CDATA[Sumankant Jha]]></dc:creator><pubDate>Mon, 08 Jun 2026 18:03:19 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a00a1fde3eebc2e2095024e/b73ca4fd-ea0b-437a-b6be-b5d2add36285.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<hr />
<h2>Why DNS failures are deceptive</h2>
<p>When DNS breaks in a Kubernetes cluster, it rarely announces itself clearly. You don't see a message that says "CoreDNS is throttled." What you see instead is: service timeouts, app-level connection errors, "host not found" in application logs, and engineers spending hours suspecting the wrong component.</p>
<p>DNS is the nervous system of a Kubernetes cluster. Every pod-to-pod call starts with a DNS resolution. If that resolution is slow or fails intermittently, the symptoms propagate upward and look exactly like application bugs or network instability.</p>
<blockquote>
<p><strong>Key insight:</strong> CoreDNS is not a background process. It is critical infrastructure that every pod in your cluster queries on every TCP connection. Treating it as a "set and forget" component is how production incidents happen.</p>
</blockquote>
<h2>Root causes — what actually goes wrong</h2>
<p>There are four common failure modes. Understanding which one you're hitting before running a single command will cut your debug time in half.</p>
<table>
<thead>
<tr>
<th>Root cause</th>
<th>What you see</th>
<th>Why it happens</th>
</tr>
</thead>
<tbody><tr>
<td>CPU throttling</td>
<td>Intermittent timeouts under load</td>
<td>CPU limits set too low on CoreDNS — queries queue and time out during bursts</td>
</tr>
<tr>
<td>Too few replicas</td>
<td>DNS fails after a pod restart</td>
<td>Default 2 replicas — one crash removes 50% DNS capacity instantly</td>
</tr>
<tr>
<td>ndots:5 storm</td>
<td>High DNS query rate, slow external calls</td>
<td>Every short hostname triggers 5 search-domain lookups before resolving</td>
</tr>
<tr>
<td>No NodeLocal cache</td>
<td>Baseline latency on all DNS calls</td>
<td>Every pod query hits CoreDNS directly — no per-node caching layer</td>
</tr>
</tbody></table>
<hr />
<h2>The ndots:5 problem explained</h2>
<p>The default <code>ndots:5</code> setting means: if a hostname has fewer than 5 dots, try every search domain suffix before falling back to a direct lookup.</p>
<h3>What happens when you call <code>api.stripe.com</code> with ndots:5</h3>
<pre><code class="language-plaintext">1. api.stripe.com.default.svc.cluster.local   → NXDOMAIN ❌
2. api.stripe.com.svc.cluster.local            → NXDOMAIN ❌
3. api.stripe.com.cluster.local                → NXDOMAIN ❌
4. api.stripe.com                              → 34.206.0.1 ✓
</code></pre>
<p><strong>4 DNS queries instead of 1</strong> — for every external call, from every pod, at every request. At scale with hundreds of pods, this becomes a DNS query storm.</p>
<h3>Fix: set ndots:2</h3>
<pre><code class="language-plaintext">1. api.stripe.com.default.svc.cluster.local   → NXDOMAIN ❌
2. api.stripe.com                              → 34.206.0.1 ✓
</code></pre>
<p>Cuts DNS query volume by ~60% in large clusters. No CoreDNS restart needed.</p>
<hr />
<h2>Step-by-step: diagnosing the issue</h2>
<p>Run these commands in order. Each one narrows down the cause before you move to the next.</p>
<h3>Step 1 — Confirm it is a DNS problem</h3>
<pre><code class="language-bash"># Test DNS resolution directly
kubectl exec -it &lt;pod-name&gt; -n &lt;namespace&gt; -- nslookup kubernetes.default

# Test service-to-service name resolution
kubectl exec -it &lt;pod-name&gt; -n &lt;namespace&gt; -- \
  curl http://&lt;service&gt;.&lt;namespace&gt;.svc.cluster.local

# Check DNS config inside the pod
kubectl exec -it &lt;pod-name&gt; -n &lt;namespace&gt; -- cat /etc/resolv.conf
</code></pre>
<p><strong>What to look for:</strong></p>
<ul>
<li><p>Intermittent failure → DNS problem</p>
</li>
<li><p>Consistent failure → check Service and Endpoints instead</p>
</li>
<li><p><code>ndots:5</code> in resolv.conf → query multiplication problem (fix in Step 4)</p>
</li>
</ul>
<hr />
<h3>Step 2 — Inspect CoreDNS health</h3>
<pre><code class="language-bash"># Pod status
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CPU and memory usage
kubectl top pods -n kube-system -l k8s-app=kube-dns

# Logs — look for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 \
  | grep -i "SERVFAIL\|timeout\|i/o"
</code></pre>
<p><strong>Red flags in logs:</strong></p>
<ul>
<li><p><code>SERVFAIL</code> → query is failing at CoreDNS</p>
</li>
<li><p><code>i/o timeout</code> → upstream resolver unreachable</p>
</li>
<li><p>CPU near limit → throttling is the cause</p>
</li>
<li><p>Repeated pod restarts → crash or OOM</p>
</li>
</ul>
<hr />
<h2>Step-by-step: fixing the issue</h2>
<h3>Fix 1 — Remove CPU limits, scale replicas</h3>
<p>CPU limits on CoreDNS cause throttling at peak load. Removing limits allows CoreDNS to burst freely. Scaling replicas ensures resilience if one pod restarts.</p>
<pre><code class="language-bash"># Scale CoreDNS — target 1 replica per 10–15 nodes
kubectl scale deployment coredns --replicas=5 -n kube-system

# Edit resource config — remove the limits block
kubectl edit deployment coredns -n kube-system
</code></pre>
<p>Under <code>resources:</code>, keep requests and delete the <code>limits</code> block entirely:</p>
<pre><code class="language-yaml">resources:
  requests:
    cpu: 100m
    memory: 70Mi
  # Delete the limits section entirely
</code></pre>
<hr />
<h3>Fix 2 — Set ndots:2 in your pod spec</h3>
<p>Add this to every Deployment or Pod spec where external DNS calls are made:</p>
<pre><code class="language-yaml">spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
      - name: single-request-reopen
  dnsPolicy: ClusterFirst
</code></pre>
<p><strong>Impact:</strong> Reduces DNS query volume by ~60% in large clusters. No CoreDNS restart needed.</p>
<hr />
<h3>Fix 3 — Deploy NodeLocal DNSCache</h3>
<p>This is the highest-impact fix. A DaemonSet runs a DNS cache on every node — most queries are answered locally without ever reaching CoreDNS.</p>
<pre><code class="language-bash"># Deploy NodeLocal DNSCache DaemonSet
kubectl apply -f \
  https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

# Verify DaemonSet is running on all nodes
kubectl get daemonset nodelocaldns -n kube-system

# Should show one pod per node — DESIRED == READY
</code></pre>
<hr />
<h3>Fix 4 — Add DNS monitoring</h3>
<p>CoreDNS exposes Prometheus metrics by default. Add these to your dashboards before you need them.</p>
<pre><code class="language-plaintext"># Scrape endpoint on every CoreDNS pod
http://&lt;coredns-pod-ip&gt;:9153/metrics

# Alert: p99 DNS latency over 100ms
coredns_dns_request_duration_seconds{quantile="0.99"} &gt; 0.1

# Alert: any SERVFAIL responses
coredns_dns_responses_total{rcode="SERVFAIL"} &gt; 0

# Alert: upstream failures
coredns_forward_request_failures_total &gt; 0
</code></pre>
<hr />
<h2>Quick cluster health check</h2>
<p>Run this bundle right now on any cluster to get a baseline reading:</p>
<pre><code class="language-bash"># 1. CPU usage — is CoreDNS near its limit?
kubectl top pods -n kube-system | grep coredns

# 2. Replica count — is it enough for your cluster?
kubectl get deployment coredns -n kube-system

# 3. Any errors in recent logs?
kubectl logs -n kube-system -l k8s-app=kube-dns \
  --tail=50 | grep -i "error\|fail\|timeout"

# 4. Is NodeLocal DNSCache deployed?
kubectl get daemonset nodelocaldns -n kube-system 2&gt;/dev/null \
  || echo "Not deployed"
</code></pre>
<hr />
<h2>Summary — the complete fix checklist</h2>
<table>
<thead>
<tr>
<th>#</th>
<th>Action</th>
<th>Impact</th>
</tr>
</thead>
<tbody><tr>
<td>✅ 1</td>
<td>Remove CPU limits on CoreDNS (keep requests)</td>
<td>Eliminates throttling under burst load</td>
</tr>
<tr>
<td>✅ 2</td>
<td>Scale CoreDNS to 1 replica per 10–15 nodes</td>
<td>Removes single point of failure</td>
</tr>
<tr>
<td>✅ 3</td>
<td>Set <code>ndots:2</code> in pod spec</td>
<td>Cuts DNS query volume by ~60%</td>
</tr>
<tr>
<td>✅ 4</td>
<td>Deploy NodeLocal DNSCache</td>
<td>Highest latency reduction — queries resolve per-node</td>
</tr>
<tr>
<td>✅ 5</td>
<td>Alert on p99 &gt; 100ms and SERVFAIL &gt; 0</td>
<td>Catch issues before users do</td>
</tr>
</tbody></table>
<hr />
<h2>The mental model that changes everything</h2>
<p>CoreDNS is not a default add-on you configure once.</p>
<p>It is a critical database that every pod queries on every connection.</p>
<p><strong>Right-size it. Monitor it. Scale it proactively — before it becomes a production incident.</strong></p>
<hr />
<p><em>Found this useful? Save it for your next DNS debugging session. If you've hit a DNS issue in production that's not covered here, share it in the comments — every cluster has its own flavor of chaos.</em></p>
<hr />
]]></content:encoded></item><item><title><![CDATA[Kubernetes Is Lying to You — Here's How to Make It Talk]]></title><description><![CDATA[A production engineer's honest guide to diagnosing CrashLoopBackOff, OOMKilled, Pending pods, and Node NotReady — with real terminal output and the…]]></description><link>https://skj02.hashnode.dev/kubernetes-is-lying-to-you-here-s-how-to-make-it-talk</link><guid isPermaLink="true">https://skj02.hashnode.dev/kubernetes-is-lying-to-you-here-s-how-to-make-it-talk</guid><dc:creator><![CDATA[Sumankant Jha]]></dc:creator><pubDate>Sun, 10 May 2026 16:11:23 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a00a1fde3eebc2e2095024e/cdf3a66c-f8bf-4cf8-a756-ee77c415031c.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<hr />
<p>I've been on-call when a pod restart count hit 40. I've watched a node flip NotReady at 2 AM and taken down three services with it. I've spent 45 minutes debugging a "networking issue" that turned out to be a label typo on a service selector.</p>
<p>Kubernetes doesn't lie to you — but it doesn't exactly make things easy either. It gives you cryptic status messages, events that disappear after an hour, and logs that only exist for the container that's already dead.</p>
<p>This guide is what I wish I had when I started. Not a list of commands — but the <em>reasoning</em> behind each command, the mental model for reading what Kubernetes is actually telling you, and the exact terminal steps I use in production when something breaks.</p>
<p>We'll cover four of the most common — and most misdiagnosed — failures:</p>
<ol>
<li><p><strong>CrashLoopBackOff</strong> — your pod keeps dying and you don't know why</p>
</li>
<li><p><strong>OOMKilled</strong> — Kubernetes silently murdered your container</p>
</li>
<li><p><strong>Pending pods</strong> — your pod refuses to schedule and says nothing useful</p>
</li>
<li><p><strong>Node NotReady</strong> — the machine underneath is sick</p>
</li>
</ol>
<hr />
<h2>Before we start: the three commands you always run first</h2>
<p>Before you do anything else in Kubernetes troubleshooting, run these three. Every time. In this order.</p>
<pre><code class="language-bash">kubectl get pods -n &lt;namespace&gt;
kubectl describe pod &lt;pod-name&gt; -n &lt;namespace&gt;
kubectl logs &lt;pod-name&gt; -n &lt;namespace&gt; --previous
</code></pre>
<p>The first tells you <em>what</em> is wrong. The second tells you <em>where</em> it went wrong. The third shows you <em>what the dead container saw</em> before it died. Together, they answer 70% of production issues.</p>
<p>Now let's go deeper.</p>
<hr />
<h2>1. CrashLoopBackOff — "My pod keeps restarting and I have no idea why"</h2>
<h3>What you see</h3>
<pre><code class="language-bash">$ kubectl get pods -n production

NAME                          READY   STATUS             RESTARTS   AGE
api-deployment-7d9f8b-xk2pl   0/1     CrashLoopBackOff   23         18m
</code></pre>
<p>That restart count of 23 means Kubernetes has tried to start your container 23 times. Each time it crashed. Kubernetes is now applying exponential backoff — waiting longer between each restart attempt. That's the "BackOff" part of the name.</p>
<h3>What's actually happening behind the scenes</h3>
<p>When Kubernetes starts a container, it runs your defined <code>command</code> or the image's entrypoint. If that process exits with a non-zero code — for any reason — Kubernetes marks the container as failed and increments the restart counter. It doesn't know <em>why</em> your process exited. It just knows it did.</p>
<p>The critical insight: <strong>the current running container knows almost nothing</strong>. It just started. The container that holds the clues is the <em>previous</em> one — the one that already crashed.</p>
<h3>The diagnostic flow</h3>
<p><strong>Step 1 — Read the current logs first (quick check)</strong></p>
<pre><code class="language-bash">$ kubectl logs api-deployment-7d9f8b-xk2pl -n production

Error: Cannot connect to database at postgres-service:5432
Connection refused
</code></pre>
<p>Sometimes you get lucky and the error is right here. Most times the container crashes so fast there are no current logs. Move to step 2.</p>
<p><strong>Step 2 — Read the PREVIOUS container's logs (this is the real one)</strong></p>
<pre><code class="language-bash">$ kubectl logs api-deployment-7d9f8b-xk2pl -n production --previous

[2024-01-15 02:14:33] INFO  Starting application server...
[2024-01-15 02:14:33] INFO  Loading configuration from /etc/config/app.yaml
[2024-01-15 02:14:33] FATAL Missing required environment variable: DATABASE_URL
[2024-01-15 02:14:33] ERROR Application startup failed. Exiting.
</code></pre>
<p>There it is. The <code>--previous</code> flag tells kubectl to fetch logs from the terminated container, not the currently running one. This is the most commonly skipped step in K8s debugging.</p>
<p><strong>Step 3 — Check the events section in describe</strong></p>
<pre><code class="language-bash">$ kubectl describe pod api-deployment-7d9f8b-xk2pl -n production

...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Pulled     23m                kubelet            Successfully pulled image
  Normal   Created    23m                kubelet            Created container api
  Normal   Started    23m                kubelet            Started container api
  Warning  BackOff    2m (x18 over 20m)  kubelet            Back-off restarting failed container
</code></pre>
<p>The Events section is a timeline of everything Kubernetes did to this pod. Read it from top to bottom. The <code>BackOff</code> warning confirms we're in CrashLoopBackOff and shows how long it's been happening.</p>
<p><strong>Step 4 — Check the last state of the container</strong></p>
<pre><code class="language-bash">$ kubectl get pod api-deployment-7d9f8b-xk2pl -n production -o yaml | grep -A 10 "lastState"

    lastState:
      terminated:
        containerID: docker://a3f9b2...
        exitCode: 1
        finishedAt: "2026-03-15T02:14:34Z"
        reason: Error
        startedAt: "2026-03-15T02:14:33Z"
</code></pre>
<p>The <code>exitCode: 1</code> tells you the app exited with a general error. If you see <code>exitCode: 137</code>, stop — that's OOMKilled (we cover that next). If you see <code>exitCode: 0</code>, the app is exiting cleanly, which means your liveness probe is probably the problem, not the app itself.</p>
<h3>Common root causes and their fixes</h3>
<p><strong>Missing environment variable:</strong> Add it to your deployment's <code>env</code> section or reference a Secret/ConfigMap.</p>
<pre><code class="language-yaml">env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: database-url
</code></pre>
<p><strong>Liveness probe killing the container before it's ready:</strong> If your app takes 30 seconds to start but your liveness probe fires after 10 seconds, Kubernetes will kill it before it's healthy. Fix this with <code>initialDelaySeconds</code>.</p>
<pre><code class="language-yaml">livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30   # give the app time to start
  periodSeconds: 10
  failureThreshold: 3
</code></pre>
<p><strong>External dependency not reachable:</strong> Use an init container to wait for the dependency before your main container starts.</p>
<pre><code class="language-yaml">initContainers:
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z postgres-service 5432; do echo waiting for database; sleep 2; done']
</code></pre>
<hr />
<h2>2. OOMKilled — "Kubernetes silently killed my container"</h2>
<h3>What you see</h3>
<pre><code class="language-bash">$ kubectl get pods -n production

NAME                        READY   STATUS      RESTARTS   AGE
worker-6b9c4d-p8mwz         0/1     OOMKilled   5          12m
</code></pre>
<p>Or sometimes it shows as <code>CrashLoopBackOff</code> with OOMKilled buried inside.</p>
<h3>What's actually happening behind the scenes</h3>
<p>OOMKilled stands for Out Of Memory Killed. When your container's memory usage exceeds the <code>limits.memory</code> value in its spec, the Linux kernel's OOM (Out Of Memory) killer terminates the process immediately. No warning. No graceful shutdown. No logs. It's a hard kill — exit code 137.</p>
<p>This is not a Kubernetes decision. Kubernetes sets the cgroup memory limit, but the kernel does the killing. That's why it's so abrupt and why your application has no chance to write a final log message.</p>
<h3>The diagnostic flow</h3>
<p><strong>Step 1 — Confirm it's actually OOMKilled</strong></p>
<pre><code class="language-bash">$ kubectl describe pod worker-6b9c4d-p8mwz -n production

Containers:
  worker:
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 28 Mar 2026 02:10:11 +0000
      Finished:     Mon, 28 Mar 2026 02:11:43 +0000
    Limits:
      memory:  256Mi
    Requests:
      memory:  128Mi
</code></pre>
<p>Exit code 137 = OOMKilled. Always. If you see 137, skip the application logs — the app didn't write anything. The kernel killed it mid-execution.</p>
<p><strong>Step 2 — Check how much memory it was actually using before the kill</strong></p>
<pre><code class="language-bash">$ kubectl top pod worker-6b9c4d-p8mwz -n production

NAME                    CPU(cores)   MEMORY(bytes)
worker-6b9c4d-p8mwz     142m         251Mi
</code></pre>
<p>It was using 251Mi against a limit of 256Mi — right at the edge. This tells you the limit is probably just too tight for the workload.</p>
<p><strong>Step 3 — Check if it's a leak or a limit issue</strong></p>
<p>Look at the restart timestamps in <code>kubectl describe</code>. If the pod runs for hours and then OOMKills, that's a <strong>memory leak</strong> — memory is growing slowly over time. If it OOMKills within seconds of starting, that's a <strong>limit too low</strong> — the app's baseline memory already exceeds the limit.</p>
<p><strong>Step 4 — Check the cgroup memory stats directly on the node (advanced)</strong></p>
<pre><code class="language-bash"># SSH into the node first, then:
$ cat /sys/fs/cgroup/memory/kubepods/pod&lt;pod-uid&gt;/memory.usage_in_bytes
268435456  # bytes — convert to MB: 256MB (exactly at the limit)

$ cat /sys/fs/cgroup/memory/kubepods/pod&lt;pod-uid&gt;/memory.oom_control
oom_kill_disable 0
under_oom 0
oom_kill 1         # oom_kill: 1 confirms the kernel fired the OOM killer
</code></pre>
<h3>Fixes that actually work in production</h3>
<p><strong>If the limit is genuinely too low:</strong> Increase it. But first, profile what the app actually needs under normal load.</p>
<pre><code class="language-yaml">resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"   # give 2x headroom over requests
</code></pre>
<p><strong>If you want to guarantee the pod survives node memory pressure:</strong> Set <code>requests == limits</code>. This gives the pod "Guaranteed" QoS class — Kubernetes kills BestEffort and Burstable pods first when a node is under memory pressure. Guaranteed pods are the last to be evicted.</p>
<pre><code class="language-yaml">resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "512Mi"   # requests = limits = Guaranteed QoS
</code></pre>
<p><strong>If it's a memory leak:</strong> Horizontal scaling buys you time but doesn't fix the leak. Profile your application. For Python, use <code>memory_profiler</code>. For Java, take a heap dump. For Go, use <code>pprof</code>. Leaks don't fix themselves.</p>
<hr />
<h2>3. Pending Pods — "My pod is stuck and not scheduling"</h2>
<h3>What you see</h3>
<pre><code class="language-bash">$ kubectl get pods -n staging

NAME                      READY   STATUS    RESTARTS   AGE
ml-job-5f8d9c-kx7vp       0/1     Pending   0          8m
</code></pre>
<p>A pod that stays in <code>Pending</code> hasn't been scheduled to any node yet. The scheduler looked at your cluster and couldn't find a suitable place to put it.</p>
<h3>The single most useful command for Pending pods</h3>
<pre><code class="language-bash">$ kubectl describe pod ml-job-5f8d9c-kx7vp -n staging

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  8m    default-scheduler  0/3 nodes are available:
                                                      1 node(s) had taint {node-role: gpu} that
                                                      the pod didn't tolerate,
                                                      2 Insufficient memory.
</code></pre>
<p>The scheduler tells you <em>exactly</em> why it couldn't schedule the pod. Read this message carefully — it lists every node and why each one was rejected.</p>
<h3>Diagnosing each failure type</h3>
<p><strong>"Insufficient memory" or "Insufficient CPU"</strong></p>
<pre><code class="language-bash"># See actual allocated resources across all nodes
$ kubectl describe nodes | grep -A 8 "Allocated resources"

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1850m (92%)   2400m (120%)
  memory             3.2Gi (89%)   4.1Gi (114%)
</code></pre>
<p>92% CPU requested across the cluster. Your new pod has nowhere to go. You either need to scale the cluster or reduce the pod's resource requests.</p>
<p><strong>"Node had taint that pod didn't tolerate"</strong></p>
<pre><code class="language-bash"># See what taints are on your nodes
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

NAME        TAINTS
node-1      [map[effect:NoSchedule key:node-role value:gpu]]
node-2      &lt;none&gt;
node-3      &lt;none&gt;
</code></pre>
<p>A taint is a node saying "don't put pods here unless they explicitly tolerate me." Fix it by adding a toleration to your pod spec.</p>
<pre><code class="language-yaml">tolerations:
  - key: "node-role"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
</code></pre>
<p><strong>"PersistentVolumeClaim not bound"</strong></p>
<pre><code class="language-bash">$ kubectl get pvc -n staging

NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ml-data-pvc   Pending                                      fast-ssd       9m
</code></pre>
<p>The PVC is also Pending. The pod can't schedule until the volume is ready.</p>
<pre><code class="language-bash">$ kubectl describe pvc ml-data-pvc -n staging

Events:
  Warning  ProvisioningFailed  2m   persistentvolume-controller
           storageclass.storage.k8s.io "fast-ssd" not found
</code></pre>
<p>The StorageClass doesn't exist. Either create it or fix the typo in your PVC definition. This is one of the most common causes — a simple typo in <code>storageClassName</code>.</p>
<hr />
<h2>4. Node NotReady — "The machine underneath is sick"</h2>
<h3>What you see</h3>
<pre><code class="language-bash">$ kubectl get nodes

NAME        STATUS     ROLES    AGE   VERSION
node-1      Ready      worker   45d   v1.30.0
node-2      NotReady   worker   45d   v1.30.0
node-3      Ready      worker   45d   v1.30.0
</code></pre>
<p>A NotReady node is serious. Kubernetes will start evicting pods from it after the <code>pod-eviction-timeout</code> (default 5 minutes). Those pods will be rescheduled to other nodes — which may then also go under pressure.</p>
<h3>Diagnosing NotReady step by step</h3>
<p><strong>Step 1 — Read the node conditions</strong></p>
<pre><code class="language-bash">$ kubectl describe node node-2 | grep -A 20 "Conditions:"

Conditions:
  Type                 Status  Reason                  Message
  ----                 ------  ------                  -------
  MemoryPressure       False   KubeletHasSufficientMemory  kubelet has sufficient memory
  DiskPressure         True    KubeletHasDiskPressure  kubelet has disk pressure   ← HERE
  PIDPressure          False   KubeletHasSufficientPID kubelet has sufficient PID
  Ready                False   KubeletNotReady         PLEG is not healthy
</code></pre>
<p><code>DiskPressure: True</code> — the node is running out of disk space. That's what triggered the NotReady state. Kubernetes marks a node NotReady when kubelet can't maintain a healthy state, which disk pressure causes by preventing kubelet from writing its own state files.</p>
<p><strong>Step 2 — SSH into the node and check disk</strong></p>
<pre><code class="language-bash">$ ssh ubuntu@node-2

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   48G  200M  99% /
/dev/sda2        20G  1.2G   18G   7% /var/lib/kubelet

$ du -sh /var/lib/containerd/*
2.1G    /var/lib/containerd/io.containerd.content.v1.content
14G     /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
</code></pre>
<p>14GB of overlay snapshots — container image layers that were never cleaned up. The disk is 99% full because nobody set up image garbage collection.</p>
<p><strong>Step 3 — Clean up the disk (immediate fix)</strong></p>
<pre><code class="language-bash"># Remove unused images (safe — only removes images not used by any container)
$ crictl rmi --prune
Deleted: sha256:a3f9b2...
Deleted: sha256:7c4d81...
Deleted: sha256:ee2f19...

# Or if using docker runtime:
$ docker system prune -f
$ docker image prune -a -f

# Check disk again
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   31G   19G  62% /
</code></pre>
<p><strong>Step 4 — Check kubelet status after cleanup</strong></p>
<pre><code class="language-bash">$ systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
   Active: active (running) since Mon 2024-01-15 02:45:11 UTC; 2min ago

# If kubelet is crashed, restart it:
$ systemctl restart kubelet

# Watch it recover:
$ kubectl get nodes --watch
NAME      STATUS     ROLES    AGE   VERSION
node-2    NotReady   worker   45d   v1.28.0
node-2    Ready      worker   45d   v1.28.0   ← recovery
</code></pre>
<h3>Preventing disk pressure permanently</h3>
<p>Set kubelet's garbage collection thresholds in <code>/var/lib/kubelet/config.yaml</code>:</p>
<pre><code class="language-yaml">evictionHard:
  nodefs.available: "10%"       # evict pods when disk &lt; 10%
  imagefs.available: "15%"      # evict when image disk &lt; 15%
imageGCHighThresholdPercent: 85 # start GC when image disk &gt; 85%
imageGCLowThresholdPercent: 80  # GC down to 80%
</code></pre>
<hr />
<h2>The mental model that ties all of this together</h2>
<p>Every Kubernetes failure follows the same investigation pattern. When something breaks, your job is to answer four questions in order:</p>
<p><strong>What</strong> is the symptom? — <code>kubectl get pods</code> or <code>kubectl get nodes</code></p>
<p><strong>Where</strong> did it fail? — <code>kubectl describe pod/node</code> — always read the Events section to the bottom</p>
<p><strong>Why</strong> did it fail? — logs (<code>--previous</code> for crashed containers), node metrics (<code>df -h</code>, <code>free -m</code>), or resource constraints (<code>kubectl top</code>)</p>
<p><strong>When</strong> did it start? — timestamps in Events, restart counts, and duration. A pod that's been failing for 2 minutes is different from one that's been failing for 2 hours.</p>
<p>Get fast at these four questions and you'll resolve 90% of production Kubernetes incidents in under 10 minutes.</p>
<hr />
<h2>What's next</h2>
<p>If you found this useful, follow along. Every post in this series is built from real incidents, not documentation.</p>
<hr />
<p><em>Sumankant Jha — Senior DevOps &amp; SRE Engineer at HCL Technologies, Based in Delhi, India.</em></p>
<p><em>Connect on LinkedIn:</em> <a href="https://linkedin.com/in/skjha0207"><em>linkedin.com/in/skjha0207</em></a> <em>Hashnode:</em> <a href="https://hashnode.com/@skj02"><em>hashnode.com/@skj02</em></a></p>
]]></content:encoded></item></channel></rss>