Every Kubernetes tutorial ends at the same place: you have a deployment, a service, and an ingress. Your application is running. kubectl get pods shows healthy green status. The tutorial declares success and moves on. What the tutorial does not cover is the next twelve months, during which you will learn that running Kubernetes in production is a fundamentally different discipline than deploying to Kubernetes.

We have been running Kubernetes in production at Harbor Software for two years across three clusters (staging, production, and a data processing cluster). We run 40 services, handle roughly 2 million requests per day, and have survived two major incidents that were entirely self-inflicted Kubernetes misconfigurations. One was a cascading OOM kill that took down 80% of our pods in production. The other was a liveness probe misconfiguration that turned a 30-second database failover into a 5-minute complete outage. Here is what they do not tell you.

Resource Requests and Limits Are Not Optional

The single most common cause of production incidents in Kubernetes is missing or incorrect resource requests and limits. Without them, a single misbehaving pod can consume all CPU or memory on a node, starving every other pod running on that node. This is how one microservice’s memory leak becomes a cluster-wide outage.

# Bad: no resource specifications - the pod can consume unlimited resources
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          image: harbor/api:v1.2.3
          # No resources block = unlimited = dangerous

# Good: explicit requests and limits based on observed usage
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          image: harbor/api:v1.2.3
          resources:
            requests:
              cpu: 250m      # Guaranteed: 0.25 CPU cores
              memory: 256Mi  # Guaranteed: 256 MiB
            limits:
              cpu: 1000m     # Maximum: 1 CPU core
              memory: 512Mi  # Maximum: 512 MiB (OOMKilled if exceeded)

Requests tell the Kubernetes scheduler how much CPU and memory this pod needs guaranteed. The scheduler uses requests to decide which node has enough available capacity to run the pod. If you set requests too low, pods get scheduled onto nodes that cannot actually handle their workload under load, leading to contention and degraded performance. If you set them too high, you waste cluster capacity because the scheduler reserves more resources than the pod actually uses, leaving nodes underutilized while new pods cannot be scheduled.

Limits are the hard ceiling. If a pod exceeds its memory limit, it gets OOMKilled (the kernel kills the process, Kubernetes restarts the container). If it exceeds its CPU limit, it gets throttled (the kernel limits its CPU time, the process runs slower but is not killed). Memory limits should be set at approximately 1.5x to 2x the observed P99 memory usage to provide headroom for spikes without allowing unbounded growth. CPU limits are more contentious in the Kubernetes community; some teams omit CPU limits entirely and rely only on requests for scheduling, which avoids throttling at the cost of potential noisy-neighbor issues on shared nodes.

How to Set Resources Correctly

Do not guess. Do not copy values from Stack Overflow or blog posts. Your application’s resource needs are unique. Deploy the application, run it under realistic load, observe actual usage over 48-72 hours, and set values based on data:

1. Deploy with generous initial resources (2 CPU, 2Gi memory) and no limits to establish a baseline without artificial constraints
2. Run for 48-72 hours under production-like traffic, including peak periods
3. Query Prometheus for actual usage metrics
4. Set requests to the P95 usage value (guarantees sufficient resources 95% of the time)
5. Set memory limits to P99 + 30% (provides headroom for occasional spikes)
6. Set CPU limits to 4x the P95 (allows burst capacity for request processing spikes)
7. Monitor for two weeks. Watch for OOMKills and CPU throttling. Adjust as needed.

# Prometheus query: P95 memory usage over 48 hours
quantile_over_time(0.95,
  container_memory_working_set_bytes{
    namespace="production",
    container="api"
  }[48h]
)

# Prometheus query: P95 CPU usage over 48 hours
quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{
    namespace="production",
    container="api"
  }[5m])[48h:5m]
)

# Prometheus query: OOMKill events (should be zero)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled",
  namespace="production", container="api"}

Liveness and Readiness Probes Will Kill Your Service

Probes are the mechanism Kubernetes uses to determine if your pod is healthy and ready to serve traffic. Misconfigured probes are the second most common cause of production incidents we have experienced, and they are insidious because they make Kubernetes actively work against you: the platform’s self-healing behavior becomes self-destructive.

Liveness probes answer the question “is the container alive?” If a liveness probe fails, Kubernetes kills the container and restarts it. This is the correct response to a deadlock, an infinite loop, or an irrecoverable state. It is absolutely not the correct response to temporary slowness, a downstream dependency being unavailable, or a garbage collection pause.

Readiness probes answer the question “is the container ready to handle requests?” If a readiness probe fails, the pod is removed from the service’s endpoint list. Traffic stops flowing to it. The container is not killed; it stays running and can recover on its own. When the readiness probe starts passing again, the pod is added back to the endpoint list.

# Dangerous configuration that WILL cause outages:
livenessProbe:
  httpGet:
    path: /health         # Checks database, cache, everything
    port: 8080
  initialDelaySeconds: 5  # Not enough time for slow startups
  periodSeconds: 5        # Checked every 5 seconds
  timeoutSeconds: 1       # 1 second timeout is too aggressive
  failureThreshold: 1     # ONE failure = restart. Absurdly aggressive.

# Correct configuration:
livenessProbe:
  httpGet:
    path: /healthz        # LIGHTWEIGHT check: is the process alive and responsive?
    port: 8080            # Does NOT check dependencies. Returns 200 if the
  initialDelaySeconds: 30 # HTTP server can respond. That is all.
  periodSeconds: 10
  timeoutSeconds: 5       # Generous timeout - GC pauses can cause brief stalls
  failureThreshold: 3     # Three consecutive failures before declaring dead

readinessProbe:
  httpGet:
    path: /readyz         # FULL check: database connection, cache, external deps
    port: 8080            # Returns 200 only when the pod can do useful work
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3     # Three failures before removing from traffic

The critical insight: liveness and readiness probes must check different things. The liveness probe should only verify that the process is alive and can respond to HTTP requests. A simple route that returns 200 OK with no dependency checks. The readiness probe should verify that the process can do useful work: can it reach the database? Is the cache connected? Are external API dependencies accessible?

We had an incident that perfectly illustrates why this separation matters. Our database performed a planned failover (30 seconds of unavailability). All API pods had a liveness probe that checked the database connection. All liveness probes failed simultaneously. Kubernetes killed all pods simultaneously. Now we had zero running pods, so no traffic could be served even after the database came back online 30 seconds later. Kubernetes started new pods, but each pod took 15 seconds to start up, and with all pods starting at once, the database was flooded with connection attempts. Total outage duration: 5 minutes, caused by a 30-second database failover.

After splitting the probes, the same scenario works correctly. The database fails over. Readiness probes fail on all pods, so they are removed from traffic (users see a maintenance page from the load balancer). Liveness probes continue to pass because the process is alive. When the database recovers 30 seconds later, readiness probes start passing, pods are added back to traffic, and service resumes. Total impact: 30 seconds of degraded service, handled gracefully.

Pod Disruption Budgets Prevent Self-Inflicted Outages

When Kubernetes needs to evict pods during node draining (for maintenance, upgrades, or autoscaler scale-down), it will do so without regard for your application’s availability unless you explicitly tell it not to via PodDisruptionBudgets:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: production
spec:
  minAvailable: 2        # Always keep at least 2 pods running during voluntary disruptions
  selector:
    matchLabels:
      app: api

Without a PDB, draining a node that runs 3 of your 4 API pods will evict all 3 simultaneously, leaving only 1 pod to handle all traffic. With a PDB specifying minAvailable: 2, Kubernetes evicts pods one at a time, waiting for each replacement to become ready before evicting the next one. The process takes longer, but your application maintains capacity throughout.

We set minAvailable to replicas - 1 for most services. For critical path services (API gateway, authentication service), we use maxUnavailable: 1 which has the same effect but reads more intuitively: “at most one pod can be unavailable at any time.”

Network Policies: Default Deny

By default, every pod in a Kubernetes cluster can communicate with every other pod on any port. This is convenient for development and catastrophic for security. If an attacker compromises one pod, they have network access to every other service, every database, and every internal API in your cluster.

# Step 1: Default deny all traffic in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}       # Applies to ALL pods in this namespace
  policyTypes:
    - Ingress
    - Egress

---
# Step 2: Allow specific communication paths
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress-from-nginx
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - port: 8080
          protocol: TCP

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-egress-to-db-and-dns
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - port: 5432
          protocol: TCP
    - to:  # CRITICAL: allow DNS resolution (easy to forget, causes mysterious failures)
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

Start with default deny in every namespace, then add explicit allow rules for each legitimate communication path. The DNS egress rule is easy to forget and will cause extremely confusing failures if missing (pods start, pass health checks locally, but cannot resolve any hostnames, so all external connections fail with DNS resolution errors).

Autoscaling: HPA and Cluster Autoscaler Together

Horizontal Pod Autoscaler (HPA) scales your pods based on metrics. Cluster Autoscaler scales your nodes based on pending pods. You need both, and they need to be configured to work together, or you will either run out of node capacity when HPA tries to scale up, or pay for idle nodes that Cluster Autoscaler does not remove quickly enough.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

The behavior section is critical and almost always omitted from tutorials. Without it, HPA uses aggressive defaults that cause thrashing: scaling up too fast (adding 10 pods when 2 would suffice, wasting resources) or scaling down too fast (removing pods during a brief traffic dip, only to need them again 30 seconds later, causing a scale-up/scale-down oscillation). Our configuration limits scale-up to 50% increase per minute and scale-down to 25% decrease per minute with a 5-minute stabilization window that prevents scaling down after a brief dip.

Observability: The Three Pillars Plus One

Running Kubernetes without proper observability is flying blind at scale. At minimum, you need metrics, logs, traces, and Kubernetes-specific health signals:

• Metrics: Prometheus + Grafana. Scrape kubelet metrics, kube-state-metrics (for deployment/pod/node status), node-exporter (for node-level CPU/memory/disk), and your application metrics. We have four dashboard categories: cluster health, namespace resource usage, deployment status, and application-specific business metrics.
• Logs: Centralized logging is non-negotiable. We use Loki + Promtail for cost-effective log aggregation that integrates natively with Grafana. Every log line includes the pod name, namespace, container name, and node name as labels, making it easy to correlate logs with Kubernetes events.
• Traces: OpenTelemetry with Tempo for distributed trace storage. When a request traverses 5 services, traces show you exactly where time is spent. Without traces, debugging latency in a microservice architecture is guesswork.
• Kubernetes events: kubectl get events shows scheduling decisions, OOMKills, probe failures, image pull errors, and other cluster-level events. We stream events to our logging system so they are searchable and correlatable with application logs.

Secrets Management: Not Base64

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with kubectl access can decode them. For production workloads, use an external secrets manager and sync secrets into Kubernetes:

# External Secrets Operator: pulls secrets from AWS Secrets Manager
# and creates native Kubernetes Secrets from them
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-secrets
  namespace: production
spec:
  refreshInterval: 1h       # Re-sync from Secrets Manager every hour
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: api-secrets        # Name of the K8s Secret to create
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/api/database-url
    - secretKey: REDIS_URL
      remoteRef:
        key: production/api/redis-url
    - secretKey: JWT_SECRET
      remoteRef:
        key: production/api/jwt-secret

The External Secrets Operator is the bridge between your external secrets store and Kubernetes. It runs as a controller in your cluster, periodically reads secrets from your provider (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager), and creates or updates Kubernetes Secrets. Your application pods consume the Kubernetes Secrets normally via environment variables or volume mounts, unaware that the values originate from an external store.

Deployment Strategy: Rolling Updates Done Right

The default Kubernetes rolling update strategy replaces pods one at a time. For most services this is fine, but the default configuration has aggressive settings that can cause brief availability dips during deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Create at most 1 extra pod during update
      maxUnavailable: 0  # Never reduce below desired count
  template:
    spec:
      terminationGracePeriodSeconds: 30  # Give pods 30s to finish requests
      containers:
        - name: api
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]  # Wait for load balancer to drain

The maxUnavailable: 0 setting ensures that during a rolling update, Kubernetes never reduces the number of running pods below the desired replica count. Combined with maxSurge: 1, this means Kubernetes creates one new pod, waits for it to become ready, then terminates one old pod. This is slower than the default (which allows some pods to be unavailable) but ensures zero-downtime deployments.

The preStop sleep is a subtle but important detail. When Kubernetes decides to terminate a pod, two things happen simultaneously: the pod is removed from the service’s endpoint list (so the load balancer stops sending new traffic) and the pod receives a SIGTERM signal. The problem is that endpoint removal is asynchronous: the load balancer might not remove the pod from its target list instantly. The 5-second sleep in the preStop hook gives the load balancer time to complete the removal before the application starts shutting down. Without this sleep, the application might reject in-flight requests that the load balancer is still routing to it during the brief window between SIGTERM and endpoint removal.

The terminationGracePeriodSeconds sets the maximum time Kubernetes will wait for a pod to shut down gracefully. After this period, Kubernetes sends SIGKILL (force kill). Set this to be longer than your longest expected request processing time plus the preStop delay. For an API that might have requests taking up to 10 seconds, a 30-second grace period (5 seconds sleep + 25 seconds for in-flight requests to complete) is appropriate.

Conclusion

Kubernetes is a powerful platform that solves real problems in orchestrating containerized workloads at scale. But its power comes with operational complexity that tutorials dramatically undersell. Resource management, probe configuration, disruption budgets, network policies, observability, autoscaling, and secrets management are not optional extras you add later. They are the foundation of a reliable production deployment, and missing any one of them will eventually cause a production incident.

If you are just starting with Kubernetes in production, prioritize in this order: get resource requests and limits right first (prevents the broadest class of incidents). Then configure liveness and readiness probes correctly (prevents self-inflicted outages during dependency failures). Then add PDBs and network policies (prevents outages during maintenance and limits blast radius of compromises). Then invest in observability (enables debugging everything else). Then configure autoscaling (optimizes cost and handles traffic variability). Each layer builds on the previous one, and together they form a production-grade Kubernetes practice.