Kubernetes Best Practices in 2025: Production-Ready Deployments

Kubernetes has become the de facto standard for container orchestration, but running production workloads takes more than kubectl apply. This guide covers the practices that separate a cluster that survives contact with real traffic from one that pages you at 3 AM — resource management, security hardening, high availability, cost control, and observability.

Resource Management

1. Always Set Resource Requests and Limits

Why it matters: Requests drive scheduling decisions; limits prevent one workload from starving its neighbours on the same node.

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"

Best practices:

• Set requests based on actual observed usage, not guesses — use metrics
• Leave headroom between requests and limits for burst traffic
• Use the Vertical Pod Autoscaler to surface right-sizing recommendations
• Monitor OOMKilled events; they mean your memory limit is too low

2. Implement Horizontal Pod Autoscaling (HPA)

Scale pods based on actual demand:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Going beyond CPU:

• Scale on custom metrics (requests/sec, queue depth) — these usually correlate better with real load than CPU does
• Use KEDA for event-driven scaling from queues, streams, and external sources
• Consider scheduled scaling for known, predictable traffic patterns

3. Use Pod Disruption Budgets (PDB)

Protect availability during voluntary disruptions like node drains and cluster upgrades:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: critical-app

Security Hardening

1. Enable Network Policies

By default, every pod in a cluster can talk to every other pod. Network policies let you move toward zero-trust networking — start by denying everything, then allow what's needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

2. Use Pod Security Standards

Pod Security Standards replaced the removed PodSecurityPolicy API. They're applied per namespace via labels:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Security levels:

• Privileged: Unrestricted — avoid in production
• Baseline: Prevents known privilege escalations
• Restricted: Heavily restricted, follows hardening best practices — the right default for production workloads

3. Implement RBAC Properly

Follow the principle of least privilege:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
subjects:
- kind: ServiceAccount
  name: app-sa
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Best practices:

• Never bind cluster-admin to application service accounts
• Create separate service accounts per application — the default service account is a shared blast radius
• Use namespaces for isolation boundaries
• Audit bindings regularly with a tool like rbac-lookup

4. Scan Images for Vulnerabilities

Integrate scanning into CI so vulnerable images fail the build rather than reaching the cluster:

# Report on HIGH and CRITICAL findings
trivy image --severity HIGH,CRITICAL myapp:latest

# Fail the pipeline on CRITICAL findings
trivy image --exit-code 1 --severity CRITICAL myapp:latest

Commonly used scanners:

• Trivy — broad coverage of OS packages and language dependencies
• Grype — fast, pairs well with Syft for SBOM generation
• Snyk — commercial, developer-workflow oriented
• Kubescape — scans cluster configuration against frameworks like NSA-CISA hardening guidance

High Availability

1. Multi-Zone Deployments

A deployment with three replicas that all land on one node is not highly available. Topology spread constraints make placement explicit:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 6
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: critical-app
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: critical-app

Note the deliberate asymmetry: zone spread is DoNotSchedule (a hard requirement), host spread is ScheduleAnyway (a preference). Making both hard requirements can leave pods permanently unschedulable.

2. Implement Health Checks

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3

What each probe is for:

• Liveness: Restarts a container that has deadlocked. A failing liveness probe kills the pod.
• Readiness: Removes a pod from Service endpoints while it can't serve traffic — without killing it.
• Startup: Gives slow-booting applications time before liveness checks begin.

Cost Optimization

1. Right-Size Your Resources

Over-provisioned requests are the single most common source of Kubernetes waste — the scheduler reserves capacity nobody uses.

# Goldilocks surfaces VPA recommendations in a dashboard
kubectl create ns goldilocks
helm install goldilocks fairwinds-stable/goldilocks

# Review VPA recommendations
kubectl get vpa -n production

2. Use Cluster Autoscaling

Cluster Autoscaler adds and removes nodes based on pending pods. It's configured through your cloud provider's flags or Helm values rather than a custom resource — consult the docs for your platform, since the mechanism differs between EKS, GKE, and AKS.

The settings that matter most are the scale-down delay (how long an underused node waits before removal) and the total node ceiling.

3. Leverage Spot and Preemptible Instances

Karpenter provisions nodes directly in response to pending pods, and can mix spot and on-demand capacity:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized

Where the savings come from:

• Spot instances are heavily discounted against on-demand — cloud providers advertise up to around 90% off — in exchange for interruption risk. Suitable for stateless, replicated, interruption-tolerant workloads.
• ARM nodes (such as AWS Graviton) are marketed on better price-performance than comparable x86 instances. The catch is that your images must be built for arm64.
• Bin-packing improves utilization by fitting more pods per node — which only works if your requests are accurate. See the previous section.

Observability

1. The Three Pillars

Metrics — Prometheus + Grafana

# ServiceMonitor for automatic scraping
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: metrics
    interval: 30s

Logs — Loki or Elasticsearch

Structured logs are searchable; plain text is not. Emit JSON in production:

logging.pattern.json: '{"time":"%d{yyyy-MM-dd HH:mm:ss}","level":"%level","msg":"%msg"}'

Traces — Tempo or Jaeger

# OpenTelemetry collector
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    exporters:
      tempo:
        endpoint: tempo:4317
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [tempo]

2. Use OpenTelemetry

OpenTelemetry has become the vendor-neutral standard for instrumentation, which means switching backends no longer means re-instrumenting your applications. Auto-instrumentation can inject agents without code changes:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"

Sampling matters: tracing every request in a high-throughput service produces data volumes nobody will pay to store. The example above samples 25%.

GitOps and CI/CD

1. Implement GitOps with Argo CD or Flux

Declarative deployments reconciled from Git:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  project: default
  source:
    repoURL: https://github.com/org/repo
    targetRevision: HEAD
    path: k8s/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

What this buys you:

• Git as the single source of truth — cluster state matches a reviewable commit
• Automatic drift detection — manual kubectl edit gets reverted
• Rollbacks via git revert rather than remembering what the previous state was
• An audit trail that already exists, for free

2. Progressive Delivery

Argo Rollouts shifts traffic gradually and can halt automatically on bad metrics:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: app
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 30
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 10m}
  analysis:
    templates:
    - templateName: success-rate
    args:
    - name: service
      value: app

The analysis template is the important part. A canary that pauses but never checks anything is just a slower deploy.

Tool Landscape

| Category | Tools | Why | |----------|-------|-----| | Cluster management | Rancher, K9s | Multi-cluster ops and terminal UI | | Security | Kubescape, Trivy | Config and image scanning | | Cost | Kubecost, OpenCost | Cost attribution per namespace/workload | | Networking | Cilium, Calico | eBPF-based networking and policy enforcement | | Service mesh | Istio, Linkerd | mTLS, traffic shifting, observability | | GitOps | Argo CD, Flux | Declarative deployments | | Scaling | Karpenter, KEDA | Node provisioning and event-driven scaling | | Backup | Velero | Cluster and volume disaster recovery |

Common Anti-Patterns to Avoid

1. Running as root — set a non-root user with a UID above 1000
2. No resource requests — the scheduler is flying blind
3. Single replica — no rolling update can be zero-downtime with one pod
4. The latest tag — you can't roll back to a tag that moved
5. Secrets committed to Git — use sealed-secrets or external-secrets
6. No health checks — broken pods stay in the load balancer rotation
7. Everything in default — no isolation, no policy boundary
8. Liveness probes that check dependencies — turns blips into outages

Production Checklist

• [ ] Resource requests and limits configured
• [ ] Horizontal Pod Autoscaler enabled
• [ ] Pod Disruption Budgets in place
• [ ] Network policies configured (and enforced by your CNI)
• [ ] Pod Security Standards enforced
• [ ] RBAC scoped to least privilege
• [ ] Liveness and readiness probes defined separately
• [ ] Multi-zone topology spread configured
• [ ] Monitoring and alerting set up
• [ ] Log aggregation configured
• [ ] Distributed tracing implemented
• [ ] GitOps workflow established
• [ ] Backup strategy defined
• [ ] Disaster recovery plan actually tested
• [ ] Cost monitoring enabled

Our Take

Most Kubernetes production incidents don't come from exotic failures. They come from the basics: no resource requests, a liveness probe wired to the database, a single replica behind a rolling update, or a latest tag that can't be rolled back.

If you're new to Kubernetes, start with our beginner's guide to Kubernetes and come back here when you're ready to run production workloads.