Kubernetes Cluster Architecture: Planning for Production

Why Architecture Matters

Before spinning up your first cluster, it's worth spending time on architecture. The decisions you make now will affect everything from day-to-day operations to disaster recovery. I've seen teams rush into Kubernetes only to rebuild their clusters from scratch six months later.

This post covers the key architectural decisions for a production Kubernetes cluster: control plane design, node sizing, networking, and capacity planning.

Control Plane High Availability

For production workloads, a single control plane node is a non-starter. If it goes down, you can't deploy, scale, or manage your cluster. The standard approach is three control plane nodes, distributed across failure domains.

*Three control plane nodes minimum - provides quorum for etcd
*Spread across availability zones or physical racks
*Load balancer in front for API server access
*Dedicated nodes (don't schedule workloads on control plane)

yaml

# Control plane node configuration
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.29.0
controlPlaneEndpoint: "k8s-api.internal:6443"
etcd:
  local:
    dataDir: /var/lib/etcd
apiServer:
  extraArgs:
    audit-log-path: /var/log/kubernetes/audit.log
    audit-log-maxage: "30"
    audit-log-maxbackup: "10"
controllerManager:
  extraArgs:
    bind-address: "0.0.0.0"
scheduler:
  extraArgs:
    bind-address: "0.0.0.0"

Node Sizing Strategy

The eternal question: fewer large nodes or many small nodes? Both approaches have trade-offs. I've landed on a middle ground that balances resource utilization with blast radius.

*Medium-sized nodes (8-16 cores, 32-64GB RAM) offer the best balance
*Large nodes improve bin-packing but increase blast radius when they fail
*Small nodes waste resources on kubelet and system overhead
*Consider separate node pools for different workload types

Start with nodes sized for your largest expected pod, then adjust based on actual utilization patterns. Over-provisioning early is better than under-provisioning.

Networking Decisions

Kubernetes networking can be complex. The CNI (Container Network Interface) you choose affects performance, security features, and operational complexity. Here's what I considered:

*Cilium - eBPF-based, great observability, built-in network policies
*Calico - Mature, flexible, good for hybrid environments
*Flannel - Simple, good for learning, limited features
*Pod CIDR sizing - plan for growth, /16 is usually safe

yaml

# Cilium configuration for production
apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  name: cilium-config
spec:
  ipam:
    mode: kubernetes
  kubeProxyReplacement: strict
  hubble:
    enabled: true
    relay:
      enabled: true
    ui:
      enabled: true
  bpf:
    masquerade: true
  loadBalancer:
    mode: dsr

Capacity Planning

Plan for growth from day one. It's much easier to scale out than to migrate to a larger cluster. I use these guidelines:

*Target 60-70% average utilization (leaves room for bursts)
*Plan for 2x current workload as minimum headroom
*Set up cluster autoscaler early, even if not needed immediately
*Monitor etcd size - large clusters can hit etcd limits

Lessons Learned

After running Kubernetes in my homelab for several years, here are the architectural decisions I'd make again:

*Invest in control plane HA from the start - retrofitting is painful
*Choose your CNI carefully - migrating later is very disruptive
*Use separate node pools for stateful vs stateless workloads
*Plan your IP address space generously - running out is a crisis
*Document everything - future you will thank present you