Kubernetes From First Principles

The Problem: Containers Alone Aren't Enough

Containers solved the packaging problem. With Docker, you can bundle your application and all its dependencies into a single, portable image. You can run that image on any machine that has a container runtime. But containers by themselves don't solve the operational problem.

Imagine you have a web application made up of 15 microservices, each running in a container. You have a fleet of 20 servers. Now answer these questions:

• Where does each container run? You need to decide which machine gets which container based on CPU, memory, disk, and GPU availability. This is the scheduling problem.
• What happens when a machine dies? The containers on that machine are gone. You need to detect the failure and restart those containers on healthy machines. This is the self-healing problem.
• How do containers find each other? Your frontend needs to talk to your API, which needs to talk to your database. Container IPs change every time they restart. This is the service discovery problem.
• How do you deploy a new version without downtime? You can't just stop everything and restart. You need to gradually roll out new containers while draining old ones. This is the rolling update problem.
• How do you scale up when traffic spikes? You need to spin up more container replicas and distribute traffic across them. This is the scaling problem.
• How do you manage configuration and secrets? Database passwords, API keys, feature flags — these need to be injected into containers without baking them into images. This is the configuration management problem.
• How do you handle persistent storage? Containers are ephemeral — their filesystems disappear when they stop. Databases and stateful services need storage that survives container restarts. This is the storage orchestration problem.

You could build custom scripts to solve each of these problems individually. But they interact in complex ways, and at scale, the combinatorial complexity becomes unmanageable. This is exactly the problem Kubernetes was designed to solve.

Before and After Kubernetes

A Brief History: From Borg to Kubernetes

Kubernetes didn't appear out of nowhere. It was born from over a decade of experience running containers at massive scale inside Google.

Borg is Google's internal cluster management system, in use since the mid-2000s. At its peak, Borg manages hundreds of thousands of jobs across millions of machines in dozens of data centers. Every Google product you use — Search, Gmail, YouTube, Maps — runs on Borg. It solved scheduling, fault tolerance, service discovery, and resource management long before Docker made containers mainstream.

In 2014, Google decided to open-source a new system inspired by the lessons learned from Borg (and its successor, Omega). Three Google engineers — Joe Beda, Brendan Burns, and Craig McLuckie — started the Kubernetes project. The name comes from the Greek word for "helmsman" or "pilot."

The key insight behind making it open-source was strategic: Google believed that if the industry standardized on an orchestration platform, it would commoditize infrastructure and reduce the competitive advantage of AWS (which at the time was locking customers in with proprietary services). By giving away the orchestration layer, Google made it easier for workloads to move between clouds.

In 2015, Kubernetes 1.0 was released and Google donated the project to the newly formed Cloud Native Computing Foundation (CNCF). Since then, it has become the de facto standard for container orchestration, with every major cloud provider offering a managed Kubernetes service.

The API Object Model

Kubernetes has a single, unified API. Most cluster resources — Pods, Services, network policies, storage volumes, and more — are represented as API objects. Containers themselves are runtime processes on nodes, described through Pod specs and reported through Pod status. Every object has four key parts:

• apiVersion — which API group and version this object belongs to (e.g., apps/v1, v1)
• kind — what type of object this is (e.g., Pod, Deployment, Service)
• metadata — identifying information: name, namespace, labels, annotations, UID
• spec — your desired state: what you want this object to look like

Most objects also have a status field, which is the actual current state as observed by Kubernetes. Controllers work to make the real world match spec, while status records what they currently observe.

The Essential Objects

Labels, Selectors, and the Loose-Coupling Model

Kubernetes objects are connected to each other not by direct references, but by labels and selectors. A label is a key-value pair attached to an object (e.g., app: web, tier: frontend, version: v2). A selector is a query that matches objects by their labels.

This is how a Service knows which Pods to route traffic to: it has a selector like app: web, and it automatically discovers all Pods with that label. This is how a ReplicaSet knows which Pods it owns. This is how network policies select which Pods they apply to. Labels are the glue of Kubernetes.

This design is intentional: it enables loose coupling. A Service doesn't need to know the names or IPs of individual Pods. It just says "give me everything labeled app: web." Pods can come and go, and the Service automatically adapts. This loose coupling is what makes Kubernetes resilient and flexible.

The Deployment → ReplicaSet → Pod Relationship

The Two Planes

Every Kubernetes cluster has a control plane and a data plane. The control plane makes decisions about the cluster: scheduling, detecting failures, responding to events. The data plane runs the actual workloads: your application containers.

In a self-managed cluster, the control plane components typically run on dedicated control plane nodes (usually 3 for high availability). In a managed Kubernetes service (like EKS, GKE, or the one you're building), the cloud provider operates the control plane for you — but users still interact with it through the Kubernetes API endpoint while running workloads on the data plane nodes.

Control Plane Components at a Glance

Data Plane Components at a Glance

The Reconciliation Loop: Kubernetes' Core Pattern

The single most important architectural pattern in Kubernetes is the reconciliation loop (also called the "control loop" or "observe-diff-act" loop). Every controller in Kubernetes follows this exact pattern:

1. Observe: Watch the API server for the current state of the objects you care about (e.g., "how many Pods exist with label app=web?")
2. Diff: Compare the current state against the desired state (e.g., "the Deployment spec says 3 replicas, but only 2 exist")
3. Act: Take the minimum action needed to drive current state toward desired state (e.g., "create 1 new Pod")
4. Repeat: Go back to step 1 and do it again. Continuously. Forever.

This pattern makes Kubernetes self-healing by design. If a Pod crashes, a node goes down, or someone manually deletes a resource, the relevant controller will detect the drift and correct it. There's no central orchestrator issuing commands — instead, many independent controllers each manage their own slice of the world, all converging toward the declared desired state.

The API Server: The Single Source of Truth

The kube-apiserver is the only component that talks directly to etcd. Every other component — the scheduler, controllers, kubelet, even kubectl — interacts with the cluster exclusively through the API server's REST API.

This design is intentional and important:

• Single point of serialization: All reads and writes to cluster state go through one gateway, making it possible to enforce authentication, authorization, validation, and admission control consistently.
• Watch mechanism: The API server implements an efficient event-streaming mechanism. Components can "watch" for changes to specific resource types and get notified in near-real-time when objects are created, updated, or deleted. This is how controllers know when to reconcile.
• Optimistic concurrency: Every object has a resourceVersion field. When you update an object, you must include the resourceVersion you read. If someone else modified the object in between, the update is rejected with a 409 Conflict. This prevents lost updates without locking.

Authentication & Authorization

Every request to the API server is first authenticated (who are you?) and then authorized (can you do this?). Kubernetes supports multiple authentication methods: X.509 client certificates, bearer tokens, OIDC tokens, webhook token review, and service account tokens (used by pods to talk to the API server).

Authorization is handled by RBAC (Role-Based Access Control) in most clusters. You define Roles (what operations are allowed on which resources) and bind them to users or service accounts with RoleBindings. RBAC is namespace-scoped (Roles and RoleBindings) or cluster-wide (ClusterRoles and ClusterRoleBindings).

Admission Controllers

After authentication and authorization, the request passes through admission controllers. These are plugins that can mutate (modify the request) or validate (accept/reject the request) API objects. There are two phases:

• Mutating admission: Can modify the object before it's persisted. Example: injecting default resource limits, adding sidecar containers (Istio does this), setting default storage classes.
• Validating admission: Can only accept or reject. Example: enforcing that all containers must have resource limits, rejecting pods that try to run as root.

Both phases support webhooks: you can run your own admission logic as an external HTTP server, and the API server will call it for every relevant request. This is extremely powerful for enforcing custom policies.

The Request Flow Through the API Server

etcd: The Cluster's Memory

etcd is a distributed, strongly consistent key-value store. It's the authoritative source of truth for the entire cluster state. Every API object — every Pod, Deployment, Service, Secret, ConfigMap, and CustomResourceDefinition — is stored in etcd as a key-value pair.

How Data is Stored

Objects are stored under a key hierarchy. For example, a Pod named "web-1" in namespace "production" is stored at: /registry/pods/production/web-1. The value is the serialized API object (Protocol Buffers by default, though JSON is also possible). The API server abstracts this away — clients never need to know etcd's key layout.

Raft Consensus

etcd uses the Raft consensus algorithm to replicate data across its cluster members. Here's how it works in simplified terms:

• Leader election: One etcd node is the leader at any time. All writes go through the leader. If the leader fails, the remaining nodes hold an election to choose a new one. This election completes in milliseconds.
• Log replication: When the leader receives a write, it appends it to its log and sends it to followers. Once a majority (quorum) of nodes have acknowledged the write, it's considered committed.
• Quorum: With 3 nodes, quorum is 2 (can tolerate 1 failure). With 5 nodes, quorum is 3 (can tolerate 2 failures). This is why etcd clusters are always odd-numbered.
• Strong consistency: Any read from the leader (or a linearizable read from a follower) returns the latest committed data. There is no "eventual consistency" in etcd — if a write is acknowledged, subsequent reads will see it.

The Watch Mechanism

etcd provides an efficient watch API. Clients can open a long-lived gRPC stream and receive notifications whenever keys matching a prefix are modified. The API server uses this to implement its own watch mechanism: controllers watch the API server, and the API server watches etcd. This creates a near-real-time event notification system without polling.

Watches are also resumable: if the connection drops, the client can reconnect and specify a resourceVersion to resume from, so it doesn't miss any events.

The Scheduler: How Pods Get Placed on Nodes

When a Pod is first created (e.g., by a ReplicaSet controller), it has no spec.nodeName set. The scheduler watches for these "unscheduled" Pods and assigns each one to a suitable node. The process has two phases:

Phase 1: Filtering (Predicates)

The scheduler eliminates nodes that cannot run the Pod. Each filter plugin returns "fits" or "doesn't fit." A node must pass all filters to remain a candidate. Common filters:

• NodeResourcesFit: Does the node have enough allocatable CPU and memory to satisfy the Pod's resource requests?
• NodeAffinity: Does the node match the Pod's nodeAffinity rules (e.g., "only run on nodes with GPU=true label")?
• TaintToleration: Does the Pod tolerate the node's taints? Taints are "repel rules" on nodes — a node tainted with gpu=true:NoSchedule will reject pods that don't explicitly tolerate that taint.
• PodTopologySpread: Does scheduling here violate the Pod's spread constraints (e.g., "spread evenly across availability zones")?
• NodePorts: If the Pod uses hostPort, is that port available on this node?
• VolumeBinding: Can the Pod's requested PersistentVolumeClaims be satisfied on this node?

Phase 2: Scoring (Priorities)

Among the nodes that passed filtering, the scheduler ranks them. Each scoring plugin gives each node a score from 0 to 100. The scores are weighted and summed. The node with the highest total score wins. Common scoring plugins and strategies:

• NodeResourcesFit: Can favor emptier nodes to spread load, or fuller nodes to bin-pack workloads and reduce the number of machines in use.
• InterPodAffinity: Scores based on whether co-located pods would satisfy affinity or violate anti-affinity rules.
• PodTopologySpread: Prefers placements that keep replicas balanced across zones, nodes, or other topology domains.
• ImageLocality: Prefers nodes that already have the required container images cached locally (avoids image pull latency).
• Preferred affinity and taint handling: Soft preferences such as preferred node affinity and tolerated taints can influence the final ranking without acting as hard filters.

The Controller Manager: Reconciliation at Scale

The kube-controller-manager is a single binary that runs dozens of independent control loops. Each loop is responsible for a specific type of API object and drives the cluster toward the desired state for that object type.

How Controllers Work Internally

Each controller uses a pattern called Informer + Work Queue:

1. Informer: A client-side cache that watches the API server via a long-lived connection. When an object is created, updated, or deleted, the Informer receives the event and updates its local cache. It also calls registered event handlers.
2. Event handler: On receiving an event (add/update/delete), the handler extracts the object's key (namespace/name) and adds it to a work queue. It does not do any work here — just queues the item.
3. Work queue: A rate-limited, deduplicating queue. If multiple events arrive for the same object before the worker processes it, only one reconciliation runs. This prevents thundering herd problems.
4. Worker: Pulls items from the work queue and calls the Reconcile() function. This function reads the object from the Informer cache, compares desired state with actual state, and takes action (create/update/delete sub-resources). If the reconciliation fails, the item is re-queued with exponential backoff.

Key Controllers in kube-controller-manager

Leader Election

In a highly available (HA) setup, you run multiple replicas of the controller manager. But only one should be actively reconciling at a time to avoid conflicts. Kubernetes uses leader election via a Lease object in etcd: one instance holds the lease and acts as leader. Others are on standby. If the leader dies, a standby instance acquires the lease within seconds. This same pattern applies to the scheduler and the CCM.

Kubelet: The Node Agent

The kubelet runs on every node in the cluster. It is responsible for:

• Pod lifecycle management: Watches the API server for Pods assigned to its node (via spec.nodeName). Creates, starts, stops, and restarts containers. Handles init containers, sidecar containers, and ephemeral containers.
• Health checking: Runs liveness, readiness, and startup probes. Restarts containers that fail liveness probes. Removes Pods from Service endpoints when readiness probes fail.
• Resource management: Enforces CPU and memory limits via cgroups. Evicts Pods when the node runs out of disk or memory (eviction thresholds are configurable).
• Volume management: Mounts ConfigMaps, Secrets, PersistentVolumeClaims, and projected volumes into Pod containers.
• Status reporting: Periodically updates the Node object's status (allocatable resources, conditions, addresses) and each Pod's status (phase, container states, IPs) in the API server.
• Node heartbeats: Sends periodic heartbeats (Lease objects) to the API server to signal that the node is alive. The node controller in the control plane uses these to detect node failures.

Pod Startup Sequence

When the kubelet picks up a new Pod, here's what happens in order:

CRI: Container Runtime Interface

Kubernetes doesn't run containers directly. It delegates to a container runtime via a gRPC interface called the Container Runtime Interface (CRI). This abstraction means Kubernetes can work with any CRI-compliant runtime.

The CRI has two main services: RuntimeService (create/start/stop/remove containers and sandboxes) and ImageService (pull/list/remove images). The kubelet calls these gRPC methods, and the runtime handles the low-level details of creating Linux namespaces, setting up cgroups, mounting filesystems, and executing the container process.

What Happens Under the Hood: Containers are Linux Processes

A container is not a VM. It's a regular Linux process that runs in isolation using two kernel features:

• Namespaces: Provide isolation. A container commonly gets its own PID namespace (can't see host processes), network namespace (own IP stack), mount namespace (own filesystem), IPC namespace, and UTS namespace (own hostname). User namespaces exist too, but they are optional and not universal in Kubernetes deployments.
• cgroups (Control Groups): Limit resources. A container's cgroup restricts how much CPU and memory the process can use, and can also participate in other kernel-level resource controls. This is the core mechanism behind Kubernetes resources.limits.

When you set resources.limits.memory: 256Mi in a Pod spec, the kubelet tells the container runtime to create a cgroup with a memory limit of 256 MiB. If the process exceeds this, the Linux kernel's OOM killer terminates it, and the kubelet restarts the container.

What Happens When a Node Dies

The Kubernetes Networking Model

Kubernetes imposes a few fundamental networking rules. These are non-negotiable — every network implementation must satisfy them:

• Every Pod gets a unique IP address. No two Pods in the cluster share an IP.
• All Pods can communicate with all other Pods without NAT. If Pod A has IP 10.244.1.5 and Pod B has IP 10.244.2.8, Pod A can reach Pod B by simply connecting to 10.244.2.8. No port mapping, no NAT translation. This massively simplifies application networking.
• All Nodes can communicate with all Pods (and vice versa) without NAT.
• The IP that a Pod sees for itself is the same IP that others see for it. No hidden NAT that could confuse applications.

These rules define what is needed, but not how to implement it. The implementation is delegated to CNI plugins.

CNI: Container Network Interface

CNI is a specification for configuring network interfaces in Linux containers. When a Pod is created, the kubelet calls the CNI plugin to:

1. Create a virtual ethernet (veth) pair: one end in the Pod's network namespace, one end on the host
2. Assign an IP address to the Pod from the node's Pod CIDR range
3. Set up routes so traffic to/from the Pod is properly forwarded
4. Configure any network policies (firewall rules)

Popular CNI plugins include Calico (BGP-based, supports network policies), Cilium (eBPF-based, high performance), Flannel (simple overlay network), and AWS VPC CNI (assigns real VPC IPs to Pods). For your managed K8s service, your choice of CNI plugin (or building your own) is a critical architectural decision — it determines how Pod IPs relate to your cloud's VPC networking.

Services: Stable Endpoints for Pods

Pods are ephemeral — they get created and destroyed constantly, and their IPs change each time. A Service provides a stable, load-balanced endpoint that always routes to the right Pods.

Service Types

kube-proxy: The Network Plumber

kube-proxy runs on every node and implements the Service abstraction. It watches the API server for Service and EndpointSlice objects and programs the node's network stack accordingly. It commonly operates in iptables or IPVS modes, though some environments use newer dataplanes or replace kube-proxy entirely:

• iptables mode (default): Creates iptables rules in the PREROUTING and OUTPUT chains. When a packet is destined for a ClusterIP, iptables performs DNAT (destination NAT) to rewrite the destination IP to a randomly selected backend Pod IP. This happens entirely in the kernel — no userspace proxy involved.
• IPVS mode: Uses Linux IPVS (IP Virtual Server) instead of iptables. IPVS is a transport-layer load balancer built into the kernel. It's more efficient for clusters with thousands of Services because it uses hash tables instead of sequential iptables rules. Supports multiple load balancing algorithms: round-robin, least connections, source hash, etc.

How a Service Routes Traffic

DNS: Service Discovery

Kubernetes runs an internal DNS server (CoreDNS) as a Deployment in the kube-system namespace. Every Service automatically gets a DNS record:

• <service-name>.<namespace>.svc.cluster.local → ClusterIP (A record)
• For headless Services (no ClusterIP): DNS returns the individual Pod IPs directly
• For ExternalName Services: DNS returns a CNAME to the external hostname

Every Pod's /etc/resolv.conf is configured to use CoreDNS as its nameserver, with search domains so you can refer to services in the same namespace by just their name (e.g., curl http://web/).

What "Managed" Means

In a managed Kubernetes service, the cloud provider runs the control plane (API server, etcd, scheduler, controller manager, CCM) on behalf of the customer. The customer still uses that control plane through the Kubernetes API, but usually only manages the worker nodes (or in serverless-style offerings like EKS Fargate or GKE Autopilot, not even that).

The value proposition is clear: customers get a production-grade Kubernetes API without having to operate etcd, handle control plane upgrades, manage certificates, or worry about API server availability. They interact with Kubernetes the same way — kubectl apply works identically — but the operational burden of the control plane is entirely on the cloud provider.

The Key Architectural Decisions

Typical Managed K8s Architecture

The Provisioning Flow

When a customer requests a new Kubernetes cluster through your managed service, here's what your platform needs to do:

Why CCM Exists

Originally, cloud-specific logic was embedded directly inside kube-controller-manager and kubelet. This was problematic: cloud providers had to fork the Kubernetes codebase or wait for upstream releases to include their changes. It also bloated the core with code that was irrelevant to non-cloud deployments.

The CCM was introduced to extract all cloud-dependent code into a separate, pluggable binary. Now, the core Kubernetes components are cloud-agnostic, and each cloud provider maintains their own CCM that implements a well-defined interface.

When you run Kubernetes with an external CCM:

• kube-controller-manager and kubelet are configured with --cloud-provider=external so in-tree cloud logic is disabled
• The API server remains cloud-agnostic and talks to the CCM through normal Kubernetes APIs
• The CCM runs as a Deployment or static pod (or standalone binary on control plane nodes)
• The CCM needs credentials to your cloud's API, though CSI drivers, CNIs, or autoscalers may also need cloud credentials depending on your design

The cloud.Interface

To implement a CCM, you implement the cloud.Interface from the k8s.io/cloud-provider package. The exact surface evolves across Kubernetes releases, so the snippet below is a simplified, partial sketch of the major concepts rather than a verbatim copy of the latest source:

Each sub-interface (LoadBalancer, InstancesV2, Zones, Routes) defines the methods the CCM calls when it needs to interact with your cloud. Let's look at each one.

The Four Controller Loops

How CCM Provisions a Load Balancer

Building Your CCM: Practical Guide

Here's a conceptual CCM skeleton for your cloud. The actual method set, imports, and command wiring vary a bit across Kubernetes releases, so treat this as pseudocode that shows the shape of the integration rather than copy-paste-ready source.

The Certificate Hierarchy

Kubernetes security relies heavily on TLS certificates and, in many core control-plane paths, mutual TLS (mTLS). But not every interaction uses client certificates: Kubernetes also uses bootstrap tokens, service account tokens, OIDC, and other auth mechanisms. You still need a robust PKI (Public Key Infrastructure) before you can start the core control plane safely.

Certificates You Need to Generate

The Bootstrap Process

Here's the complete sequence for standing up a Kubernetes cluster from scratch. For a managed service, you'll automate every step of this.

kubeadm: The Reference Implementation

kubeadm is the official tool for bootstrapping Kubernetes clusters. While you'll likely build your own provisioning system for a managed service, understanding kubeadm's flow is valuable because it implements the same bootstrap process described above.

• kubeadm init: Generates certificates, starts the control plane as static pods (manifests in /etc/kubernetes/manifests/ that the kubelet auto-starts), installs CoreDNS, and outputs a kubeadm join command with a bootstrap token.
• kubeadm join: Uses the bootstrap token to connect to the API server, validates the CA via a discovery token hash, requests a client certificate via TLS bootstrapping, and starts the kubelet.

For a managed service, you'll replace kubeadm with your own automation — but the underlying certificate generation, etcd bootstrapping, and component startup sequence is the same.