EXERCISE
1See how major companies implement leader election at scale. Learn from production deployments handling millions of requests.
Challenge: Kubernetes clusters need highly available control plane.
Solution: Run 3 or 5 master nodes. Use leader election for controllers.
Implementation:
Kubernetes uses leader election with leases stored in etcd.
Components using leader election:
How it works:
3 master nodes running controller-manager. Only one is active leader.
Leader holds lease in etcd. Updates every 10 seconds.
Leader crashes? Lease expires after 15 seconds. Another controller becomes leader.
User experience: Zero downtime. Kubernetes continues scheduling pods and managing resources.
What is etcd? Distributed key-value store used by , CloudFlare, and others.
Challenge: Store critical cluster state. Must be highly available and consistent.
Solution: etcd cluster with Raft consensus.
Setup: 5 etcd nodes. Raft elects one leader.
Leader handles writes. Followers replicate data.
Leader crashes? Followers elect new leader in 1-2 seconds.
Why this matters: etcd stores state. If etcd is down, entire Kubernetes cluster is read-only or down.
With Raft leader election: etcd achieves 99.99%+ .
Challenge: Each EBS volume distributed across many storage servers. Must maintain consistency.
Solution: Leader election per volume region.
Implementation:
Each volume area has a primary (leader) and followers.
Primary uses lease-based leader election.
Primary orders all reads and writes for consistency.
Primary fails? Follower detects lease expiration. Promotes itself to primary in seconds.
User experience: Brief I/O pause (seconds), then volume continues working.
Scale: Amazon EBS handles millions of volumes, each with its own leader election.
Split-brain: Two nodes both think they are leader. Catastrophic for data integrity.
How it happens:
Network partition splits cluster into two groups.
Each group cannot communicate with the other.
Both groups elect their own leader.
Now you have two leaders making conflicting decisions.
Example disaster:
Bank database splits. Leader A processes withdrawal. Leader B processes withdrawal.
Network heals. Both withdrawals recorded. Account overdrawn incorrectly.
Solution: Require majority (quorum) to elect leader.
Example: 5 nodes. Need 3 votes to become leader.
Network partition: 3 nodes in one group, 2 in another.
Group of 3: Can elect leader (has majority).
Group of 2: Cannot elect leader (no majority). Goes read-only.
Result: Only one leader ever exists.
This is why production systems use odd numbers (3, 5, 7 nodes). Ensures clear majority.
Problem: Lease-based election depends on time. Clocks can .
Scenario:
Leader thinks: "My lease expires at 10:00:30"
Follower thinks: "Leader lease expired at 10:00:25" (clock 5 seconds ahead)
Follower becomes leader while old leader still thinks it is leader. Split-brain!
Solution 1: NTP (Network Time Protocol) keeps clocks synchronized within milliseconds.
Solution 2: Use logical clocks instead of wall-clock time. Raft does this.
Best practice: Assume clocks can skew. Design for safety anyway.
