EXERCISE
1See how major companies implement leader election at scale. Learn from production deployments handling millions of requests.
Save
Challenge: clusters need highly available control plane.
Solution: Run 3 or 5 master nodes. Use leader election for controllers.
Implementation:
Kubernetes uses leader election with leases stored in etcd.
Components using leader election:
How it works:
3 master nodes running controller-manager. Only one is active leader.
Leader holds lease in etcd. Updates every 10 seconds.
Leader crashes? Lease expires after 15 seconds. Another controller becomes leader.
User experience: Zero downtime. Kubernetes continues scheduling pods and managing resources.
What is etcd? Distributed key-value store used by Kubernetes, CloudFlare, and others.
Challenge: Store critical cluster state. Must be highly available and consistent.
Solution: etcd cluster with Raft consensus.
Setup: 5 etcd nodes. Raft elects one leader.
Leader handles writes. Followers replicate data.
Leader crashes? Followers elect new leader in 1-2 seconds.
Why this matters: etcd stores state. If etcd is down, entire Kubernetes cluster is read-only or down.
With Raft leader election: etcd achieves 99.99%+ .
Challenge: Each EBS volume distributed across many storage servers. Must maintain consistency.
Solution: Leader election per volume region.
Implementation:
Each volume area has a primary (leader) and followers.
Primary uses lease-based leader election.
Primary orders all reads and writes for consistency.
Primary fails? Follower detects lease expiration. Promotes itself to primary in seconds.
User experience: Brief I/O pause (seconds), then volume continues working.
Scale: Amazon EBS handles millions of volumes, each with its own leader election.
Split-brain: Two nodes both think they are leader. Catastrophic for data integrity.
How it happens:
Network partition splits cluster into two groups.
Each group cannot communicate with the other.
Both groups elect their own leader.
Now you have two leaders making conflicting decisions.
Example disaster:
Bank splits. Leader A processes withdrawal. Leader B processes withdrawal.
Network heals. Both withdrawals recorded. Account overdrawn incorrectly.
Solution: Require majority (quorum) to elect leader.
Example: 5 nodes. Need 3 votes to become leader.
Network partition: 3 nodes in one group, 2 in another.
Group of 3: Can elect leader (has majority).
Group of 2: Cannot elect leader (no majority). Goes read-only.
Result: Only one leader ever exists.
This is why production systems use odd numbers (3, 5, 7 nodes). Ensures clear majority.
Problem: Lease-based election depends on time. Clocks can .
Scenario:
Leader thinks: "My lease expires at 10:00:30"
Follower thinks: "Leader lease expired at 10:00:25" (clock 5 seconds ahead)
Follower becomes leader while old leader still thinks it is leader. Split-brain!
Solution 1: NTP (Network Time Protocol) keeps clocks synchronized within milliseconds.
Solution 2: Use logical clocks instead of wall-clock time. Raft does this.
Best practice: Assume clocks can skew. Design for safety anyway.
Amazon provides library: DynamoDB Lock Client for leader election.
How to use:
AmazonDynamoDBLockClient client = new AmazonDynamoDBLockClient(
AmazonDynamoDBClientBuilder.defaultClient(),
"LeaderElection"
);
LockItem lock = client.acquireLock("my-service-leader");
if (lock != null) {
// I am the leader
while (doLeaderWork()) {
lock.sendHeartbeat(); // Maintain lease
}
lock.release();
}
Benefits: Proven, tested, handles edge cases. No need to implement from scratch.
What is ZooKeeper? Distributed coordination service. Used by Hadoop, Kafka, HBase.
Provides: Leader election, distributed locks, .
How leader election works:
Applications create ephemeral sequential nodes in ZooKeeper.
with lowest sequence number becomes leader.
Leader connection dies? ZooKeeper deletes ephemeral node. Next lowest becomes leader.
Example:
/election/leader-0000000001 (leader)
/election/leader-0000000002 (follower)
/election/leader-0000000003 (follower)
Leader crashes:
ZooKeeper deletes leader-0000000001
leader-0000000002 becomes new leader
Common problems:
Flapping leaders: Leader changes every few minutes. Usually network instability or resource exhaustion.
Stuck without leader: Election keeps failing. Usually split configuration or network partition.
Multiple leaders: Split-brain. Critical bug. Check quorum settings and network partition handling.
How to debug:
Logs: Track leadership changes. "Became leader", "Lost leadership", "Election started".
Metrics: Time spent without leader. Election duration. Leadership tenure.
Alerts: Alert when leadership changes more than once per hour (unusual).
Start with proven libraries: DynamoDB Lock Client, etcd client, ZooKeeper. Do not build your own unless necessary.
Use odd numbers: 3, 5, or 7 nodes for clear majority.
Monitor leadership: Track current leader, election frequency, leadership duration.
Test failure scenarios: Regularly kill leaders. Verify automatic recovery works.
Lease durations: Balance detection speed vs false positives. 30-60 seconds is typical.
Heartbeat frequency: Half of lease duration (15-30 seconds if lease is 60 seconds).
These patterns are battle-tested across billions of requests daily. Learn from them.