EXERCISE
1Leader election is powerful but not always the right choice. Understand the trade-offs, failure modes, and alternatives.
Irony: Leader election solves recursion but creates a new single point of failure: the leader itself.
What happens when leader is slow?
Leader becomes bottleneck. All decisions flow through leader. System performance capped by leader capacity.
Example:
Database cluster with leader-follower. Leader handles all writes.
Load increases. Leader saturated. Write throughput cannot increase. Adding followers does not help (they only handle reads).
Solution: Shard data across multiple leader-elected groups. More leaders = more write capacity.
Trade-off: Increased complexity.
Split-brain remains possible if leader election is misconfigured.
Real incident (anonymized company):
Setup: 3-node cluster. Quorum = 2 nodes.
Network partition: Node A isolated. Nodes B and C together.
Expected: B and C elect new leader. A goes read-only.
Bug: Faulty quorum config. Both sides thought they had majority. Two leaders.
Result: Data corruption. Conflicting writes to database. Hours of manual reconciliation.
Lesson: Test network partition scenarios religiously. Use proven libraries.
Leader election adds latency to system startup and recovery.
Without leader election: Service starts, immediately begins working. Startup: 2 seconds.
With leader election: Service starts, waits for election, then begins working. Startup: 10-15 seconds.
During leader failure: Election takes 5-15 seconds. System in degraded mode during this time.
Trade-off: Slightly slower startup and recovery for automatic failover.
Scenario: Web API servers. Stateless. Any server can handle any request.
Do you need leader election? No.
Better approach: Load balancer distributes traffic. Servers independently handle requests. No coordination needed.
Adding leader election here: Unnecessary complexity. Bottleneck. No benefit.
Scenario: Startup with 100 users. Single database. Single .
Do you need leader election? Not yet.
Better approach: Focus on product. Manual failover acceptable at this scale.
Adding leader election: Over-engineering. Time better spent on .
When to add: 10,000+ users, requirements, 24/7 operations.
Scenario: Social media feed. Eventual consistency acceptable. Multiple writers okay.
Do you need leader election? Probably not.
Better approach: Multi-master . Conflict resolution through CRDTs or last-write-wins.
Example: Cassandra. No leader. All nodes equal. Writes go to any .
Trade-off: Give up strong consistency for higher availability and write throughput.
Scenario: Financial transactions. Strong consistency required.
Leader election option: Leader handles all writes. Guarantees consistency.
Better option: Use managed database with built-in high (AWS RDS Multi-AZ, Google Cloud SQL).
Why better: teams already solved this problem. Do not reinvent the wheel.
