EXERCISE
1Learn the leader-worker pattern that breaks the infinite monitoring loop. Understand how systems automatically promote workers to leaders.
Instead of one Orchestrator, we have multiple Orchestrator instances.
One is the Leader. Others are Workers.
Architecture:
[Orchestrator Leader] ← Coordinates workers
↓
[Orchestrator Worker 1] [Orchestrator Worker 2] [Orchestrator Worker 3]
↓ each monitors
[API Servers]
Now we have redundancy and coordination.
Workers (Orchestrator Worker 1, 2, 3):
Do the actual work. Watch API servers. If API crashes, spin up replacement.
Why multiple workers? Redundancy. If one worker crashes, others continue.
Leader (Orchestrator Leader):
Does NOT monitor servers directly. Instead, monitors the workers.
If a worker crashes, Leader spins up a new worker.
Coordinates work distribution among workers (optional).
This is where leader election solves everything.
Scenario: Leader crashes at 2 AM.
Step 1: Workers notice. "Leader has not sent heartbeat for 30 seconds. Leader is dead."
Step 2: Workers initiate leader election among themselves.
Step 3: Election algorithm runs. One worker is chosen as new Leader.
Step 4: Chosen worker promotes itself to Leader role.
Step 5: Other workers recognize the new Leader.
Step 6: System continues operating normally.
Total downtime: 30-60 seconds. Fully automatic. No human intervention.
Key insight: Workers monitor each other through leader election.
No need for "Who monitors the leader?" because workers monitor the leader by being ready to replace it.
The recursion stops here.
Kubernetes uses this exact pattern.
Leader: Kubernetes Controller Manager (runs on master node)
Workers: Kubernetes nodes running pods
What happens:
Leader manages cluster state. Schedules pods. Handles failures.
Leader node crashes? Another master (worker) becomes leader through election.
Kubernetes keeps running. Pods keep traffic. Nobody gets paged.
This is why can achieve extremely high .
PostgreSQL with Patroni (high-availability setup):
Leader: Primary database (handles writes)
Workers: Standby replicas
Primary database crashes?
No manual failover. No data loss (with synchronous replication). Self-healing.
How do workers know leader is alive?
Leader sends periodic heartbeat: "I am alive!" message every 10 seconds.
Workers track last heartbeat time.
If 30 seconds pass without heartbeat: Leader is dead. Start election.
Why 30 seconds? Balance between:
Production systems typically use 15-60 second timeouts.
When leader dies, workers do this:
Election takes 5-15 seconds typically.
During election, system operates in degraded mode but does not crash.
Scenario: 3 workers, 1 leader.
Time 0:00: Leader crashes. Workers elect Worker 1 as new leader.
Time 0:05: Worker 2 crashes. New leader (Worker 1) notices. Spins up Worker 4.
Time 0:10: New leader (Worker 1) crashes. Workers 3 and 4 elect Worker 3 as leader.
System keeps healing itself. No human intervention needed.
As long as at least one worker survives, system recovers.
Before leader election: Human on-call gets paged. Logs in. Investigates. Manually restarts failed component. Takes 15-60 minutes.
With leader election: System detects failure. Elects new leader. Continues operating. Takes 30-60 seconds. Human learns about it in morning summary email.
This is the difference between 99.9% uptime and 99.99% uptime.
Self-healing systems enabled by leader election are the foundation of modern high-availability infrastructure.