Master the art of protecting your most valuable asset: data. Learn backup strategies, replication techniques, and disaster recovery patterns that prevent catastrophic data loss.
Complete lesson & earn 250 PX
EXERCISE
1Understand why database failures are fundamentally different from API server crashes. Data loss can destroy businesses overnight.
EXERCISE
2Redundancy means keeping multiple copies of data. Simple concept. Massive impact. Understanding the fundamentals prevents disaster.
EXERCISE
3Learn practical backup approaches that protect against data loss. Start simple, scale as needed.
EXERCISE
4Backups protect against yesterday mistakes. Replication protects against right-now failures. Learn how to maintain live redundant copies.
EXERCISE
5Numbers tell the story. Learn key metrics that quantify system reliability and help you make informed decisions.
EXERCISE
6Learn how major companies implement redundancy at scale. These battle-tested patterns prevent disasters.
API Servers: Stateless
Your application has 5 API servers behind a load balancer. One server crashes at 3 AM.
What happens?
Nothing dramatic. Load balancer detects the crash. Stops sending traffic to that server. Other 4 servers continue handling requests. Users never notice.
In the morning: You spin up a new API server. Add it to the load balancer. Done. Total impact: Zero.
Why so easy? API servers hold no data. They process requests and return responses. Any server can handle any request. They are stateless.
Now your database crashes. Disk fails. Data corrupted.
What happens?
Complete outage. Every API becomes useless. They cannot read user data. Cannot write new data. Application is dead.
Spin up new database? Empty database. No users. No posts. No orders. No transaction history. Nothing.
Your business just lost everything.
GitLab Database Incident (2017):
Engineer accidentally deleted production database. 300GB of data gone. Backup systems had failed silently for months. They recovered only 6 hours of data from a delayed backup. Thousands of customer projects lost forever.
Impact: Trust shattered. Customers left. Reputation damaged permanently.
Code Spaces Shutdown (2014):
Hacker deleted their AWS resources including database backups. All customer data gone. Company had no recovery option.
Result: Business closed permanently. Could not operate without data.
Your E-commerce Store:
Database crashes during Black Friday. Disk corrupted. No backups.
Lost: Customer accounts, order history, payment information, inventory records, everything.
Outcome: Business cannot function. Customers cannot log in. Orders cannot process. Revenue stops. Company dies.
Data is the oil of your business. Everything else can be rebuilt.
Database? Cannot rebuild. User data is unique. Transaction history is unique. Once gone, gone forever.
API server crash:
Database crash:
Every production system MUST have database redundancy.
Not optional. Not "nice to have." Mandatory.
You can skip fancy . You can delay optimizations. You cannot skip data redundancy.
One database failure should never destroy your business.
Redundancy = Multiple Copies
You have critical data. Instead of storing it once, store it multiple times in different places.
Example:
Your database has 1 million user records. You create a second database with identical 1 million records.
Primary database crashes? Switch to the second database. Users keep logging in. Business continues.
That is redundancy.
Single Point of Failure:
One database. One disk. Disk fails. Everything gone.
Multiple Points of Survival:
Three databases. One disk fails. Two others keep running. Data safe.
Redundancy eliminates single points of failure.
Intentional Redundancy (Good):
You deliberately create copies for disaster recovery.
Example: Master database + Replica database. Both have same data on purpose.
Purpose: Protection. If master fails, replica takes over.
Accidental Redundancy (Bad):
Same data stored in multiple places by mistake. Often inconsistent.
Example: Customer email stored in Users table, Orders table, and Support Tickets table. One place updates, others do not. Now you have three different emails for same customer.
Problem: Inconsistency. Which email is correct? Nobody knows.
We want planned, synchronized copies for disaster recovery.
Not accidental duplicates that cause inconsistency.
Entire Database: Copy everything to another server.
Tables: Replicate specific tables to different databases.
Rows: Backup individual records across systems.
Files: Store database files in multiple locations.
Level depends on criticality. More critical = more redundancy.
"If this database disappears right now, can I recover?"
No redundancy: Answer is "No." Business dies.
With redundancy: Answer is "Yes." Switch to backup. Business survives.
Netflix:
Stores data across multiple AWS regions. US-East region goes down? Switches to US-West. Users keep watching. Nobody notices.
Redundancy cost: Millions in infrastructure.
Cost of outage without redundancy: Billions in lost revenue and reputation.
Redundancy is always cheaper than disaster.
Data redundancy is not expense. It is insurance.
You pay for redundancy hoping you never need it. But when disaster strikes, it saves everything.
Businesses without redundancy eventually face catastrophe. Businesses with redundancy survive and thrive.
How it works:
Day 0 (Sunday): Take complete backup of entire database. 100GB stored.
Day 1 (Monday): Backup only changes from Monday. Maybe 2GB of new/modified data.
Day 2 (Tuesday): Backup only Tuesday changes. Another 2GB.
Continue daily.
Total storage: 100GB (full) + 7 days × 2GB (incremental) = 114GB
Recovery: Restore full backup, then apply each incremental backup in order.
Storage efficient: Not storing full 100GB every day.
Recovery point: If database crashes Wednesday, you lose only Wednesday data. Tuesday backup available.
Acceptable loss: Most businesses can afford losing one day of data versus losing everything.
How it works:
Every Sunday, take complete snapshot of database regardless of size.
Why weekly complete backups matter:
Incremental backups can corrupt. If Monday incremental is corrupted, you cannot restore Tuesday onward.
Weekly complete backup is safety net. Even if daily incrementals fail, you have last Sunday full backup.
Example:
Week 1: Full backup Sunday (100GB)
Daily incrementals Monday-Saturday (2GB each)
Week 2: Full backup Sunday (105GB)
Daily incrementals Monday-Saturday
If Wednesday incremental corrupts: Restore Sunday full backup. Lose only 3 days instead of entire database.
The ultimate protection: Store backup in different geographic location.
Scenario: Your data center is in California. Earthquake destroys entire facility. Servers gone. Disks gone. Everything gone.
Without cross-region backup: Business ends.
With cross-region backup: Copy of database in Virginia data center. Switch to Virginia. Business continues.
Weekly process:
Bandwidth consideration: 30GB upload once per week is manageable.
Cost: Extra storage + bandwidth. Worth every penny.
Amazon RDS ( Service):
Offers automatic cross-region . Your database in us-east-1? Enable replica in eu-west-1.
AWS region failure: Happened in 2017. Multiple data centers down for hours.
Customers with cross-region redundancy: Switched regions. Stayed online.
Customers without: Complete outage. Lost revenue.
Local disk on same server: Fast but dangerous. Server dies, backup dies.
Separate backup server: Better. Database server crashes, backup survives.
Cloud storage (S3, Google Cloud Storage): Excellent. Cheap, reliable, geographically distributed.
Multiple cloud providers: Paranoid level protection. AWS down? Backup on Google Cloud. Both down? Extremely unlikely.
How long to keep backups?
Daily incrementals: Keep 7 days. Older than week? Delete.
Weekly complete: Keep 4 weeks (one month).
Monthly archives: Keep 12 months (one year).
Balance: Storage cost vs recovery options.
Backup you never tested is worthless.
Horror story: Company had backups for years. Database crashed. Tried restoring backup. Backup corrupted. All backups corrupted. Backup process had bug nobody noticed.
Best practice: Quarterly backup restoration test.
If restore fails, fix immediately. Backup system is broken.
Pessimism wins: Assume disaster will happen. When? Unknown. Could be today.
Prepared businesses survive. Unprepared businesses become cautionary tales.
Backups are boring. Disasters are exciting in the worst way. Choose boring.
Problem with daily backups: Lose up to 24 hours of data.
Example:
Last backup: Yesterday 11 PM
Database crashes: Today 10 PM
Data loss: 23 hours of user activity
For critical systems, this is unacceptable.
Replication: Maintain live, continuously updated copy of database.
Every write to main database immediately copies to replica.
Result: Replica is always up-to-date (within seconds).
Master Database (Primary):
Handles all writes. All reads (optionally). The "source of truth."
Replica Database (Standby):
Receives copies of all data changes. Can handle reads (optionally). Ready to become master if needed.
Visualization:
Application → Master Database (writes)
↓ replication
Replica Database (standby)
Synchronous Replication:
Write completes only after replica confirms.
Advantage: Zero data loss. Master and replica always identical.
Disadvantage: Slower writes. Network latency to replica adds delay.
Write completes immediately. Replication happens after.
Advantage: Fast writes. No latency penalty.
Disadvantage: Replica slightly behind. Master crashes before replicating? Recent writes lost.
Financial transactions: Synchronous. Cannot lose money data.
Social media posts: Asynchronous. Losing last few seconds acceptable.
Trade-off: Data safety vs performance.
Critical: Replica is standby only. Not user traffic.
Why?
Its only job: Stay synchronized with master. Be ready to take over instantly.
Serving user reads from replica? Increases load. Slows replication. Defeats the purpose.
Use separate read replicas for that (different pattern, different lesson).
Disaster scenario: Master database crashes. Disk failure. Unrecoverable.
Failover process:
Downtime: Seconds to minutes (depending on automation).
Data loss: None (synchronous) or minimal (asynchronous).
PostgreSQL Streaming Replication:
Master sends WAL (Write-Ahead Log) stream to replica. Replica applies changes in real-time.
Configuration:
-- On master
ALTER SYSTEM SET wal_level = replica;
ALTER SYSTEM SET max_wal_senders = 3;
-- On replica
CREATE SUBSCRIPTION my_subscription
CONNECTION 'host=master-db port=5432 dbname=mydb'
PUBLICATION my_publication;
Result: Replica stays synchronized. Failover ready.
Production pattern: Multiple replicas.
Application → Master
↓ replication
Replica 1 (US-East)
Replica 2 (US-West)
Replica 3 (Europe)
Master crashes: Promote closest replica. Other replicas sync from new master.
Maximum redundancy: Even if two replicas fail, third survives.
Lag: How far behind replica is from master.
Synchronous: Lag = 0 seconds (by definition).
Asynchronous: Lag = seconds to minutes.
Dangerous lag: Hours behind. Replica not keeping up.
Causes: Network issues. Replica overloaded. Configuration problems.
Monitor replication lag constantly. Alert if exceeds threshold (e.g., 60 seconds).
Infrastructure: 2-3x database servers.
Storage: 2-3x disk space.
Network: Bandwidth for replication traffic.
Worth it? Absolutely. Compare to cost of catastrophic data loss.
Companies that skip replication: Eventually learn expensive lesson.
"Our system is reliable" means nothing without numbers.
"Our system has 99.95% uptime" means something concrete.
Metrics help you:
Definition: Average time a component operates before failing.
Formula:
MTBF = Total Operating Time / Number of Failures
Example:
Database ran for 8,760 hours (one year). Failed 3 times.
MTBF = 8,760 hours / 3 failures = 2,920 hours per failure
Translation: On average, database fails every 2,920 hours (about 4 months).
Higher MTBF = More reliable
Server A: MTBF = 10,000 hours (fails once per year)
Server B: MTBF = 1,000 hours (fails every 6 weeks)
Server A is clearly more reliable.
Use case: Choosing hardware. Buy server with higher MTBF.
Definition: Average time to restore system after failure.
Formula:
MTTR = Total Downtime / Number of Failures
Example:
System failed 4 times last year. Downtimes: 2 hours, 1 hour, 3 hours, 2 hours.
Total downtime = 8 hours
MTTR = 8 hours / 4 failures = 2 hours per recovery
Translation: On average, takes 2 hours to fix system after failure.
Lower MTTR = Faster recovery
Team A: MTTR = 15 minutes (well-prepared, automated failover)
Team B: MTTR = 4 hours (manual process, hunting for problems)
Team A recovers 16x faster.
Use case: Measure operational efficiency. Optimize recovery procedures.
Definition: Percentage of time system is operational and accessible.
Formula:
Availability = (Uptime / Total Time) × 100%
Example:
One year = 8,760 hours
System down for 43.8 hours total
Uptime = 8,760 - 43.8 = 8,716.2 hours
Availability = (8,716.2 / 8,760) × 100% = 99.5%
90% (One Nine): Down 36.5 days/year. Unacceptable for production.
99% (Two Nines): Down 3.65 days/year. Barely acceptable.
99.9% (Three Nines): Down 8.76 hours/year. Standard for most businesses.
99.99% (Four Nines): Down 52.6 minutes/year. High availability. Requires redundancy.
99.999% (Five Nines): Down 5.26 minutes/year. Extreme availability. Expensive. Used by critical systems (911, hospital systems).
Google Search: Targets 99.99%. You almost never see "Google is down."
Netflix: Targets 99.99%. Millions watching. Downtime = millions lost.
Your startup: 99.9% probably sufficient. Don focus on five nines yet.
Question: How much downtime can business tolerate?
E-commerce during holidays: Very little. Every minute = lost sales.
Target: 99.99% (52 minutes/year)
Internal tool used by 50 employees: Some downtime acceptable.
Target: 99.5% (43 hours/year)
Match availability target to business impact.
99%: 1 database server. Cheap. Frequent outages.
99.9%: Master + replica. Moderate cost. Rare outages.
99.99%: Master + multiple replicas + auto-failover + . Expensive. Very rare outages.
99.999%: Multi-region, multiple data centers, complex . Very expensive. Almost never down.
Each additional nine costs roughly 10x more.
System A:
System B:
Which is better?
System B fails more often but recovers faster. Better availability despite more frequent failures.
Lesson: MTTR matters more than MTBF for availability. Fast recovery beats preventing all failures.
Start with business requirements:
Metrics turn reliability from vague goal into measurable outcome.
Active Redundancy: Multiple components work simultaneously.
Example: Load balancer with 3 API servers. All 3 serve traffic actively.
One fails? Other 2 continue seamlessly. No switchover needed.
Passive Redundancy: Backup component waits on standby.
Example: Master database + replica. Replica does nothing until master fails.
Master fails? Promote replica to master. Switchover required.
Active redundancy: Stateless components. Easy to add/remove. Load can distribute.
Use for: API servers, load balancers, caches.
Passive redundancy: Stateful components. Data synchronization complex. One leader needed.
Use for: Databases, message queues, coordination services.
Single Region Risk: Natural disaster, power grid failure, ISP outage affects entire region.
Multi-Region Solution: Deploy across multiple geographic locations.
Architecture:
United States (Primary)
├─ US-East Database (Master)
├─ US-East API Servers
└─ US-East Load Balancer
Europe (Replica)
├─ EU-West Database (Replica)
├─ EU-West API Servers
└─ EU-West Load Balancer
US region fails: Traffic routes to Europe. Database replica promoted. Service continues.
Problem: Serve 200+ million subscribers. Cannot tolerate regional outages.
Solution:
Data: Replicated across 3 AWS regions (US-East, US-West, EU-West)
Traffic routing: DNS-based global . User connects to nearest healthy region.
Failure scenario: US-East AWS region has outage (happened 2017, 2020).
Result: Netflix automatically routes users to US-West. Users keep watching. Most never notice.
Cost: 3x infrastructure. Worth it for business continuity.
Multi-AZ Deployment:
Master database in Availability Zone 1. Replica in Availability Zone 2 (different data center).
Automatic failover: Master fails? RDS promotes replica. Updates DNS. Application reconnects automatically.
Downtime: 60-120 seconds. Fully automated.
Developer experience: Simple checkbox "Enable Multi-AZ." AWS handles complexity.
Design: Database replicated across continents. Data synchronized globally.
Write: Goes to nearest replica. Replicated worldwide using Paxos consensus.
Read: Served from nearest replica. Low latency everywhere.
Failure: Any single data center dies. Others continue. Users never notice.
Trade-off: Higher latency for writes (global consensus). Perfect consistency.
Level 0 - No Redundancy:
Level 1 - Backups:
Level 2 - Replication:
Level 3 - Multi-Region:
Level 4 - Global Distribution:
Startup MVP: Level 1. Daily backups sufficient. Focus on product.
Growing SaaS: Level 2. Customers expect reliability. Replication worth cost.
Enterprise product: Level 3. SLAs require high availability. Multi-region necessary.
Global platform: Level 4. Millions of users. Downtime costs millions.
Start simple. Upgrade as business grows.
Question: Is redundancy worth the cost?
Calculate downtime cost:
Revenue/hour × Downtime hours = Lost revenue
Example:
E-commerce site: $10,000/hour revenue
Expected downtime without redundancy: 20 hours/year
Cost of downtime: $200,000/year
Redundancy cost: $10,000/year
Savings: $190,000/year
ROI: 1,900%
For most businesses, redundancy pays for itself many times over.
Netflix Chaos Monkey: Randomly terminates servers in production.
Purpose: Verify systems actually survive failures.
Result: Engineers forced to build resilient systems. Netflix can handle massive failures.
Your version:
Pessimism wins: Assume everything will fail eventually.
Redundancy is not paranoia. It is pragmatism.
Every major tech company learned this lesson the hard way. Now they build redundancy from day one.
Your system will face disasters. The only question: Will you survive them?
API servers can crash and restart easily because they are stateless. But databases hold your data—lose that and your business stops. Data redundancy means keeping multiple copies so you can always recover from disasters.