EXERCISE
1Large systems use circuit breakers extensively. Understanding production patterns and tools helps you implement them effectively.
Save
Netflix pioneered circuit breaker patterns at massive scale.
Hystrix (now in maintenance mode) was their circuit breaker library. Handled billions of requests daily across thousands of services.
Key :
Circuit breaking: Automatic failure detection and circuit opening
Fallbacks: Built-in fallback mechanism
Real-time : Dashboard showing all circuit states
Thread pool isolation: Each service call uses separate thread pool, preventing one slow service from exhausting all threads
Metrics: Detailed success/failure/timeout tracking
Resilience4j (Java): Hystrix successor. Lighter weight, functional approach.
Polly (.NET): Circuit breakers for C# applications.
Opossum (): Circuit breaker for /Node backends.
Istio (Service Mesh): Circuit breaking at infrastructure level, not application code.
All provide similar functionality: automatic failure detection, fallbacks, monitoring.
Traditional: Each service implements circuit breaker logic in code.
Service Mesh (Istio, Linkerd): Circuit breakers in infrastructure layer.
Benefits:
Language agnostic: Works regardless of programming language
Centralized configuration: One place to manage all circuit breakers
Automatic application: No code changes needed in services
Consistent behavior: All services use same circuit breaker logic
Example Istio configuration:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: profile-service-circuit-breaker
spec:
host: profile-service
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
http1MaxPendingRequests: 50
maxRequestsPerConnection: 2
outlierDetection:
consecutiveErrors: 5
interval: 30s
baseEjectionTime: 60s
Translation: After 5 consecutive errors within 30 seconds, stop sending traffic for 60 seconds.
All services calling profile-service get this protection automatically.
Large systems use circuit breakers at multiple levels.
Application Layer:
Service A → [Circuit Breaker] → Service B
Application code checks circuit breaker before making call.
Layer:
Client → [API Gateway with Circuit Breaker] → Service A → Service B
API Gateway protects backend from excessive traffic during failures.
Infrastructure Layer (Service Mesh):
Service A → [Sidecar Proxy with Circuit Breaker] → Service B
Traffic intercepted at network level, circuit breaker applied transparently.
Amazon:
Circuit breakers protect checkout flow. If recommendation service fails, checkout still works. Users complete purchases without recommendations.
Uber:
If surge pricing calculation fails, rides still happen at base price. Circuit breaker prevents pricing issues from blocking rides.
Twitter:
If timeline ranking service fails, show chronological timeline. Circuit breaker ensures users still see tweets, just without personalization.
Challenge: In , multiple instances of Service A call Service B.
Question: Should circuit breakers be per-instance or global?
Per-Instance Circuit Breakers:
Each Service A instance has own circuit breaker. Instance 1 might open circuit while Instance 2 keeps calling.
Advantage: Localized failure handling
Disadvantage: Service B still gets hammered by other instances
Global Circuit Breakers:
Shared circuit breaker state across all Service A instances. When one instance opens circuit, all instances stop calling.
Advantage: Complete protection for Service B
Disadvantage: Requires coordination (shared or cache)
Hybrid Approach:
Per-instance circuit breakers + global trip mechanism. Any instance can trigger global "Service B is down" signal.
Early days (pre-2010): Manual service disable flags. Engineers manually flipped switches.
Netflix era (2012-2018): Automated circuit breakers in application code. Hystrix popularized the pattern.
Service mesh era (2018-present): Circuit breakers in infrastructure. Configuration over code.
Future: ML-based adaptive circuit breakers. Automatically tune thresholds based on historical patterns.
Start simple: Manual flags in database. Graduate to automation.
Monitor everything: Know when circuits open, how long they stay open, when they recover.
Test regularly: Chaos engineering. Intentionally break services to verify circuit breakers work.
Tune continuously: Adjust thresholds based on real behavior, not guesses.
Combine patterns: Circuit breakers + retries + timeouts + fallbacks = resilient system.
Circuit breakers are not silver bullets. They are one tool in your resilience toolkit. Combined with , health checks, retries, and monitoring, they create systems that gracefully handle failures instead of collapsing completely.