Design Patterns — How Microservices Actually Survive in Production
1. Why Patterns Become Necessary (Not Optional)
By the time a system reaches the failure modes discussed earlier, one thing becomes clear. The system is not failing because developers wrote incorrect logic. It is failing because the architecture does not handle distributed behaviour properly.
In a monolith, correctness is enforced by the platform itself. Transactions guarantee consistency. Method calls guarantee execution. Failures are contained within a single process.
Once the system becomes distributed, those guarantees disappear.
What replaces them is not another framework or tool. It is a set of design patterns that define how the system should behave under uncertainty.
These patterns are not optimizations. They are control mechanisms. Without them, the system behaves unpredictably under load and failure.
2. The Core Problem: You No Longer Control Execution
Before going into patterns, it is important to understand what has fundamentally changed.
In a monolith, execution is controlled. When a function is called, it executes immediately. When a transaction starts, it either commits or rolls back.
In microservices, execution is indirect, a service sends a request and waits. The response may come late, may fail, or may not come at all. Multiple services may act independently without a central coordinator.
This means the system must be designed to handle incomplete, delayed, and repeated execution.
Design patterns exist to bring structure to this uncertainty.
3. Managing Distributed Transactions: The Saga Pattern
One of the first problems that appears in microservices is the loss of a single transaction boundary.
Operations that were once atomic are now spread across services. Each service updates its own data independently. There is no shared rollback mechanism. This creates situations where part of the operation succeeds while another part fails.
The Saga pattern exists to handle this exact problem. Instead of relying on a single transaction, the system breaks the operation into a sequence of steps. Each step completes independently and moves the system forward.
If a step fails, the system does not roll back automatically. Instead, it executes compensating actions to undo the completed steps.
For example, if a payment fails after an order is created, the system must explicitly cancel the order and release any reserved resources.
What makes Saga important is not just the sequence, but the responsibility it introduces.
The system must now:
- Track the state of each step
- Decide what to do on failure
- Ensure compensation is reliable
This shifts the responsibility of consistency from the database to the application.
4. Controlling Failure Propagation: The Circuit Breaker
In distributed systems, failures rarely stay isolated. When one service slows down or becomes unavailable, other services continue calling it. These calls wait, retry, and eventually consume resources. This is how small failures turn into system-wide issues.
The Circuit Breaker pattern is designed to stop this behaviour early.
Instead of continuously calling a failing service, the system monitors failures. When failures cross a threshold, it stops making calls temporarily. During this period, the system can either return a fallback response or fail fast.
Internally, this changes how the system behaves under stress. Instead of allowing threads to block indefinitely, the system actively prevents resource exhaustion.
Circuit Breaker Pattern does not fix the failing service. It prevents the failure from spreading.
5. Isolating Resource Usage: The Bulkhead Pattern
One of the key reasons cascading failures occur is shared resource usage. When multiple operations share the same thread pool or connection pool, a slow dependency can consume all available resources.
The Bulkhead pattern addresses this by isolating resources.
Different parts of the system are given separate pools. For example, calls to external services may use a different thread pool than internal processing. This ensures that a failure in one area does not affect others.
The idea is similar to compartments in a ship. If one section is flooded, the entire ship does not sink.
In microservices, this isolation prevents one failing dependency from taking down the entire system.
6. Handling Communication Flexibility: Event-Driven Architecture
One of the underlying causes of failure in microservices is tight coupling through synchronous communication. When one service calls another and waits for a response, it becomes dependent on that service’s availability and performance.
Event-driven architecture changes this interaction model. Instead of direct calls, services communicate through events. One service emits an event, and others react to it independently.
This removes immediate dependency between services. Internally, this introduces a different execution model.
- Operations become asynchronous
- Services process events at their own pace
- Failures do not immediately block upstream systems
However, this also introduces complexity in tracking system state, since operations are no longer linear.
Event-driven systems trade immediate consistency for resilience and scalability.
7. Managing Entry Complexity: The API Gateway
As systems grow, the number of services increases. Without control, clients would need to interact with multiple services directly. This creates complexity in communication, security, and version management.
The API Gateway pattern provides a single entry point into the system.
Instead of clients calling multiple services, they interact with the gateway. The gateway routes requests to the appropriate services and may aggregate responses.
This simplifies client interaction and centralizes cross-cutting concerns like authentication, rate limiting, and logging.
Internally, this reduces coupling between clients and services, but introduces a new critical component that must be highly available.
8. The Hidden Requirement: Idempotency
Across all these patterns, one requirement keeps appearing repeatedly—idempotency.
In distributed systems, operations may be retried, messages may be delivered multiple times, failures may lead to reprocessing. If operations are not idempotent, repeated execution leads to inconsistent results.
For example, processing the same payment twice must not result in duplicate transactions. This requires designing operations in a way where repeating them produces the same outcome.
Idempotency is not a pattern on its own, but it is a foundational requirement for all distributed behavior.
9. From Production Perspective
In production environments, introducing these patterns is not an optimization step, it is a stabilization phase.
Systems that initially move to microservices often experience failures similar to those described earlier—latency issues, retries, partial states, and cascading failures.
These patterns are introduced gradually as those issues appear.
- Saga handles incomplete workflows
- Circuit Breaker prevents overload
- Bulkhead protects resources
- Event-driven architecture reduces tight coupling
Over time, the system becomes more resilient, not because failures disappear, but because the system learns to contain and manage them.
Summary
Microservices do not work reliably by default, they require deliberate design to handle distributed behaviour.
The patterns discussed are not optional improvements. They are mechanisms that define:
- how the system handles failure
- how it maintains consistency
- how it protects itself under load
Without these patterns, microservices systems remain fragile.
With them, the system becomes capable of operating under real-world conditions where delays, failures, and retries are normal.
Circuit Breaker: The State Machine That Prevents Cascades
A circuit breaker is the implementation of the decision to stop calling a service that is already failing. It exists because retrying a broken downstream service makes the failure worse — it adds load to a system that is already overloaded and keeps threads blocked in the caller. The circuit breaker gives the downstream service time to recover by stopping traffic at the source.
The three-state model — CLOSED, OPEN, HALF_OPEN — maps directly to observable production behaviour. Production systems use Resilience4j for this. The implementation below shows the state transitions explicitly, which is what matters when diagnosing why a circuit breaker is not recovering as expected.
// Java 11+ -- circuit breaker state transitions; production use Resilience4j
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class CircuitBreaker {
private static final Logger log = LoggerFactory.getLogger(CircuitBreaker.class);
public enum State { CLOSED, OPEN, HALF_OPEN }
private final int failureThreshold;
private final long openDurationMs;
private final AtomicInteger failures = new AtomicInteger(0);
private final AtomicLong openedAt = new AtomicLong(0);
private final AtomicReference<State> state = new AtomicReference<>(State.CLOSED);
public CircuitBreaker(int failureThreshold, long openDurationMs) {
this.failureThreshold = failureThreshold;
this.openDurationMs = openDurationMs;
}
public boolean allowRequest() {
State s = state.get();
if (s == State.CLOSED) return true;
if (s == State.OPEN) {
long elapsed = System.currentTimeMillis() - openedAt.get();
if (elapsed > openDurationMs) {
// Transition to HALF_OPEN: allow one probe request through.
state.compareAndSet(State.OPEN, State.HALF_OPEN);
log.info("circuit HALF_OPEN -- probing downstream after {}ms", elapsed);
return true;
}
return false; // still OPEN -- fail fast, no downstream call
}
return true; // HALF_OPEN -- let the probe through
}
public void recordSuccess() {
failures.set(0);
if (state.compareAndSet(State.HALF_OPEN, State.CLOSED)) {
log.info("circuit CLOSED -- downstream recovered");
}
}
public void recordFailure() {
int f = failures.incrementAndGet();
if (f >= failureThreshold && state.compareAndSet(State.CLOSED, State.OPEN)) {
openedAt.set(System.currentTimeMillis());
log.warn("circuit OPEN after {} failures", f);
} else if (state.compareAndSet(State.HALF_OPEN, State.OPEN)) {
openedAt.set(System.currentTimeMillis());
log.warn("probe failed -- circuit back to OPEN");
}
}
}Code language: Java (java)The HALF_OPEN state is the most important one to understand in production. It represents one probe request being allowed through. If that probe succeeds, the circuit closes and normal traffic resumes. If it fails, the circuit reopens and waits again. Without this state, a circuit breaker that reopened too early would immediately re-trigger the cascade it was protecting against.
Interview Questions
What happens if a circuit breaker transitions to HALF_OPEN, the probe request is sent, and the downstream service fails again?
Root Cause: HALF_OPEN allows exactly one request through to test recovery. If that request fails, the circuit must return to OPEN — not retry immediately. Internal Behaviour: The failure recorded in HALF_OPEN triggers another state transition back to OPEN, restarting the wait duration. If the open duration is fixed (not exponential), the circuit will keep probing at a fixed cadence even if the downstream service remains unavailable indefinitely. Production Impact: A flapping circuit — repeatedly HALF_OPEN then OPEN — produces periodic error spikes in the upstream service as probe requests fail. These appear as transient errors in dashboards, making the downstream failure look intermittent when it is actually total. Fix: Use exponential backoff on the open duration in Resilience4j (waitDurationInOpenState combined with automaticTransitionFromOpenToHalfOpenEnabled). This reduces probe frequency over time, matching the expected recovery curve of a restarting service.
What is the difference between retry with exponential backoff and a circuit breaker, and when does each make a failure worse rather than better?
Root Cause: Retry is designed for transient failures — a single request that failed due to a temporary network issue. A circuit breaker is designed for sustained downstream unavailability. They solve different problems. Internal Behaviour: Retry with no circuit breaker on a persistently unavailable service multiplies the load on that service. Each caller retries, amplifying request volume at exactly the moment the downstream service is least able to handle it — a retry storm. A circuit breaker without retry does not recover from genuinely transient single-request failures. Production Impact: Retry storms collapse downstream services that are recovering. They can extend outages by minutes or hours by preventing restart under load. Fix: Use both together with correct ordering — circuit breaker wraps the retry. The retry handles transient failures within a single call attempt. The circuit breaker detects sustained failure patterns and stops all calls until recovery is confirmed. In Resilience4j, wrap a Retry inside a CircuitBreaker decorator, not the reverse.
