Data Consistency — Why It Becomes the Hardest Problem in Micro Services
1. The Moment Consistency Stops Being Guaranteed
Up to this point in the series, the shift from monolith to microservices has already changed how the system behaves under load and failure. But there is one area where the change is deeper than it first appears – that area is data.
In a monolith, consistency is something you rarely think about explicitly. It is handled by the database – Transactions ensure that changes either happen completely or not at all. Once a transaction is committed, every part of the system sees the same state. This creates a very strong mental model, there is one source of truth, and it is always correct.
When you move to microservices, this model quietly breaks. There is ideally no single database anymore, each service owns its own data, each one updates its state independently. There is no shared transaction boundary that ties them together. At that point –
consistency is no longer guaranteed by the system – it becomes something you have to design.
2. Why a Single Source of Truth No Longer Exists
In a distributed system, data is intentionally split across services. Order Service stores orders – Payment Service stores transactions – Inventory Service stores stock levels. Each service is responsible for its own data and does not directly control others.
This separation is necessary for independence and scalability, but it introduces a new challenge. There is no single place where the complete state exists in real time.
If you want to understand the full picture of a request, you have to look across multiple services. And those services may not be perfectly in sync at any given moment.
For a short period of time, different parts of the system may have different versions of the truth, this is not a failure – it is a natural outcome of distribution.
3. Understanding Eventual Consistency in Practice
To deal with this, distributed systems rely on a concept called eventual consistency. Instead of guaranteeing that all services are consistent at the same moment, the system guarantees that they will become consistent over time.
When one service updates its data, it communicates that change to others, often through events or messages. Those services process the update when they receive it. This introduces a delay between when something happens and when every part of the system reflects that change. For example, an order may be created immediately, but inventory updates and notifications may happen slightly later. During that window, the system is temporarily inconsistent.
The important shift here is not technical—it is conceptual.
You move from expecting immediate correctness everywhere to accepting that correctness emerges over time.
4. When Delays Become Real Problems
While eventual consistency works in theory, it introduces challenges in real systems, the delay between updates can create confusion if not handled properly.
A user might see an order confirmed, but inventory might still show old data. A payment might be processed, but the order status may not yet reflect it. If the system is not designed carefully, these temporary inconsistencies can lead to incorrect decisions or poor user experience.
The problem is not that data is wrong, it is that different parts of the system are seeing it at different stages. This requires designing flows that can tolerate and correctly handle these intermediate states.
5. The Hidden Risk: Duplicate Operations
As soon as communication between services becomes asynchronous, another issue appears –
- Messages may be delivered more than once
- Requests may be retried
- Network failures may cause uncertainty about whether an operation completed.
This means the same action can be executed multiple times.
For example, a payment request might be processed twice if the system retries after a timeout but the original request was actually successful. This is not a rare edge case, it is a normal behaviour in distributed systems.
Handling this requires designing operations in a way that repeating them does not change the final outcome.
6. Why Idempotency Becomes Critical
This is where idempotency becomes essential. An idempotent operation is one that produces the same result no matter how many times it is executed.
In distributed systems, Idempotency is not just a good practice—it is a requirement.
If a payment request is received multiple times, the system must recognize that it has already been processed and avoid applying it again. This usually requires maintaining identifiers for operations and checking whether they have already been handled.
Without idempotency, the system cannot guarantee correctness in the presence of retries and duplicates.
7. Data Consistency Is Now a Flow, Not a State
One of the biggest mindset changes in microservices is how you think about data.
In a monolith, data is a state. You query it, and you get the current truth.
In microservices, data is a flow. Information moves from one service to another over time. The system’s overall state is the result of multiple updates happening across different services.
Understanding the system means understanding how data flows, not just where it is stored.
This is why designing events, message handling, and update sequences becomes as important as designing the data model itself.
8. Reconciling the System Over Time
Even with careful design, inconsistencies can still happen. Messages can be delayed. Services can fail temporarily. Updates may not propagate immediately.
To handle this, many systems introduce reconciliation processes. These are background mechanisms that periodically check for inconsistencies and correct them. For example, the system might verify that all completed payments have corresponding orders, or that inventory levels match expected values.
This is not a workaround, it is part of how distributed systems maintain long-term correctness.
Consistency is not enforced in a single step, it is maintained over time.
9. From Production Perspective
From a production perspective, data consistency is where most real-world issues appear, not because the system is broken, but because it behaves differently than expected.
Teams often assume that once an operation is completed, the entire system reflects that change immediately. In distributed systems, that assumption no longer holds. The first signs of issues are usually subtle:
- Temporary mismatches in data
- Duplicate processing of requests
- Delayed updates across services
These issues become visible under load, retries, or partial failures.
Stabilizing the system requires:
- Designing idempotent operations
- Handling delayed updates gracefully
- Introducing reconciliation where necessary
Over time, the system becomes reliable not because it avoids inconsistency, but because it manages it effectively.
Summary
Data consistency in microservices is fundamentally different from monoliths.
You move from:
- immediate consistency to eventual consistency
- a single source of truth to distributed data ownership
- guaranteed transactions to managed workflows
Consistency is no longer automatic. It is designed through:
- controlled data flow
- idempotent operations
- delayed synchronization
- reconciliation processes
Understanding this shift is critical, because in distributed systems, correctness is not something you get for free, it is something you build, maintain, and continuously verify.
Implementing an Idempotent Saga Step in Java
The Saga pattern replaces a distributed transaction with a sequence of local transactions, each paired with a compensating action. The critical property each step must have is idempotency — the same step executed twice must produce the same result as executing it once. Without idempotency, retries after partial failures corrupt state.
The pattern below shows the structure of a saga step with idempotency enforced at the database layer using a deduplication key. This is the minimum implementation required for safe retry behaviour in a distributed system.
// Java 11+ -- idempotent saga step with compensation; deduplication at persistence layer
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class PaymentSagaStep {
private static final Logger log = LoggerFactory.getLogger(PaymentSagaStep.class);
private final PaymentRepository repo;
public PaymentSagaStep(PaymentRepository repo) {
this.repo = repo;
}
// idempotencyKey is the saga correlation ID -- same key = same outcome, no double charge.
// The DB has a UNIQUE constraint on (idempotency_key, status).
public void execute(String orderId, long amountCents, String idempotencyKey) {
if (repo.existsByIdempotencyKey(idempotencyKey)) {
log.info("saga step already executed orderId={} key={} -- skipping", orderId, idempotencyKey);
return; // idempotent: safe to call again after a retry or timeout
}
try {
repo.chargeAndRecord(orderId, amountCents, idempotencyKey);
log.info("payment recorded orderId={} amount={}", orderId, amountCents);
} catch (Exception e) {
log.error("payment step failed orderId={} -- compensation required", orderId, e);
throw e; // caller triggers the compensating transaction
}
}
// Compensation: reverse the charge if a downstream step in the saga fails.
public void compensate(String orderId, String idempotencyKey) {
if (!repo.existsByIdempotencyKey(idempotencyKey)) {
log.info("nothing to compensate orderId={} key={}", orderId, idempotencyKey);
return;
}
repo.reverseCharge(orderId, idempotencyKey);
log.info("payment reversed orderId={}", orderId);
}
}Code language: Java (java)The idempotencyKey is the linchpin. It must be generated before the saga starts, stored with the saga state, and passed to every step. If the caller retries after a timeout — not knowing whether the step succeeded — the duplicate check at the persistence layer ensures no double execution. The UNIQUE constraint in the database is the final enforcement layer that makes the check reliable even under concurrent retries.
Interview Questions
What happens if a saga compensation step itself fails midway through reverting a partially committed distributed transaction?
Root Cause: Compensating transactions are not guaranteed to succeed. Network failures, database unavailability, or downstream service downtime can prevent a compensation from completing. Internal Behaviour: If the orchestrator does not retry compensations with persistence, the saga is left in an intermediate state — some steps executed, some compensated, some neither. No ACID rollback is available because each step committed to its own database. Production Impact: Data inconsistency that is invisible at the application level. The order service shows cancelled, the payment service shows charged, and the inventory service shows reserved — all three in different states. Fix: Persit the saga state machine in a durable store. Compensations must be retried with exponential backoff until they succeed. Design compensations to be idempotent for the same reason as forward steps. For steps that cannot be compensated (external payment processor calls), use a dedicated reconciliation job to detect and resolve inconsistencies out of band.
What issues arise when a saga step is not idempotent and the orchestrator retries it after a timeout with no response from the downstream service?
Root Cause: A timeout does not indicate failure — it indicates that the caller did not receive a response within the deadline. The downstream step may have succeeded, failed, or be in progress. Without idempotency, a retry on a step that already succeeded executes it twice. Internal Behaviour: For a payment step, this means the customer is charged twice. For an inventory reservation, the stock is decremented twice. The second execution has no context from the first and applies the same mutation again. Production Impact: Duplicate charges, over-reservation, or double-sending of notifications. These errors are often discovered by customers before operations teams. Fix: Enforce idempotency at the persistence layer with a unique constraint on the idempotency key. Generate the key at the saga start, store it in the saga state record, and pass it through every step. The check must be inside the same transaction as the mutation — a separate lookup followed by an insert has a race condition under concurrent retries.
