Circuit Breaker Pattern: Distributed Fault Tolerance and State Machine Mechanics
A circuit breaker can limit repeated calls to a failing dependency and reduce cascading load. Closed, Open, and Half-Open are a common implementation model; thresholds and transitions are design choices.
Publication & corrections: infostoa · Editorial standards
Individual author and specialist reviewer information is not provided for this article.
1. When a slow dependency holds caller resources
A slow or unresponsive remote dependency can leave callers holding threads or connections until their timeouts expire. If concurrent calls accumulate, other work that needs those resources may also be affected. A circuit breaker is one way to reject further calls temporarily after observed failures reach a configured threshold, reducing waits and retry load. The affected scope and recovery outcome depend on the call graph and policy; a single dependency fault does not inevitably disable an entire system.
An electrical-breaker metaphor; it does not depict a specific software library’s state transitions or policy.
2. The Three-State Finite State Machine: Closed, Open, and Half-Open Transitions
A circuit breaker limits calls to a dependency when a chosen failure signal suggests that repeated attempts are wasting resources or amplifying an outage. Fowler’s 2014 article presents a simple implementation and credits Michael Nygard’s book *Release It!* with popularizing the pattern; Fowler is not the originator. Closed, Open, and Half-Open form a common state model, but implementations may add states or use different transition rules.
3. Possible Threshold Policies: Windows, Latency, and Recovery Delays
Threshold policy depends on the implementation and failure signal. Fowler’s simple Ruby example does not use a 50% rate: it counts Timeout::Error failures, opens at a threshold of five, and resets the counter after a successful call. The 50% rate appears later as a separate, more sophisticated alternative. Neither value is a recommended default. Microsoft likewise describes thresholds within an observation period without prescribing one universal window or sample count. Choose the window, minimum sample, latency rule, and counted error classes for the actual traffic; keep expected business rejections and invalid input out of the dependency-failure signal.
Consider a hypothetical policy to make the decision concrete: evaluate a 50% threshold only after at least 10 eligible dependency calls, excluding expected business rejections. Five failures among nine calls do not meet the minimum sample, so the breaker stays unchanged while observation continues. Five timeouts and five successes among ten eligible calls meet the example threshold, so this policy opens the breaker. These invented values are not recommendations; verify that the chosen library supports the same window, minimum-sample, and error-filter rules.
Retry and circuit breaking have different jobs. Retry assumes a transient failure may succeed on another attempt; a breaker blocks calls when repeated failures suggest they are unlikely to succeed. When combined, retry logic should stop when the breaker reports a non-transient open state. Otherwise callers may keep re-issuing work while the breaker is trying to reduce load, amplifying the incident.
Fast rejection after a breaker opens is only part of the protection. If the protected remote call has an excessively long timeout, many requests can still occupy threads or connections before the threshold is reached. Both Microsoft and Fowler describe how waiting for remote calls can consume resources and contribute to cascading failure. Check that the call timeout and failure aggregation work within the system’s resource budget.
4. Operational choices after calls are blocked
The design question is the response contract after calls are blocked: can a read return explicitly stale data, can a safe write be queued, or must the caller return an error? Half-Open can admit a limited number of trial calls so a recovering dependency is not flooded at once; the permitted volume and reopening rule depend on the implementation. The Microsoft pattern describes reopening on a failed trial in its state model, while Fowler’s simple example resets its wait after a failed trial. Neither should be generalized into one invariant for all libraries.
Record state transitions and observe successful as well as failed calls to check whether the threshold tracks the dependency. Where independent shards or providers exist, avoid letting one breaker block healthy resources. Sharing state across instances is an architecture choice based on consistency needs and the latency and availability costs of shared coordination, not a requirement of the pattern. For example, combining failures from independent data shards in one counter can block calls to a healthy shard when only one is failing. The breaker’s failure domain should match the dependency boundary.