Circuit Breaker

In distributed systems, calls to remote services fail for many reasons: slow networks, timeouts, temporary unavailability. Short-lived faults can be smoothed over with the Retry pattern, which retries after a brief pause. But some failures, like partial connectivity loss or a full outage, take longer to resolve, and retrying them only wastes resources. The system needs to recognize those quickly and stop trying.

This matters under load, where one failing component can cascade: callers block waiting for timeouts, tying up memory and threads until they are exhausted and other parts of the system break too. The fix is to fail fast when success is unlikely, rather than wait for every call to time out.

Implementation

The Circuit Breaker pattern detects failures and stops repeated, unsuccessful attempts. When failures are detected, the circuit trips and subsequent calls fail immediately. This isolates faults, lets the failing service recover, and prevents cascading failures across the system.

A circuit breaker acts as a proxy for operations that may fail. It tracks recent failures and uses that signal to decide whether to forward a call or reject it right away.

The circuit breaker can be seen as a state machine that mediates remote calls:

• Closed: Forwards requests to the remote service, distributing them across the active pool URLs in round-robin. If failures are detected, the affected URL moves to the Open state.
• Open: Stops routing requests to the failing URL, reducing pressure on the struggling service. After a configurable delay, the URL switches to Half-Open to test whether it has recovered.
• Half-Open: Tests the URL with traffic again. On success, the URL returns to the active pool, provided the configured criteria are met. On failure, it reverts to Open and the delay restarts, so a recovering service is not overwhelmed.

In HARP, the circuit breaker operates at the level of a remote's URL pool. You declare which failures trip the breaker with break_on: http_5xx and network_error are the usual choices, while http_4xx is best left out, since a 4xx is a client error rather than the upstream failing. You set min_pool_size to define how many active URLs must remain before HARP falls back. When the active pool drops below that size, HARP activates any configured fallback pool, and if no viable URL is left it returns an HTTP 503 to the client.

To limit the errors that reach clients, you can configure a probe that periodically checks endpoint health. HARP probes every URL, including those currently outside the active pool, so a recovered endpoint can rejoin as soon as it is healthy. The circuit breaker is disabled by default and is enabled by selecting a liveness algorithm, with naive and leaky (a leaking bucket) as the documented options.