Retry

In distributed systems, calls to remote resources and services can fail for reasons that are short-lived, such as a brief network glitch, a momentary timeout, or a service that is busy for an instant. Many of these faults clear on their own within a short window.

The Retry pattern handles these transient failures by automatically attempting a failed request again, with the goal of eventually succeeding. A single retry, or a small number of spaced retries, often turns a request that would have surfaced as an error into one that quietly succeeds, so a temporary problem on the remote side never reaches your users.

When to retry

Retrying only helps when the fault is likely to be temporary. Connection resets, timeouts, and "service busy" style responses are good candidates. A retry should be spaced out rather than immediate, so the remote service is given time to recover instead of being hit again at once.

Retrying is the wrong tool when a failure is not transient. A malformed request or a permission error will fail again no matter how many times it is sent, so it should be returned to the caller without retrying. Likewise, when a remote service is down for a longer period, repeated retries waste resources and can keep pressure on a service that is already struggling.

When retrying is not enough

When a fault is sustained rather than brief, retrying stops helping, and hammering a service that will not recover just wastes resources. The circuit breaker handles this by failing fast and shedding load until the service is healthy again. Retry and circuit breaking are complementary: the first absorbs brief hiccups, the second protects the system from sustained outages.

How HARP handles transient failures

HARP applies this resilience at the proxy layer, outside your application code. Rather than blindly retrying the same failing endpoint, it steers traffic to a healthy one: it spreads calls across an active pool of remotes, falls back to a secondary pool when the active one thins out, and uses health-check probes and the circuit breaker to keep failing endpoints out of rotation until they recover.