Health Monitoring

Calling a remote API that is slow, degraded, or down is one of the most common sources of failure in a distributed system. Health monitoring is the practice of continuously assessing whether a remote service is fit to receive traffic, so that requests can be steered away from unhealthy endpoints before they turn into errors for your users.

HARP monitors remote health in two complementary ways: passively, by observing the responses that flow through the proxy, and actively, with a healthcheck probe that requests a known URL on a regular schedule. An endpoint whose error rate is too high, or whose probe stops succeeding, is marked unhealthy and taken out of the active pool until it recovers.

Active healthcheck probes

A probe periodically requests a specific URL on each remote and uses the response to decide whether the endpoint is healthy. You configure the HTTP method, the path, any headers to send, and a timeout.

proxy:
  endpoints:
    - name: httpbins
      port: 4001
      remote:
        endpoints:
          - { url: "https://api1.example.com/" }
          - { url: "https://api2.example.com/" }
        probe:
          method: GET
          path: /health
          headers: { x-probe: "true" }
          timeout: 10

See the proxy documentation for the full set of options.

Reacting to unhealthy remotes

Detecting a failing remote is only useful if the proxy can act on it. HARP combines health monitoring with two reliability mechanisms.

When you declare several remote URLs, they are used in a round-robin pool. You can tag some of them as a fallback pool and set a minimum size for the active pool. If the number of healthy active endpoints drops below min_pool_size, the fallback endpoints take over.

proxy:
  endpoints:
    - name: httpbins
      port: 4001
      remote:
        min_pool_size: 2
        endpoints:
          - { url: "https://api1.example.com/" }
          - { url: "https://api2.example.com/" }
          - { url: "https://api3.example.com/", pools: [fallback] }
          - { url: "https://api4.example.com/", pools: [fallback] }

The Circuit Breaker works hand in hand with this. When failures are detected on a URL, configurable through break_on (for example http_5xx or network_error), the breaker opens and traffic stops flowing to that URL. After a delay it moves to a half-open state to test recovery, then closes again once the URL responds normally. If every URL is unavailable, the proxy returns a 503 to the client until a circuit closes. The circuit-breaker documentation covers the states and thresholds in detail. For short-lived faults, the Retry pattern can resolve the call before any of this is needed.

Monitoring the proxy itself

Health monitoring also applies to HARP as a running service. It exposes a /healthz endpoint that liveness and readiness probes, such as Kubernetes', can call to confirm the process is healthy, and it publishes Prometheus metrics so you can watch how the proxy and its remotes behave over time from your existing monitoring stack.