X402.Facilitator.Failover (X402 v0.9.0)

Copy Markdown View Source

Multi-endpoint failover policy for X402.Facilitator.

A facilitator client may be started with :fallbacks — additional endpoints tried in order when the primary fails — and a :failover policy. Each endpoint carries its own URL, auth, and transport settings; a per-endpoint circuit breaker (kept in the facilitator process) skips endpoints that recently failed for :cooldown_ms, so a dead primary does not add a timeout to every call until it recovers.

What fails over

  • verify, supported, list_resources, and search_resources fail over on transport errors, timeouts, and 5xx responses. They never fail over on 4xx or on a facilitator that answered (a verification failure is an answer, not an outage).
  • settle fails over only when the request provably never reached the endpoint: connection refused, DNS resolution failure, unreachable host or network. It never fails over on a TLS alert, a timeout, a closed connection, or a 5xx — the facilitator may have broadcast the settlement before the failure, and retrying elsewhere could settle the same authorization twice (or, for nonce-consuming schemes, surface a confusing "already used" error while the money has moved). Callers that need at-least-once settlement should retry the same endpoint and reconcile through the pending-settlement flow. With fallbacks configured, settlement disables per-endpoint HTTP retries so an ambiguous attempt cannot be hidden by a later failure. Without fallbacks, existing single-endpoint retry behavior is unchanged.

Every failover emits [:x402, :facilitator, :failover] with metadata :operation, :from and :to (endpoint URLs), :reason (the X402.Facilitator.Error type), and :status.

Summary

Types

breaker()

Open circuits: endpoint URL to the monotonic millisecond it reopens.

endpoint()

One facilitator endpoint (the primary or a fallback).

operation()

Facilitator operation names.

policy()

Validated failover policy.

Functions

build_endpoints(primary, fallbacks)

Builds the fallback endpoint list, inheriting unset transport settings from primary.

failover?(arg1, error)

Whether error from operation should be retried on the next endpoint.

order(endpoints, breaker, policy, now_ms)

Orders endpoints for an attempt: healthy ones first, then those in cooldown.

run(endpoints, operation, request, trip)

Runs request against each candidate endpoint in turn.

undelivered?(reason)

Whether a transport error reason proves the request never reached the server.

Types

breaker()

@type breaker() :: %{optional(String.t()) => integer()}

Open circuits: endpoint URL to the monotonic millisecond it reopens.

endpoint()

@type endpoint() :: %{
  url: String.t(),
  finch: term(),
  auth: term(),
  max_retries: non_neg_integer(),
  retry_backoff_ms: non_neg_integer(),
  receive_timeout_ms: non_neg_integer()
}

One facilitator endpoint (the primary or a fallback).

operation()

@type operation() ::
  :verify | :settle | :supported | :list_resources | :search_resources

Facilitator operation names.

policy()

@type policy() :: %{max_attempts: pos_integer() | nil, cooldown_ms: non_neg_integer()}

Validated failover policy.

Functions

build_endpoints(primary, fallbacks)

(since 0.9.0) 
@spec build_endpoints(endpoint(), [keyword()]) :: [endpoint()]

Builds the fallback endpoint list, inheriting unset transport settings from primary.

Examples

iex> primary = %{url: "https://a", finch: F, auth: nil, max_retries: 2, retry_backoff_ms: 100, receive_timeout_ms: 5_000}
iex> X402.Facilitator.Failover.build_endpoints(primary, [[url: "https://b", auth: nil, receive_timeout_ms: 1_000]])
[%{url: "https://b", finch: F, auth: nil, max_retries: 2, retry_backoff_ms: 100, receive_timeout_ms: 1_000}]

failover?(arg1, error)

(since 0.9.0) 
@spec failover?(operation(), term()) :: boolean()

Whether error from operation should be retried on the next endpoint.

Examples

iex> alias X402.Facilitator.Error
iex> X402.Facilitator.Failover.failover?(:verify, %Error{type: :http_error, status: 503})
true
iex> X402.Facilitator.Failover.failover?(:verify, %Error{type: :http_error, status: 400})
false
iex> X402.Facilitator.Failover.failover?(:verify, %Error{type: :timeout})
true
iex> X402.Facilitator.Failover.failover?(:settle, %Error{type: :timeout})
false
iex> X402.Facilitator.Failover.failover?(:settle, %Error{type: :http_error, status: 502})
false
iex> X402.Facilitator.Failover.failover?(:settle, %Error{type: :transport_error, reason: %{reason: :econnrefused}})
true
iex> X402.Facilitator.Failover.failover?(:verify, {:hook_halted, :before_verify, :nope})
false

order(endpoints, breaker, policy, now_ms)

(since 0.9.0) 
@spec order([endpoint()], breaker(), policy(), integer()) :: [endpoint()]

Orders endpoints for an attempt: healthy ones first, then those in cooldown.

Endpoints whose circuit is open are still tried — last — so an outage of every endpoint degrades to the plain single-endpoint behaviour rather than failing without a request. The list is capped at policy.max_attempts.

Examples

iex> endpoints = [%{url: "https://a"}, %{url: "https://b"}, %{url: "https://c"}]
iex> breaker = %{"https://a" => 1_000}
iex> policy = %{max_attempts: nil, cooldown_ms: 100}
iex> X402.Facilitator.Failover.order(endpoints, breaker, policy, 500) |> Enum.map(& &1.url)
["https://b", "https://c", "https://a"]

iex> endpoints = [%{url: "https://a"}, %{url: "https://b"}]
iex> X402.Facilitator.Failover.order(endpoints, %{"https://a" => 1_000}, %{max_attempts: nil, cooldown_ms: 100}, 2_000) |> Enum.map(& &1.url)
["https://a", "https://b"]

iex> endpoints = [%{url: "https://a"}, %{url: "https://b"}, %{url: "https://c"}]
iex> X402.Facilitator.Failover.order(endpoints, %{}, %{max_attempts: 2, cooldown_ms: 100}, 0) |> Enum.map(& &1.url)
["https://a", "https://b"]

run(endpoints, operation, request, trip)

(since 0.9.0) 
@spec run(
  [endpoint()],
  operation(),
  (endpoint() -> {:ok, term()} | {:error, term()}),
  (String.t() ->
     :ok)
) ::
  {:ok, term()} | {:error, term()}

Runs request against each candidate endpoint in turn.

request receives an endpoint and returns {:ok, result} | {:error, reason}. The first success, or the first error that is not eligible for failover (failover?/2), is returned; an eligible error trips the endpoint's breaker through trip and is retried on the next candidate. The last candidate's error is returned as-is. With a single endpoint the request runs exactly once.

undelivered?(reason)

(since 0.9.0) 
@spec undelivered?(term()) :: boolean()

Whether a transport error reason proves the request never reached the server.

Accepts a connection-establishment POSIX/DNS reason, invalid transport options, or a transport error struct (Mint.TransportError) wrapping one. TLS alerts do not identify when they occurred and are ambiguous.

Examples

iex> X402.Facilitator.Failover.undelivered?(:econnrefused)
true
iex> X402.Facilitator.Failover.undelivered?(%{reason: :nxdomain})
true
iex> X402.Facilitator.Failover.undelivered?(%{reason: {:tls_alert, {:handshake_failure, ~c"bad"}}})
false
iex> X402.Facilitator.Failover.undelivered?(:timeout)
false
iex> X402.Facilitator.Failover.undelivered?(%{reason: :closed})
false
iex> X402.Facilitator.Failover.undelivered?(:econnreset)
false