# Client failover concepts

> **For AI agents:** the complete documentation index is at [llms.txt](https://questdb.com/docs/llms.txt). Every page is available as markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` request header.

How QuestDB clients detect a failed primary and transparently switch to a healthy peer using multi-host addr lists, host-health classification, role filtering, and zone-aware selection.

<EnterpriseNote>
  Client failover is most useful with QuestDB Enterprise
  [primary-replica replication](/docs/high-availability/overview/). OSS users
  with a single instance gain limited benefit from multi-host configuration.
</EnterpriseNote>

When a QuestDB cluster fails over from one primary to another — whether through
a planned [in-place promotion](/docs/high-availability/failover/), a rolling
upgrade, or an unplanned outage — clients with a
single hard-coded address must be reconfigured and restarted. A failover-aware
client instead carries the full list of peers and walks that list automatically
when the current connection breaks.

This page explains the model. The user-facing knobs and examples live in
the [Configuration](/docs/high-availability/client-failover/configuration/)
page.

## What failover does

You give the client a comma-separated list of endpoints:

```
addr=node-a:9000,node-b:9000,node-c:9000
```

The client picks one, connects, and uses it until that connection breaks. When
it breaks, the client walks the rest of the list, classifies what it found at
each host, and either reconnects or surfaces a failure to your code. The exact
loop that drives this depends on whether you are ingesting (long-lived
background reconnect) or querying (per-request retry budget). Both loops share
the same primitives described here.

## Host health model

For every entry in `addr`, the client tracks two attributes: a **state** and a
**zone tier**.

### State

The state records what the client most recently observed when it tried that
host.

| State | When the client moves a host here |
|---|---|
| `Healthy` | The last connect attempt succeeded. |
| `Unknown` | The host has not been tried in this round, or its classification was reset. |
| `TransientReject` | The server returned `421` with `X-QuestDB-Role: PRIMARY_CATCHUP` — it is a primary that is still catching up after promotion. Expected to recover. |
| `TransportError` | TCP/TLS handshake failed, an HTTP upgrade returned a transient error code, or an established connection broke mid-stream. |
| `TopologyReject` | The server returned `421` with any role other than `PRIMARY_CATCHUP` (`PRIMARY`, `REPLICA`, `STANDALONE`, or an unrecognised token), or — on egress — a successfully-upgraded host whose `SERVER_INFO` role does not satisfy the requested `target=` filter. The host will not become usable without a topology change. |

A lower state in the table above is preferred when the client picks the next
host to try.

### Zone tier

Each host is also classified relative to the client's configured `zone=`:

| Zone tier | Meaning |
|---|---|
| `Same` | Server's advertised zone matches the client's `zone=` (case-insensitive), or `zone=` is unset, or `target=primary`. |
| `Unknown` | Server has not advertised a zone yet. |
| `Other` | Server advertised a different zone. |

Zone information is advertised by the server on a successful upgrade and
on `421` rejects. Once observed, the client remembers a host's zone tier for
the lifetime of that client — it persists across rounds and reconnects until
the host re-advertises a different zone.

`target=primary` collapses every host's zone tier to `Same` — writers must
follow the primary regardless of geography. Ingress is currently zone-blind in
both storage modes, so the `zone=` key is silently accepted on ingress
connections and only takes effect on egress.

### Selection priority

When the client needs to pick the next host, it sorts by the tuple `(state,
zone_tier)` lexicographically — state first, zone second. So a known-good host
in another zone wins against an untried local host. Within a tied bucket, the
order in your `addr=` list is preserved verbatim.

The client does **not** shuffle, randomise, or load-balance across peers.
Cluster-level load balancing is the responsibility of QuestDB's server-side
coordinators. If you need a different first-pick distribution across many
simultaneously-starting clients, rotate the connect string at deployment time.

## Sticky-Healthy across rounds

Once the client lands on a `Healthy` host, that host stays the priority pick on
the next round of failover — provided its zone tier is still `Same`. This
avoids unnecessary churn after a short blip: a momentary network glitch
doesn't promote a different node into the active slot just because it
happened to be probed first.

`Healthy` hosts in another zone are reset to `Unknown` between outages rather
than kept sticky. Otherwise a once-healthy cross-zone host would lock the
client out of probing local hosts after they recover.

## Role filter (`target=`)

The `target=` key controls which server role the client is willing to bind to:

| `target=` | STANDALONE | PRIMARY | REPLICA | PRIMARY_CATCHUP |
|---|---|---|---|---|
| `any` (default) | accept | accept | accept | accept (transient) |
| `primary` | accept | accept | reject (topology) | accept (transient) |
| `replica` | reject (topology) | reject (topology) | accept | reject (topology) |

`PRIMARY_CATCHUP` is a primary that has been promoted but has not yet caught
up to its predecessor's WAL — the client treats it as transient and retries
the same host (with a fresh round, no exponential backoff) until it becomes a
full `PRIMARY`. On an ingress sender this retry has no deadline; the producer
is bounded by buffer capacity rather than by elapsed time.

A `421 Misdirected Request` response **without** an `X-QuestDB-Role` header
is treated as a generic transport error, not a role reject — the client walks
to the next host but does not pin the rejecting host as topology-unreachable.

`target=replica` is intended for read-side workloads that explicitly want to
spread query load across read-only peers (see the egress flow below).

## Two failover contexts

Failover applies to both directions of QWP traffic, but the two contexts have
very different goals.

### Ingress (writes)

The ingress reconnect loop sits inside the store-and-forward I/O thread. It
runs continuously in the background, retrying through outages while the
producer keeps appending to the local buffer. There is no wall-clock give-up:
the loop retries an outage of any length, and what bounds your tolerance is
buffer capacity (`sf_max_total_bytes` and disk), not a timer.

- Initial backoff: `100 ms`
- Maximum backoff: `5 s`
- Per-outage budget: **none**. `reconnect_max_duration_millis` bounds only the
  blocking sync initial connect, and the running loop never consults it.
- Jitter: **equal-jitter** `[base, 2·base)` — non-zero lower bound damps
  reconnect storms when many producers share a cluster
- Inter-host pause within a round: **none** — the client walks the full
  address list as fast as `auth_timeout_ms` allows, paying one backoff
  sleep at round exhaustion

See the [store-and-forward concepts](/docs/high-availability/store-and-forward/concepts/)
page for how the reconnect loop interacts with the disk-backed segment ring.
Replay across a primary failover delivers at-least-once — see
[Delivery semantics](/docs/concepts/delivery-semantics/) for the DEDUP
requirement on affected tables.

### Egress (queries)

The egress failover loop wraps each `execute()` call on the read-side query
client. It is interactive: a slow failover is worse than a clear error, so
the budget is short:

- Initial backoff: `50 ms`
- Maximum backoff: `1 s`
- Total wall-clock budget: `30 s` (`failover_max_duration_ms`)
- Attempt cap: `8` (`failover_max_attempts`)
- Jitter: **full-jitter** `[0, base)` — a single-user query benefits from the
  lowest expected recovery time, and one client per workload removes the
  thundering-herd concern

The egress loop also respects the `target=` role filter and prefers same-zone
hosts when `zone=` is set.

## Error classification

Errors fall into three buckets, which drive the loop's response. The
classification depends on the client and whether it is making an initial
connection, reconnecting an established sender, or running a query.

### Terminal — bypass failover

The client surfaces the error to your code immediately. Retrying every host
will not help.

| Condition | Why terminal |
|---|---|
| HTTP `401` / `403` on a caller-awaited initial connection or query | After any eligible one-time token replacement, rejected credentials end the endpoint walk because credentials are cluster-wide. |
| Protocol-version failure in C, C++, and Rust 7.3.0+, or Python 6.1.0+ | Terminal during initial connection and established sender reconnects; authentication retries do not make version failures retryable. |
| Server-status reject (SF) | Application-layer reject; replay reproduces the same response. |

An established C, C++, or Rust 7.3.0+ sender, or Python 6.1.0+ sender, instead
retries HTTP `401` / `403` with capped backoff. The rejection ends the current
endpoint walk, not the reconnect loop. See
[Authentication is cluster-wide](#authentication-is-cluster-wide) for the
phase rules and other clients' behavior.

### Topology — handled inside the round

The host is demoted in the priority lattice; the client walks to the next host
within the same round. No exponential backoff is consumed.

- `421` + `X-QuestDB-Role: PRIMARY_CATCHUP` → `TransientReject`
- `421` + any other non-empty role, including unrecognised tokens → `TopologyReject`
- `SERVER_INFO.Role` does not match the requested `target=` (egress only)

If every host in a round role-rejects, ingress pays one fixed backoff sleep
(reset to `InitialBackoff`, no doubling) and starts a fresh round; egress
fails the current `execute()` call.

### Transient — enter backoff

TCP/TLS errors, `auth_timeout_ms` expiry, mid-stream send or receive failures,
`404` / `426` / `503` on upgrade, and retryable errors decoding server messages
are treated as transport failures. The client records `TransportError` and
walks to the next host.

Java also treats protocol-version mismatches as transport failures so that
a rolling upgrade does not lock out compatible peers. This is not a shared
retry rule: protocol-version failures remain terminal in the native/Python
clients listed above.

The Go query client doesn't resubmit after a server response it can't read
that may have ended the statement; see
[Query failover](/docs/connect/clients/go/#query-failover).

When a round exhausts with transient errors, the client sleeps for the
backoff interval and starts the next round. On the ingress sender the rounds
continue indefinitely; on the egress query client they are bounded by
`failover_max_attempts` and `failover_max_duration_ms`, which apply per
`execute()`.

## Mid-stream demotion

If a connection breaks mid-stream — for example, the receive pump throws after
a successful upgrade — the client marks the failed host as `TransportError`
**before** picking the next host. Without this ordering, the sticky-Healthy
rule would re-pick the same just-failed host as the priority candidate, and
the next attempt would target the broken node again.

This invariant only matters when you are reading client source code or
debugging a custom implementation. As a user, you observe it as "failover
moves off a broken node on the very next attempt, with no exponential delay
when at least one peer is healthy."

## Authentication is cluster-wide

A `401` or `403` on the HTTP upgrade ends the current endpoint walk, after
any eligible one-time token replacement: the client does not retry other
hosts. This does not necessarily terminate a sender's reconnect loop. The
assumption is that auth credentials are configured identically across the
cluster, so a credential failure against one node is a credential failure
against all of them. Trying every peer would spam their audit logs without
recovering.

In Java, a token provider can supply a new token for one same-host retry
after an eligible `401`. A Java sender that has connected before also
treats a rejected credential like an outage: it
keeps retrying with backoff, buffering rows, and reports each failed round to
its error handler, so ingestion resumes once QuestDB accepts the credential
again, such as after a token renewal. See
[Authentication failures and token providers](/docs/connect/wire-protocols/qwp-client-behavior/#authentication-failures-and-token-providers).

C, C++, and Rust **7.3.0 or later**, and Python **6.1.0 or later**, also retry
HTTP `401` / `403` with capped backoff once a sender has connected, including
with fixed credentials. They retain unacknowledged rows and report
`auth_failed` / `AuthFailed` connection events. A caller-awaited initial
connection still fails on rejected credentials. With a token provider, an
eligible `401` can first trigger one same-host retry if the token changes;
a `403` cannot. These authentication exceptions do not extend to
protocol-version failures, which remain terminal in the native/Python
clients.

The Node.js client also treats a rejected credential like an outage, for a
sender that has connected before and uses store-and-forward or was started
with `async`: it keeps retrying with backoff, buffering rows, and reports each
rejected attempt as an `auth-failed`
[connection event](/docs/connect/clients/nodejs/#connection-events). So does
a browser sender that authenticates with `sessionBootstrap` and was started
with `initialConnectMode: "async"`. A browser cannot see why an upgrade
failed, so a browser sender that authenticates with
[`auth`](/docs/connect/clients/browser/#credentials-for-applications-on-another-origin)
retries a rejected credential like an outage once it has connected, whatever
its mode, and reports `endpoint-attempt-failed` instead. For the same reason,
such a rejection does not end the endpoint walk: the browser client tries the
other hosts, as it would for an unreachable one.

If your deployment has per-host credentials, that is unsupported and outside
the failover model — split the workload into one connect string per credential.

## Next steps

- [Configuration](/docs/high-availability/client-failover/configuration/) —
  the connect-string keys and examples for each context.
- [Store-and-forward concepts](/docs/high-availability/store-and-forward/concepts/) —
  how the ingress failover loop interacts with the disk-backed substrate.