High availability and replicas

Puddle keeps no state of its own between requests: a replica holds no cache of tables, namespaces or metadata, and every request reads what it needs from the metastore and the warehouse. Whether you can run more than one replica depends only on the metastore.

SQLite: exactly one replica

SQLite serializes all writes through one connection inside the process. The guarantees below (atomic location checks, transactions that don’t lose updates) rely on that and don’t hold between two processes, so never point two puddle processes at the same database file, not even briefly:

• In Kubernetes, run one pod with strategy: Recreate (or a StatefulSet with one replica), so an update stops the old pod before it starts the new one. The default rolling update runs both at once.
• Keep the file on a local disk (a block volume, not NFS or another network file system): puddle runs SQLite in WAL mode, which needs shared memory between the processes that open the file.

A restart is downtime: requests that arrive while puddle is stopped fail (connection refused, which Iceberg’s clients don’t retry). It lasts as long as puddle takes to drain, stop and start, usually a few seconds.

PostgreSQL: multiple replicas

With type: postgresql, any number of replicas can share one database. Everything that must be atomic is done in the database, so it holds across replicas:

• Commits swap a table’s (or view’s) metadata pointer with a compare-and-swap on the current metadata location. Of two commits based on the same metadata, one wins; puddle retries the other against the new metadata (up to 4 times, as Java’s catalog does), and if its requirements still hold it lands too, otherwise the client gets 409 CommitFailedException and retries.
• Multi-table transactions (/transactions/commit) lock every table they change (SELECT … FOR UPDATE, in a fixed order, so two transactions can’t deadlock) and then check and swap all pointers in one database transaction. A concurrent single-table commit, drop or rename either lands before (and the transaction fails with 409) or after.
• Location checks (no two tables or views, or namespaces with a location, may overlap) take a transaction-scoped advisory lock per warehouse, so of two concurrent creates at overlapping locations at most one succeeds, whichever replica they reach. Only writes that set or change a location take it; ordinary commits don’t.
• Namespaces: concurrent property updates don’t lose each other’s keys (row lock), and dropping a namespace while a child namespace, table or view is being created in it either fails (Namespace … is not empty) or makes the create fail (NoSuchNamespaceException); nothing is left orphaned (foreign keys).
• Startup: puddle creates its schema (postgresql.schema, default puddle) at startup under a session advisory lock, so replicas starting at the same time don’t race.

Size the connection pool for the whole deployment: max-open-conns (default 10) is per replica, so replicas × max-open-conns must fit within the database’s max_connections (with room for anything else that connects).

Replicas are interchangeable: put them behind any load balancer, no session affinity needed. What each replica does keep for itself:

• JWKS: each replica fetches and caches its issuers’ signing keys (refresh every 15 minutes, and on an unknown kid at most every 30 seconds per issuer). N replicas make N times the requests to the IdP.
• Purges in progress: a drop with purgeRequested=true deletes the table’s files in the replica that handled it, possibly after the response (see Timeouts and purge). A replica that stops abandons its unfinished purges; see below.

Rolling restarts and shutdown

On SIGTERM a replica stops accepting connections, waits for requests in flight, and then for its background purges, all within server.shutdown-timeout (default 25s), and exits (non-zero if work was cut off; see Graceful shutdown).

• Keep shutdown-timeout below the platform’s grace period: in Kubernetes terminationGracePeriodSeconds (default 30s); for plain Docker, docker stop waits only 10s unless you pass -t (or run the container with --stop-timeout).

• A replica closes its listener as soon as it gets SIGTERM, and a load balancer may still send it new connections for a moment. In Kubernetes, delay the signal until the pod is out of the Service’s endpoints with a preStop sleep (the image has no shell, so use the built-in sleep action, Kubernetes 1.30 and later) and add it to the grace period:

  spec:
    terminationGracePeriodSeconds: 40
    containers:
      - name: puddle
        lifecycle:
          preStop:
            sleep:
              seconds: 5

• Requests still running at the end of shutdown-timeout are cut off, and their clients see a connection error. Requests that reach a replica after it closed its listener are refused. Iceberg’s clients retry neither; they do retry a 503 (and a 500) on idempotent requests, such as a request that ran into its deadline. A commit cut off this way may or may not have landed; as after any network failure, the client has to reload the table to find out. Keep requests from being cut off by giving shutdown-timeout more than the request deadline (55s with the default write-timeout) where the platform allows it.

• An abandoned purge is logged (purge: abandoned at shutdown, with the table and its metadata location). The table is already dropped; its remaining files stay in the warehouse until you delete them (see Timeouts and purge).

Readiness: /readyz fails when the replica can’t reach the metastore, so a load balancer stops sending it traffic; /healthz only says the process is alive. See Probes.