Skip to content
Configuration

Configuration

Puddle is configured with a single YAML file passed via -c:

puddle -c /path/to/puddle.yaml

A config file is required — puddle won’t boot without one, and it must declare at least one warehouse. ${VAR} and $VAR references are expanded from the process environment before the YAML is parsed; any unset variable is a hard error at boot.

This document covers every top-level block. A minimal config that boots is just:

puddle.yaml
warehouses:
  default:
    location: file:///tmp/puddle-warehouse

Top-level blocks:

  • server — listen address.
  • logging — log level and format.
  • metastore — where catalog metadata is stored.
  • warehouses — one or more named warehouses.
  • external-warehouses — warehouses managed by other metastores, available for registerTable.
  • authn — bearer-token validators (off by default).
  • authz — policy backend (allow-authenticated, OPA).

server

puddle.yaml
server:
  addr: ":8089"               # default
  read-header-timeout: 10s    # default
  read-timeout: 60s           # default
  write-timeout: 60s          # default
  idle-timeout: 120s          # default
  shutdown-timeout: 25s       # default
  max-request-body: 8MiB      # default

The Iceberg REST API is mounted at /api/catalog/, so clients point their catalog uri at http://<host>/api/catalog.

HTTP timeouts

The four *-timeout knobs are forwarded to Go’s http.Server and bound how long the process will wait on a misbehaving (or merely slow) client:

  • read-header-timeout — reading request headers.
  • read-timeout — reading the full request, headers + body. Caps slow-body attacks where a client drips body bytes indefinitely.
  • write-timeout — writing the response. Long-running catalog operations (commit retry, STS round-trip during credential vending) live under this budget; tune up if your vending path is slow.
  • idle-timeout — bound how long an idle keep-alive connection stays open.

A value of 0s is accepted literally as “no limit” — useful for debugging, dangerous in production. Negative values are rejected at config-load.

Request body size

max-request-body caps the size of incoming request bodies; the limit is installed as a http.MaxBytesReader wrap before route dispatch, so every JSON-bodied endpoint inherits it without per-handler wiring. Over-limit requests get a 400 BadRequestException with a request body exceeds maximum size of N bytes message.

The default of 8 MiB is comfortably above any realistic metadata.json payload — typical commits are tens of KB, and the largest pathological cases observed in the wild are still well under a megabyte. Bump it for unusual workloads; the trade-off is that a higher cap allows larger memory spikes per request.

Accepts a bare integer (bytes) or a value with a unit suffix: B, KB/KiB, MB/MiB, GB/GiB. Matching is case-insensitive. Zero, negative, and unknown-unit values are rejected at config-load — the field exists to impose a limit, so disabling it via config is misconfiguration.

Graceful shutdown

On SIGTERM, SIGINT, or SIGHUP, puddle stops accepting new connections and waits up to shutdown-timeout for in-flight requests to finish before exiting. If the deadline elapses with requests still in flight, puddle exits non-zero so the platform sees the unclean shutdown.

SIGHUP is treated as “shut down cleanly” rather than “reload config” — config changes are restart events, not in-process refreshes.

Tune shutdown-timeout to fit comfortably inside your platform’s termination grace window. On Kubernetes, that’s terminationGracePeriodSeconds (default 30s) — the 25s default above leaves a few seconds of headroom before the kubelet sends SIGKILL. If you raise the K8s grace period, raise this in step.

Probes

Two endpoints sit on the top-level mux for orchestrators:

  • GET /healthz — liveness. Returns 200 once the process is past startup. The contract is “this process is responsive enough to write a 200”; nothing more. Any failure here is a reason to restart the pod.
  • GET /readyz — readiness. Returns 200 with {"status":"ready"} when the metastore pings successfully; returns 503 with {"status":"unready","checks":{"metastore":"<error>"}} otherwise. The metastore ping has a 2 s context deadline so a wedged backend never stalls the probe.

Both are unauthenticated by design — kubelets don’t carry credentials, and the responses leak only “process up” / “DB reachable”, no PII or table contents. Lock the surface down via NetworkPolicy or a separate listener if your environment requires it.

A minimal Kubernetes probe block:

livenessProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /readyz, port: http }
  periodSeconds: 5
  failureThreshold: 3

/readyz only checks the metastore, not the warehouse storage — S3 latency / throttling is a write-path concern that is better surfaced on the actual catalog operation than as flapping pod readiness.

logging

puddle.yaml
logging:
  level: info     # debug | info | warn | error
  format: text    # text | json

Both fields default to the values shown above. Logs go to stderr.

metastore

Where puddle keeps the catalog’s own state (which tables exist, what their current metadata locations are). Table data and the metadata files themselves live in the warehouse, not here.

puddle.yaml
metastore:
  type: memory    # memory | sqlite | postgresql

type selects the backend; the matching sub-block (if any) carries its settings.

memory

Non-persistent. State is lost on shutdown. Useful for tests, demos, and the RCK harness.

puddle.yaml
metastore:
  type: memory

sqlite

File-backed. Suitable for single-node deployments.

puddle.yaml
metastore:
  type: sqlite
  sqlite:
    path: ./tmp/puddle.db   # required; ":memory:" for ephemeral

path is required. Use :memory: for an ephemeral in-process DB (tests).

postgresql

Shared Postgres. Suitable for multi-replica deployments where several puddle instances point at one database.

puddle.yaml
metastore:
  type: postgresql
  postgresql:
    url: "postgres://puddle:${PUDDLE_PG_PASSWORD}@db:5432/puddle?sslmode=require"
    schema: puddle              # default: "puddle"
    max-open-conns: 10          # default: 10
    max-idle-conns: 5           # default: 5
    conn-max-lifetime: 1h       # default: 1h

url must use the postgres:// (or postgresql://) URL form. Use ${VAR} env expansion for the password to keep secrets out of YAML.

schema is auto-created on first start. Must match [a-zA-Z_][a-zA-Z0-9_]*.

warehouses

warehouses: is a map from warehouse name to its configuration. At least one entry is required. Clients address a specific warehouse by name (e.g. via the warehouse=<name> query parameter).

Names must match [a-zA-Z0-9_-]+. config, oauth, and tokens are reserved and rejected at boot.

puddle.yaml
warehouses:
  default:
    location: file:///tmp/puddle-warehouse
  prod:
    location: s3://my-warehouse
    s3: { ... }      # see below

location is a URL whose scheme picks the storage backend.

Supported schemes:

  • file:// — local filesystem. No sub-block. Suitable for dev, CI, and single-node deployments where data lives on local disk.

    puddle.yaml
    warehouses:
  default:
    location: file:///tmp/puddle-warehouse

  • s3:// (also s3a://, s3n://) — S3 or any S3-compatible store: AWS S3, MinIO, Ceph RGW, Cloudflare R2, Tigris. Requires an s3: sub-block.

    puddle.yaml
    warehouses:
  prod:
    location: s3://my-warehouse
    s3:
      region: us-east-1                       # required
      endpoint: https://s3.amazonaws.com      # omit for AWS, set for MinIO/Ceph
      path-style-access: false                # true for MinIO

      # Catalog's own S3 credentials. Leave empty to defer to the
      # AWS default credential chain (env vars, profile, IRSA,
      # instance metadata).
      access-key-id: ${AWS_ACCESS_KEY_ID}
      secret-access-key: ${AWS_SECRET_ACCESS_KEY}
      session-token: ""

      # Optional toggles, all default off / empty.
      use-arn-region: false
      checksum-enabled: false
      acl: ""                                 # e.g. bucket-owner-full-control
      write-storage-class: ""                 # STANDARD | STANDARD_IA | ...

    region, endpoint, and path-style-access are forwarded to clients automatically — PyIceberg / Spark / Trino pick them up without further configuration.

Vended credentials

When enabled, puddle mints short-lived, prefix-scoped session tokens via STS AssumeRole and hands them to clients on every table load. The catalog’s own credentials never leave the server. Reads get a read-only token; writes get a read-write token. Default policy templates scope every token to the table’s own prefix, so a token for db.events cannot read or write db.orders.

puddle.yaml
    s3:
      ...
      vended-credentials:
        enabled: true
        role-arn: arn:aws:iam::123456789012:role/IcebergTableAccess
        duration-seconds: 3600              # default; AWS allows [900, 43200]
        sts-endpoint: ""                    # defaults to s3.endpoint
        sts-region: ""                      # defaults to s3.region
        external-id: ""                     # cross-account
        read-policy-template: ""            # text/template; default scopes to table prefix
        write-policy-template: ""           # default extends read with PutObject + DeleteObject

Vending requires explicit access-key-id + secret-access-key on the warehouse (the catalog uses them to call STS). Works the same against AWS, MinIO, and Ceph (RGW); only role-arn and the optional sts-endpoint differ. See Credential vending for the per-target operator runbook (trust policies, role creation, template overrides, troubleshooting).

external-warehouses

external-warehouses: declares storage locations belonging to warehouses managed by other metastores, that you want to register tables from. Each entry has the same shape as a routed warehouse — a location URL and the matching per-scheme sub-block — and the same naming and validation rules apply across the union of warehouses: and external-warehouses:.

puddle.yaml
warehouses:
  default:
    location: file:///tmp/puddle-warehouse

external-warehouses:
  legacy:
    location: s3://legacy-bucket/x
    s3:
      region: us-east-1
      access-key-id: ${LEGACY_AWS_ACCESS_KEY_ID}
      secret-access-key: ${LEGACY_AWS_SECRET_ACCESS_KEY}

authentication

authn: is opt-in. Omit it entirely and every request runs anonymously — fine for local dev. Configure it and every request must present a valid bearer token in the Authorization header or get a 401.

Both kinds of token can be configured at once: tokens that look like a JWT are sent to the JWT validator, everything else to the static-token list.

puddle.yaml
authn:
  static-tokens: [ ... ]    # see below
  jwt: { ... }              # see below

Per-request attributes set by either validator (e.g. groups) are visible to authorization policy.

static-tokens

A list of bearer tokens, each tied to a principal name and free-form attributes.

puddle.yaml
authn:
  static-tokens:
    - token: ${PUDDLE_ADMIN_TOKEN}
      principal: admin
      attrs:
        groups: [admins]
        team: platform
    - token: ${PUDDLE_CI_TOKEN}
      principal: ci-runner
      attrs:
        groups: [ci]

token: should always use ${VAR} env-var expansion so the literal secret stays out of source control. attrs: is forwarded verbatim to authorization policy — the keys are operator-chosen.

Configuration errors hard-fail at boot: empty token or principal, duplicate tokens, and tokens whose value would be misrouted to the JWT validator.

jwt

OIDC-style validation against one or more issuers. Each request’s JWT is routed to its issuer by exact match on the iss claim.

puddle.yaml
authn:
  jwt:
    issuers:
      - issuer: https://login.example.com/
        audience: puddle                       # string or list of strings
        algorithms: [RS256]                    # restrict to specific algs
        # JWKS source — pick one (default: OIDC discovery)
        jwks-uri: https://login.example.com/.well-known/jwks.json
        # jwks: |                              # inline alternative
        #   { "keys": [ ... ] }
        claims:                                # claim → attrs.<key> mapping
          email: email
          groups: groups

If neither jwks-uri nor jwks is given, puddle does OIDC discovery (GET <issuer>/.well-known/openid-configuration) and follows the returned jwks_uri.

claims: is an allowlist: only the listed JWT claims are passed through to authorization policy. Anything not in the map is dropped.

Debugging: who am I?

GET /api/catalog/v1/whoami echoes back the identity of the calling token: subject, issuer, and the attribute map authorization will see. Useful for confirming static-token attrs: are wired the way you expect, or seeing which JWT claims actually surface under your claims: allowlist. With authn: unconfigured the response contains "anonymous": true.

$ curl -s -H "Authorization: Bearer $TOKEN" \
    http://localhost:8089/api/catalog/v1/whoami | jq
{
  "sub": "admin",
  "issuer": "static:admin",
  "attrs": { "groups": ["admins"], "team": "platform" }
}

authorization

authz: selects the policy that gates every request after authentication. Decisions are logged with the stable event="authz.decision" discriminator regardless of mode.

puddle.yaml
authz:
  mode: allow-authenticated   # allow-authenticated (default) | opa
  opa: { ... }                # required when mode is opa; see below

allow-authenticated

Default mode. Permit any authenticated request. With authn: unconfigured this collapses to allow-all so the dev path keeps working without auth.

puddle.yaml
authz:
  mode: allow-authenticated

opa

Delegate every decision to Open Policy Agent. OPA runs as a separate process (sidecar, daemonset, or shared service); puddle calls its data API on every request.

puddle.yaml
authz:
  mode: opa
  opa:
    url: http://localhost:8181/v1/data/puddle/rbac/allow

url points at the OPA data API endpoint that returns the allow/deny decision. The wire shape puddle posts is documented by a JSON Schema at authz/opa/input.schema.json — point your editor or check-jsonschema at it when iterating on policy fixtures with opa eval -i.

A runnable starter policy with admin / writer / reader roles, plus a working puddle.yaml, lives in examples/opa/. For container-based stacks, see examples/compose/ — the opa-local, trino-rustfs-opa, and trino-rustfs-opa-vending presets all exercise this code path against a real OPA sidecar.

Environment variable expansion

${VAR} and $VAR in any string field are expanded from the process environment before YAML parsing. An unset variable is a hard error — config-load will fail listing every missing name.

puddle.yaml
authn:
  static-tokens:
    - token: ${PUDDLE_ADMIN_TOKEN}
      principal: admin
warehouses:
  prod:
    location: s3://my-warehouse
    s3:
      region: us-east-1
      access-key-id: ${AWS_ACCESS_KEY_ID}
      secret-access-key: ${AWS_SECRET_ACCESS_KEY}

Use this for every secret. Don’t paste literal tokens into config files.