Deploying Tabularia on Kubernetes

Operational notes for whoever moves Tabularia from infrastructure/docker-compose.yml (the development stack) to Kubernetes. The compose file is the reference for which services exist and how they talk to each other; it is not a production manifest: it runs dev servers with hot reload and bind-mounts the sources.

Everything below marked verified was exercised on 2026-09-12 against the real stack (Scaleway object storage, managed ClickHouse).

Helm chart

A chart that applies everything on this page lives in infrastructure/helm/tabularia. It encodes the constraints as templates — beat cannot be scaled, the root filesystem is read-only, /tmp is sized per role — and refuses to render when a required secret is missing. Its README explains each decision, and values-production.example.yaml is a filled-in starting point.

This page remains the reference for why; the chart is how.

To try Tabularia on one computer first, without knowing Kubernetes, follow Try Tabularia on your computer with minikube. To try the chart as a company would run it (several nodes, enforced network policies, a private registry with its own certificate authority), follow A company-like cluster on one computer.

Components

Use the image default command for the gateway and the engine API. The compose file overrides them with --reload, which must not reach production.

• Gateway: two workloads from one image, told apart by APP__ROLE. With api a process only answers requests, and can be replicated freely: one process uses one core (about 95 requests a second), and sessions, the sign-in throttle and the editor's presence notice live in Postgres. With orchestrator it only does the background work — schedules, flow executions, reconciliation, clean-up — and is not exposed. Several orchestrators share the work through the database: a schedule slot and a queued execution are each taken by exactly one, and a unique index keeps a flow from having two executions running. all (the default, one process doing both) is what the development compose file runs.
• Schema. Migrations (Alembic) are a step of their own: python -m app.db.migra, run by an init container in the Helm chart before the application starts, under a Postgres advisory lock. The application never migrates in production; it checks that the database is not behind its code and refuses to start if it is (a newer database is fine: that is the previous version during a rolling update). It seeds the admin user. See database-migrations.md; upgrading an installation, step by step: upgrading.md.
• Database connections. Each gateway and orchestrator process keeps its own pool, up to APP__GATEWAY_THREADS × 1.25 + DB__MAX_OVERFLOW (25 with twelve threads): size Postgres' max_connections for the sum.
• Beat: one replica. Two beats enqueue every periodic task twice.
• Postgres holds all control-plane metadata (users, groups, permissions, versioned flows, encrypted connection credentials, runs, audit log): back it up.
• Valkey is the Celery broker and the step-cache index. Its content is disposable: losing it drops queued tasks and makes the cache start cold, nothing else.
• One installation per Valkey database. The queues and keys have fixed names. Two installations on the same Valkey and the same externalServices.redis.db (staging and production on one managed Redis, both on db 0) share them: one's workers run the other's tasks, with the wrong key and storage, and Administration → Settings and → Workers mix the two. Give each its own db number, or its own instance.
• One release per namespace. The namespaces of the users' code are named after the release's namespace, and the Jobs, Secrets and images made at run time after run ids: two releases in one namespace collide (the second install fails on the namespaces; with createNamespaces: false they share them, and follow each other's Jobs).

Networking

• Public: the frontend and the gateway, on one host: / to the frontend, /api to the gateway (not stripped; the gateway knows its prefix through APP__ROOT_PATH). The browser calls the API relatively (NUXT_PUBLIC_API_BASE=/api). Routing the gateway's own paths instead cannot work: they are also pages of the frontend.
• The notebooks' host (with scripts): a second host name, /api to the gateway alone, where the marimo notebooks' apps and editors are served (scripts.notebooks.host, NOTEBOOKS__ORIGIN): there the gateway answers only for them, so a notebook's code cannot use the session of whoever opens it. A sibling of the interface's host, with its certificate. See An address of their own.
• Internal: the engine API is reached only by the gateway, through ENGINE__BASE_URL (cluster DNS, e.g. http://engine:8000). Keep it off the ingress: it answers only requests carrying the gateway's internal token, derived from SECURITY__FERNET_KEY (nothing to configure), and a NetworkPolicy should admit only the gateway, the orchestrator and the workers — the chart ships one, on by default.
• The client's address: the gateway reads X-Forwarded-For from the right, past the proxies it trusts (--forwarded-allow-ips, the chart's gateway.trustedProxies). Without it every client seems to be the ingress controller, and the sign-in throttle, keyed by (address, email), lets anyone lock the administrator out. The default trusts every private range, so any pod of the cluster could choose its address: narrow it to the pod CIDR of your ingress controller. The application never reads the header itself: the audit log, the sessions and the throttle all use the address the server resolved.

Ingress settings the application depends on:

Egress. The engine connects to databases that users configure as connections: that is the product, and also a server-side request surface. Restrict the engine pods' egress with a NetworkPolicy to the object storage, ClickHouse, Valkey and the approved database hosts, and block cloud metadata endpoints such as 169.254.169.254. The chart does it from networkPolicy.egressCIDRs (entries are CIDRs or {cidr, except}), and always carves the metadata address out of a range that holds it, 0.0.0.0/0 included. With the list empty no egress policy is rendered.

Administration → Cluster (clusterView.enabled, on by default) shows the pods of the release and of the users' code, their state, CPU and memory, and the warning events, read with read-only rights in those namespaces (no logs, no exec); CPU and memory need metrics-server.

Users' code (scripts.enabled, containers.enabled) runs as Jobs in namespaces of its own, never in the release's. The gateway (…-scaler) and the orchestrator (…-orchestrator) are the only pods with a Kubernetes API token, and only the orchestrator may create the image builds, in their privileged namespace. Those namespaces get a NetworkPolicy: the internet and the other namespaces, not the pods of the release's namespace, the private ranges (unless listed in networkPolicy.userCodePrivateCIDRs) nor the cloud metadata. It needs a network plugin that enforces egress rules. See Python and R in flows and The Container node.

Backend image (backend/Dockerfile)

One Dockerfile for the four backend roles and for both development and production. The local docker-compose.yml builds the very same file with WITH_CHDB=true and WITH_DEV=true (test dependencies); production builds it with the defaults. Production variants:

chDB alone accounts for almost 600 MB, pandas and numpy included, which nothing else in the application needs. In production the ClickHouse dialect is covered by the external ClickHouse engine, so the lightweight variant is the default. Without chDB the engine picker shows it as not configured, and a flow that still asks for it fails with a clear message. For reference, the development image is 1.41 GB.

Verified with --read-only and a tmpfs /tmp: engine API healthy, run worker ready, beat started; every available engine producing identical results on real data; CSV and Excel exports written to /tmp; database ingest from PostgreSQL and ClickHouse working in the lightweight variant.

• runs as uid/gid 10001, no shell login, no build tools, no uv, no test dependencies;
• compatible with a read-only root filesystem: the only writable path is /tmp (TMPDIR=/tmp, HOME=/tmp); uploads, intermediate parquet files, DuckDB and Polars spills and chDB state all go there;
• MALLOC_ARENA_MAX=2 is preset to limit memory fragmentation from Polars/Arrow threads.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities: { drop: ["ALL"] }
volumeMounts:
  - { name: tmp, mountPath: /tmp }
volumes:
  - name: tmp
    emptyDir: { sizeLimit: 50Gi }   # size per role, see below

Sizing /tmp

Without a sized /tmp volume, a large file lands in the container's ephemeral storage and the kubelet evicts the pod.

With the ClickHouse engine in s3 transport for a flow, in both its development and production engine, flow execution never touches local disk: the ClickHouse server reads sources from and writes results to the bucket. Uploads, exports, database refreshes and database/S3 outputs still use /tmp whatever the engine.

Gateway image (gateway/Dockerfile)

Same rules as the backend image: one Dockerfile for development and production, uid/gid 10001, compatible with a read-only root filesystem, /tmp the only writable path. The gateway itself writes no files, so a small emptyDir on /tmp is enough. Test dependencies are included only with --build-arg WITH_DEV=true, which the local compose sets. Production size: 199 MB.

Verified read-only against an empty PostgreSQL: the migration step creates the schema, the admin user is seeded, login works. The securityContext shown for the backend applies unchanged.

Worker lifecycle

• Graceful shutdown. A run can last up to the Celery task limit (3600 s). On SIGTERM a worker stops taking tasks and waits for the running ones; with Kubernetes' default terminationGracePeriodSeconds: 30, every deploy would kill in-flight runs (the chart sets 3600 s for the run workers, 120 s for the preview workers). Tasks are not redelivered (acks_late is off, on purpose: outputs in append mode are not idempotent). Keep the grace period above the task limit (raised from Administration → Settings too), or accept that deploys interrupt the longer runs.
• A worker killed mid-run (OOMKilled: on cgroup v2 the kernel kills the whole container; an emptyDir past its sizeLimit; its node lost) leaves its runs started in Celery's results. The engine tells them apart from running ones by the worker's own record in Valkey (gone, or started again after the run began) and reports them failed, «The worker running it stopped before the end», within about half a minute, instead of waiting for ENGINE__RUN_STALE_TIMEOUT_SECONDS.
• Gateway and engine. Stopping, they wait 5 s (preStop) for the Service to drop them, then finish the requests in flight (a preview waits up to 290 s, an upload as long as it takes) for up to their grace period minus 15 s: gateway.terminationGracePeriodSeconds and engineApi.terminationGracePeriodSeconds, 300 s. The orchestrator's is 60 s: it closes the executions it carries as interrupted and stops their steps already with a worker (revoked), so a flow run again does not write on top of them.
• Memory. Set a memory limit per pod. The per-child memory cap is derived from the container limit and CELERY__WORKER_CONCURRENCY, so the limit is also the tuning knob.

Probes

The gateway's and the engine's answer on the event loop, apart from the threads serving requests, so a busy pod is not taken for a dead one; the chart gives every probe 5 s. Workers and beat expose no HTTP port; rely on the restart policy.

Configuration

Settings are environment variables with __ as the nesting delimiter. Every one of them, with its default, its bounds and the values.yaml key the chart sets it from, is in the configuration reference; the commented starting point is infrastructure/.env.example. Values marked secret belong in Kubernetes Secrets.

Comments in env files. Not every tool strips a trailing # comment from a value: docker run --env-file keeps it as part of the value, and the gateway then refuses to start on APP__TIMEZONE — worse, a secret with a trailing comment would be silently wrong. infrastructure/.env.example keeps every comment on its own line; if you derive ConfigMaps or Secrets from an existing .env, check that no value carries a comment.

The Helm chart sets the variables above from named values; any other one goes through extraEnv, by its variable name (secrets through existingSecret). The Helm value column of the configuration reference says which is which.

SSO behind real hostnames. OIDC__REDIRECT_URI is https://<host>/api/auth/sso/callback and must be registered, identical, on the identity provider; OIDC__POST_LOGIN_URL is https://<host>/auth/callback. <host> is the address people open Tabularia at: a sign-in started at another name of the same server (an IP, an internal alias) loses its cookie on the way back and fails with tx_missing (the gateway logs both addresses). The gateway allows a minute between its clocks and the identity provider's; keep NTP on the nodes. Step by step, with Keycloak: Single sign-on with Keycloak.

Frontend

Build the production image from frontend/Dockerfile.prod; frontend/Dockerfile is the development server with hot reload and must not reach production. The production image compiles the app with nuxt build and ships only the compiled output, served by Nitro on port 3000. Size: 176 MB.

• One image for every environment. The public settings are read at startup, so no rebuild per environment: NUXT_PUBLIC_API_BASE (public gateway URL, called by the browser), NUXT_PUBLIC_BUCKET (must equal STORAGE__BUCKET) and, optionally, NUXT_PUBLIC_GRAFANA_URL.
• No server-side API calls. Server rendering never calls the gateway; every API call comes from the browser. The frontend pods need no route to the gateway.
• Same hardening as the other images: uid/gid 10001, read-only root filesystem, /tmp the only writable path; the securityContext shown for the backend applies unchanged.
• Content-Security-Policy from the frontend itself. Every page carries one: scripts only from /_nuxt plus Nuxt's own inline scripts, allowed by hash; fetch only to the site and to NUXT_PUBLIC_API_BASE; frames only the site and NUXT_PUBLIC_GRAFANA_URL. Nothing to configure on the ingress. The session cookie is written by the gateway, HttpOnly (no script on the page can read it) and SameSite=Lax, and Secure when the page is served over HTTPS — behind a proxy that terminates TLS, through X-Forwarded-Proto, which ingress-nginx sets.

Verified read-only: the login page renders, a gateway URL set at startup ends up in the served page, compiled assets are served, and / redirects to /login without a session.

Known application limits that affect operations

• Flow executions are not resumed. An execution whose orchestrator stops or dies is closed as failed, with a message that says so — at once on a clean stop, after ORCHESTRATOR__DEAD_AFTER_SECONDS (a minute) when the process is killed. The steps already done have written their output; the flow has to be launched again. Rollouts of the orchestrator make this visible in the run history.
• Versions before the queue must not run next to this one. A gateway from before APP__ROLE existed ran the scheduler in process and knows nothing of the queue: stop it before starting the new version (the chart's README has the two commands).
• The interface and the gateway must be on the same domain. The browser's session is a cookie the gateway writes (HttpOnly, SameSite=Lax), and the interface's server must see it too — the reference ingress serves both from one host, with the gateway under /api. Two hosts of one domain work with AUTH__COOKIE_DOMAIN; two unrelated domains leave the browser without a session.
• Removing someone from an SSO group takes effect at their next sign-in, or at once if an administrator closes their sessions (Audit page). Disabling the account, or changing its password, closes them by itself.