Tabularia · Runtime Architecture

Tabularia · Runtime Architecture An architecture diagram generated by Archify. Web UI · Nuxt 3 · :3000 · Architecture component Web UI Nuxt 3 · :3000 Gateway API · FastAPI · JWT + RBAC · Docker Compose · dataprep-network · public · N replicas Gateway API FastAPI · JWT + RBAC public · N replicas Observability · VictoriaMetrics · Grafana · Docker Compose · dataprep-network Observability VictoriaMetrics · Grafana PostgreSQL · metadata · run queue · Docker Compose · dataprep-network · PostgreSQL PostgreSQL metadata · run queue Engine API · FastAPI · private network · Docker Compose · dataprep-network › private network · no published ports · FastAPI Engine API FastAPI · private network Orchestrator · APP__ROLE=orchestrator · Docker Compose · dataprep-network › private network · no published ports · same image · scales Orchestrator APP__ROLE=orchestrator same image · scales Valkey · broker + cache index · Docker Compose · dataprep-network › private network · no published ports Valkey broker + cache index Preview worker · Celery · preview queue · Docker Compose · dataprep-network › private network · no published ports Preview worker Celery · preview queue Run worker · run · ingest · export · Docker Compose · dataprep-network › private network · no published ports · Polars · DuckDB · chDB Run worker run · ingest · export Polars · DuckDB · chDB Celery beat · cache eviction · stats · Docker Compose · dataprep-network › private network · no published ports Celery beat cache eviction · stats Object storage · MinIO · Scaleway S3 · Docker Compose · dataprep-network Object storage MinIO · Scaleway S3 ClickHouse cloud · optional engine · Architecture component · ClickHouse ClickHouse cloud optional engine External databases · Postgres · ClickHouse · MySQL… · Architecture component External databases Postgres · ClickHouse · MySQL… HTTPS + JWT metadata · enqueues runs claims · beats runs the steps proxy · RBAC scrapes /metrics enqueues tasks preview queue celery queue periodic tasks parquet in/out reads parquet SQL over HTTPS s3(): reads and writes parquet ADBC ingest · writes tables Docker Compose · dataprep-network private network · no published ports Legend Frontend Backend Database Cloud Security Message bus External

Control plane

  • • The gateway is the only exposed service: JWT login, per-project RBAC, append-only audit log — and it can run in several copies
  • • Metadata (users, versioned flows, runs, datasources, encrypted connections, data contracts) lives in PostgreSQL
  • • Schedules and flow runs belong to the orchestrator role: the queue is the runs table, claimed with a conditional update, so no schedule fires twice

Data plane

  • • Every preview and every run is a Celery task: the preview queue stays responsive during long runs
  • • Engines run in-process in the worker (Polars, DuckDB, chDB); cloud ClickHouse is an optional remote engine
  • • A data contract is checked by the worker on what it just wrote, before the snapshot swap: a blocking violation refuses the publication

Storage and boundaries

  • • One bucket: datasets/ (snapshots), cache/ (steps), out/ (results); MinIO locally, Scaleway S3 in the cloud
  • • Source databases are ingested as streaming Arrow → parquet; Output nodes write datasources, tables or S3
  • • VictoriaMetrics scrapes the Engine API /metrics, celery-exporter, cAdvisor and node-exporter; Grafana shows them