Architecture
Lucille Order Center is a three-tier internal web application with a fourth long-running deployable: a background worker that owns every outbound supplier side effect. Two short-lived workloads — the Sysco inbound-poll CronJob and the Prisma migration Job — run from the same image as the API.
Component diagram
┌──────────────────────────────┐
│ Browser (desktop, internal) │
└──────────────┬───────────────┘
│ HTTPS, cert-manager certs (withSSL: true)
┌─────────────────┴─────────────────┐
▼ ▼
┌───────────────────────┐ CORS REST │ ┌────────────────────────┐
│ lucille-frontend │ + cookie │ │ lucille-api │
│ React 18 + Vite 5 │─────────────┼─►│ Fastify v4 + TS │
│ nginx static, 2 pods │ orders.<d> │ │ 2 pods, port 3000 │
└───────────────────────┘ │ └───┬────────┬───────────┘
api.<d> │ │ BullMQ enqueue
Prisma │ │
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ PostgreSQL │ │ Redis (HA + │
│ schema app │ │ Sentinel 3+3) │
└──────┬───────┘ └─────── ─┬─────────┘
│ │ consume
│ ┌────────▼──────────────┐
└──────────┤ lucille-worker │
│ BullMQ, 2 pods │
│ same image as API │
└───┬───────┬───────┬───┘
│ │ │
Sysco SFTP ◄───X12 850─────┘ │ └───► SMTP relay
(+ inbound 855/997 poll) │ (Costco email)
│
DO Spaces ◄──PDF upload────┘──► SMTP relay
lucille-po-pdfs(-dev) (Central Kitchen PDF)
┌──────────────────────┐ ┌────────────────────────┐ ┌─────────────────────┐
│ Zitadel (OIDC) │ │ lucille-sysco-poll │ │ lucille-migrate │
│ roles: chef, admin │ │ CronJob */15 * * * * │ │ Job, migrate deploy│
└──────────────────────┘ └────────────────────────┘ └─────────────────────┘
Three Docker images cover six workloads. lucille/api (built from apps/api/Dockerfile) runs the API,
the worker, the CronJob and the migration Job; lucille/frontend and lucille/docs are nginx images
carrying the built SPA and this documentation site. The API and the SPA are on separate subdomains —
api.<domain>, orders.<domain> and docs.<domain> — because one hostname can only route to one
backend Service (decisions D5 and D10).
The tiers
Frontend SPA — apps/frontend
React 18 and TypeScript, built with Vite 5, served as static files by nginx. shadcn/ui-style components over Radix primitives and Tailwind CSS. TanStack Query owns server state (catalog, orders, audit events); Zustand owns the ephemeral cart. No Redux, no SSR — there is nothing to server-render for an authenticated internal tool, and Vite's dev loop is materially faster to work in.
The access token lives in memory only, never in localStorage: the API hands it over in the URL
fragment of its redirect to {frontendOrigin}/auth/callback, which is never sent to a server and
never appears in an access log (decision D12). The refresh token lives in an httpOnly, Secure,
SameSite=Strict cookie the API sets and reads.
In development the SPA calls /api/* and Vite proxies it to http://localhost:3000, so the browser
stays on one origin. In the cluster there is no proxy: VITE_API_BASE_URL is baked into the bundle as
https://api.<domain>, so calls are genuinely cross-origin and rely on the API's CORS configuration
(FRONTEND_ORIGIN, credentials: true).
API server — apps/api
Fastify v4 and TypeScript, one deployable container, two replicas, stateless. It owns all business logic: catalog reads and writes, order placement, supplier resolution, the audit trail, admin operations, and the OIDC code exchange.
Request bodies, query strings and route parameters are validated with the Zod schemas from
packages/types through apps/api/src/http/validate.ts — not with Fastify's AJV/JSON Schema
integration as the original plan sketched. Decision D9 records the reason: the SPA already codes against
those Zod schemas, so validating with them means one definition instead of a JSON Schema on the server
and a Zod schema on the client that can drift apart.
The API performs no outbound supplier I/O. It writes rows and enqueues jobs. That separation is what makes order placement fast and predictable regardless of how slow Sysco's SFTP server is on a given afternoon.
Background worker — apps/worker
BullMQ consumers in a separate Kubernetes Deployment, two replicas, built from the same image as the
API. It owns every side effect that leaves the cluster: generating and dropping X12 850 EDI files,
sending order emails, rendering PDFs with puppeteer-core against system Chromium, uploading to object
storage, and performing back-order reroutes.
The image's default command is not the API server but a mode dispatcher,
node apps/api/dist/entrypoint.js, which selects a process from the environment: WORKER_MODE=true
starts the worker, CRON_MODE=true runs one Sysco poll, MIGRATE_MODE=true runs
prisma migrate deploy, and the default is the Fastify server. This exists because the Chrono
regular-deployment and regular-job templates at v0.1.0 render no container command; without the
dispatcher the worker Deployment would start a second copy of the API. An explicit
node apps/worker/dist/worker.js still works wherever a template does render command — the CronJob
does.
The worker exposes port 3001 with three routes (/healthz, /readyz, /metrics) and no ingress. The
image runs as the non-root pptruser (UID 1001) because it launches Chromium; the matching pod-level
securityContext is declared in infra/*/lucille-worker.yaml but is not rendered by
regular-deployment v0.1.0, so today the image's USER is what actually applies.
Shared types — packages/types
Zod schemas plus the TypeScript types inferred from them, consumed by the API, the worker and the
frontend. Enumerations that exist as CHECK constraints in PostgreSQL — order status, line-item status,
supplier type, audit event type, X12 acknowledgement codes — are declared once here and re-used
everywhere, which is why a status string cannot be misspelled in one tier only. Queue names and the
deterministic deduplication ids (dispatch:<orderId>:<supplierType>,
reroute:<lineItemId>:<supplierId>) live here too, so producer and consumer cannot drift.
Request flow: placing an order
1. SPA POST /api/orders { restaurantId, notes, lines[] }
Authorization: Bearer <access token>
X-Idempotency-Key: <uuid> (optional)
2. API auth plugin: verify JWT via JWKS, extract role + restaurant assignments
3. API replay check against idempotency_keys in Postgres (not Redis — D11)
4. API validate body with the Zod schema → 400
5. API restaurant not assigned to caller → 403
restaurant does not exist → 404
restaurant inactive → 409
6. API resolve every catalogItemId to a supplier mapping
unknown item id → 404 + id list
inactive item → 409 + id list
no active supplier mapping → 422 + id list
7. API one Prisma transaction:
insert orders (status 'pending')
insert order_line_items
insert supplier_dispatches (one per distinct supplier)
insert supplier_dispatch_line_items (the join rows)
insert audit_events ORDER_PLACED + SUPPLIER_DISPATCH_CREATED
8. API after the commit, for each dispatch:
queue.add('dispatch', payload, { deduplication: { id } })
enqueue failure → that dispatch is marked 'failed', audit
SUPPLIER_DISPATCH_FAILED (stage 'enqueue')
9. API orders.status := 'dispatching' (or 'failed' if nothing was enqueued)
audit ORDER_STATUS_CHANGED, orders_placed_total incremented
10. API 201 { orderId, status }
11. SPA poll GET /api/orders/:id until status leaves pending/dispatching
Enqueuing deliberately happens after the transaction commits. A queued job that referenced an uncommitted row would be a race; a committed order whose job never reached Redis is a visible, retryable dispatch an administrator can act on.
Job flow: dispatching to a supplier
1. Worker picks up a `dispatch` job from `supplier-dispatch`
2. Worker parse job.data with the shared Zod schema
invalid shape → job.discard(), permanent failure (the API's bug)
3. Worker load supplier_dispatches row by id
row missing → discard, permanent failure
status already submitted/confirmed → return { skipped: true }
4. Worker branch on supplierType
sysco → build X12 850, reserve ISA13, SFTP put
costco → render HTML table, SMTP send
central_kitchen → render HTML, Chromium → PDF, Spaces put, SMTP send
5. Worker in one transaction: dispatch → 'submitted' (+ reference, submitted_at),
its line items → 'submitted', audit SUPPLIER_SUBMITTED
6. Worker recompute orders.status → completed | partial | failed | dispatching
on throw: supplier_dispatch_failures_total{supplier_type,terminal} is incremented and
BullMQ retries with exponential backoff from DISPATCH_BACKOFF_MS (1s, then 2s)
up to DISPATCH_ATTEMPTS (3). A failure classified permanent is discarded
immediately. On the terminal attempt the dispatch becomes 'failed', its lines
'dispatch_failed', an admin alert email is sent, and the job is retained for
24 hours (removeOnFail: { age: 86400 }) for manual retry.
Step 3 is the worker's idempotency guard: a plain status read, not a SELECT … FOR UPDATE. Double
dispatch is prevented in layers — BullMQ's deduplication id suppresses a duplicate enqueue, a single
dispatch worker at concurrency 1 means two attempts never overlap, and the status check makes a replay a
no-op.
The inbound direction is a CronJob rather than a webhook, because Sysco pushes files onto SFTP and has
nothing to call. Every 15 minutes lucille-sysco-poll opens one SFTP session, lists the inbound
directory, downloads new X12 855 and 997 files one at a time, hands each to the shared inbound
processor, and only then moves it into the archive — unparseable files land in {archive}/failed/ so a
harmless list entry is never replayed forever.
Why each choice
| Decision | Reasoning |
|---|---|
| Node.js + TypeScript | One language across API, worker and frontend; strong libraries for SFTP, EDI, email and PDF |
| Fastify over Express | Lower overhead, first-class hooks and serialisation, better TypeScript ergonomics |
| Zod validation over Fastify/AJV JSON Schema | One schema shared with the SPA rather than two that can drift (D9) |
| React + Vite over Next.js | No SSR requirement for an authenticated desktop-first internal tool; Vite dev loop is fast |
| Radix + Tailwind (shadcn/ui pattern) | Accessible unstyled primitives without design-system lock-in on an internal tool |
| Prisma | Schema-first and type-safe with migrations built in; parameterised queries remove SQL injection as a class |
| PostgreSQL | Relational data with real constraints; CHECK constraints keep statuses honest at the storage layer |
| BullMQ over pg-boss | A first-class deduplication API, retry/backoff semantics and queue observability, which is exactly the hard part here |
| Separate worker deployment | Supplier I/O is slow and failure-prone; isolating it keeps the API's latency and failure profile clean, and lets the two scale independently |
| Redis only as queue backend | No application caching in Redis, so a Redis outage delays dispatch but never loses or corrupts data |
| Idempotency keys in Postgres, not Redis | A replay record survives a Sentinel failover and is auditable (D11) |
puppeteer-core + system Chromium | Avoids bundled-browser downloads at image build time and keeps the image reproducible |
| Zitadel (OIDC) | Provisioned from a Chrono service template; gives PKCE, refresh tokens and project roles without building identity |
| DO Spaces for PDFs | The PDF is an audit artefact, not application state; object storage is the right home and presigned URLs serve it |
| One shared image, four entrypoints | API, worker, CronJob and migration Job share all business code; one build, one scan, one dispatcher |
Scale and availability
The proof of concept is small: around five restaurant locations, ten chefs, three suppliers and roughly 120 catalog items, with one to five orders per restaurant per day. Even if every restaurant submitted simultaneously — around 25 concurrent submissions — that is comfortably inside one API pod's capacity, and two replicas exist for rolling deploys rather than for throughput.
- API: 2 replicas, stateless, trivially horizontally scalable. Rate limiting is per authenticated
user, in-process, so the effective limit is
RATE_LIMIT_MAXper pod. - Worker: 2 replicas. There is one
supplier-dispatchconsumer atconcurrency: 1— for every supplier type, not just Sysco. The reason is recorded inapps/worker/src/worker.ts: the producer writes all dispatches to one queue, and BullMQ offers no server-side filter, so a per-lane concurrency scheme would mean workers repeatedly fetching and re-delaying each other's jobs. Serialising everything is correct and, at ~25 dispatches a day, fast enough.backorder-rerouteruns atEMAIL_WORKER_CONCURRENCY(default 4) because it only touches Postgres and Redis. - Database: shared PostgreSQL cluster, schema
app. No pool size is set in code — Prisma's default applies unlessconnection_limitis added toDATABASE_URL. - Redis: HA with Sentinel, 3 Redis plus 3 Sentinel replicas.
REDIS_SENTINELSandREDIS_SENTINEL_NAMEswitch the client into Sentinel mode;REDIS_URLalone is the local default. - Availability: no SLA is defined for the proof of concept. Deploys are standard Kubernetes
RollingUpdate, so they are zero-downtime for the API, frontend and docs. A worker rollout briefly runs old and new pods together, which is safe because dispatch is idempotent at both the queue and the database layer.
Where to go next
- Data model — the storage layer in full.
- API reference — every endpoint.
- Queues and jobs — the worker's contract in detail.
- Observability — the log fields and metrics these flows emit.
- Deployment — how these six workloads reach the cluster.