Deployment and infrastructure

Every cloud resource this project needs comes from the Chrono service-template catalog. Nothing is hand-rolled where a template exists, which is what keeps the infrastructure reviewable as a set of small declarative files under infra/<env>/, rendered through their named template and synced to Kubernetes by ArgoCD.

This is the Chrono two-layer model. Layer 1 is the project config in chrono-shared-infra (projects/lucille-order-center/), which declares the repository, the base domain per environment and which directory each environment reads. Layer 2 is this repository's infra/dev/ and infra/prod/, one YAML file per resource. Layer 2 changes need no Terraform run and no PR against chrono-shared-infra — a commit to main is enough.

Nothing is deployed yet

The Layer-1 project config is open as a pull request and has not been merged. Until it is, there is no ApplicationSet and no namespace, so nothing in infra/ deploys. Merging it is a Terraform apply against the live cluster and was deliberately left to a human.

Two rules apply to every file in infra/<env>/: name: must equal the filename without .yaml (the ApplicationSet builds the Helm value path from it, and a mismatch fails silently), and none of them may set namespace, argocdProject, environment or domains — the ApplicationSet injects all four. Files reference domain keys, so hosts.default.subdomain: api resolves against the injected domains.default.

Templates used

Eleven files per environment, identical in infra/dev/ and infra/prod/ except where noted.

Resource	File / template	Configuration
PostgreSQL	lucille-postgres.yaml / postgres-db	Provider shared-db; database lucille_order_center; user lucille-api with readwrite on schema app
Redis	lucille-redis.yaml / redis	Chart-default HA topology: 3 Redis + 3 Sentinel replicas, redis_exporter, PodDisruptionBudget. Writes nothing to Vault and sets no password
Object storage	lucille-storage.yaml / objectstorage-bucket	Provider shared-spaces; private bucket lucille-po-pdfs-dev in dev, lucille-po-pdfs in prod; keys remapped to S3_*
Identity	lucille-zitadel.yaml / zitadel	Instance shared (namespace zitadel) → org lucille-org → project lucille-order-center (projectRoleAssertion: true) → OIDC app lucille-web-app
Error tracking	lucille-sentry.yaml / sentry-project	Projects lucille-api (node) and lucille-frontend (javascript-react), shared by both environments
API	lucille-api.yaml / regular-deployment	2 replicas, port 3000, /healthz, ingress + SSL, metrics.enabled: true, subdomain api
Worker	lucille-worker.yaml / regular-deployment	2 replicas, port 3001, no ingress, WORKER_MODE=true
Frontend	lucille-frontend.yaml / regular-deployment	2 replicas, port 80, ingress + SSL, subdomain orders
Docs	lucille-docs.yaml / regular-deployment	This site on nginx, port 80, ingress + SSL, subdomain docs; 1 replica in dev, 2 in prod
Inbound poll	lucille-sysco-poll.yaml / regular-cronjob	*/15 * * * *, command: node apps/worker/dist/jobs/sysco-poll.js
Migrations	lucille-migrate.yaml / regular-job	prisma migrate deploy selected by MIGRATE_MODE=true — not an ArgoCD hook, see below

Domains and subdomains

The base domain is the managed lucille-order-center.cc.chrono-backbone.com zone, not the lucille.internal the original brief offered as a fallback. Every ingress here sets withSSL: true, which drives a cert-manager letsencrypt ClusterIssuer, and Let's Encrypt cannot issue a certificate for a non-public .internal name — all three sites would be left without a valid certificate.

The three externally reachable workloads are on separate subdomains (decisions D5 and D10); the plan's shared orders subdomain for both the API and the SPA cannot route, because one hostname resolves to one backend Service.

Workload	dev	prod
API	api.dev.lucille-order-center.cc.chrono-backbone.com	api.lucille-order-center.cc.chrono-backbone.com
Frontend (SPA)	orders.dev.lucille-order-center.cc.chrono-backbone.com	orders.lucille-order-center.cc.chrono-backbone.com
Docs	docs.dev.lucille-order-center.cc.chrono-backbone.com	docs.lucille-order-center.cc.chrono-backbone.com

If the client brings a real domain, only domains.default in the Layer-1 environment files changes — nothing under infra/ needs editing, because these files reference the domain key rather than its value.

The workloads

API — lucille-api

name: lucille-api
template: regular-deployment

image:
  repository: registry.digitalocean.com/chrono-shared/lucille/api
  tag: 0.1.0-init # owned by .github/workflows/build-and-deploy.yml — do not hand-edit

replicas: 2
port: 3000
healthCheckEndpoint: /healthz

enableIngress: true
withSSL: true
hosts:
  default:
    subdomain: api

metrics:
  enabled: true
  path: /metrics

resources:
  requests: { memory: 512Mi, cpu: 250m }
  limits: { memory: 1Gi, cpu: 1000m }

env:
  SENTRY_ENVIRONMENT: dev

secrets:
  - lucille-api
  - lucille-postgres
  - lucille-redis
  - lucille-spaces
  - lucille-zitadel
  - lucille-smtp
  - lucille-sentry

env is resolved after envFrom, so a key listed there shadows the same key coming from Vault. Keep each key in exactly one of the two places.

Worker — lucille-worker

Same image, different mode. It is also the only workload that needs a hardened security context, because it launches Chromium.

name: lucille-worker
template: regular-deployment

image:
  repository: registry.digitalocean.com/chrono-shared/lucille/api
  tag: 0.1.0-init

replicas: 2
port: 3001 # /healthz, /readyz, /metrics — no ingress
healthCheckEndpoint: /healthz
enableIngress: false

# NOT RENDERED by regular-deployment v0.1.0 — declarative intent only.
command: node apps/worker/dist/worker.js
securityContext:
  runAsNonRoot: true
  runAsUser: 1001 # pptruser in apps/api/Dockerfile
  seccompProfile:
    type: RuntimeDefault
containerSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ['ALL']

resources:
  requests: { memory: 512Mi, cpu: 250m }
  limits: { memory: 1Gi, cpu: 1000m }

env:
  WORKER_MODE: 'true'
  PUPPETEER_EXECUTABLE_PATH: /usr/bin/chromium
  PUPPETEER_SKIP_DOWNLOAD: 'true'
  SENTRY_ENVIRONMENT: dev

Two template gaps, one closed and one open

regular-deployment v0.1.0 renders neither command/args nor any securityContext. Left as-is this Deployment would run the image's default command — a second copy of the API rather than the worker.

That is closed in the application layer: the API image's CMD is node apps/api/dist/entrypoint.js, a dispatcher that reads APP_MODE / WORKER_MODE / CRON_MODE / MIGRATE_MODE from the environment. env.WORKER_MODE: "true" is rendered, so the worker starts correctly.

The securityContext gap is still open. The image's USER pptruser (UID 1001) is all that applies today; runAsNonRoot, seccompProfile, allowPrivilegeEscalation: false and capabilities.drop: [ALL] are not enforced at the pod level. The keys are kept in the file so the intent is not lost. The fix is to add gated command/args + securityContext support to the shared chart — additive and backwards compatible, since every key would be wrapped in an if.

The memory limit accounts for Chromium: a 512 MiB limit will kill a PDF render mid-flight. See Central Kitchen PDF.

Frontend — lucille-frontend

name: lucille-frontend
template: regular-deployment

image:
  repository: registry.digitalocean.com/chrono-shared/lucille/frontend
  tag: 0.1.0-init

replicas: 2
port: 80
healthCheckEndpoint: /healthz # served by apps/frontend/nginx.conf

enableIngress: true
withSSL: true
hosts:
  default:
    subdomain: orders
    path: /

resources:
  requests: { memory: 64Mi, cpu: 50m }
  limits: { memory: 128Mi, cpu: 200m }

No secrets and no runtime env: Vite inlines VITE_* variables at build time, so VITE_API_BASE_URL, VITE_SENTRY_DSN, VITE_SENTRY_ENVIRONMENT and VITE_APP_VERSION are Docker build arguments passed by CI. A change to the API URL requires a rebuild, not a redeploy — worth knowing before debugging why a redeploy did not pick it up.

Sysco inbound poll — lucille-sysco-poll

name: lucille-sysco-poll
template: regular-cronjob

image:
  repository: registry.digitalocean.com/chrono-shared/lucille/api
  tag: 0.1.0-init

cronSchedule: '*/15 * * * *'
command: node apps/worker/dist/jobs/sysco-poll.js

secrets:
  - lucille-api
  - lucille-postgres
  - lucille-redis
  - lucille-spaces
  - lucille-zitadel
  - lucille-smtp
  - lucille-sentry
  - lucille-sysco-sftp

regular-cronjob does render command (as sh -c), unlike the deployment and job templates, so the explicit entrypoint is used here. concurrencyPolicy: Forbid, backoffLimit: 2 and activeDeadlineSeconds: 600 are fixed by the chart, which is exactly what is wanted: polls must not overlap on Sysco's session-limited SSH server. A failed run reports to Sentry and exits; the next run is 15 minutes away and no data is lost, because files stay on Sysco's server until they are both processed and archived.

Database migration — lucille-migrate

name: lucille-migrate
template: regular-job

image:
  repository: registry.digitalocean.com/chrono-shared/lucille/api
  tag: 0.1.0-init

# Selects the migrate branch of the image's entrypoint dispatcher. This is the
# mechanism that actually makes the Job migrate.
env:
  MIGRATE_MODE: 'true'

secrets:
  - lucille-postgres
  - lucille-api

The chart sets backoffLimit: 0, ttlSecondsAfterFinished: 60 and restartPolicy: Never. Because the Job spec changes whenever image.tag changes, each deploy produces a new Job that runs the migration once.

This is not an ArgoCD PreSync hook

The plan asked for lucille-migrate to run as an ArgoCD PreSync hook so the schema is migrated before the API pods roll out. That is not expressible in the two-layer model, for two independent reasons, both recorded in infra/README.md and in the header of infra/*/lucille-migrate.yaml:

1. regular-job v0.1.0 renders neither command/args nor any metadata annotations. The command: and the argocd.argoproj.io/hook annotations kept in the file are declarative intent the chart ignores.
2. Even with annotation support it would not help: every file in infra/<env>/ becomes its own ArgoCD Application. A PreSync hook inside the lucille-migrate Application can only order that Application's own resources — it can never gate the separate lucille-api Application's rollout. Cross-Application ordering needs sync waves on the ApplicationSet, which is Layer 1 and is not exposed per service file.

What actually happens: MIGRATE_MODE=true selects the migrate branch of apps/api/dist/entrypoint.js, which shells out to prisma migrate deploy and exits with the CLI's exit code, rather than starting the API server and never terminating. prisma is a runtime dependency of @lucille/api, so the CLI survives the production-only dependency install; if it ever became a devDependency the dispatcher exits 78 (EX_CONFIG) with an explanatory message instead of failing obscurely. migrate deploy is idempotent and takes a Postgres advisory lock, so two simultaneous starts are safe.

Interim procedure: run the migration from CI against the newly built image before the API tag is promoted, or run it once by hand. Ordering is a CI concern today, not an ArgoCD one.

Because the migration and the new pods are not strictly ordered, migrations must be written additive-first: add a nullable column, deploy code that writes it, backfill, tighten the constraint in a later migration. prisma migrate deploy is backward-compatible by design, so a brief window where new pods meet an un-migrated schema is survivable — but it is not free, and closing it is tracked.

Secrets from Vault

No credential is in source, in a Docker image, or in a Kubernetes manifest. Every secret lives in Vault and is synced into a Kubernetes Secret, which the templates mount as envFrom through their secrets[] list.

Vault secret	Produced by	Key names
lucille-postgres	lucille-postgres.yaml	DB_LUCILLE_API_HOST / _PORT / _DATABASE / _USERNAME / _PASSWORD / _SCHEMA / _SSLMODE / _URI
lucille-spaces	lucille-storage.yaml	Remapped to S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_ENDPOINT, S3_BUCKET, S3_REGION
lucille-zitadel	lucille-zitadel.yaml	Remapped to ZITADEL_CLIENT_ID, ZITADEL_CLIENT_SECRET, ZITADEL_ISSUER
lucille-sentry	lucille-sentry.yaml	Backend DSN (SENTRY_DSN), shared by API, worker, CronJob and migration Job
lucille-sentry-frontend	lucille-sentry.yaml	SPA DSN — needed at build time, so it is not mounted into any pod. build-and-deploy.yml currently reads the optional repo/environment variable vars.VITE_SENTRY_DSN rather than Vault; unset means Sentry is disabled in the browser bundle
lucille-redis	manual	REDIS_URL (and REDIS_SENTINELS / REDIS_SENTINEL_NAME / REDIS_PASSWORD if used)
lucille-api	manual	App config: session/JWT settings, EDI ISA identifiers, FRONTEND_ORIGIN, rate limits
lucille-smtp	manual	SMTP_HOST / SMTP_PORT / SMTP_USERNAME / SMTP_PASSWORD / SMTP_FROM_ADDRESS
lucille-sysco-sftp	manual	SYSCO_SFTP_HOST / _PORT / _USERNAME / _KEY_B64 / _KEY_PASSPHRASE / the three directories

DATABASE_URL does not exist in Vault

The postgres-db template has no key-remapping option, so it publishes the connection string as DB_LUCILLE_API_URI — the prefix comes from the user CR name — and Prisma only reads DATABASE_URL. infra/README.md leaves this as an open decision for whoever owns apps/api; the code has since resolved it. normaliseEnv() in apps/api/src/config/index.ts aliases the template's names onto the canonical ones before the schema is parsed, and writes DATABASE_URL back onto process.env because Prisma reads it directly:

alias('DATABASE_URL', ['DB_LUCILLE_API_URI', 'DB_LUCILLE_API_URL', 'POSTGRES_URI']);
alias('REDIS_URL', ['REDIS_URI', 'REDIS_CONNECTION_STRING']);
alias('S3_ACCESS_KEY_ID', ['SPACES_ACCESS_KEY_ID', 'AWS_ACCESS_KEY_ID']);

An explicitly-set canonical name always wins, so nothing here can override deliberate configuration, and no operator has to hand-maintain a duplicate key.

The Spaces key must be full-access

The bucket relies on a bucket policy to enforce private-only access, and limited-access Spaces keys are incompatible with bucket policies — you cannot apply a policy to a bucket using a limited-access key, and you cannot create a limited-access key for a bucket that already has one. The key in lucille-spaces must therefore be a full-access key; access is scoped by the bucket policy instead.

Two more manual steps no template can perform: enabling the application-level "User Roles Inside Access Token" toggle on the lucille-web-app OIDC application in the Zitadel console (its project-level counterpart, "Assert Roles on Authentication", is managed here as projectRoleAssertion: true), and creating the four hand-maintained Vault secrets above. If the token toggle stays off, the chef and admin roles are silently absent from access tokens and every authorization check misbehaves.

The suppliers.config JSONB column stores Vault secret names, never credential values; the worker resolves them against the injected environment at run time.

CI/CD pipeline

Two GitHub Actions workflows.

ci.yml — pull requests and pushes to main

Job	Runs
checks	pnpm lint, pnpm typecheck, pnpm build (all Turborepo root tasks, so a new package is covered automatically)
test	postgres:16 + redis:7.2-alpine service containers, then pnpm db:generate, db:deploy, db:seed, pnpm test
images	Builds all three Dockerfiles without pushing, so image breakage is caught on the pull request

pnpm test includes the docs integrity check, node apps/docs/scripts/check-docs.mjs, which enforces front-matter keys, sidebar reachability and a 300-word prose minimum per page. docusaurus build runs with onBrokenLinks: 'throw' and onBrokenAnchors: 'throw', so a dead cross-reference fails CI.

There is deliberately no Playwright/e2e job. Decision D4 keeps responsive coverage at three viewports in scope, but no Playwright config is committed yet, and a job that cannot pass is worse than no job. It gets added when the suite exists and can start its own stack.

build-and-deploy.yml — dev automatically, prod on demand

push to main (application code)           →  environment = dev
Actions → Build & Deploy → Run workflow   →  environment = dev | prod

1. setup    resolve the environment and its base domain
2. build    matrix over lucille/api, lucille/frontend, lucille/docs
              tag = <package version>-<short sha>, e.g. 0.1.0-a1b2c3d
              docker build from the repo root, docker push to the DO registry
              frontend also gets --build-arg VITE_API_BASE_URL=https://api.<base-domain>
                and the three other VITE_* build args
3. deploy   yq rewrites image.tag in every infra/<env>/*.yaml whose image.repository
              matches an image this run built, then commits to main with [skip ci]
            ArgoCD picks the commit up and syncs

paths-ignore covers infra/**, .github/**, docs/** and **/*.md, so the deploy job's own tag-bump commit cannot re-trigger a build — belt and braces with [skip ci].

Environment	Trigger	Reads	Gate
dev	Automatic on every push to main touching app code	infra/dev/	None
prod	Manual workflow_dispatch with environment: prod	infra/prod/	The deploy job declares environment: prod, so required reviewers on that GitHub Environment turn promotion into an approval gate

Both environments live on the same branch (main) and are separated by directory, because this repo has one long-lived branch. That is why promotion is a workflow run rather than a branch merge.

Promotion rebuilds the images rather than re-tagging the artefact that passed CI in dev: a prod dispatch runs the same build matrix against the current main. The tag is deterministic (<version>-<short sha>), so if main has not moved the tag is identical even though the digest comes from a fresh build. Promoting an existing digest instead — a registry re-tag, skipping build when the target tag already exists — is the obvious hardening and is not implemented.

Deploys are Kubernetes RollingUpdate, giving zero-downtime rollout for the API, frontend and docs; the worker's rollout briefly runs old and new pods together, which is safe because dispatch is idempotent at both the queue and the database layer.

Images

Image	Base	Contents
lucille/api	node:20-slim	API, worker, CronJob and migration code, Prisma client, system Chromium
lucille/frontend	nginx:1.27-alpine	Built SPA static files
lucille/docs	nginx:1.27-alpine	This documentation site

All three build with the repository root as their Docker context, because each installs from the root pnpm workspace and copies packages/types.

The base images are Node 20 (node:20-slim) while the development pod runs Node 22 — decision D7: the images follow the brief's Node 20, .nvmrc pins CI to 20, and the code targets ES2022 / Node ≥ 20, so the two never disagree about language level. Debian rather than Alpine is a hard requirement for the API image: the Chromium binary needs glibc, and musl will not run it.

The API image is a four-stage build — full dependency install, production-only dependency tree, TypeScript build, and a runtime stage carrying only dist, the production node_modules, the Prisma schema and the generated client. It runs as pptruser (UID 1001) under dumb-init, so signals reach Node and BullMQ can drain in-flight jobs on SIGTERM. Its CMD is the entrypoint dispatcher, which is what lets one image serve four workloads.

The frontend and docs images share a pattern: build with pnpm on node:20-slim, copy the static output into nginx, and expose a /healthz location that returns 200 without touching the filesystem — the same path both Dockerfiles use for their HEALTHCHECK and both service definitions pass as healthCheckEndpoint.

location = /healthz {
    access_log off;
    add_header Content-Type text/plain;
    return 200 'ok';
}

# SPA history fallback — every unknown path renders the app shell.
location / {
    try_files $uri $uri/ /index.html;
}

The docs image adds $uri.html to that fallback for Docusaurus's pre-rendered pages; the frontend also serves /assets/ with a one-year immutable cache, since Vite hashes those filenames.

Network and TLS

• All external traffic is HTTPS, terminated at the cluster ingress with cert-manager certificates (withSSL: true → the letsencrypt ClusterIssuer).
• Pod-to-pod traffic stays inside the namespace; the worker has no ingress at all.
• isolate_namespace: true is set for both environments. The NetworkPolicy template's third egress rule allows all ports to 0.0.0.0/0, so outbound SFTP (22) and SMTP (587) still work — that was checked before choosing it.
• The Sysco SFTP connection uses SSH key authentication; SYSCO_SFTP_KEY_B64 holds the base64 key and the decoded buffer never reaches the logger.
• SMTP uses TLS — STARTTLS by default, implicit TLS when SMTP_SECURE=true.
• Security headers are set by nginx in the frontend and docs images: X-Content-Type-Options: nosniff, Referrer-Policy: strict-origin-when-cross-origin, and X-Frame-Options: DENY on the SPA. No Content-Security-Policy is configured anywhere in this repository — not in the nginx configs, not in the service definitions. Adding one is outstanding work, either in apps/frontend/nginx.conf or as an ingress-level plugin.

Where to go next

• Observability — what metrics.enabled exposes and which signals are still gated off.
• Local development — the same stack, locally.
• Testing — what CI runs before any of this happens.
• Architecture — why there are six workloads and three images.