Observability

Four signals: structured logs to stdout, Prometheus metrics, OpenTelemetry traces, and Sentry errors. Each of the last three is gated by its own environment variable, and two of the three are off by default:

Signal	Gate	Default	Implemented in
Logs	`LOG_LEVEL`	`info` (always on)	`apps/api/src/logging/index.ts`
Metrics	`METRICS_ENABLED`	`true`	`apps/api/src/observability/metrics.ts`
Traces	`OTEL_ENABLED`	`false`	`apps/api/src/observability/telemetry.ts`
Errors	`SENTRY_DSN` (non-empty)	unset → Sentry never starts	`apps/api/src/observability/sentry.ts`

The API and the worker share all four implementations because they run from the same image; the worker re-exports the API's logger (apps/worker/src/logging.ts) and calls the same startTelemetry() / initSentry() pair from apps/worker/src/instrumentation.ts. The frontend carries Sentry only.

Structured logging with pino

Both processes log line-delimited JSON on stdout, collected by the standard Kubernetes pipeline and ingestible by Loki without a parser.

Every line carries three base fields — service (lucille-api, or lucille-worker whenever WORKER_MODE is true), version (APP_VERSION) and env (NODE_ENV) — plus pino's own level, time and msg. Two formatting choices matter when you write queries:

level is the string label ("info", "error"), not pino's numeric level, because Loki and Sentry both expect a label.
time is ISO-8601 (pino.stdTimeFunctions.isoTime), not epoch milliseconds.

The context field vocabulary

LogContext in apps/api/src/logging/index.ts is the full list of standard context fields. All are optional, and undefined values are stripped rather than serialised as null.

Field	Attached by
`requestId`	Fastify `genReqId` (honours an inbound `x-request-id`) via an `onRequest` child logger
`userId`	The authenticated caller's Zitadel `sub`
`restaurantId`	Any log line scoped to one restaurant
`orderId`	Order placement, dispatch, reroute, inbound correlation
`lineItemId`	Line-level operations (back-order, reroute)
`supplierId`	Any supplier-scoped operation
`supplierType`	`sysco`, `costco`, `central_kitchen`
`dispatchId`	Anything touching one `supplier_dispatches` row
`jobId`	Worker job lifecycle and handler lines
`jobName`	The BullMQ job name (`dispatch`, `reroute`)
`queue`	`supplier-dispatch`, `backorder-reroute`
`attempt`	1-based attempt number inside a handler
`component`	Added by the worker's `componentLogger('dispatch' \| 'pdf' \| 'health' \| …)`
`err`	Serialised error object (`errorKey: 'err'`)

A dispatch failure therefore reads as one self-contained record:

{
  "level": "error",
  "time": "2026-08-13T09:32:00.123Z",
  "service": "lucille-worker",
  "version": "0.1.0-a1b2c3d",
  "env": "production",
  "component": "dispatch",
  "jobId": "412",
  "attempt": 3,
  "orderId": "af31c9b2-7d44-4f0e-9a1e-2b6f0c7d9e10",
  "dispatchId": "5f0e1d2c-3b4a-4c6d-8a55-9c2a7f114c6d",
  "supplierId": "1111aaaa-2222-bbbb-3333-cccc4444dddd",
  "supplierType": "sysco",
  "msg": "supplier dispatch failed terminally",
  "err": {
    "type": "Error",
    "message": "connect ETIMEDOUT 203.0.113.10:22",
    "stack": "Error: connect ETIMEDOUT …"
  }
}

How the fields get there

Per-request and per-job child loggers, so no call site has to remember to attach context:

// API — Fastify binds requestId itself (requestIdLogLabel: 'requestId'); the
// onRequest hook rebinds it onto a child so downstream children inherit it.
app.addHook('onRequest', async (request) => {
  request.log = request.log.child({ requestId: request.id as string });
});

// Worker — one child logger per handler invocation.
const log = componentLogger('dispatch', {
  orderId: payload.orderId,
  dispatchId: payload.dispatchId,
  supplierId: payload.supplierId,
  supplierType: payload.supplierType,
  jobId: job.id,
  attempt,
});

LOG_LEVEL controls verbosity (debug locally, info in the cluster) and the level is forced to silent when NODE_ENV=test. pino-pretty is attached automatically — but only when the process is not production, not test, and stdout is a TTY, so piping logs to a file or a collector always yields JSON.

Two logging rules the code enforces rather than documents:

Credentials cannot be logged. redact matches a fixed key list — password, secret, token, authorization, accessToken, refreshToken, privateKey, passphrase, clientSecret, secretAccessKey, SYSCO_SFTP_KEY_B64 — at the top level, one and two levels deep, and as a lower-cased req.headers.* path, replacing the value with [redacted].
Never log a full EDI interchange at info. Raw interchanges go to vendor_documents, which is the right place for them; logging them duplicates large payloads into the log pipeline.

Prometheus metrics

prom-client, on a dedicated Registry rather than the global default one — which is why default process metrics are registered explicitly by collectRuntimeMetrics() and carry the prefix lucille_ (lucille_process_cpu_seconds_total, lucille_nodejs_eventloop_lag_seconds, …).

Two exposition points, both suppressed entirely when METRICS_ENABLED=false:

Endpoint	Served by	Scraped today?
`GET /metrics` on port 3000	Fastify, registered in `app.ts`	Yes — `infra/*/lucille-api.yaml` sets `metrics.enabled: true`, `path: /metrics`
`GET /metrics` on port 3001	the worker's `node:http` health server	No — `infra/*/lucille-worker.yaml` has no `metrics` block

That second row matters: the dispatch and reroute counters are incremented in the worker process, so until a metrics block is added to the worker service definition they are exposed but never collected. Both endpoints are excluded from request logging, from the HTTP duration histogram and from rate limiting.

The metrics that exist

Every metric below is registered in apps/api/src/observability/metrics.ts and incremented somewhere in the source; there are no others.

Name	Type	Labels	Incremented by
`orders_placed_total`	counter	`restaurant_id`	API, after an order commits (`domain/orders.ts`)
`supplier_dispatch_duration_seconds`	histogram	`supplier_type`, `outcome`	Worker, per attempt; `outcome` is `success`/`failure`
`supplier_dispatch_failures_total`	counter	`supplier_type`, `terminal`	Worker; `terminal` is `"true"`/`"false"` as a string
`backorder_reroutes_total`	counter	`outcome`	Worker; `outcome` is `rerouted` or `no_alternate`
`http_request_duration_seconds`	histogram	`method`, `route`, `status`	API `onResponse` hook; `route` is the route pattern
`queue_jobs_total`	counter	`queue`, `outcome`	Worker lifecycle events; `outcome` is `completed`/`failed`
`queue_jobs_deduplicated_total`	counter	`queue`	The `deduplicated` QueueEvents subscriber, in both processes

Note the two places the code deviates from the original plan's label sketch: dispatch failures are labelled terminal (did this attempt exhaust the retries?) rather than a free-text reason, because a reason label is unbounded cardinality and the reason already lives in supplier_dispatches.failure_reason and in the logs; and reroutes carry only outcome, not a from/to supplier pair, for the same reason.

export const supplierDispatchDurationSeconds = new Histogram({
  name: 'supplier_dispatch_duration_seconds',
  help: 'Wall-clock duration of a supplier dispatch job, by supplier type and outcome',
  labelNames: ['supplier_type', 'outcome'] as const,
  buckets: [0.1, 0.5, 1, 2, 5, 10, 30, 60, 120],
  registers: [registry],
});

The dispatch buckets are chosen for the workload: an email send is sub-second, a PDF render is a few seconds, and an SFTP handshake against a slow partner can be tens of seconds. Buckets that stopped at one second would hide exactly the tail that matters. http_request_duration_seconds uses a much tighter set (0.005 … 5) because an API request that takes five seconds is already pathological.

Label cardinality is deliberately low: supplier_type has three values rather than an unbounded supplier_id, and restaurant_id is bounded by the number of locations, which is single digits.

Alerting — designed, not yet implemented

No Prometheus rule, Grafana dashboard or Sentry alert rule is defined in this repository. Slack and PagerDuty notifications are switched on for the prod environment in the Layer-1 project config (they are off in dev, to keep a churn-prone environment out of the alert channels), but nothing routes these metrics to them yet. The conditions below are the intended first set, recorded so the work is not lost:

Proposed condition	Why it is worth alerting on
`supplier_dispatch_failures_total{terminal="true"}` rising for one `supplier_type`	A channel is down — usually Sysco's SFTP or the SMTP relay
`supplier_dispatch_duration_seconds` p95 rising	A partner is degrading before it fails outright
`backorder_reroutes_total{outcome="no_alternate"}` rising	Back-orders are arriving with nowhere to reroute to
`queue_jobs_deduplicated_total` spiking	Something is enqueuing repeatedly — a retry loop or a client bug
`queue_jobs_total{outcome="failed"}` over `completed`	The worker is failing broadly rather than on one supplier

There is also no bull-monitor dashboard: infra/*/lucille-redis.yaml uses the redis template's default HA topology, which ships redis_exporter and a PodDisruptionBudget, not a queue UI. Queue depth today is inspected with redis-cli or from the queue_jobs_* counters.

OpenTelemetry tracing

Both processes use OpenTelemetry Node.js auto-instrumentation through @opentelemetry/sdk-node and @opentelemetry/auto-instrumentations-node, exporting over OTLP/HTTP.

if (config.observability.otelEnabled) {
  sdk = new NodeSDK({
    resource: resourceFromAttributes({
      [ATTR_SERVICE_NAME]: config.observability.otelServiceName, // defaults to serviceName
      [ATTR_SERVICE_VERSION]: config.appVersion,
      [ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: config.env,
    }),
    traceExporter: new OTLPTraceExporter({ url: `${endpoint}/v1/traces` }),
    instrumentations: [
      getNodeAutoInstrumentations({
        '@opentelemetry/instrumentation-fs': { enabled: false },
        '@opentelemetry/instrumentation-http': {
          ignoreIncomingRequestHook: (request) =>
            (request.url ?? '').startsWith('/healthz') ||
            (request.url ?? '').startsWith('/metrics'),
        },
      }),
    ],
  });
  sdk.start();
}

startTelemetry() is called as the first statement of apps/api/src/instrumentation.ts and apps/worker/src/instrumentation.ts, both of which are the first import of their entrypoint. The instrumentations patch libraries at require time, so a module already imported is never patched — this ordering is the whole reason those files exist. Filesystem spans are disabled as pure noise, and probe and scrape traffic is dropped so it does not dominate the trace volume.

What that buys without writing any span code: inbound HTTP spans on Fastify, outbound HTTP spans, PostgreSQL spans through the pg instrumentation, and Redis command spans. shutdownTelemetry() is awaited on SIGTERM in both processes so the last batch is flushed.

Variable	Purpose
`OTEL_ENABLED`	Master switch. Defaults to `false` and is `false` in `.env.example`
`OTEL_EXPORTER_OTLP_ENDPOINT`	Collector base URL; `/v1/traces` is appended. Omit it to use the SDK default
`OTEL_SERVICE_NAME`	Optional override; falls back to `lucille-api` / `lucille-worker`

Tracing is not switched on in the cluster yet

Neither infra/dev/* nor infra/prod/* sets OTEL_ENABLED or OTEL_EXPORTER_OTLP_ENDPOINT in a service definition's env block, and the default is false — so no traces are being exported in either environment. Turning it on means adding both keys (to the env block, or to the hand-maintained lucille-api Vault secret) once a collector endpoint is known. The code path is complete and needs no change.

Sentry

Two Sentry projects are provisioned from the Chrono sentry-project template: lucille-api (platform node) and lucille-frontend (platform javascript-react). The API, the worker, the CronJob and the migration Job all run from the API image and therefore share the backend DSN from the lucille-sentry Vault secret. Dev and prod share both projects and are separated by Sentry's own environment tag, which is why SENTRY_ENVIRONMENT is set per environment in each service definition's env block.

Initialisation is conditional on a DSN being present. No DSN, no initialisation — not a disabled client, not a no-op transport, but the SDK never starting:

export function initSentry(): boolean {
  const config = getConfig();
  const dsn = config.observability.sentryDsn;
  if (!dsn) {
    getLogger().debug('SENTRY_DSN is not set — Sentry is disabled');
    return false;
  }
  Sentry.init({
    dsn,
    environment: config.observability.sentryEnvironment, // SENTRY_ENVIRONMENT, else NODE_ENV
    release: config.appVersion,
    tracesSampleRate: config.observability.sentryTracesSampleRate, // default 0.1
    // OpenTelemetry owns tracing; letting Sentry instrument too double-patches.
    skipOpenTelemetrySetup: true,
    initialScope: { tags: { service: config.serviceName } },
  });
  return true;
}

That is what keeps local development quiet: .env.example ships SENTRY_DSN= empty and docker-compose.yml sets SENTRY_DSN: '' for both the API and the worker, so no local Sentry container is needed. DSNs are never hard-coded.

What actually reaches Sentry, and with what context:

Source	Captured	Tags on the event
API error handler	5xx `AppError`s and unhandled errors (4xx are logged only)	`requestId`, `userId`, `route`, `service`
Worker `failed` event	Every job failure, including non-terminal attempts	`queue`, `jobId`, `jobName`, `attemptsMade`, `service`
Worker `error` event	Consumer-level errors	`queue`, `service`
Worker process handlers	`unhandledRejection`, `uncaughtException`	`component: 'worker'`
SPA	Unhandled render/runtime errors via `Sentry.ErrorBoundary`	Sentry user id, `role`, `restaurantId`

The frontend SDK is opt-in at build time: VITE_SENTRY_DSN is inlined by Vite, so an empty build arg means every helper in apps/frontend/src/lib/sentry.ts is a no-op and the app falls back to its local error boundary. Its tracesSampleRate is a fixed 0.1 and browserTracingIntegration() is enabled.

No Sentry alert rules are defined in infra/ — the template creates projects and writes DSNs, and nothing more. Routing terminal dispatch failures, unparseable inbound EDI, Chromium crashes and 997 rejections to a channel is still to be configured in Sentry itself; all four already produce a captured exception with the tags above, so the rules have something to match on.

Health checks

The split between the two probes is deliberate and identical in both processes.

Endpoint	Cost	Checks	Status codes
`GET /healthz` (API 3000)	No external I/O	Returns service, version, `uptimeSeconds`	Always 200
`GET /readyz` (API 3000)	One `SELECT 1` + one `PING`	`{ database, redis }`	200 or 503
`GET /healthz` (worker 3001)	No external I/O	Same payload shape; `/health` and `/` alias to it	Always 200
`GET /readyz` (worker 3001)	Postgres, Redis, Chromium	Named checks plus a `failed[]` list in the body	200 or 503

/healthz deliberately touches neither Postgres nor Redis. If it did, a 30-second database blip would fail liveness on every pod at once and Kubernetes would restart the whole fleet — turning a recoverable dependency wobble into an outage and killing in-flight SFTP uploads for nothing. Liveness answers one question: is this process still able to serve a request?

/readyz is the deeper check and the one to curl when something looks wrong. The worker's version also verifies that the Chromium binary at PUPPETEER_EXECUTABLE_PATH is executable, so a pod that could not render a PDF is visibly not-ready rather than discovering it on the day's first Central Kitchen dispatch. A check that throws is recorded as a failed check with its message in detail, never as a 500.

One gap worth knowing: both service definitions pass a single healthCheckEndpoint: /healthz to the regular-deployment template, so /readyz is not currently wired to a Kubernetes readiness probe — it is there for humans, dashboards and kubectl exec. Pointing the readiness probe at it is a service-definition change, not a code change.

Where to go next

Deployment — where metrics.enabled, SENTRY_ENVIRONMENT and the Vault secrets are configured.
Queues and jobs — the job lifecycle these metrics count.
Troubleshooting — reading these signals when something breaks.

Structured logging with pino​

The context field vocabulary​

How the fields get there​

Prometheus metrics​

The metrics that exist​

Alerting — designed, not yet implemented​

OpenTelemetry tracing​

Sentry​

Health checks​

Where to go next​