Troubleshooting

Every entry below is a failure this system is specifically known to be susceptible to — either drawn from the project's research briefings or read directly out of the error paths in the code and the developer scripts. Each one is written as symptom, cause, fix — because in most of these cases the symptom points somewhere other than the cause.

The local-environment entries come first, because that is where most days start. Everything from Chromium crashes mid-render onward is runtime behaviour that applies equally in the cluster.

native-services.sh reports a service as down

Symptom. ./scripts/native-services.sh status prints down for one or more of the four services, or prints nothing useful at all.

Cause. Several possibilities, and the script deliberately hides most of them: every command in start and stop ends in || true, so a daemon that refused to start does not fail the script. The usual reasons:

1. The port is already bound. Something else holds 5432, 6379, 9000/9001 or 1025/8025 — often a docker compose up of the same stack. pg_ctlcluster and redis-server fail silently; MinIO and Mailpit exit immediately after nohup backgrounds them.
2. A stale PostgreSQL cluster. If the machine or container was killed rather than shut down, the Debian cluster can be left with a postmaster.pid that pg_ctlcluster 15 main start refuses to start over.
3. Not root. apt-get, the writes to /usr/local/bin and the su postgres -c … calls all need it. sudo is not optional.
4. Wrong architecture. The MinIO and Mailpit download URLs are hard-coded to linux-amd64. On arm64 the download succeeds and the binary then fails with Exec format error.

Fix. Read the logs the script does keep — ${LOG_DIR:-/var/log}/minio.log and ${LOG_DIR:-/var/log}/mailpit.log for those two, and the Debian cluster log under /var/log/postgresql/ for Postgres. Then:

# Who is on the port?
ss -ltnp | grep -E ':(5432|6379|9000|9001|1025|8025)\b'

# Stop the compose stack before using the native path (or the other way round).
docker compose down

# Restart just the background pair after freeing a port.
sudo ./scripts/native-services.sh stop
sudo ./scripts/native-services.sh start

Note that start re-runs install_packages first, and that set -euo pipefail means a failed apt-get or a failed download aborts before anything is started — if the script exits early with a curl or apt error, nothing came up at all. If you only want the binaries, use the install subcommand.

MinIO or Mailpit will not pick up new settings

Symptom. You changed MINIO_DATA or LOG_DIR and re-ran start, and nothing changed. Or stop appeared to succeed while the process kept running.

Cause. Both processes are tracked purely by executable name. start skips the launch entirely when pgrep -x minio (or mailpit) already matches, so an instance started with the old arguments keeps running; stop uses pkill -x, which only matches that exact process name, so an instance started under a different name — a wrapper script, a container shim — survives.

Fix. Stop it explicitly and confirm it is gone before starting again:

sudo ./scripts/native-services.sh stop
pgrep -a -f 'minio|mailpit'          # must print nothing
sudo MINIO_DATA=/srv/minio ./scripts/native-services.sh start

There are no PID files to clean up: the script writes none.

lucille_test does not exist, so every test fails to connect

Symptom. pnpm test in apps/api or apps/worker fails immediately with a Prisma connection error naming lucille_test.

Cause. Both suites are integration suites against a real database, and both default to postgresql://lucille:lucille@localhost:5432/lucille_test?schema=app. That database is created either by scripts/postgres-init/01-create-databases.sql, which compose runs only on a completely empty data volume, or by bootstrap_databases in native-services.sh. A postgres-data volume created before that SQL file existed will not have it.

Fix. Recreate the volume, or create the database by hand:

docker compose down -v && docker compose up -d      # re-runs scripts/postgres-init/*.sql
# …or, without recreating anything:
docker compose exec postgres psql -U lucille -d lucille_order_center \
  -c 'CREATE DATABASE lucille_test OWNER lucille;'

Then apply the schema: TEST_DATABASE_URL=… pnpm --filter @lucille/api db:deploy. Note that prisma migrate reset is not the escape hatch here — Prisma 6.19 guards it (decision D13) and the project does not bypass that guard.

A test run seems to hang before any test executes

Symptom. pnpm --filter @lucille/api test sits there. Eventually it prints [test] could not acquire the test-database lock within 900000ms — running anyway and proceeds.

Cause. Expected behaviour, not a bug. apps/api/tests/global-setup.ts takes a PostgreSQL session-level advisory lock for the whole run, so a second run — from another terminal, another agent, or a watch process — waits rather than truncating the shared database mid-transaction. It polls for up to 15 minutes, then warns and continues anyway.

Fix. Wait, or find the other run. Because the lock is session-scoped it cannot go stale: a killed run drops its connection and the lock with it. If you need to confirm nothing else holds it:

docker compose exec postgres psql -U lucille -d lucille_test \
  -c "SELECT pid, granted FROM pg_locks WHERE locktype = 'advisory';"

Also remember the suite is deliberately serialised — pool: 'forks', singleFork: true and fileParallelism: false — so it is slow by design, and retry: 0 means a flake is a real bug.

seed-zitadel.ts exits before doing anything

Symptom. The script prints ZITADEL_ADMIN_PAT is required. plus a five-step list, and exits with status 64.

Cause. Exactly what it says: the seed authenticates every Management API call with a Personal Access Token belonging to a machine user that holds IAM_OWNER, and it refuses to start without one. The comment block at the top of the script also mentions ZITADEL_ADMIN_USERNAME/ZITADEL_ADMIN_PASSWORD; that path is not implemented — only ZITADEL_ADMIN_PAT is read.

Fix. Create the token in the console (Users → Service Users → New, Bearer; grant IAM_OWNER under Organisation → Managers; then Personal Access Tokens → New) and re-run:

ZITADEL_ADMIN_PAT=<token> pnpm --filter @lucille/api exec tsx ../../scripts/seed-zitadel.ts

Zitadel did not become healthy … within 120000ms

Symptom. The seed script prints waiting for Zitadel followed by a row of dots, then fails with that message.

Cause. It polls ${ZITADEL_ISSUER}/debug/healthz every two seconds for two minutes. Either the container is not up, or ZITADEL_ISSUER points somewhere else than the instance. Two specific traps:

• docker compose up -d was run but Zitadel is crash-looping, usually because its own zitadel database is missing (see the lucille_test entry above — the same init script creates both).
• You are on the native service path. scripts/native-services.sh does not cover Zitadel at all, so there is nothing on 8080 to seed and this script cannot be used there.

Fix. docker compose logs zitadel, and check ZITADEL_ISSUER (it defaults to http://localhost:8080; trailing slashes are stripped for you). On the native path, skip the script — the API's token verification is exercised by the test harness's local JWKS instead (decision D2).

Zitadel 401/403 on /management/v1/…

Symptom. The seed fails with seed-zitadel failed: Zitadel 401 on /management/v1/orgs: … or the same with 403 and a PermissionDenied body.

Cause. A 401 means the PAT is wrong, expired or revoked — including the very common case of having copied the service user's client secret rather than a Personal Access Token. A 403 means the token is valid but the service user does not hold IAM_OWNER, so it may not create organisations or projects.

Fix. Re-issue the PAT and re-check the manager grant. Everything the script does is idempotent (409/AlreadyExists is logged as exists and skipped), so re-running after fixing the token is safe and will not duplicate the org, project, roles, application, users or grants.

The seed printed no ZITADEL_CLIENT_ID

Symptom. The copy these into your .env block contains the issuer and project id but no client id or secret, and a comment pointing at a console URL.

Cause. The OIDC application already existed, so the create returned 409 and Zitadel did not re-issue credentials. There is nothing to print.

Fix. Read them from the URL the script prints, ${ZITADEL_ISSUER}/ui/console/projects/{projectId}/apps/{appId}. Remember that on this path the script also never updates the application, which is why the manual token toggle below has to be re-checked.

Login works in the browser but the API rejects every token

Symptom. The SPA completes the OIDC round-trip, but API calls return 401 Invalid or expired access token. Nothing is wrong with the clock and the token is not expired.

Cause. An issuer mismatch. apps/api/src/auth/jwt.ts verifies with jwtVerify(…, { issuer }) strictly. In docker-compose.yml the containerised api and worker are configured with ZITADEL_ISSUER: http://zitadel:8080, because that is how they reach Zitadel over the compose network — while a browser, and therefore the token's iss, uses http://localhost:8080.

Fix. Make the two agree. For browser-driven local work, run the API from pnpm --filter @lucille/api dev with ZITADEL_ISSUER=http://localhost:8080 (the config default) rather than in the container, or override the compose service's ZITADEL_ISSUER to the host-visible URL. Do not relax the issuer check.

Chromium crashes mid-render

Symptom. Central Kitchen PDF generation works most of the time and fails intermittently. Errors look like Target closed, Protocol error (Page.printToPDF): Target closed, or Navigation failed because browser has disconnected. It is worse under concurrency and nearly impossible to reproduce locally.

Cause. The default shared-memory size in a container is 64 MB, and that is not enough for Chromium. It runs out of /dev/shm partway through rendering and the renderer process dies.

Fix. Always pass --disable-dev-shm-usage, which makes Chromium write shared memory to /tmp instead:

args: ['--disable-dev-shm-usage', '--disable-gpu', '--no-sandbox', '--disable-setuid-sandbox'];

In Docker Compose the alternative is shm_size: "1g". A Kubernetes pod spec has no shm_size field; the equivalent is mounting an emptyDir with medium: Memory at /dev/shm, which is more moving parts for the same outcome. The flag is the answer.

Verify. docker compose exec worker df -h /dev/shm shows the size, and a successful render after the flag is added confirms it. Watch supplier_dispatch_failures_total{supplier_type="central_kitchen"} stop climbing.

"No usable sandbox!" on launch

Symptom. Chromium refuses to start: No usable sandbox! Update your kernel or see https://chromium.googlesource.com/…/suid_sandbox_development.md. The same image works on one node pool and not another.

Cause. Ubuntu 23.10 and later ship an AppArmor profile at /etc/apparmor.d/chrome that prevents Chrome binaries from using user namespaces. It fires even when kernel.unprivileged_userns_clone=1 is set, and it is node-OS dependent — which is why it can appear after a DOKS node pool upgrade with no change to the application.

Fix. This project already runs Chromium with --no-sandbox plus --disable-setuid-sandbox as a documented tradeoff for a trusted-content-only worker, so it is not affected on the current code path. If you are trying to move to a sandboxed Chromium:

1. Test on the actual DOKS node operating system, not locally.
2. Kubernetes user namespaces (hostUsers: false, beta from v1.33) are the supported path and remove the need for SYS_ADMIN.
3. Do not adopt ghcr.io/puppeteer/puppeteer as a shortcut — it is built for sandbox mode and requires the SYS_ADMIN capability, which conflicts with capabilities: drop: ["ALL"].

Related failure. Failed to move to new namespace: Operation not permitted means the process is running as non-root without user namespaces and without --no-sandbox. Running as root without --no-sandbox is equally unsupported. Non-root plus --no-sandbox is the combination this project uses.

Chromium will not start at all

Symptom. error while loading shared libraries: libgbm.so.1: cannot open shared object file, or a similar message naming libnss3, libatk-bridge2.0-0, libcups2, libxkbcommon0, libgtk-3-0 or libasound2.

Cause. A missing system library. Chromium needs a specific set, and missing exactly one produces a cryptic startup failure rather than a useful diagnostic.

Fix. The runtime stage of apps/api/Dockerfile installs the full list. If a library is missing after a base-image bump, add it there. Two other rules:

• Never move this image to Alpine. Alpine is musl libc; Chromium is built against glibc.
• Keep the fonts. fonts-liberation and fonts-dejavu-core are why the PDF is not full of tofu boxes.

SFTP: "no more sessions"

Symptom. Sysco dispatches begin failing in clusters. The SFTP server reports no more sessions, or connections are dropped with no error at all — the job simply hangs and then times out.

Cause. Two contributing behaviours:

1. SSH clients make an unauthenticated connection first to negotiate protocol settings, so each attempt costs more sessions than it looks like. Several concurrent jobs easily exceed the server's limit.
2. When the server drops a connection for this reason, ssh2 raises only an end event — no error — so nothing detects it as a failure.

There is a third trap: calling connect() repeatedly to test connectivity provokes the same error, and a failed connect can still eventually establish a session even after reporting failure.

Fix.

• Keep BullMQ worker concurrency at 1 on the Sysco SFTP queue. This is the primary control.
• One SftpClient instance per job execution; connect, operate, end() in a finally block, unconditionally.
• Never loop on connect(). Let BullMQ's attempts and backoff be the only retry mechanism — ssh2-sftp-client v12 removed built-in connection retry entirely.
• Use for...of with await for sequential SFTP operations, never .forEach with an async callback, which does not follow the promise chain.

SFTP: connect never resolves

Symptom. A Sysco job hangs with no error and no timeout, then BullMQ stalls it.

Cause. A corrupted or empty private key. Historically the connect promise neither resolved nor rejected in this case. Fixed in v12.x, but defensive validation is still cheap.

Fix. Validate the decoded buffer before connecting and always set a hard ceiling on the handshake:

const key = Buffer.from(env.SYSCO_SFTP_KEY_B64, 'base64');
if (key.length === 0) {
  throw new Error('SYSCO_SFTP_KEY_B64 decoded to an empty buffer');
}
await sftp.connect({ /* … */ privateKey: key, readyTimeout: 20_000 });

Also confirm the key type is one Sysco's server accepts — RSA, ECDSA and Ed25519 are supported by the client; Ed25519 is preferable for new keys but must be agreed with Sysco.

Related. An abruptly closed connection used to produce a hanging promise; v12 invalidates the internal SFTP object after any ECONNRESET so subsequent use fails immediately. Make sure the deployed version is v12.x.

Missing Zitadel role claims

Symptom. Sign-in succeeds. The token's signature verifies. Every API call returns 401 with Access token carries no Lucille Order Center role, and the API logs a warning token carries no chef/admin project role — check that both Zitadel role-assertion toggles are enabled. GET /api/auth/me fails the same way.

It is a 401 rather than a 403 on purpose: verifyAccessToken() in apps/api/src/auth/jwt.ts throws UnauthenticatedError for a token it cannot map onto a role, because from the caller's point of view the token is unusable rather than insufficient.

Cause. Almost always one of three things:

1. Only one of the two "Assert Roles" toggles is enabled. Role assertion is a two-layer setting: the project-level "Assert Roles on Authentication" (projectRoleAssertion: true) and the application-level "User Roles Inside Access Token". If either is off, roles are silently absent from the token. scripts/seed-zitadel.ts sets the project-level half on every run, and sends accessTokenRoleAssertion: true when it creates the application — but that is a create-only field there, so a re-run against an existing application never sets it, and it is not settable through the Zitadel custom resource. In practice the application-level toggle has to be flipped by hand in the console, in every environment including local: ${ZITADEL_ISSUER}/ui/console/projects/{projectId}/apps/{appId} → Token Settings.
2. The wrong claim key is being read. There are two: the generic urn:zitadel:iam:org:project:roles and the project-scoped urn:zitadel:iam:org:project:{projectId}:roles. Reading only the one the project does not emit yields an empty role set with no error thrown. rolesFromPayload() merges both — but it only looks at the project-scoped key when ZITADEL_PROJECT_ID is configured, so an unset ZITADEL_PROJECT_ID on an instance that emits only the scoped form produces exactly this symptom. Any other consumer of the token has to check both keys itself.
3. The claim is being treated as an array. It is a nested object whose top-level keys are role names. Array.isArray(claim) is false and iterating it yields nothing. Extract with Object.keys(claim) — which is what extractRoles() in packages/types does.

Fix. Check the toggles first — it is the most common cause. Then decode the access token and look at the raw claim:

# Inspect the claims of an access token (payload only, no verification).
node -e 'const [,p]=process.argv[1].split(".");console.log(JSON.parse(Buffer.from(p,"base64url")))' "$TOKEN"

Expected shape:

{
  "urn:zitadel:iam:org:project:roles": {
    "chef": { "201982826478953724": "lucille.localhost" }
  }
}

If the claim is absent entirely, it is the toggles. If it is present but your code sees no roles, it is the key or the array assumption. See authentication.

"Invalid audience" — every request returns 401

Symptom. Tokens verify cryptographically but every API call is 401 Invalid or expired access token — the same generic message every jwtVerify failure produces, which is why this one is easy to misdiagnose. Raise LOG_LEVEL=debug to see the underlying jose error.

Cause. The aud claim does not contain the API's Client ID. The SPA's authorization request is missing the audience scope.

Fix. The authorization request must include the scope below, which authorizationScopes() in apps/api/src/auth/oidc.ts appends automatically — but only when ZITADEL_PROJECT_ID is set:

urn:zitadel:iam:org:project:id:{projectId}:aud

The API cannot compensate for this — relaxing audience validation would accept tokens minted for other applications in the same Zitadel instance. Note the mirror-image trap: audience validation is only applied when ZITADEL_CLIENT_ID is configured, so an empty ZITADEL_CLIENT_ID silently skips the check instead of failing. After running the Zitadel seed, set both variables.

Intermittent 401s after working fine

Symptom. Authentication works, then every request starts failing with 401 for no apparent reason, often after days of uptime. A pod restart fixes it.

Cause. Zitadel rotates signing keys without prior notice. A verifier holding a statically fetched key starts rejecting everything the moment a rotation happens.

Fix. Use jose's createRemoteJWKSet, which resolves by the token's kid and re-fetches the key set on a miss. Create it once at module scope, not per request. Do not pre-fetch keys at boot and fail hard on an empty result — a low-traffic Zitadel instance can legitimately return an empty JWKS between rotations.

EDI 997 rejections

Symptom. An order shows as submitted in the Order Center, and Sysco has no record of it. Nothing failed anywhere.

Cause. The interchange was rejected at Sysco's gateway for a compliance error. Sysco returns a 997 Functional Acknowledgment for every interchange it receives, and a rejection is only visible there. An interchange rejected with no 997 handler is a completely silent failure.

Fix. The inbound poll checks for 997 files and treats AK5 as follows:

AK5	Meaning	Action
A	Accepted	Recorded, no action
E	Accepted with errors — compliance error	EDI_REJECTED audit event, Sentry alert, administrator notified
R	Rejected	Same as E; correct and resend

The usual root causes, in order of likelihood:

1. SE01 is wrong. It must count segments from ST through SE inclusive. Adding a segment and forgetting to bump the count is the single most common X12 defect.
2. ISA padding is wrong. The segment is exactly 106 characters; ISA06 and ISA08 are exactly 15 characters each, space-padded on the right.
3. ISA13 was reused. Control numbers must be unique and monotonically increasing per interchange.
4. ST02 does not match SE02, or GS06 does not match GE02, or ISA13 does not match IEA02.
5. Header and line statuses disagree on an inbound 855 — BAK02 = AT alongside a line ACK01 = IB.
6. Wrong partner identifiers. ISA sender and receiver IDs are location-specific and must come from Sysco's Trading Partner Implementation Guide. A guessed ID causes every interchange to bounce.

The raw interchange is always in vendor_documents.raw_content, which is where to start reading. See Sysco EDI.

Inbound EDI file cannot be parsed or correlated

Symptom. A Sentry alert for an unparseable or unmatched inbound file. The order is unchanged.

Cause. Either the file is malformed, or it references a purchase order this system does not know about — for example a legacy order placed before the Order Center existed.

Fix. Nothing is lost: the file is stored in vendor_documents with parsed_data = null and dispatch_id = null, precisely so it can be read later. Retrieve it, inspect the ISA for non-default separators, and confirm BAK03 against supplier_dispatches.reference. If the correlation is genuinely absent, the document is informational and can be left as-is.

DO Spaces: limited-access key versus bucket policy

Symptom. PutBucketPolicy fails, or creating a limited-access key for the bucket fails, or PDF uploads fail with an access error after an infrastructure change.

Cause. Limited-access Spaces keys and bucket policies are mutually exclusive. You cannot apply a bucket policy to a bucket that already uses a limited-access key, and you cannot create a limited-access key for a bucket that already has a bucket policy. Applying an IAM-style policy requires All (Buckets and Objects) permissions on the key.

Fix. The key in the lucille-spaces Vault secret must be a full-access key. Access is scoped by the bucket policy — a plain deny of public s3:GetObject — rather than by the key, with presigned URLs as the only read path. If a scoped key was provisioned by mistake, replace it with a full-access key and re-apply the policy; do not try to make the scoped key work.

Other Spaces surprises

Symptom	Cause	Fix
An S3 call returns NotImplemented	The operation is not part of the Spaces API. Object Lock, Intelligent-Tiering and S3 Select are absent	Use only operations listed in the Spaces API reference
Setting ACL: 'authenticated-read' fails	Spaces supports only private and public-read	Use private — it is what this project wants anyway
Presigned URL contains the bucket name twice	An endpoint-handling defect between AWS SDK v3.154 and v3.183	Stay on a recent v3 release (this project uses ^3.716.0) and keep a test asserting the URL shape
Uploads work locally but not against Spaces	forcePathStyle	true for MinIO, false for Spaces — Spaces uses virtual-hosted addressing
Bucket-level encryption cannot be enabled	Not supported by Spaces; SSE-C is	Private ACL plus bucket policy is sufficient for an internal audit archive

Duplicate purchase orders at a supplier

Symptom. A supplier received the same order twice.

Cause. Only one mechanism can produce this, and it is the one that looks safest: removeOnComplete: true combined with jobId-based deduplication. A job removed from the queue is no longer considered a duplicate, so a completed job's ID is freed immediately and a re-enqueue with the same ID succeeds.

Fix. The correct configuration, which this project uses:

• The structured deduplication: { id } option in Simple Mode, not a raw jobId. The key survives every automatic retry and is released only on terminal completion or failure.
• removeOnComplete: { age: 3600, count: 1000 } and removeOnFail: { age: 86400 } — never true.
• A PostgreSQL check on the dispatch row before any side effect, because BullMQ's stall detection re-queues a job that was mid-execution when its worker crashed, and that re-queue bypasses queue-level deduplication entirely.

Also note that job.remove() clears the deduplication key, so admin tooling must never remove an active dispatch job without re-checking the database state.

An order exists but was never dispatched

Symptom. An order sits in pending. Its supplier_dispatches rows exist in pending. No job is in Redis.

Cause. The process crashed between the Prisma transaction committing and queue.add() returning. Rare, but real — BullMQ offers no atomicity across Redis and PostgreSQL.

Fix. A pending dispatch older than a few minutes with no corresponding job is the detectable symptom, so re-enqueue it: the deduplication key is long gone, and the worker's idempotency check makes a re-enqueue safe even if a job did in fact run. The durable fix is a transactional outbox — write the pending marker inside the order transaction and have a relay move it into Redis. See queues and jobs.

Failed jobs pile up, or a count limit seems ignored

Symptom. The failed set is larger than removeOnFail.count allows, and lowering the limit changes nothing.

Cause. BullMQ auto-removal is lazy: the completed set is pruned only when a job completes, and the failed set only when a job fails. On a low-volume queue that can be a long wait.

Fix. Expected behaviour, not a bug. Call queue.clean() on worker startup, or schedule a periodic cleanup.

Verification

The local-environment, Zitadel and audience entries on this page were checked against the source rather than against the plan: scripts/native-services.sh, scripts/seed-zitadel.ts, scripts/postgres-init/01-create-databases.sql, docker-compose.yml, apps/api/src/auth/jwt.ts, apps/api/src/auth/oidc.ts, apps/api/src/config/index.ts, apps/api/vitest.config.ts, apps/api/tests/global-setup.ts, apps/worker/vitest.config.ts and IMPLEMENTATION.md. Quoted error strings, exit codes, timeouts and toggle paths come from those files. The Chromium, SFTP, EDI 997 and DO Spaces entries further down originate in the research briefings under docs/research/ and describe third-party behaviour that cannot be verified from this repository alone.

Where to go next

• Local development — the stack these local failures happen in, including the Docker-free fallback and the Zitadel seed.
• Queues and jobs — the deduplication and retention settings in full.
• Authentication — the claim shape and the two toggles.
• Sysco EDI — segment-level detail behind 997 rejections.
• Observability — the signals that surface these failures.