Central Kitchen — PDF purchase order

Central Kitchen is Lucille's own facility, acting as a supplier to the individual restaurants. Orders for it are delivered as a PDF purchase order attached to an email. The PDF is also archived in object storage, because it is the artefact the audit trail points at and the file a chef can re-download.

Receiving-side order management for Central Kitchen staff is explicitly out of scope; this system only sends purchase orders to them.

The dispatch handler

supplier-dispatch job, supplierType = 'central_kitchen'
    │
    ├─ shared prologue (payload parse, load, idempotency check, line-item
    │  intersection, active-supplier check) — see supplier integrations
    ├─ resolveEmailSupplierConfig(supplier) → emailAddress, ccAddresses, facilityName?
    ├─ renderPurchaseOrderHtml({ orderNumber, orderDate, restaurantName,
    │      restaurantLocation, supplierName: facilityName ?? supplier.name,
    │      deliveryNotes, placedByEmail, lineItems, generatedAt })
    ├─ renderHtmlToPdf(html)                     → Buffer
    ├─ uploadPdf(key, pdf)                       → { bucket, key, url }
    │     key: central-kitchen/{orderId}/{objectKeyTimestamp}.pdf
    │     ContentType: application/pdf, ACL: private
    ├─ buildCentralKitchenOrderEmail(...)        → { subject, html, text }
    ├─ sendMail({ to, cc?, subject, html, text, attachments: [PO_{orderNumber}.pdf] })
    └─ markSubmitted():
         supplier_dispatches.status = 'submitted', reference = messageId,
           pdf_url = uploaded.url, submittedAt, attempt, failureReason = null
         order_line_items.status = 'submitted'
         audit_events SUPPLIER_SUBMITTED { supplierId, attempt, pdfUrl, pdfKey,
                                           pdfBytes, subject, reference, supplierType }
         recomputeOrderStatus(orderId)

objectKeyTimestamp is the ISO instant with -, : and . stripped — 20260813T140530123Z — which is sortable and filename-safe. The attachment is named PO_{sanitised orderNumber}.pdf (an underscore, and the order number, not the order UUID); sanitizeFilenameToken collapses anything outside A-Za-z0-9._- to -. The email subject is the same Purchase Order – {restaurantName} – {orderNumber} used by the Costco channel, and the covering message names the attached file so the reader looks for it. The body also repeats the line-item table: the PDF is the document of record, but a reader should not have to open an attachment to see what was ordered.

Order matters: the PDF is uploaded before the email is sent. If the upload fails there is no email for an artefact nobody can show, and the job retries cleanly. The cost is that a retry re-renders and re-uploads under a new timestamped key, leaving the previous object behind — deliberate, and cheap at a few tens of kilobytes per order. Overwriting a fixed key would destroy the artefact of an attempt that may already have been emailed. supplier_dispatches.pdf_url always names the object attached to the email that actually succeeded.

The PDF template

pdf/po-template.ts returns one self-contained HTML document. Two hard rules: everything is inline (no stylesheet links, no webfonts, no images by URL, because renderHtmlToPdf waits for networkidle0 and a single external reference turns a 200 ms render into a timeout in a pod with no egress), and every interpolated value is escaped with escapeHtml.

Section	Fields
Header	Lucille / Order Center brand block; Purchase Order; the order number; the order date
Meta	Supplier (config.facilityName → supplier name → literal Central Kitchen), Ship to (restaurant name and location on separate lines), Order date
Line table	#, Item, Supplier SKU (monospaced), Qty (right-aligned), Unit, Unit size
Total line	{n} line items, singular for one
Notes	Delivery notes block, white-space: pre-line, omitted entirely when there are none
Footer	Placed by {email} or Placed via Lucille Order Center, and Generated {yyyy-MM-dd HH:mm UTC}

Note the differences from the email table: the PDF has a line-number column and calls the SKU column Supplier SKU, and a missing unit size renders as an em dash — rather than the email's -. Dates are formatted in UTC and the clock is injected as generatedAt, so identical input yields identical bytes.

HTML to PDF with puppeteer-core

The worker uses puppeteer-core with system-installed Chromium, never the puppeteer package with its bundled browser download. The launch arguments are exported so tests can assert them:

export const CHROMIUM_LAUNCH_ARGS: readonly string[] = [
  '--disable-dev-shm-usage',
  '--disable-gpu',
  '--no-sandbox',
  '--disable-setuid-sandbox',
];

export const DEFAULT_PDF_MARGIN = {
  top: '18mm',
  right: '14mm',
  bottom: '18mm',
  left: '14mm',
} as const;

renderHtmlToPdf checks that the binary is executable, launches, renders and always closes:

if (!isChromiumAvailable(executablePath)) throw new ChromiumUnavailableError(executablePath);
browser = await puppeteer.launch({
  executablePath,
  headless: true,
  args: [...CHROMIUM_LAUNCH_ARGS],
  timeout,
});
const page = await browser.newPage();
page.setDefaultTimeout(timeout);
await page.setContent(html, { waitUntil: 'networkidle0', timeout });
const pdf = await page.pdf({
  format: options.format ?? 'A4',
  margin: options.margin ?? { ...DEFAULT_PDF_MARGIN },
  printBackground: true,
  landscape: options.landscape ?? false,
  timeout,
});

A browser is launched and closed per call, inside a finally that swallows and logs a close failure. PDF jobs are rare, a per-call browser cannot leak state between orders, and a crashed render cannot poison the next one. An orphaned Chromium holds roughly 100 MB against a 1 GB pod limit, so the close is not optional.

Variable	Default	Used for
PUPPETEER_EXECUTABLE_PATH	/usr/bin/chromium	The binary to launch, and what isChromiumAvailable probes with X_OK
PUPPETEER_TIMEOUT_MS	30000	Launch timeout, page default timeout, setContent timeout and page.pdf timeout
PUPPETEER_SKIP_DOWNLOAD	set in the image	Stops any bundled-browser download during install

isChromiumAvailable is also one of the worker's /readyz checks (alongside PostgreSQL and Redis; /healthz deliberately touches nothing external), so a pod that cannot render a PDF is visibly not-ready rather than discovered by the first PDF job of the day. A missing binary raises ChromiumUnavailableError, which the dispatch handler classifies as permanent: every retry in this pod would fail identically.

The flags, and why each one is there

Flag	Why
--disable-dev-shm-usage	The single most important flag in Kubernetes. A container's default /dev/shm is 64 MB, which is not enough for Chromium; this writes shared memory to /tmp instead. Without it you get intermittent mid-render crashes. A pod spec has no shm_size; the alternative is an emptyDir with medium: Memory
--disable-gpu	There is no GPU in the pod
--no-sandbox and --disable-setuid-sandbox	A documented trade-off, not a default. The container runs as non-root pptruser, so the setuid sandbox cannot initialise anyway, and the user namespaces a real sandbox needs are not available

On --no-sandbox: disabling the sandbox is genuinely dangerous when rendering untrusted pages. Here the risk is bounded by content provenance — this renderer only ever loads server-controlled HTML built by po-template.ts from database values, every one of them escaped, never a user-supplied URL and never user-supplied markup. That property is the whole justification; if it ever changes, the sandbox must come back. It is recorded here so nobody later assumes the flag was accidental.

The image and the non-root user

The API, worker and CronJob share one image, apps/api/Dockerfile, a four-stage build whose runtime stage is node:20-slim:

FROM node:20-slim AS runtime
ENV NODE_ENV=production \
    LANG=C.UTF-8 \
    PNPM_HOME=/pnpm \
    PATH=/pnpm:$PATH \
    PUPPETEER_SKIP_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium

RUN apt-get update && apt-get install -y --no-install-recommends \
      chromium ca-certificates openssl dumb-init \
      fonts-liberation fonts-dejavu-core \
      libasound2 libatk-bridge2.0-0 libatk1.0-0 libcairo2 libcups2 \
      libdbus-1-3 libdrm2 libgbm1 libglib2.0-0 libgtk-3-0 libnspr4 \
      libnss3 libpango-1.0-0 libx11-6 libxcb1 libxcomposite1 \
      libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 \
    && rm -rf /var/lib/apt/lists/*

# Non-root user. UID 1001 matches the Kubernetes securityContext runAsUser.
RUN groupadd --system --gid 1001 pptruser \
 && useradd --system --uid 1001 --gid pptruser --create-home --shell /usr/sbin/nologin pptruser
USER pptruser

Decision	Reason
Debian node:20-slim, not Alpine	Alpine is musl libc; the Chromium binary is built against glibc
PUPPETEER_SKIP_DOWNLOAD=true	No bundled-browser download during install, so image builds do not depend on Google's CDN
System chromium via apt	Reproducible and patched by the base image's security updates
Explicit shared library list	Missing even one produces a cryptic startup failure. libgbm1 and libnss3 are the classic offenders
Fonts installed	Without them the rendered PDF is full of tofu boxes
Non-root pptruser, UID 1001	Matches the intended pod runAsUser, and is part of what makes --no-sandbox defensible
dumb-init as entrypoint	PID 1 signal handling and child reaping — relevant because Chromium forks child processes
Multi-stage, --no-install-recommends	The Chromium binary dominates the image size; the production dependency tree is installed separately with --prod

The image's default CMD is apps/api/dist/entrypoint.js, a dispatcher that starts the API, the worker, the CronJob or the migration based on APP_MODE / WORKER_MODE / CRON_MODE / MIGRATE_MODE — see deployment.

Kubernetes security context — intent versus reality

The worker's securityContext is currently declarative intent only

infra/dev/lucille-worker.yaml declares exactly this:

securityContext:
  runAsNonRoot: true
  runAsUser: 1001 # pptruser in apps/api/Dockerfile
  seccompProfile:
    type: RuntimeDefault
containerSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ['ALL']

but the file's own header records that the shared regular-deployment template at v0.1.0 renders neither command/args nor any securityContext. The command gap was resolved by making the image's entrypoint branch on WORKER_MODE=true (which is rendered, via env). The securityContext gap is still open and tracked in infra/README.md. In practice the pod inherits the image's USER pptruser (UID 1001), so it does run non-root — but that is the image's doing, not an enforced pod policy. Treat the seccomp profile and dropped capabilities as designed but not yet enforced.

The resources block is rendered by the chart: requests 512Mi / 250m, limits 1Gi / 1000m. The Deployment runs replicas: 2; two replicas are safe only because both the BullMQ deduplication key and the handler's supplier_dispatches.status check stand between them and a duplicate purchase order.

AppArmor on recent Ubuntu nodes

Ubuntu 23.10 and later ship an AppArmor profile that prevents Chrome for Testing binaries from using user namespaces, producing No usable sandbox! at launch even when kernel.unprivileged_userns_clone=1. It does not affect the --no-sandbox path used here, but it does block a future migration to genuinely sandboxed operation, so test on the actual DOKS node operating system before relying on one.

Object storage

apps/api/src/storage/index.ts owns the S3 client, the upload and the presigning; the worker imports it as @lucille/api/storage. DigitalOcean Spaces in the cluster, MinIO locally, one code path.

Setting	Default	Notes
S3_BUCKET	lucille-po-pdfs	bucketName: lucille-po-pdfs-dev on dev — Spaces names are globally unique
S3_ENDPOINT	unset	http://minio:9000 locally; https://{region}.digitaloceanspaces.com for Spaces
S3_REGION	us-east-1	e.g. nyc3 for Spaces
S3_FORCE_PATH_STYLE	false	true locally for MinIO; Spaces is virtual-hosted
S3_PRESIGN_EXPIRY_SECONDS	3600	Default lifetime of a presigned GET

Credentials are only attached to the client when both S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY are present. The infra layer maps the Spaces secret's keys onto these S3_* names, and normaliseEnv also accepts the SPACES_* and AWS_* aliases.

The key, the URL, and the presigned link

This is the part that is easy to get wrong, so plainly:

1. uploadPdf writes central-kitchen/{orderId}/{timestamp}.pdf with ACL: 'private' and returns { bucket, key, url }, where url is the canonical, non-presigned object URL built by objectUrl() — {endpoint}/{bucket}/{key} when path-style, otherwise https://{bucket}.{host}/{key}.
2. supplier_dispatches.pdf_url stores that full canonical URL, not the bare key. It is not publicly readable; nothing serves it directly.
3. The chef-facing download is GET /api/orders/:id/dispatches/:dispatchId/pdf. getDispatchPdfLink in apps/api/src/domain/orders.ts loads the order under the caller's access rules, finds the dispatch, recovers the key with keyFromUrl(dispatch.pdfUrl) and returns presignPdf(key) — a { url, expiresAt } pair for a short-lived presigned GET, one hour by default, generated per request. A dispatch with no pdfUrl, or a URL no key can be derived from, is a 404.

That is the only access path. No PDF is ever public, and no presigned URL is ever persisted.

Spaces specifics worth knowing

Everything in this table comes from docs/research/digitalocean-spaces-s3-api-compatibility.md, which is the source the storage module cites.

Point	Detail
Signature version	Spaces supports SigV4, which SDK v3 uses by default. No configuration needed
Addressing style	Virtual-hosted is correct for Spaces (forcePathStyle: false); MinIO needs path-style — hence the env flag
ACLs	Only private and public-read exist. authenticated-read and the rest of the AWS ACL model do not. private is exactly what is wanted
Presigned URLs and the CDN	Presigned URLs bypass the Spaces CDN and mixing them can double the bandwidth charge. Irrelevant here — private artefacts served from the origin
Multipart upload	Supported, and irrelevant: a PO PDF is far below the 5 GB single-PUT limit
Object tagging	PutObjectTagging / DeleteObjectTagging are supported (an earlier assumption that they were not is outdated). Bucket-level DO tags are not
Bucket policies	Supported on standard buckets, which is what the Chrono template provisions; not on Cold Storage buckets
Rate limits	800 operations per second on a new bucket — orders of magnitude above one PDF per order
Encryption	SSE-C is supported; bucket-level encryption settings are not
Unsupported operations	Return a standard NotImplemented error. Avoid Object Lock, Intelligent-Tiering and S3 Select

The access key must be a full-access key

Limited-access Spaces keys and bucket policies are mutually exclusive. You cannot apply a bucket policy to a bucket that already uses a limited-access key, and you cannot create a limited-access key for a bucket that already has one.

infra/dev/lucille-storage.yaml therefore states in its header that the key written to the lucille-spaces Vault secret must be a full-access key, because a deny-public-read policy is applied to enforce private-only access. The failure mode is confusing — the policy call is rejected, not the key.

Note that the bucket policy document itself is not checked into this repository; the service definition sets acl: private and the policy is applied at provisioning time. The recommended policy (a plain deny of s3:GetObject for Principal: "*", leaving presigned URLs as the only access path) is in the research brief. Designed but not verified from this repository: no file here proves which policy is live on the dev bucket.

One historical SDK note from the same brief: endpoint handling changed between AWS SDK v3.154 and v3.183 and introduced a "double bucket" defect in presigned URLs. Both apps/api and apps/worker depend on @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner at ^3.716.0, which is well clear of it.

Failure handling

Failure	Classification	Behaviour
Chromium crash or render timeout	Retryable	Retried by BullMQ; the browser is closed in a finally either way
Chromium binary missing or not executable	Permanent	ChromiumUnavailableError → job.discard(), dispatch failed immediately. It is a deployment fault, and /readyz already reports the pod not-ready
Missing shared library, /dev/shm exhaustion	Retryable	Presents as a launch or mid-render crash — see troubleshooting
PutObject failure	Retryable	No email is sent without an archived PDF
SMTP failure	As Costco	4xx and connection errors retry; 5xx or no accepted recipient is permanent
Invalid suppliers.config	Permanent	SupplierConfigError; there is no fallback for emailAddress

On terminal failure the dispatch goes to failed, its lines to dispatch_failed, the order is rolled up to partial or failed, and an alert naming the supplier, order, dispatch, attempt and reason goes to ADMIN_ALERT_EMAIL.

Configuration

centralKitchenSupplierConfigSchema accepts one key more than the Costco schema:

Key	Required	Notes
emailAddress	yes	Must be a valid email address; no environment fallback
ccAddresses	no, defaults to []	Every entry must be a valid email address
facilityName	no	1–200 characters. When set it replaces the supplier name in the PDF Supplier block — useful once Lucille operates more than one production facility

Automated test coverage

• apps/worker/tests/pdf/po-template.test.ts — the template: a standalone document that fetches nothing over the network, every field present, optional fields degrading gracefully (no notes block, no location line, generic footer, em dash for a missing unit size), Central Kitchen as the default supplier name, HTML escaping of every interpolated field, and determinism including UTC formatting.

• apps/worker/tests/pdf/renderer.test.ts — the renderer against real Chromium: the configured executable path resolves, a missing binary yields ChromiumUnavailableError rather than a Puppeteer stack trace, CHROMIUM_LAUNCH_ARGS contains all four flags, DEFAULT_PDF_MARGIN is asserted exactly, and a rendered buffer starts with the %PDF- magic bytes.

• apps/worker/tests/handlers/central-kitchen.test.ts — the whole chain with every production adapter: a real Chromium render, a real upload to MinIO, a real email through Mailpit. The assertions are on the bytes — the stored object begins with %PDF- and the attachment Mailpit decoded is byte-identical to it — and a second case proves the upload happens before the send, so a send failure still leaves the PDF stored.

• apps/api/tests/orders.query.test.ts — the presigned download endpoint: a 200 whose url contains the key and an X-Amz-Signature with a future expiresAt, a 404 when the dispatch has no PDF, and a 404 when the dispatch belongs to a different order. apps/api/tests/rbac.test.ts pins the route to the chef role.

See testing.

Local development

MinIO stands in for Spaces. The API creates the bucket at startup when NODE_ENV is not production (ensureBucketExists, which logs and continues if it cannot). Browse uploaded PDFs in the MinIO console at http://localhost:9001 (minioadmin / minioadmin), and read the outbound email with its attachment in Mailpit at http://localhost:8025.

Note that S3_FORCE_PATH_STYLE=true locally and false against Spaces, which also changes the shape of the canonical URL written to pdf_url. That is the one place where the local emulator and production genuinely differ, so presigned-URL behaviour is worth verifying against a real Spaces bucket before a production release.

Verification

The claims on this page were checked against apps/worker/src/handlers/central-kitchen.ts, apps/worker/src/pdf/po-template.ts, apps/worker/src/pdf/renderer.ts, apps/worker/src/handlers/supplier-dispatch.ts, apps/worker/src/supplier-config.ts, apps/api/src/storage/index.ts, apps/api/src/mail/index.ts, apps/api/src/config/index.ts, apps/api/src/domain/orders.ts, apps/api/src/routes/orders.ts, packages/types/src/supplier.ts, apps/api/Dockerfile, infra/dev/lucille-worker.yaml, infra/dev/lucille-storage.yaml, the tests under apps/worker/tests/pdf/, and apps/worker/tests/handlers/central-kitchen.test.ts. The one claim not verifiable from this repository is which bucket policy is live on the dev Space; it is called out inline above.

Where to go next

• Troubleshooting — /dev/shm crashes, "No usable sandbox", and the Spaces key conflict.
• Deployment — the shared image, the entrypoint dispatcher and the secrets.
• Supplier integrations — the shared retry, idempotency and failure path.
• Local development — MinIO and Mailpit.