# 0023. Shipping the server, and backups

- Status: Accepted
- Date: 2026-10-28

## Context

M9 ships the hub to people who are not developers, on whatever they already have:
- a Raspberry Pi or mini PC at the shop;
- a $5 VPS;
- Docker;
- a hosting platform.

ADR 0011 fixes the shape (one container, one process, one port, one data folder, amd64 and
arm64), and ADR 0012 fixes the cost (SQLite per shop, no required services, a resource budget,
backups to any S3-compatible storage). What remained was how to package, release and back up.

## Decision

**What runs.**
- The server is one esbuild bundle. `scripts/package-server.ts` puts it in a folder with:
  - the built back office;
  - better-sqlite3's JavaScript and its prebuilt binary for the platform, and nothing else from
    `node_modules`.
- The Docker image and the release tarballs are that folder.
- better-sqlite3 13 ships prebuilt binaries for Linux, macOS and Windows on x64 and arm64, so its
  install script is not allowed to run. No compiler is needed anywhere.
- The server starts without `--enable-source-maps`. Loading the map costs about 75 MB, half the
  idle budget. `NODE_OPTIONS=--enable-source-maps` turns it on for debugging.

**The image.**
- **Base:** `node:22-bookworm-slim`. It is glibc, like the hosts the release tarball targets.
  Alpine would save a few MB but makes the image and the tarball differ.
- **User:** it runs as `node`.
- **Data:** in `/data`.
- **Size:** about 78 MB compressed, of which about 3 MB is OpenKedai.
- **Health check:** none in the image, so idle stays idle. The Compose file adds one each minute.

**Releases.** A `v*` tag runs `release.yml`:
- Each architecture is built on a native runner (no QEMU), checked against the resource budget,
  and pushed by digest.
- A stress test runs, compressed to 6 minutes.
- The digests are merged into one multi-arch image:
  - on GitHub Container Registry, and on Docker Hub when it is configured;
  - tagged with the version, the minor version and `latest`. Pre-releases move neither.
- The installer's tarballs are copied out of the same images and attached to a GitHub release
  with `SHA256SUMS`.
- Runners come from repository variables (`LINUX_RUNNER`, `ARM_RUNNER`), so moving to another
  runner provider is a settings change.

**The Linux installer** (`deploy/install.sh`):
- **Node.js:** it downloads its own Node.js 22 from nodejs.org, rather than using the
  distribution's, which varies from release to release.
- **Checks:** both downloads are checked against their published checksums.
- **Layout:** the hub goes in `/opt/openkedai/releases/<time>`, with a symlink to the current
  one; data in `/var/lib/openkedai`, settings in `/etc/openkedai/openkedai.env`.
- **Service:** a hardened systemd service runs as its own user.
- **Updates:** running the script again updates the hub and keeps one previous release.

**Hosting platforms.**
- **Fly.io:** the image, a volume, and stopping when idle.
- **Render:** a blueprint building the image, on a paid instance, because only those have disks.
- **Railway:** `railway.json`; the volume is added by hand until there is a template.
- **DigitalOcean:** a Droplet with cloud-init running the installer. App Platform keeps no files
  between deploys, so it can't hold the data.
- **Platform address:** on Render, Railway and Fly the hub takes its public address from the
  platform's variables, and leaves mDNS off.

**Backups.**
- **Snapshots:**
  - Every `OPENKEDAI_BACKUP_INTERVAL_HOURS` (24 by default), the hub copies every SQLite
    database in the data folder with SQLite's online backup.
  - The copies go to `<data>/backups/<time>/`, gzipped, with a manifest written last; the newest
    `OPENKEDAI_BACKUP_KEEP` (7) are kept.
- **S3, when the owner sets a bucket:** each snapshot is uploaded with a small built-in Signature V4
  `PUT`, checked against AWS's worked examples. No AWS SDK is added (image size, dependencies).
- **No Litestream:** continuous replication would add a second process and a binary, against ADR
  0011. It isn't needed: every device holds the log too. So ADR 0012's backups that "stream" to
  S3-compatible storage are uploads of each snapshot.
- **Part of the server, not a plugin:** ADR 0018 says the hub's own features use the plugin
  kernel where practical. Backups don't, because they copy every database, including the hub's
  registry, which plugins can't see.
- **Before migrations:** before migrations change a shop's existing database, a plain copy is kept
  in `<data>/backups/pre-migration/`.
- **After a restore:**
  - Each copy of a shop's database (in a snapshot, or before migrations) gets a new **log id**,
    and the hub sends its log id with every page of events.
  - A device that sees a new log id knows the hub was put back from a backup. It sends all its own
    events and catalog again, and reads the hub's log from the start. The hub and the device each
    keep one copy of every event.
  - So sales made after the snapshot come back from the devices that made them, and every device
    gets the others'.
  - A first version compared the length of the hub's log with what each device had seen. It
    missed a device when another device's resent events made the log longer first. The log
    length is still checked, for a hub that lost its last writes without a restore.

## Consequences

- **Every target met, measured:** on a release, the budget measures an image of about 79 MB, a
  cold start of 0.7 s, 119 MB at idle, 0.4% of a core idle, 110 MB with three devices selling
  30 times faster than the brief's busy shop, and 2.4 KB sent per order.
- **A server bug that only shows in production builds:** without source maps, its stack trace
  points into the bundle. Turn the maps on to read it.
- **Images and tarballs need the repository public:** they are on GitHub (Container Registry,
  releases, raw files). While it is private, they need a token, or a local build
  (`docker build -f deploy/Dockerfile .`).
- **Restoring loses some changes:** back office changes after the snapshot (accounts, settings,
  devices paired since) are lost. Devices paired since must pair again; their sales then follow.
- **Old devices after a restore:** a device older than this change reads on correctly after a
  restore, but doesn't resend its own events. Update devices before restoring a hub.
- **Every restore costs one full resend:** even a snapshot that is up to date has a new log id,
  so each device sends and reads everything once. It is rare, and duplicates are skipped.
- **A copy made some other way** (a disk image, `cp` of the data folder) keeps the old log id.
  It is treated like a power cut: the devices notice if the log is shorter than what they've
  seen, which can miss a device. Use the hub's snapshots to restore.
- **Power cuts:** the log-length check covers a hub that loses its last writes in a power cut.
  Its databases use `synchronous=NORMAL`, so a Raspberry Pi losing power can drop the latest
  transactions. Its log then ends before what devices have seen, and they send theirs again.