Deploying the relay (`serve remote`)

One relay serves one or more pools; each pool uses exactly one relay URL. The relay stores signed, age-encrypted memory envelopes and public membership metadata in its databases: by default one SQLite file, relay.db, or PostgreSQL (see Databases), where several relay processes can serve the same pools. It loads no device identity and never needs or receives memory decryption keys — you can run it on any host you trust for availability, not confidentiality (details in Security).

Databases

Without configuration the relay keeps everything in <data_dir>/relay.db (SQLite) and initializes it on the first start. For PostgreSQL, create an empty database owned by the relay's database user and set MERKHAIN_RELAY_DATABASE_URL=postgres://… (or database.relay.control.url in config.json); one database then hosts every relay role. Larger deployments add pool and replay shards. Layouts, first start of a sharded relay, explicit upgrades (merkhain migrate up --relay), several relay processes behind a load balancer, moves, backups and failover: Databases.

Transport

Clients accept https:// relays, and http:// only on loopback. The relay itself refuses a plain-HTTP listener on a non-loopback address. Pick one:

Native TLS: MERKHAIN_REMOTE_LISTEN=0.0.0.0:443, MERKHAIN_REMOTE_TLS_CERT=/abs/fullchain.pem, MERKHAIN_REMOTE_TLS_KEY=/abs/privkey.pem (or remote_tls_cert_file/remote_tls_key_file in config.json). A renewed certificate, or changed file paths in config.json, are served from the next TLS handshake; a keypair that does not load keeps the previous one in service (logged). Switching TLS on or off needs a restart.
Reverse proxy on the same host: keep the default remote_listen 127.0.0.1:7782 and terminate TLS in Caddy/nginx. A proxy on another host needs "remote_listen": "0.0.0.0:7782", "remote_trusted_proxy": true in config.json. The proxy must forward path, query, body and all Merkhain-* headers unchanged (requests are signed over them; if a proxy answers the membership checkpoint route with its own error status, owner devices fall back to one membership revision about every 36 hours, which grows the membership history, so fix the proxy when the log says so; a redirect or a 200 page from the proxy gets no fallback and writers wait), and must allow request bodies of 8 MiB (the relay's limit; a key rotation in a 1,024-device pool sends about 2–3 MB). Relays and clients older than the 1,024-device limit reject pools with more than 256 devices, so upgrade the relay and every client before a pool grows past 256 devices.

Caddy example: relay.example.com { reverse_proxy 127.0.0.1:7782 }.

The operator token (relay-operator-token in the config directory) can be exchanged for an 8-hour browser session at POST /ui/api/v1/auth/login. Set remote_public_origin (or MERKHAIN_REMOTE_PUBLIC_ORIGIN) to the https origin browsers use, e.g. https://relay.example.com; with native TLS or remote_trusted_proxy login is disabled until it is set, and an http:// origin is refused. Only a plain-HTTP loopback relay may omit it for local development. Forwarded headers (X-Forwarded-*, Forwarded) are never used. A browser session can read relay metadata (pools, public team labels, members, membership history, enrollment status, content-free sizes) and issue single-use creation grants: a bzg_… credential that lets one local instance create one pool of the granted kind within 10 minutes, without ever seeing the operator token. Hand it over as the fragment #relay=<origin>&grant=<grant> (never a query string). A session cannot bootstrap a pool itself, and membership changes still need owner-signed device requests. Sessions end on logout and on every relay restart; issued grants survive restarts until they expire.

Relay Administration, the relay's browser UI, is served at <remote_public_origin>/ui/ (sign in with the operator token). It shows relay health, teams, members and computers, and invitation status, and can issue and revoke permissions to create a team. Allow someone to create a team opens a review of the public label, relay and one-use, ten-minute scope; Issue permission then produces a team creation link to this relay's public recipient page, /ui/create-team, with the grant in the URL fragment. Send it privately to the team's owner. That page needs no operator sign-in and makes no requests; its Open My Memory opens the owner's local merkhain UI (default http://127.0.0.1:7781, changeable on that page, loopback only) at /ui/sharing/handoff#…, and the owner can instead paste the link into My Memory's Create a team. Invite, cancel and remove are links to the owner's My Memory, where the signed-in owner confirms; Relay Administration itself never changes membership or reads team memory.

Reading the console

Overview separates the last successful health check, published statistics freshness, workers and maintenance loops, request traffic, and storage/runtime details. Worker progress is not evidence of backups or remaining quota. Route 4xx counts describe client refusals (including authentication and validation); 5xx describe server failures. Neither identifies the cause by itself. Traffic counters reset with the process.

Teams, computers, invitations and creation permissions show the loaded range. Filters and sorting affect only loaded records and stay in page memory; load more to include more records. A failed refresh retains the last successful inventory marked stale. A successful refresh replaces it with the first page. Public team labels are not private team names. Computer records are grouped by public person ID, not inferred identities.

Permission revocation requires confirmation and a successful server response. A used or reserved permission may no longer be revocable; refresh the inventory to inspect its state. Revoking a permission never removes an existing team. If issuance loses its response, inspect the refreshed inventory before deliberately issuing another: the secret cannot be retrieved. Revoke the unused permission or wait for expiry. The issued link lives only in the page until opened, dismissed or navigated away from; copied links remain the recipient’s responsibility.

Settings reports server, browser access, operator session and storage facts read-only. Disk capacity, storage quotas, backup status and service limits are Not reported; that does not mean unlimited or healthy. Configuration changes remain server-side. Managed Personal hosting does not expose this standalone operator console.

systemd

sudo install -m 0755 merkhain /usr/bin/merkhain     # or install the .deb/.rpm
sudo install -Dm644 share/systemd/merkhain-relay.service /etc/systemd/system/merkhain-relay.service
sudo systemctl daemon-reload && sudo systemctl enable --now merkhain-relay
curl -fsS https://relay.example.com/v1/health          # {"api_version":"v1","mode":"remote",...}

The unit uses a dynamic, unprivileged user, StateDirectory=/var/lib/merkhain-relay, strict sandboxing, and loopback by default; its comments show the native-TLS variant (certificates passed with LoadCredential) and the PostgreSQL variant: put MERKHAIN_RELAY_DATABASE_URL=postgres://… into a root-owned /etc/merkhain/relay-database.env (mode 0600) and uncomment EnvironmentFile= with systemctl edit merkhain-relay. After an upgrade, run the new binary's migrate up --relay (the old unit may keep serving meanwhile), then restart the unit (see Databases).

Container

The image is gcr.io/distroless/static + the static binary, non-root (UID 65532), state in /var/lib/merkhain:

go run ./packaging/release -targets linux/amd64,linux/arm64 -packages off
docker build -f packaging/container/Dockerfile -t merkhain-relay .
docker run -d --name merkhain-relay -p 443:7782 -v merkhain-relay:/var/lib/merkhain \
  -v /etc/merkhain/tls:/tls:ro -e MERKHAIN_REMOTE_TLS_CERT=/tls/fullchain.pem -e MERKHAIN_REMOTE_TLS_KEY=/tls/privkey.pem \
  merkhain-relay
docker cp merkhain-relay:/var/lib/merkhain/config/relay-operator-token .

The key/cert files must be readable by UID 65532. Behind a proxy instead, mount a config.json with remote_trusted_proxy: true at /var/lib/merkhain/config/config.json. packaging/container/smoke.sh checks the image (plain HTTP refused, TLS health, token persisted, no identity created).

With PostgreSQL, pass the URL to every container that uses the relay's configuration and keep the volume for the operator token. Migrations run in a one-off container from the same image (the entrypoint is the merkhain binary):

docker run --rm -e MERKHAIN_RELAY_DATABASE_URL=postgres://bz:secret@db/relay merkhain-relay migrate up --relay
docker run -d --name merkhain-relay -p 443:7782 -v merkhain-relay:/var/lib/merkhain \
  -e MERKHAIN_RELAY_DATABASE_URL=postgres://bz:secret@db/relay … merkhain-relay

migrate up is needed before the first start of a sharded relay and after every upgrade (with the new image, while the old container keeps serving); a single, empty database is also initialized by the first start. For shards, mount a config.json with the database.relay section and run migrate up --relay and shard register … the same way (Databases). packaging/ci/postgres-smoke.sh runs the container smoke against PostgreSQL.

Operator token

On first start the relay writes <config_dir>/relay-operator-token (owner-only; systemd: /var/lib/merkhain-relay/config/relay-operator-token). It is only needed to create pools (merkhain pool create --operator-token-file FILE); all other requests are authenticated by device signatures and pool membership. Hand it only to people who may create pools. Rotate it by deleting the file and restarting; existing pools are unaffected.

Over plain HTTP (a loopback relay) pool create first asks the relay to prove it holds the operator token, bound to the address the client connected to, and sends the token only after that proof checks out. Another local account that binds the relay's port while it is stopped therefore never receives the token. Relays older than this check refuse the challenge, so create pools on them over https:// or with a single-use creation grant. Creation grants are not proven this way; over HTTPS the TLS certificate authenticates the relay. The relay answers the proof only on plain-HTTP connections it accepted on a loopback address, never over native TLS or on a non-loopback listener: an answer lets its holder test token guesses offline. A reverse proxy on the same host forwards to a loopback socket, so the relay still answers requests that reach it through the proxy; keep the generated token there.

Storage quotas

The relay refuses to store more envelope bytes or records than its quotas allow, per pool, per member (a user, counting all of the user's devices and every record the user ever stored in that pool) and per shard (all pools stored in one pool database). The defaults are 16 GiB and 10,000,000 records per pool, 4 GiB and 2,000,000 records per member, and 64 GiB and 40,000,000 records per shard; set them in config.json and restart:

{"relay_quota": {"pool_max_bytes": 8589934592, "pool_max_records": -1, "member_max_bytes": 1073741824, "member_max_records": 0,
  "shard_max_bytes": 34359738368, "shard_max_records": 0}}

In the default layout the pools share one SQLite database with the control catalog and replay data, so keep shard_max_bytes well below the free disk space: once the pools reach it, appends to every pool of that shard are refused while the relay's control operations keep their room. Keep pool_max_bytes below the shard quota so a single pool cannot take every other pool's share. On PostgreSQL, appends to different pools commit concurrently, so the shard quota can be exceeded by at most one envelope (512 KiB) per append in flight; the pool and member quotas are exact.

0 keeps a limit's default and -1 removes it; with both shard limits at -1, appends skip reading the shard's usage. The relay keeps that running total either way, so it is right when a limit is turned on: removing the limits does not remove the cost of maintaining it per append, which on PostgreSQL includes appends to pools of one stripe (about 1 in 64 pairs) committing one after the other. An append over a quota gets 409 with reason quota_exceeded and a message naming the quota, the usage and the limit; nothing is stored. Retries of records already stored always succeed. Devices keep refused records in their outbox and upload them on a later sync once the quota is raised. Records are never deleted, so usage only grows; the operator console's storage figures show where it goes.

member_max_devices (default 1024, the most devices a pool's membership can list) limits something else: the distinct devices a non-owner user can claim enrollments with in one pool, because each leaves a row that is never deleted (see Enrollment evidence). Cancelling, expiry and pruning do not free any of it. Above the limit a claim of a device the user has no enrollment of in the pool gets 409 device_quota_exceeded; a device the user already has one of can always enroll again, and a device claimed with a member invitation can come back as one device above the limit. Pool owners are not limited: their own devices and member invitations do not count, and the relay still accepts a membership change an owner signs. No command adds another user's device that way, so to let a user continue, raise the limit and restart; -1 removes it and with it this bound on disk use. Older relays do not enforce the limit, so upgrade every relay process first. The relay never writes this key to config.json itself, so a rollback only needs attention if you set it: older relays refuse the unknown key, so remove it first.

Repairing a damaged usage total

The shard total lives in relay_shard_usage and triggers keep it equal to the pools' counters for every relay version. Only a manual edit or a partial restore can damage it. A total missing a row is refused rather than trusted: appends get a retryable 503, also with the shard limits off, pool moves fail so that running them again resumes, and the log says that a shard's storage usage total is wrong if a repair fails. Nothing is stored until it is repaired.

A current relay repairs it by itself, in its next statistics round, and logs that it rebuilt the total. A relay older than this table cannot: after a rollback its appends to the affected pools fail with an internal error naming relay_shard_usage until the total is rebuilt. Back up first, then rebuild it from the counters on that database. PostgreSQL holds off counter writers while it runs; run the SQLite one while the relay's writes can wait for it.

-- relay_shard_usage rebuild: sqlite
BEGIN IMMEDIATE;
DELETE FROM relay_shard_usage;
INSERT INTO relay_shard_usage (stripe, records, bytes)
	SELECT 0, COALESCE(SUM(record_seq), 0), COALESCE(SUM(record_bytes), 0) FROM relay_pool_counters;
COMMIT;

-- relay_shard_usage rebuild: postgres
BEGIN;
LOCK TABLE relay_pool_counters IN SHARE ROW EXCLUSIVE MODE;
DELETE FROM relay_shard_usage;
INSERT INTO relay_shard_usage (stripe) SELECT generate_series(0, 63);
UPDATE relay_shard_usage SET
	records = (SELECT COALESCE(SUM(record_seq), 0) FROM relay_pool_counters),
	bytes = (SELECT COALESCE(SUM(record_bytes), 0) FROM relay_pool_counters)
	WHERE stripe = 0;
COMMIT;

Do not recreate a missing row with zeros or by hand: the total would look whole and read low, and the shard quota would admit what it should refuse. Dropping the table or its triggers is no repair either: a later migration or restart of a current relay needs them.

Backup and monitoring

relay.db holds every pool's log and membership history. Back it up online with sqlite3 relay.db "VACUUM INTO '/backup/relay-$(date +%F).db'", or stop the relay and copy relay.db*. Restoring an older copy makes clients report relay_mismatch / a membership fork for records they already have — keep backups current or avoid restoring old ones. PostgreSQL relays: pg_dump, and for shards a consistent set (Databases); prefer synchronous replication (Databases).
Health: GET /v1/health (unauthenticated). The relay logs transport mode and errors, never tokens or content.
Membership sync: devices refresh a pool's membership with a conditional request (no body when nothing changed) and catch up in pages of compact transitions (relay-v1, "Membership changes"). The relay stores each transition when it accepts a membership change; history accepted before the upgrade is served as full snapshots. Clients and relays can be upgraded in any order: older clients keep using single snapshots, and new clients fall back to them against an older relay.
Enrollment evidence: when a cancelled enrollment or an expired claim is pruned, the relay keeps one small row per device, user and pool (a tombstone: IDs and timestamps, no offer, secret, claim or approval), so that device needs a fresh enrollment before it can be admitted. Tombstones have no expiry, because signed admissions have no age limit and an expiring tombstone would let a stale sponsor admit the device. A tombstone also stays when the device later enrolls again, so the count is not a list of refused devices. A tombstone's ID, tomb_<pool>_<device>_<user> with an empty user when it names none, is outside the enrollment ID format, so no offer, request or pool move can collide with it. The offer quotas limit how fast tombstones are created, and relay_quota.member_max_devices how many devices a non-owner user can leave them for (see Storage quotas); the operator admin views count them as tombstone, apart from other enrollment states, and an admission refused because of one names the device. Upgrade every relay process before relying on this: an older process still prunes enrollments it finds without leaving a tombstone, though it keeps and honors existing ones. An older relay's pool move onto a shard where another pool still holds an enrollment's ID fails, and an upgraded one fails with a named error, instead of overwriting the row. A relay keeps an enrollment's locator until the row is gone, so this takes a row an older relay left without one: cancel the move or pick another shard; the other pool's next offer prunes the row if it expired. Enrollments pruned before the upgrade left no evidence and cannot be reconstructed. If a sponsor may have been offline during such a cancellation, remove any unexpected device with merkhain device remove.
Sizing: every authenticated request writes a nonce. SQLite has one writer per database; PostgreSQL writes different pools and nonces concurrently. Measured local SQLite latency figures are in the team acceptance report; see Databases.

Deploying the relay (serve remote)