Operations: status, repair, recovery, backup

Status and repair

merkhain status            # asks serve local; falls back to reading the database directly ("source": "direct")
merkhain status --direct   # skip the service

The report is content-free (IDs, counts, times, codes) and ends with diagnoses[], each with a severity, the affected IDs and a remediation. A direct report is strictly read-only: it never creates the data directory or database (it fails with not_found before the first capture) and never applies schema migrations (a database with pending migrations is reported as db_schema_not_current; the next normal open applies them, see Upgrades; a database a newer binary migrated is ahead, reported as db_schema_ahead (info), and healthy). Recovery actions are bounded and never delete data or contact a provider or relay:

Status, repair and serve local (with its browser UI) run on either local database. The report's database section names the backend (driver; for PostgreSQL also endpoint, postgres://host[:port]/dbname without credentials), the server version, the read-only migration state of the local role (schema), and the process's connection pool (connections). The file sizes, WAL checkpoint progress (files, checkpoint_starved) and writer-lock waits (locks) are SQLite figures and appear only on SQLite; a PostgreSQL report runs no PRAGMA and inspects no database file.

The full diagnosis table is in Troubleshooting.

Slow transactions

Every process (serve local, MCP servers, hooks, CLI commands) appends a record to data/slow-tx.jsonl for each write transaction that held the writer lock for at least 250 ms (MERKHAIN_SLOW_TX_HOLD), waited for it for at least 1 s (MERKHAIN_SLOW_TX_WAIT), or failed busy (a hook that exits 75 records how long it waited). A record names the transaction (op, e.g. store.Store.CommitCapture), the process (serve local, mcp, hook claude Stop, cli status, ...) and its pid, the phase durations (acquire_ms, begin_ms, body_ms, commit_ms, wait_ms, hold_ms), the rows changed, the WAL frames written, and the outcome. serve local also logs each one as a WARN. The file is appended without a lock and rotated to slow-tx.jsonl.1 at 512 KiB. status (and status --direct) lists the slowest holders of the last 24 hours by process and op under database.slow_transactions. Migrations take the same lock and are recorded as migrate.Up (and migrate.Down, migrate.Force). The log covers SQLite databases only: a PostgreSQL database has no writer lock, and its own log_lock_waits and pg_stat_activity show long transactions.

Recovery

Your device keys live only in the key store. If you lose your only owner device, an owner recovery export is the only way back into your pools (protocol).

merkhain recovery export --out ~/merkhain-recovery.age            # prompts for a new recovery passphrase
merkhain recovery export --out r.age --recipient age1…             # or encrypt to an age X25519 key
merkhain recovery inspect ~/merkhain-recovery.age                 # verify the file; prints pools/epochs, no secrets

Store the file and its passphrase in different places. Export again after creating or joining a pool, after a relay URL change, after a pending rotation completes, and when replacing the exporting device (removing that device makes its export useless). Rotations the device survives need no new export.

A successful export (or restore) records when it was made, for which user and which pools it covers — no secrets — in recovery-export.json next to identity.json. The browser UI's Sharing and Settings pages read it: they warn while there is no export, or when pools were created or joined since the last one, and otherwise show its date. Exporting and restoring stay in the CLI.

Restore on a new machine into an empty profile:

merkhain recovery restore ~/merkhain-recovery.age [--key-store file]
merkhain sync status

It restores the identity, pool keys up to the current epoch (newer keys come from the relay mailbox) and the project mapping, then history syncs from the relay. Exit 75 means the relay was unreachable; retry. Private memories that were never published are not in the export — back up the database for those.

Backup

The local database (database_path in merkhain config) contains captured events, all decrypted memories and the index. Back up a SQLite database while running:

sqlite3 "<database_path>" "VACUUM INTO '/backup/merkhain-$(date +%F).db'"

or stop serve local and copy merkhain.db, merkhain.db-wal and merkhain.db-shm together. Restore by stopping the service and putting the copy back as merkhain.db (delete stale -wal/-shm). Keys are not in the database; pair with the recovery export above. Never sync the database file between devices — use pools.

A PostgreSQL instance database is backed up with pg_dump while running; see Databases.

Upgrades

The local database (and the Personal sync relay) upgrade by themselves: the next normal open after a binary upgrade (any command, a capture hook, the MCP server, serve local) applies pending migrations in one transaction under the migration lock. Other processes wait briefly for that lock instead of being stopped; a hook whose budget runs out fails retryably (exit 75). Opening a database that is already current, or ahead (migrated by a newer binary), only reads the migration ledger and takes no write lock. Processes of the previous version keep working against the upgraded database, so restart serve local when convenient. Standalone relay servers migrate explicitly with merkhain migrate up --relay, run with the new binary before rolling out the relay processes. The procedure is in Install; migrations, databases below the baselines, migrate down/force, PostgreSQL, transfer from SQLite, relay shards, pool and bucket moves and failover are in Databases.

Upgrade order for the security release

These are deliberate refusals that nothing relaxes for compatibility. What changes for an existing setup, and what to do:

• Upgrade relays, then owner and sponsoring devices, then the rest. Uploads wait (membership_stale) on a device when no other owner device has signed within 72 hours, which is immediate after an upgrade if the pool's owner has not signed recently; they resume after an upgraded owner device syncs. Nothing queued is lost.
• Development builds with the 72-hour check but no checkpoint support. Builds from commit 310d8247 up to the checkpoint change were never released and are not kept compatible: owner devices do not add revisions for them. Such a writer stops uploading (membership_stale) about 72 hours after an upgraded owner device last signed a revision, until the writer is upgraded. Nothing queued is lost. This is separate from older relays: on a relay without the checkpoint route, owner devices still renew with revisions.
• Enrollment across versions. A joining device refuses an approval from a sponsor that does not anchor the pool's membership and reports sponsor_outdated: update the sponsoring device. A join already in progress stays pending until the code's local expiry, because a relay's refusal is unsigned; start again with a new code if it expires. An older joining device cannot read the new approval, so update it before joining.
• Remote PostgreSQL instance database. One that is not on a Unix socket or loopback address must use sslmode=verify-full (with sslrootcert), or the local service refuses to open it. Fix the URL, or reach the server through a Unix socket or a loopback tunnel. Relay databases are not affected.
• Removed devices on an older relay. A device believes only a signed removal, and an older relay refuses removed devices without serving it, so such a device keeps retrying and reads as offline instead of not_member. Upgrade the relay.
• Moving pools during the upgrade. Upgrade the relay before moving pools. From reading the mover's code, an older binary does not copy membership checkpoints, so writers that lose the evidence wait until an owner device signs again; if no owner device is online, that wait starts early. This has not been run against an older mover.
• Publishing a correction by ID. merkhain publish --pool POOL MEMORY_ID... refuses a correction whose private predecessors are not listed and names them. Review them and list them too; scripts that relied on the old implicit sharing need this.