merkhain

Troubleshooting

Start with merkhain status: each diagnosis carries its remediation. Retryable CLI failures (busy database, relay offline, pending approval) exit 75; run the command again later.

Symptom / diagnosis Cause Fix
Nothing recalled after a session; provider_not_configured provider.kind is none configure a provider in config.json (Local memory); a running serve local picks it up
worker_failed a background worker of serve local kept panicking and is no longer restarted (workers[].state is failed); the service keeps serving without it restart serve local (this ends browser sessions) and report the stack from its log, which holds no event, prompt or memory content; a configuration change also retries the configuration-driven workers
service_unreachable local status probe failed; daemon state is unknown check service.probe_reason and local permissions/authentication before checking the user service
service_not_probed direct database inspection did not probe the daemon run merkhain status without --direct to inspect the worker
service_not_running legacy code from older releases; current releases report service_unreachable instead treat as service_unreachable
provider_config_invalid provider config rejected (missing model or non-loopback URL without allow_remote), or provider_error is transport_refused: a non-loopback http:// URL without allow_insecure_http, which still loads while nothing is sent to it, jobs stay pending and serve local starts fix config.json or select another provider; a running serve local reloads it
config_reload_failed the last config.json change was not applied: invalid (does not parse or validate), restart_required (a startup-only key changed) or apply_failed (a component could not apply it). After invalid or restart_required the previous configuration stays active; after apply_failed a provider that cannot be built pauses extraction, with no fallback to the previous one, and another component keeps its previous settings read the serve local log, check the file with merkhain config validate, or restart to apply startup-only keys (Local memory)
event_filter_rules_invalid the event filter rules file does not parse or validate; a running serve local keeps its last valid rules (event_filter.rules_kept), other processes run with the filter off. Capture is unaffected merkhain rules check lists every problem with its location; fix the file and check again (Event filter)
provider_outage provider attempts failed and are scheduled to retry automatically (timeouts, rate limits, temporary network/server failures and provider conditions such as authentication, invalid_config or missing_executable retry until they clear, while the provider is paused; unclassified failures retry after 1 s, 1 min, 1 h, 6 h, 12 h and 24 h) usually nothing (a briefly unreadable login clears itself); if it persists, check the endpoint, PATH, the provider config or codex login
provider_invalid_output output violates the extraction contract inspect error_reasons, correct the contract or limits, then retry affected IDs
extraction_invalid_citations output cited events outside its request; the provider ignored the per-request citation schema keep provider.openai.response_format at json_schema with a model or endpoint that enforces it (or use codexcli), then repair retry-jobs with the affected IDs
capture_identity_conflict distinct content used one event identity (a rewritten transcript, or a transcript claimed by another session); the batch was not committed merkhain repair reconcile to find the source; upgrade merkhain if a newer release exists and report the source ID if it persists; do not reset cursors, skip the event, or overwrite events
extraction_jobs_failed jobs failed terminally: a single event too large for the provider, the seventh unclassified failure, or the fifth unusable output (provider conditions such as authentication or configuration never fail a job); failed_retry_classes says which; later jobs of those sessions wait as blocked_by_failure (jobs behind an unfinished one count as blocked) merkhain extract jobs --state failed, fix cause, repair retry-jobs
job_lease_expired a worker stopped mid-attempt (crash, kill, sleep) and its job lease expired; the attempt is counted keep serve local running; its worker reclaims expired leases automatically
extraction_backlog (info) ready jobs have waited longer than 30 min while the worker runs none if the count falls; a slow provider limits throughput (one job per session at a time)
Claude session not captured hooks not installed in the scope Claude loads, or --no-session-persistence merkhain claude install (check scope), merkhain claude import <transcript>
Codex session not captured hooks not trusted yet trust them in codex → /hooks, meanwhile merkhain codex import
capture_stalled, capture_source_truncated a transcript has unread bytes / was rewritten merkhain repair reconcile
capture_lagging (info) an active transcript has bytes not yet imported (normal while a session is writing) none; hooks and the next reconciliation import them
capture_source_unreadable transcript files exist but cannot be inspected, or a link now stands where an imported file was (it is not followed) fix file permissions for the user running merkhain, then merkhain repair reconcile
capture_import_rebind_required (warning) a Codex or pi source is paused and not read: it predates directory binding; its selected directory now leads elsewhere, so it may have come from a tree that changed since; a link or special file stands at or on the way to an imported file; or a source is stored at a symbolic link merkhain codex import --archived / pi import for its directory (with --codex-home or --sessions-dir if it was another one); a link planted in the tree stays refused; a file the given directory no longer has is settled as missing; a hook does not resume it. An import also reads every transcript in that directory it has not seen, which can start a large extraction backlog, and --archived adds archived rollouts; to unpause only sources already tracked, name each one as follows. For a source no import holds that the import skips: merkhain capture reconcile --agent AGENT --session ID --format FORMAT --path FILE, with the agent's own session ID and the format status lists (codex-rollout, pi-session, claude), since without --format the file is read as generic records. A source stored at a link stays paused while the link is there, and naming the link reads it once: to keep reading, name the file it leads to; the link's own entry clears only when you remove the link or put a regular file there. Naming does not change a source shown as bound: restore the regular file or directory, or remove what stands there so it reports missing
capture_source_missing (info) a transcript no longer exists at its recorded path; everything imported before is kept none for cleaned-up sessions; if transcripts moved, merkhain claude import / codex import them
capture_reconcile_failed the last reconciliation pass failed for some sources merkhain repair reconcile shows per-source error codes; unsupported = unknown transcript format, busy = retry
capture_rejected_records transcript records the parser does not support (e.g. a newer agent format), malformed or oversized; they were skipped, later records were captured, and their content was not stored upgrade merkhain; merkhain capture reconcile --all lists rejected_total and last_rejection (offset, reason) per source
extraction_input_error a job's captured events could not be loaded (storage error or event-count mismatch); the attempt is recorded and retried with backoff usually clears on retry; if the job ends failed, merkhain repair retry-jobs and report the job ID
extraction_storage_error a finished extraction result could not be saved to the local database; the worker retried storing it, then recorded the attempt, which is retried with backoff usually clears on retry; if it persists, check free disk space and the database file's permissions, then merkhain repair retry-jobs
MCP tools missing in the agent server not registered, or merkhain not on the agent's PATH claude mcp list; use an absolute command path
MCP search result has "match": "any" no record contains every word of the query, so records matching any word are returned (plain multi-word queries only) normal; use fewer, more distinctive words for precise hits
Recall finds nothing for a pulled project your clone is not mapped to the shared project merkhain project alias <PROJECT_ID> <REMOTE_URL>, or search with --pool
identity_missing, sync_disabled no identity, or the service started before identity init merkhain identity init, restart serve local
identity_locked, identity_unavailable file store without passphrase file / keychain locked or no Secret Service (headless Linux) set MERKHAIN_KEY_PASSPHRASE_FILE (0600) in the service, unlock the keychain, or re-create with --key-store file; check with identity check
sync_offline relay unreachable check the URL/TLS/proxy; merkhain sync or repair retry-sync
sync_error the last sync pass failed with a non-retryable error (pool error_code) inspect merkhain sync status, fix the cause, then merkhain repair retry-sync
publish_failed some outbox entries could not be sealed for the pool (e.g. too large, invalid or foreign provenance). A relay refusal (record_conflict, not_author, invalid) is unsigned, so it never fails an entry: the entry stays sealed and is offered again after an hour inspect merkhain sync status; those records stay private and are not retried
sync_records_rejected (info) pulled records failed verification; they are counted (without their envelopes; the relay keeps those) and never applied none; tell the pool owner if the count grows
sync_records_deferred (info) pulled records wait for a key, author device, or superseded record none; they apply on a later pass once the missing piece arrives. Each sync retries a bounded share of them (round robin, at most 2000 records or 32 MiB per pool), so a large backlog can take several syncs to work through. Beyond 64 MiB of waiting envelopes per pool, further ones keep only a small placeholder and are pulled again from the relay when retried (at most 32 per sync); none are dropped locally, but they apply only once the relay serves the same envelope again
sync_auth_failed clock skew, or the proxy altered signed headers/path sync the clock (NTP), fix proxy forwarding, identity check
sync_deferred_backlog_full (pool error_code deferred_backlog_full) the pool holds 10,000 deferred records, the local limit none usually: syncing pauses before the next record that would wait too (later records of the pool are not pulled until then); retries continue, and syncing resumes where it stopped once some resolve. Corrections whose superseded record lies after the pause point can only resolve once the pull moves on, so a pool full of such corrections stays paused; nothing is cleaned up retroactively, so a backlog stored before the limit existed stays as it is and pauses syncing the same way
sync_waiting_for_key a newer epoch exists and its key has not arrived keep the device online; the key arrives from the relay mailbox
sync_membership_stale (pool error_code membership_stale) no other owner device of the pool has signed its membership in the last 72 hours, or its checkpoints could not be read (the service log says which), so this device cannot rule out that the relay hides a removal; uploads wait and stay queued, pulls continue keep sync running on an owner device of the pool (for a personal pool, another of your devices) and upgrade it if it is on an older version; owner devices confirm the membership about every 18 hours and uploads resume after their next sync. If that device is gone for good, remove it from another owner device (merkhain device remove <DEVICE_ID>; merkhain device list shows the devices, and the service log names the ones it waits for)
sync_not_member; merkhain sync exits 1 this device or user was removed (sync fails when every attached pool is not_member) final; join again with a new invitation
publish_undeliverable records were queued for a pool this device was removed from; the relay will never accept them (undeliverable per pool in status) nothing to retry; they stay in local recall
rotation_pending a removal could not reach the relay merkhain sync once the relay is reachable
sync_relay_untrusted (membership_fork, relay_mismatch) the relay served history inconsistent with what you verified (e.g. restored from an old backup) do not ignore: contact the relay operator; local data is untouched
device join / member join exits 75 the sponsor has not approved yet keep the sponsor online, or run merkhain device finalize on it
device finalize reports code_required with a hint a device join/member join was interrupted before the relay acknowledged its claim; the code's secret is never stored run the printed hint (the same join with the same code) before it expires; afterwards it is reported expired
My Memory shows "The relay has not confirmed your request" a join from the browser did not get the relay's confirmation; nothing resumes it without the code enter the same code again under Join a team (or Connect another computer → I have a pairing code) before it expires
My Memory shows "The relay refused the request" or Not accepted yet the relay refused the code (does not match, expired, cancelled or used, …); you stay signed in. The relay's answer is unsigned, so the request stays open until it expires on this computer send it again with the same code; if it keeps being refused, ask for a new invitation
Sharing shows Not accepted yet: the computer that made the code runs an older merkhain (sponsor_outdated) the inviting computer approved with an older approval format, which carries no membership anchor and is refused (#381) update merkhain on the inviting computer, then use a new invitation or pairing code
enrollment_pending (info) device pairings or member invitations are in progress finish them with merkhain device / merkhain member while the inviting device is online
enrollment_expired, invalid_code code older than 24 h, already used, or mistyped create a new invitation
checkpoint_starved a long-lived reader pins the WAL merkhain repair checkpoint; close long-running readers
db_lock_timeouts (exit 75) another process holds a long write retry; status lists the slowest recent holders under database.slow_transactions (slow transactions)
db_lock_waits (info) writes waited at least 100 ms for the writer lock none; sustained growth points to a long write transaction (another merkhain instance, a sqlite3 shell)
db_schema_not_current the local schema has pending migrations (reported by the read-only status --direct) or is dirty pending: the next normal open applies them (or merkhain migrate up --local); dirty: merkhain migrate status --local, then the named repair
db_schema_ahead (info) a newer merkhain migrated the database; this binary keeps working against it none; upgrade this binary when convenient
db_sqlite_data_left_behind PostgreSQL local database is empty while merkhain.db still holds data stop all processes, merkhain transfer local (Databases)
db_sqlite_data_left_behind_unknown PostgreSQL local database is empty and merkhain.db could not be read, so it may still hold data; sqlite_data_left_behind_reason is exposed, access_refused, unsafe_file, permission_denied, cannot_open, busy, other_schema or unreadable follow the diagnostic's remediation as the file's owner (sudo -u <owner> if you are root), such as chmod 600 "$DATA_DIR"/merkhain.db* for exposed, with DATA_DIR set to the data_dir that merkhain config prints, then merkhain status --direct; move the file away if it is not needed
"storage: database refused": accessible by other users a read-only open refuses exposed files, which it could not repair, and SQLite would add -wal and -shm files with the database's mode run the chmod 600 it names, on each file it names; migrate status also names the merkhain migrate up for that database, which tightens your own files. When the instance now uses PostgreSQL, normal commands do not open the SQLite file, so use chmod 600
"storage: database refused": owned by uid 0, not like its database another process running as root was creating this -wal or -shm file, or one was left behind run the command again; if it stays, repair the file's owner
"storage: database refused" as root root opens a SQLite database only if every directory and symbolic link on its path is root's and writable only by root, whoever owns the database sudo -u <owner> merkhain …, or move the database to such a path
"requires a migration" from serve remote or a relay command the relay binary was upgraded and its databases were not migrated yet back up, merkhain migrate up --relay with the new binary (Databases)
a hook or command exits 75 with "take migration lock" another process is applying a schema migration none: the next hook retries; long waits point to a long migration (Databases)
"target of an unfinished offline transfer" / "transfer": "copying" in migrate status the PostgreSQL target of transfer local or transfer relay was not verified yet run the same merkhain transfer again, or drop the target and point the configuration back at SQLite
Relay answers 503 unavailable for one pool or replay bucket it is being moved, or a move was interrupted after fencing merkhain move status, merkhain move resume ID (Databases)
Relay answers 503 unavailable for every signed request a replay database is unreachable, or a sharded relay's buckets have no route check the databases; register the replay shards (merkhain shard register)
Relay console signs out at random behind a load balancer operator sessions are per relay process enable session affinity for /ui/ (Databases)
Sync stalls with quota_exceeded / "storage quota exceeded" a pool, member or pool shard reached the relay's storage quota the relay operator raises relay_quota and restarts (Relay)
schema too new on start a binary from before the bridge release met a database a newer binary migrated upgrade to the bridge release or later, or restore a pre-upgrade backup
Relay exits: remote_listen ... is not loopback plain HTTP on a public address configure TLS or remote_trusted_proxy (Relay)
merkhain ui exits 75 serve local is not running start it, then merkhain ui again (Web UI)
Browser: "expired or was already used" a launch link is single-use and valid for 60 s merkhain ui (or --no-open) again
Browser: file …/merkhain/ui-launch/launch-….html not found the browser cannot read the private launch page (a sandboxed one the CLI could not recognize as what opens HTML files), or it opened the page after the 60 s it is kept merkhain ui --no-open and paste the link
merkhain ui prints "Started your browser, but could not confirm" with the link a browser that reports an error may have got the page anyway, and under WSL the CLI cannot see whether Windows read it if the UI opened, ignore the link; it is single-use, so if it says it was already used, run merkhain ui again
merkhain ui prints "Could not open a browser: …" with the link the reason follows the colon: a launch directory of another user (next row), the document portal or a desktop file the CLI refuses to run use the link, and fix what it names; --no-open skips the attempt
merkhain ui prints the link: launch directory … or …/merkhain belongs to another uid an earlier run as another user (sudo, say) created …/merkhain/ui-launch or …/merkhain remove the directory it names as its owner, or keep using --no-open
merkhain ui says it started your browser and nothing opens the CLI sees only that the browser started, not that it loaded the page run merkhain ui --no-open and open the link
Browser: "Your session ended." / sign-in page after a restart browser sessions end on sign out and when serve local (or the relay) restarts sign in again
Browser: 403 / blank page on localhost:7781 only the exact local_listen Host is accepted open the address merkhain ui prints (default 127.0.0.1:7781)
Relay Administration: "Browser sign-in is disabled on this relay" TLS or remote_trusted_proxy without remote_public_origin set remote_public_origin to the https origin, restart (Web UI)
Relay Administration: sign-in refused from the right token the page was not opened on remote_public_origin (other host name, bypassed proxy) use the configured origin exactly
macOS: "cannot be opened" / killed on launch quarantine attribute from a browser download xattr -d com.apple.quarantine $(command -v merkhain)

Logs: journalctl --user -u merkhain-local, ~/Library/Logs/merkhain-local.log, or %LOCALAPPDATA%\merkhain\serve-local.log; log_level: debug for detail. Logs are designed to exclude tokens, keys, codes and memory content; still review them before sharing.