Tracing (OpenTelemetry)

merkhain can export OpenTelemetry traces over OTLP/gRPC. Tracing is off by default: with tracing.enabled unset, no exporter is created and nothing is sent anywhere. It follows the Talos conventions: one global tracer provider per process, operation spans with stable error codes, W3C trace-context propagation, parent-based ratio sampling, and a bounded flush on exit.

Configuration

{
  "tracing": {
    "enabled": true,
    "service_name": "merkhain",
    "service_version": "v1.4.0",
    "environment": "laptop",
    "exporter": "otlp",
    "endpoint": "localhost:4317",
    "sample_rate": 1
  }
}

Precedence is the usual defaults < config.json < environment. Invalid values (unknown exporter, an endpoint without a port or with a path/query, a non-finite or out-of-range rate, a malformed boolean) fail like any other config error. merkhain help and merkhain version never read the configuration.

The standard OTEL_EXPORTER_OTLP_HEADERS / OTEL_RESOURCE_ATTRIBUTES variables are honored by the OpenTelemetry SDK, e.g. to authenticate to a hosted collector. Do not put secrets in OTEL_RESOURCE_ATTRIBUTES: resource attributes are exported with every span.

Collector setup

Any OTLP/gRPC receiver works. For a quick local look, run Jaeger and point merkhain at it:

docker run --rm -p 4317:4317 -p 16686:16686 jaegertracing/jaeger:2.10.0
MERKHAIN_TRACING_ENABLED=true merkhain status
# open http://localhost:16686, service "merkhain"

For serve local as a user service, set tracing.enabled in config.json (the hooks, merkhain mcp, and the CLI read the same file, so every process exports). The exporter is set up when a process starts: running serve and mcp processes reject changes to tracing.* and metrics.* until they restart, except tracing.sql, which applies live (Local memory). For a relay, configure its own config.json or environment; relay spans carry the client's trace context, so a sync from a laptop and the relay's handling of it appear in one trace when both export to the same backend.

Sampling

The sampler is ParentBased(TraceIDRatioBased(sample_rate)): a root span is kept with probability sample_rate, and every child follows its parent's decision. Server spans follow the sampled flag of an incoming traceparent, so a client's decision carries across the local API and the relay. Background work (extraction attempts, capture reconciliation passes, sync cycles) starts new root traces and is sampled independently; trace context is never written to durable queues.

One exception survives sampling: a write transaction that held the writer lock for at least 250 ms, waited for it for at least 1 s, or failed busy is also recorded as a storage.slow_write root span that is always exported, linked to its (possibly unsampled) transaction span, with merkhain.storage.op, merkhain.storage.wait_ms, merkhain.storage.hold_ms, merkhain.storage.rows_changed, and merkhain.storage.wal_frames. See operations for the slow-transaction log.

What is traced

Idle background work stays silent: polls that claim no extraction job create no span, storage transactions are traced only inside a traced operation, and a background planning tick is recorded only when it created or finalized jobs or failed. Sync cycles and reconciliation passes are always recorded.

Failures set the span status to a stable code (error.type): an API error code such as not_found or busy, canceled, deadline_exceeded, an HTTP status, panic, or internal. Request and worker logs written with a traced context carry otel.trace_id and otel.span_id for correlation.

Privacy

Spans carry only fixed operation names, known command/tool/route names, counts, sizes, durations, provider kind and stage, and stable codes. They never contain memory content, prompts, model responses, search terms, credentials or tokens, raw error messages, filesystem or database paths, pool/device/session/memory IDs, URLs or query strings, hosts, headers, or request bodies. The one exception is the opt-in sql switch, whose spans carry parameterized SQL statement text (the schema's queries, without bound values). Only the W3C traceparent (and tracestate) header is propagated; baggage is neither sent nor accepted. Panics are recorded as panic without their value and then re-raised.

The HTTP instrumentation is a small wrapper rather than otelhttp, whose defaults record URLs, peer addresses, and error text.

Exit, hooks, and collector outages

Spans are batched and exported asynchronously; exporting never changes a command's output or exit code, and MCP stdout is untouched. When the collector is unreachable or rejects spans, merkhain prints at most one fixed line per minute to stderr (merkhain: tracing: span export failed; …) and drops those spans.

On exit each process flushes pending spans with an uncancelled deadline, so an interrupted command still exports: 5 s for ordinary commands and services, but only 250 ms for capture event, capture protocol, claude hook, and codex hook. Hooks run inline with your agent, so a slow or unreachable collector may cost a hook at most 250 ms; spans still pending when that deadline expires are deliberately dropped.

Metrics

Retrieval metrics are recorded by serve, mcp, recall, and insights. Like tracing, OTLP export is off by default:

{ "metrics": { "enabled": true, "endpoint": "localhost:4317", "interval": "1m" } }

Independently of enabled, the same instruments appear on the opt-in Prometheus endpoint (MERKHAIN_DEBUG_LISTEN=127.0.0.1:9464 serves /metrics) next to the existing process and storage metrics, whose names are unchanged. Scrape it instead of running a collector if you prefer pull:

scrape_configs:
  - job_name: merkhain
    static_configs: [{ targets: ["127.0.0.1:9464"] }]

To receive OTLP, any collector with an otlp receiver works, e.g. an OpenTelemetry Collector forwarding to Prometheus:

receivers:  { otlp: { protocols: { grpc: { endpoint: 127.0.0.1:4317 } } } }
exporters:  { prometheus: { endpoint: 127.0.0.1:8889 } }
service:    { pipelines: { metrics: { receivers: [otlp], exporters: [prometheus] } } }

Instruments

Prometheus renders dots as underscores, appends _total to counters and the unit (_seconds, _bytes) to histograms, e.g. merkhain_memory_retrieval_requests_total.

traffic is interactive, page (a cursor follow-up), or benchmark (MERKHAIN_TRAFFIC=benchmark, set by fleetbench); exclude page and benchmark for user-facing rates. Every attribute is a closed enumeration; an unknown value is recorded as other, and each instrument is capped at 1,000 attribute sets.

Useful formulas (per operation, traffic="interactive"):

• hit rate = hits{stage="returned"} / requests{outcome="ok"}
• empty rate = sum(empty) / requests{outcome="ok"}; budget share = empty{reason="budget"} / sum(empty)
• delivery rate = deliveries{outcome="ok"} / requests{outcome="ok"}
• truncation rate = truncations{reason="any"} / requests{outcome="ok"}
• engagement rate (search) = engagements / deliveries{operation="search",outcome="ok"}

Metrics carry no query text, memory content or IDs, scopes, pools, projects, or error messages. Exporting never changes command output; metrics share the tracing exit deadline (5 s, 250 ms for hooks) and are flushed concurrently with spans. The local usage ledger behind Insights (merkhain insights, the web UI) is separate: it stays on this device and is configured under Insights → Settings.