OpenRath v2 Operations#

This guide covers production-oriented operation of OpenRath v2.0.0.

Release and upgrade#

  • Build immutable images by Git commit digest.

  • Replace every mutable image tag in deploy/kubernetes/openrath.yaml with the exact image@sha256:... produced for the release.

  • Run openrath-migrate --check before traffic and openrath-migrate as a single pre-deploy Job.

  • Database changes are additive in v2.0.0. Roll application pods back first; retain added columns and tables until the rollback window closes.

  • Take and restore-test PostgreSQL and artifact backups before an upgrade.

For timeout-controlled steps, use an async handler or isolated executor. Embedded synchronous handlers run to completion before their result is persisted.

Use LocalTrustedPolicy for explicit local trusted_host workflows. Service deployments use a fail-closed policy, governed adapter executors, a durable effect ledger, and an audit sink.

The reference server requires OPENRATH_GRANTS with explicit actions and rejects the wildcard grant. It emits redacted newline-delimited JSON audit records to stdout. Configure a collector, retention, access control, and delivery alerts for that stream.

The Kubernetes template is fail closed for egress. PostgreSQL, Redis, S3, and an HTTPS egress gateway must run in a namespace labelled openrath.io/data-plane=allowed; DNS is limited to kube-system. If the CNI supports FQDN policies, restrict provider and object-store hostnames there. Do so with explicit allowlists.

Incident runbooks#

PostgreSQL unavailable#

Stop accepting new Runs (/health/ready returns 503), keep existing pods from restart loops, restore database connectivity, verify schema and lease expiry, then let workers requeue expired leases. PostgreSQL remains the durable source of truth throughout recovery.

Redis unavailable#

Runs remain durable. Alert on signal failures and increased queue latency, restore Redis, and allow polling to continue. Do not reconstruct Run state from Redis.

Worker terminated or stuck#

Confirm the worker lease has expired, call the orphan reconciliation loop, and verify the fencing token increased. A stale worker must fail its next commit. Inspect dispatched non-idempotent ToolInvocations; they must enter NEEDS_REVIEW, not automatic retry.

Queue backlog#

Measure queued age, database lock time, provider saturation, and artifact latency. Scale replicas only after confirming PostgreSQL connection capacity. Rate-limit tenants producing disproportionate load.

Artifact store unavailable#

Keep the Run and checkpoint durable, fail or pause the affected step, and do not inline payloads beyond the configured limit. Restore object storage and verify SHA-256 before resuming.

Backup and restore exercise#

Quarterly, restore PostgreSQL and artifacts into an isolated environment, run openrath-migrate --check, fetch historical Runs and artifacts, requeue an expired lease, and verify a non-idempotent ambiguous invocation remains blocked for review.

Deployment verification#

For each deployment, record the immutable commit and image digest together with enabled provider, sandbox, and memory lifecycle checks; PostgreSQL, Redis, and S3-compatible restart drills; backup and restore; rollback; scaling; soak; SBOM; and vulnerability-scan results.

Source: tagged operations document.