# OpenRath v2 Operations

This guide covers production-oriented operation of OpenRath v2.0.0.

## Release and upgrade

- Build immutable images by Git commit digest.
- Replace every mutable image tag in `deploy/kubernetes/openrath.yaml` with the exact
  `image@sha256:...` produced for the release.
- Run `openrath-migrate --check` before traffic and `openrath-migrate` as a
  single pre-deploy Job.
- Database changes are additive in v2.0.0. Roll application pods back first;
  retain added columns and tables until the rollback window closes.
- Take and restore-test PostgreSQL and artifact backups before an upgrade.

For timeout-controlled steps, use an async handler or isolated executor.
Embedded synchronous handlers run to completion before their result is
persisted.

Use `LocalTrustedPolicy` for explicit local `trusted_host` workflows. Service
deployments use a fail-closed policy, governed adapter executors, a durable
effect ledger, and an audit sink.

The reference server requires `OPENRATH_GRANTS` with explicit actions and
rejects the wildcard grant. It emits redacted newline-delimited JSON audit
records to stdout. Configure a collector, retention, access control, and
delivery alerts for that stream.

The Kubernetes template is fail closed for egress. PostgreSQL, Redis, S3, and
an HTTPS egress gateway must run in a namespace labelled
`openrath.io/data-plane=allowed`; DNS is limited to `kube-system`. If the CNI
supports FQDN policies, restrict provider and object-store hostnames there. Do
so with explicit allowlists.

## Incident runbooks

### PostgreSQL unavailable

Stop accepting new Runs (`/health/ready` returns 503), keep existing pods from
restart loops, restore database connectivity, verify schema and lease expiry,
then let workers requeue expired leases. PostgreSQL remains the durable source
of truth throughout recovery.

### Redis unavailable

Runs remain durable. Alert on signal failures and increased queue latency,
restore Redis, and allow polling to continue. Do not reconstruct Run state from
Redis.

### Worker terminated or stuck

Confirm the worker lease has expired, call the orphan reconciliation loop, and
verify the fencing token increased. A stale worker must fail its next commit.
Inspect dispatched non-idempotent ToolInvocations; they must enter
`NEEDS_REVIEW`, not automatic retry.

### Queue backlog

Measure queued age, database lock time, provider saturation, and artifact
latency. Scale replicas only after confirming PostgreSQL connection capacity.
Rate-limit tenants producing disproportionate load.

### Artifact store unavailable

Keep the Run and checkpoint durable, fail or pause the affected step, and do
not inline payloads beyond the configured limit. Restore object storage and
verify SHA-256 before resuming.

## Backup and restore exercise

Quarterly, restore PostgreSQL and artifacts into an isolated environment, run
`openrath-migrate --check`, fetch historical Runs and artifacts, requeue an
expired lease, and verify a non-idempotent ambiguous invocation remains blocked
for review.

## Deployment verification

For each deployment, record the immutable commit and image digest together with
enabled provider, sandbox, and memory lifecycle checks; PostgreSQL, Redis, and
S3-compatible restart drills; backup and restore; rollback; scaling; soak;
SBOM; and vulnerability-scan results.

Source:
[tagged operations document](https://github.com/Rath-Team/OpenRath/blob/v2.0.0/deploy/docs/operations-v2.md).