Skip to content

production-checklist.mdx

Latest commit

 

History

History
58 lines (41 loc) · 5 KB

production-checklist.mdx

File metadata and controls

58 lines (41 loc) · 5 KB
Raw
title Production Checklist
description Checklist for deploying a self-hosted Rivet Engine to production.
skill true
We recommend passing this page to your coding agent to verify your configuration before deploying. PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the file system backend (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB.

Also review the general production checklist.

Security

  • Validate that you have an admin token configured. Generate a strong, random token for engine authentication. See Configuration.
  • Verify your admin token is not exposed publicly. Do not include the admin token in RIVET_PUBLIC_ENDPOINT or anywhere accessible to clients. See Endpoints.
  • Configure TLS termination. Ensure connections to the engine are encrypted via a reverse proxy or load balancer.

Resources

  • Set container resource limits. Recommended at least 1 CPU and 2 GB of RAM per Rivet Engine instance.
  • Configure health checks. Set up liveness and readiness probes on port 6421 at /health. Recommended timeout of 5 seconds.

Scaling

  • Configure autoscaling for the Rivet Engine. Set target CPU utilization to 70% and memory to 80% to ensure headroom for traffic spikes. In Kubernetes, this is configured via a Horizontal Pod Autoscaler (HPA).
  • Use 2+ engine nodes for redundancy. Running a single engine node is a single point of failure. Deploy at least two engine instances behind a load balancer.
  • RocksDB only supports a single node. Do not run multiple RocksDB nodes. The file system backend (RocksDB-based) is for single-node deployments. For multi-node and multi-region deployments, use the PostgreSQL backend.
  • Validate the rate limit on your serverless actor host. Actor start requests are sent from your engine instances, so they all originate from a small set of IPs. Per-IP rate limits on the actor host will throttle the engine before they would throttle end-user traffic. Size the limit to your peak actor create and wake rate, and configure platform max concurrency (e.g. on GCP Cloud Run) to match your expected concurrent actor count.

PostgreSQL

  • Use PostgreSQL for multi-node and multi-region deployments. Multiple engine nodes can share one PostgreSQL instance. Multi-node also requires NATS (see below). See PostgreSQL.
  • Raise max_connections. Each engine node opens well over a hundred connections under load. Size max_connections to at least (number of engine nodes × 150) plus headroom. PostgreSQL's default of 100 is too low. See Connection Limits.
  • Do not use a connection pooler. Rivet requires direct connections. Do not put PgBouncer, Supavisor, or RDS Proxy in front of PostgreSQL.
  • Give PostgreSQL dedicated resources. Provision dedicated CPU, memory, and fast disk, and keep autovacuum healthy. PostgreSQL is the system of record for the whole deployment.
  • Configure automated backups. Set up regular backups and point-in-time recovery, and test restoring from them.
  • Configure failover. Set up a standby replica with automatic failover to ensure high availability. A single instance is a single point of failure.
  • Use one PostgreSQL instance per region. For multi-region deployments, deploy a separate PostgreSQL instance in each region.
  • Use FoundationDB for the largest deployments. Enterprise teams running at very large scale can contact enterprise support for FoundationDB guidance.

NATS

  • NATS is required for multi-node PostgreSQL deployments. Multiple engine nodes on PostgreSQL coordinate through NATS for pub/sub and UniversalDB commit transport. Single-node deployments (one engine node, or the RocksDB file system backend) do not need NATS. See Single-Node vs Multi-Node.
  • Deploy 2+ NATS replicas. For multi-node deployments, run at least two NATS replicas for high availability.

Monitoring

  • Configure OpenTelemetry. The Rivet Engine supports exporting traces and metrics via OpenTelemetry. Set RIVET_OTEL_ENABLED=1 and RIVET_OTEL_GRPC_ENDPOINT to your collector endpoint (defaults to http://localhost:4317). Adjust RIVET_OTEL_SAMPLER_RATIO to control trace sampling (defaults to 0.001). See Configuration.
  • Set up alerts for critical metrics. Monitor engine CPU, memory, request latency, and error rates. Configure alerts to notify your team before issues become outages.

Enterprise

  • Contact enterprise support for production-ready deployments. We can help with architecture review, scaling guidance, and FoundationDB support.