diff --git a/website/src/content/docs/self-hosting/configuration.mdx b/website/src/content/docs/self-hosting/configuration.mdx index 837f4c9cbc..b2ba3f87ca 100644 --- a/website/src/content/docs/self-hosting/configuration.mdx +++ b/website/src/content/docs/self-hosting/configuration.mdx @@ -18,7 +18,7 @@ Rivet supports JSON, JSON5, JSONC, YAML, YML, and environment variable configura **Environment Variables** -Use the `RIVET__` prefix with `__` as separator to configure properties in the config. For example: set the `RIVET__database__postgres__url` environment variable for `database.postgres.url`. +Use the `RIVET__` prefix with `__` as separator to configure properties in the config. For example: set the `RIVET__POSTGRES__URL` environment variable for `postgres.url`. **Configuration Paths** diff --git a/website/src/content/docs/self-hosting/filesystem.mdx b/website/src/content/docs/self-hosting/filesystem.mdx index b7a8b86ce6..bfc9e48e9d 100644 --- a/website/src/content/docs/self-hosting/filesystem.mdx +++ b/website/src/content/docs/self-hosting/filesystem.mdx @@ -14,16 +14,14 @@ The file system backend (RocksDB-based) is recommended for single-node Rivet dep ```json Configuration-file { - "database": { - "file_system": { - "path": "/var/lib/rivet/data" - } + "file_system": { + "path": "/var/lib/rivet/data" } } ``` ```bash Environment-variables -RIVET__database__file_system__path="/var/lib/rivet/data" +RIVET__FILE_SYSTEM__PATH="/var/lib/rivet/data" ``` diff --git a/website/src/content/docs/self-hosting/postgres.mdx b/website/src/content/docs/self-hosting/postgres.mdx index dd05cd9c42..1e4151699b 100644 --- a/website/src/content/docs/self-hosting/postgres.mdx +++ b/website/src/content/docs/self-hosting/postgres.mdx @@ -5,41 +5,83 @@ skill: true --- -PostgreSQL is the recommended backend for production multi-node and multi-region self-hosted Rivet deployments. For single-node deployments, the [file system backend](/docs/self-hosting/filesystem) (RocksDB-based) is simpler. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. +PostgreSQL is the recommended backend for production self-hosted Rivet deployments. A single-node PostgreSQL deployment needs no other services. Multi-node and multi-region deployments additionally require NATS (see [Single-Node vs Multi-Node](#single-node-vs-multi-node)). For the simplest single-node setups, the [file system backend](/docs/self-hosting/filesystem) (RocksDB-based) is an alternative. Enterprise teams running at very large scale can contact [enterprise support](https://rivet.dev/sales) about FoundationDB. ## Overview -PostgreSQL is the storage and coordination backend for self-hosted Rivet deployments that run more than one engine node. Multiple engine nodes can share a single PostgreSQL instance with no extra coordination service to deploy. Rivet handles leader election, failover, and version sequencing internally. +PostgreSQL is the storage and coordination backend for self-hosted Rivet deployments. It can run a single engine node on its own, or back multiple engine nodes when paired with NATS. Rivet handles leader election, failover, and version sequencing internally. Use PostgreSQL when you need: -- **Multiple engine nodes** behind a load balancer for redundancy and horizontal scaling. +- **A durable, managed system of record** instead of local RocksDB storage. +- **Multiple engine nodes** behind a load balancer for redundancy and horizontal scaling (requires NATS). - **Multi-region deployments** (deploy one PostgreSQL instance per region, see [Multi-Region](/docs/self-hosting/multi-region)). - **High availability** with a managed or self-managed primary/replica failover setup. -For a single-node deployment, the file system backend is simpler and requires no separate database. For the largest, most demanding production workloads, FoundationDB ([enterprise](/sales)) scales further. +For the simplest single-node deployment, the file system backend requires no separate database. For the largest, most demanding production workloads, FoundationDB ([enterprise](/sales)) scales further. + +## Single-Node vs Multi-Node + +Whether PostgreSQL runs Rivet single-node or multi-node depends on whether NATS is configured for pub/sub: + +- **Single-node (no NATS):** Configure only `postgres`. Pub/sub uses the in-memory driver and UniversalDB runs in single-node mode with an in-process commit resolver. Run exactly one engine node against this deployment. +- **Multi-node (with NATS):** Configure `postgres` **and** a top-level `nats` block for pub/sub. Pub/sub runs over NATS, and UniversalDB inherits that NATS config to run in multi-node mode, using NATS for follower-to-leader commit transport. Multiple engine nodes can then share the same PostgreSQL instance. + +Configure NATS for pub/sub to make PostgreSQL multi-node. PostgreSQL inherits the pub/sub NATS config automatically, so a single top-level `nats` block drives both pub/sub and UniversalDB multi-node coordination. See [Multi-Node Configuration](#multi-node-configuration) below. + + +Do not run more than one engine node against a PostgreSQL deployment without NATS configured. Without NATS, UniversalDB runs single-node: the first node takes leadership, and any additional node cannot obtain leadership and will eventually error. + ## Basic Configuration +A single-node deployment needs only a `postgres` block: + ```json Configuration-file { - "database": { - "postgres": { - "url": "postgresql://user:password@host:5432/database" - } + "postgres": { + "url": "postgresql://user:password@host:5432/database" } } ``` ```bash Environment-variables -RIVET__database__postgres__url="postgresql://user:password@host:5432/database" +RIVET__POSTGRES__URL="postgresql://user:password@host:5432/database" ``` +## Multi-Node Configuration + +To run multiple engine nodes against the same PostgreSQL instance, add a top-level `nats` block to configure NATS for pub/sub. PostgreSQL inherits this config to run multi-node. Point every engine node at the same PostgreSQL instance and the same NATS cluster: + + + +```json Configuration-file +{ + "postgres": { + "url": "postgresql://user:password@host:5432/database" + }, + "nats": { + "addresses": ["nats-1:4222", "nats-2:4222"] + } +} +``` + +```bash Environment-variables +RIVET__POSTGRES__URL="postgresql://user:password@host:5432/database" +RIVET__NATS__ADDRESSES="nats-1:4222,nats-2:4222" +``` + + + +The top-level `nats` block configures pub/sub, and UniversalDB inherits it to enable multi-node mode. This is the recommended way to make PostgreSQL multi-node. For NATS high availability, run at least two NATS replicas. + +`postgres.nats` is for advanced setups only: set it to point UniversalDB at a different NATS cluster than the one used for pub/sub. Leave it unset and rely on inheritance from the top-level `nats` block unless you specifically need to separate the two. + ## Requirements and Recommendations ### Version @@ -88,16 +130,14 @@ Use direct connection (not connection pooler). ```json Configuration-file { - "database": { - "postgres": { - "url": "postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require" - } + "postgres": { + "url": "postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require" } } ``` ```bash Environment-variables -RIVET__database__postgres__url="postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require" +RIVET__POSTGRES__URL="postgresql://pscale_api_.:@.pg.psdb.cloud:5432/postgres?sslmode=require" ``` @@ -114,16 +154,14 @@ Use direct connection on port `5432` (not connection pooler). ```json Configuration-file { - "database": { - "postgres": { - "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable" - } + "postgres": { + "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable" } } ``` ```bash Environment-variables -RIVET__database__postgres__url="postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable" +RIVET__POSTGRES__URL="postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=disable" ``` @@ -136,20 +174,18 @@ Download the root certificate from your Supabase dashboard and specify its path. ```json Configuration-file { - "database": { - "postgres": { - "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=require", - "ssl": { - "root_cert_path": "/path/to/supabase-ca.crt" - } + "postgres": { + "url": "postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=require", + "ssl": { + "root_cert_path": "/path/to/supabase-ca.crt" } } } ``` ```bash Environment-variables -RIVET__database__postgres__url="postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=require" -RIVET__database__postgres__ssl__root_cert_path="/path/to/supabase-ca.crt" +RIVET__POSTGRES__URL="postgresql://postgres:@db..supabase.co:5432/postgres?sslmode=require" +RIVET__POSTGRES__SSL__ROOT_CERT_PATH="/path/to/supabase-ca.crt" ``` @@ -166,16 +202,14 @@ To enable SSL for Postgres, add `sslmode=require` to your PostgreSQL connection ```json Configuration-file { - "database": { - "postgres": { - "url": "postgresql://user:password@host.example.com:5432/database?sslmode=require" - } + "postgres": { + "url": "postgresql://user:password@host.example.com:5432/database?sslmode=require" } } ``` ```bash Environment-variables -RIVET__database__postgres__url="postgresql://user:password@host.example.com:5432/database?sslmode=require" +RIVET__POSTGRES__URL="postgresql://user:password@host.example.com:5432/database?sslmode=require" ``` @@ -196,24 +230,22 @@ For databases using custom certificate authorities (e.g., Supabase) or requiring ```json Configuration-file { - "database": { - "postgres": { - "url": "postgresql://user:password@host:5432/database?sslmode=require", - "ssl": { - "root_cert_path": "/path/to/root-ca.crt", - "client_cert_path": "/path/to/client.crt", - "client_key_path": "/path/to/client.key" - } + "postgres": { + "url": "postgresql://user:password@host:5432/database?sslmode=require", + "ssl": { + "root_cert_path": "/path/to/root-ca.crt", + "client_cert_path": "/path/to/client.crt", + "client_key_path": "/path/to/client.key" } } } ``` ```bash Environment-variables -RIVET__database__postgres__url="postgresql://user:password@host:5432/database?sslmode=require" -RIVET__database__postgres__ssl__root_cert_path="/path/to/root-ca.crt" -RIVET__database__postgres__ssl__client_cert_path="/path/to/client.crt" -RIVET__database__postgres__ssl__client_key_path="/path/to/client.key" +RIVET__POSTGRES__URL="postgresql://user:password@host:5432/database?sslmode=require" +RIVET__POSTGRES__SSL__ROOT_CERT_PATH="/path/to/root-ca.crt" +RIVET__POSTGRES__SSL__CLIENT_CERT_PATH="/path/to/client.crt" +RIVET__POSTGRES__SSL__CLIENT_KEY_PATH="/path/to/client.key" ``` diff --git a/website/src/content/docs/self-hosting/production-checklist.mdx b/website/src/content/docs/self-hosting/production-checklist.mdx index 201124b8fb..023eea7637 100644 --- a/website/src/content/docs/self-hosting/production-checklist.mdx +++ b/website/src/content/docs/self-hosting/production-checklist.mdx @@ -34,7 +34,7 @@ Also review the [general production checklist](/docs/general/production-checklis ## PostgreSQL -- **Use PostgreSQL for multi-node and multi-region deployments.** Multiple engine nodes can share one PostgreSQL instance; no extra coordination service is required. See [PostgreSQL](/docs/self-hosting/postgres). +- **Use PostgreSQL for multi-node and multi-region deployments.** Multiple engine nodes can share one PostgreSQL instance. Multi-node also requires NATS (see below). See [PostgreSQL](/docs/self-hosting/postgres). - **Raise `max_connections`.** Each engine node opens well over a hundred connections under load. Size `max_connections` to at least `(number of engine nodes × 150)` plus headroom. PostgreSQL's default of `100` is too low. See [Connection Limits](/docs/self-hosting/postgres#connection-limits). - **Do not use a connection pooler.** Rivet requires direct connections. Do not put PgBouncer, Supavisor, or RDS Proxy in front of PostgreSQL. - **Give PostgreSQL dedicated resources.** Provision dedicated CPU, memory, and fast disk, and keep autovacuum healthy. PostgreSQL is the system of record for the whole deployment. @@ -45,8 +45,8 @@ Also review the [general production checklist](/docs/general/production-checklis ## NATS -- **Use NATS for very high-throughput pub/sub (optional).** By default, Rivet uses PostgreSQL for pub/sub, which is sufficient for most deployments. For very high-traffic deployments, NATS can be used as a dedicated pub/sub layer. This is not needed if using RocksDB. See [Configuration](/docs/self-hosting/configuration). -- **Deploy 2+ NATS replicas.** If you use NATS, run at least two replicas for high availability. +- **NATS is required for multi-node PostgreSQL deployments.** Multiple engine nodes on PostgreSQL coordinate through NATS for pub/sub and UniversalDB commit transport. Single-node deployments (one engine node, or the RocksDB file system backend) do not need NATS. See [Single-Node vs Multi-Node](/docs/self-hosting/postgres#single-node-vs-multi-node). +- **Deploy 2+ NATS replicas.** For multi-node deployments, run at least two NATS replicas for high availability. ## Monitoring