Answers to behaviors that look like bugs but are working as designed, plus a few real footguns. This FAQ answers "why is the thing I'm running behaving this way?". For how the stack is built internally, see docker-architecture.md.
Questions are grouped by what you are doing:
- For operators: running the stack as infrastructure.
- For developers and testers: building or testing against Z3, often across several networks at once.
When an entry says "see README", the fix lives there; this FAQ only adds the diagnostic context.
That's the /ready probe doing its job, not a fault. /ready requires the configured peer minimum, a sync status close to tip, a present chain tip, a local tip age within ready_max_tip_age, and estimated lag within ZEBRA_HEALTH__READY_MAX_BLOCKS_BEHIND. Mainnet uses Zebra's default lag limit of 2; Testnet uses 4 to tolerate the observed lag=3 transition during a canonically current 225-second gap. Estimated lag comes from the best tip's header time, while tip age starts when the local height increases, so either independent check can return 503. The response body identifies insufficient peers, syncing, no tip, tip_age=..., or lag=... blocks. A fresh start from a cold chain, or a restart on a cache that's a few minutes behind, remains unhealthy until every readiness check passes.
For development, /healthy is a looser signal that only checks peer connectivity. The tracked docker-compose.override.yml.example flips the healthcheck to /healthy so Zallet (and Zaino under the indexer profile) can start without waiting for full sync. Use it for dev, never for production where you want consumers to wait for a synced node.
Don't run docker compose up -d (it starts Zallet) until Zebra reports /ready. The poller scripts/check-zebra-readiness.sh waits for exactly that. README Quick start step 3 explains the two-phase boot.
Chain state for each network is a Docker named volume, z3-<network>-chain, under Docker's data root (/var/lib/docker/volumes/ on Linux). Mainnet is roughly 300 GB. Find the exact path with:
docker volume inspect z3-mainnet-chain -f '{{.Mountpoint}}'To keep it off the OS disk, set Z3_CHAIN_DATA_PATH=/mnt/ssd/zebra-state and run ./scripts/fix-permissions.sh zebra /mnt/ssd/zebra-state before the first start.
For backups, the only thing worth keeping is the z3-<network>-zallet volume. It holds the age-encrypted wallet database and the identity that decrypts it, so the volume is self-contained: restore it and the wallet opens. Chain state is re-syncable and the RPC cookie is regenerated on boot, so neither belongs in a backup.
docker compose --env-file .env.<network> --profile "*" down keeps all volumes; down -v deletes them and forces a full re-sync. Include --profile "*" so profile-gated services (indexer, monitoring) are stopped too.
scripts/regtest-init.sh initializes the regtest wallet automatically. On mainnet and testnet, the node runs fine without an initialized wallet; set up Zallet's wallet encryption yourself, once per network, when you are ready to manage keys. A normal stack start prepares the data volume through the cookie-permissions sidecar. If you initialize the wallet first with --no-deps, prepare the volume explicitly, then generate the identity and initialize encryption:
# One-time preparation for the no-deps path.
docker run --rm -v z3-<network>-zallet:/data busybox \
sh -c 'touch /data/.z3-volume-initialized && chown 1000:1000 /data /data/.z3-volume-initialized'
docker compose --env-file .env.<network> run --rm --no-deps zallet \
--datadir /var/lib/zallet --config /etc/zallet/zallet.toml generate-encryption-identity
docker compose --env-file .env.<network> run --rm --no-deps zallet \
--datadir /var/lib/zallet --config /etc/zallet/zallet.toml init-wallet-encryption--no-deps skips starting Zebra and the volume-preparation sidecar because encryption setup does not need the node. The zero-byte marker prevents Docker from replacing the prepared ownership with platform-specific image-directory metadata on Zallet's first mount. generate-encryption-identity writes the age key into the z3-<network>-zallet volume and refuses to overwrite an existing identity, so re-running it is safe. Each network keeps its Zallet data in a separate volume, so run this once per network you use. Back up the volume (see the data question above): it holds both the wallet and the identity that decrypts it. Afterwards, use the Zallet RPC or generate-mnemonic to create or import keys.
By design. Pinning logging.driver in the compose file would override whatever logging driver you set on the Docker daemon (journald, local, awslogs, a remote collector). z3 leaves logging unset so your daemon default wins.
To bound disk growth on a 24/7 node, set a rotating default once in /etc/docker/daemon.json and restart the daemon:
{
"log-driver": "local",
"log-opts": { "max-size": "50m", "max-file": "5" }
}The local driver rotates by default and is more efficient than json-file. This applies to every container on the host, not just z3. If you want per-service control, add a logging: block in your operator-local override file.
Because docker-compose.yml doesn't set deploy.resources.limits on any service. The choice is deliberate for a node platform: a constrained limit that makes sense on a 4-core laptop will silently throttle a 32-core production host, and the bound that's right for one operator is wrong for the next.
If you want to bound a single noisy service (a bulk indexer, a one-off backfill) without hand-tuning everything else, add the limit in your operator-local override file rather than the tracked compose:
# docker-compose.override.yml (mainnet) or docker-compose.testnet.override.yml (testnet)
services:
zebra:
deploy:
resources:
limits:
cpus: "8"
memory: 8gKeep the limit generous on the service you want to win contention (Zebra), tight on the service you want to lose it (the noisy consumer). Don't add limits to services you haven't actually seen misbehave, since under-sized limits cause more outages than they prevent.
Z3's checked-in defaults stay pinned so a docker compose pull cannot silently change the node version for every operator. If you prefer fewer Z3 stack updates and accept a moving Zebra image, set the image override in your operator-local .env:
Z3_ZEBRA_IMAGE=zfnd/zebra:latestThen pull before recreating Zebra. Docker does not fetch a newer image just because the tag is named latest; run pull first or use up --pull always. When you use the documented --env-file .env.<network> commands, pass .env after the network file so the override is loaded:
docker compose --env-file .env.mainnet --env-file .env pull zebra
docker compose --env-file .env.mainnet --env-file .env up -d zebra
# Equivalent one-command form:
docker compose --env-file .env.mainnet --env-file .env up -d --pull always zebraFor testnet or regtest, replace .env.mainnet with .env.testnet or .env.regtest. If you run mainnet without --env-file, Compose auto-loads .env, so docker compose up -d --pull always zebra is enough.
The same override pattern works for other services through Z3_ZAINO_IMAGE and Z3_ZALLET_IMAGE, but do not switch those to a moving tag unless you have checked the tag keeps the variant and platform support Z3 expects.
The default Z3 stack does not require Zebra's indexer build feature or its indexer gRPC listener. Z3 runs the zallet-zaino wallet backend, which reads Zebra through its regular JSON-RPC endpoint, while the optional indexer Compose profile starts the standalone Zaino service.
Several settings use the same word for different capabilities:
| Name | What it does | Changes Zebra's stored state? | Required by default Z3? |
|---|---|---|---|
Compose --profile indexer |
Starts the standalone Zaino service for lightwalletd-compatible clients | No | No |
Zallet's [indexer] section |
Configures how Z3's zallet-zaino process reaches Zebra over JSON-RPC |
No | Yes |
Zebra's rpc.indexer_listen_addr |
Starts the zebra.indexer.rpc.Indexer gRPC listener in Zebra 6.2 |
No | No |
Zebra's Cargo indexer feature |
Adds persistent indexes that map spent outpoints and revealed nullifiers to spending transactions | Yes | No |
The gRPC listener reads Zebra's existing blocks and streams chain-tip, non-finalized-state, and mempool changes. You can enable it on a state directory or restored snapshot that Zebra previously opened without the listener; it does not change the database format, add column families, or backfill data. Set the listener in the operator-local .env:
ZEBRA_RPC__INDEXER_LISTEN_ADDR=0.0.0.0:8155Then recreate Zebra:
docker compose --env-file .env.<network> up -d zebraContainers attached to the Z3 network can then connect to zebra:8155. Z3 does not publish this unauthenticated, plaintext endpoint to the host; add an operator-local Compose port mapping only when a host-side client needs it, and do not expose it to an untrusted network.
The alternative direct-state Zallet backend is different. It shares Zebra's state directory, follows the non-finalized tip over the gRPC listener, and requires a Zebra binary built with the Cargo indexer feature. Enabling that build feature against an existing database checks and backfills its persistent spending indexes, so it can make the next startup expensive. Z3 deliberately uses zallet-zaino instead, which needs neither the shared state directory nor those additional Zebra indexes.
Overrides are opt-in. A fresh clone boots with no override file, so nothing breaks before you run any setup.
- Mainnet: Compose natively auto-loads
docker-compose.override.ymlwhen it is present and you pass no-forCOMPOSE_FILE. An absent file is not an error. - Testnet: No network overlay exists;
.env.testnetsetsCOMPOSE_FILE=docker-compose.yml, so the stack renders on a fresh clone. To add per-host customizations (pinning Zebra tolinux/arm64, addingdeploy.resources.limits), createdocker-compose.testnet.override.ymland load it explicitly, either by passing-f docker-compose.yml -f docker-compose.testnet.override.yml, or by appending the override toCOMPOSE_FILEin your operator-local.env. - Regtest:
.env.regtestsetsCOMPOSE_FILE=docker-compose.yml:docker-compose.regtest.ymlto layer the regtest overlay (addsrpc-routerand regtest-only settings) on a fresh clone. Per-host customizations work the same way as testnet: createdocker-compose.regtest.override.ymland either pass it with-for append it toCOMPOSE_FILE.
The compose merge order is left-to-right, so the override comes last and wins. The live override file is gitignored, so git pull never touches it.
Because docker compose --env-file .env.<network> replaces the auto-loaded .env for variable interpolation rather than layering on top of it. When you use --env-file, anything set only in .env is ignored for ${DOCKER_PLATFORM:-linux/amd64} substitution. This is documented inside .env itself and in docs/contract.md → How env is loaded, but it bites everyone at least once.
Two reliable workarounds:
# (a) export before the call (shell env beats env-file in compose precedence)
export DOCKER_PLATFORM=linux/arm64
docker compose --env-file .env.mainnet up -d
# (b) pass .env as a second --env-file (loaded after the network one, so its values win)
docker compose --env-file .env.mainnet --env-file .env up -dThe mainnet stack also has a third path: docker-compose.override.yml (gitignored, auto-loaded for mainnet only) can hold an explicit services.zebra.platform: linux/arm64. That bypasses env-var interpolation entirely. For testnet and regtest the override file isn't auto-loaded; see the overrides question above.
The default Z3_ZEBRA_IMAGE is multi-arch and resolves to native arm64 on Apple Silicon. If uname -m inside the Zebra container reports x86_64 instead of aarch64, you've either pinned Z3_ZEBRA_IMAGE to an amd64-only tag or exported DOCKER_PLATFORM=linux/amd64. Emulation pushes Halo2/Groth16 verification past its internal deadline and surfaces as Transaction(InternalDowncastError("...Elapsed(())")) followed by chain-tip sync stalls.
Confirm what's actually running:
docker exec z3-mainnet-zebra-1 uname -m # aarch64 = native, x86_64 = emulated
docker top z3-mainnet-zebra-1 -o cmd | head -3 # /usr/bin/qemu-x86_64 wrapper = emulatedZaino is pinned to amd64 by default (its upstream image publishes amd64 only) and runs under emulation; the workload is light enough that the CPU drain is barely noticeable next to Zebra's verifier. Zallet is multi-arch and runs natively.
Not from the pinned tag. The default Zaino image publishes linux/amd64 only (declared in z3-contract.yaml under image_platforms:). Confirm with docker buildx imagetools inspect <image>. The unknown/unknown entries in that output are OCI attestation manifests (SBOM/provenance), not real platform variants.
Two ways forward if you need native arm64:
- Build locally. Fetch the upstream source with
scripts/vendor.sh zaino, then build with the opt-in overlay:DOCKER_PLATFORM=linux/arm64 docker compose -f docker-compose.yml -f docker-compose.build.yml build zaino. - Wait for the upstream tag to gain a multi-arch publish, then bump the pin. Existing tags never gain new platform variants after the fact; only new tags do.
Leaving Zaino under emulation is fine in practice; the workload is light compared to Zebra's verifier, which runs natively. Zaino only runs under the indexer profile, so the default stack is unaffected either way.
Zaino's canonical upstream is zingolabs/zaino, published to Docker Hub as zingodevops/zainod (matching the daemon binary name). zingodevops/zaino is an alias publishing identical digests.
Because docker image inspect <tag> returns metadata for whichever variant your local store currently has cached under that tag, not the variant the running container was launched from. On an arm64 host the local cache often holds the arm64 metadata for a multi-arch tag even when a platform: linux/amd64-pinned container is actively running the amd64 variant out of the same manifest list. The tag → arch mapping is not the container → arch mapping.
For the actual running architecture, use runtime signals:
docker exec <container> uname -m # reports x86_64 or aarch64
docker top <container> -o cmd | head -3 # qemu-x86_64 wrapper = emulated
docker exec <container> sh -c 'od -An -tx1 -N20 /path/to/binary' # ELF e_machine at offset 0x12The ELF e_machine field at offset 0x12 is 0x3e for x86_64 and 0xb7 for aarch64. That's the definitive answer when uname or /proc aren't available.
Because Zebra serves block fetches and indexer streaming from the same Tokio worker pool that drives consensus block verification. There is no built-in per-client RPC rate limit and no priority queue between request handlers and the verifier. A bulk indexer in catch-up mode (Zinder, lightwalletd, a fresh Zaino) with high fetch_concurrency can sustain enough RPC pressure to slow tip sync, and in extreme cases push transaction verification past its internal deadline.
The CPU saturation is usually the consumer's choice, not Zebra's: turn the consumer's parallelism down before you reach for compose-level limits. For an indexer that's behind, a fetch_concurrency in the low single digits while it catches up is much friendlier than the defaults most clients ship with.
If you can't control the consumer, the next lever is compose-level CPU limits (the resource-limits question under For operators) so the consumer can never starve the verifier.
Because regtest is meant to look like classic local dev, where username/password auth is the convention every existing tutorial and client library assumes. The regtest overlay (docker-compose.regtest.yml) disables Zebra's cookie auth and adds an rpc-router sidecar that authenticates with zebra / zebra against Zebra, so the same RPC client works without juggling cookie files.
Cookie auth stays the default for mainnet and testnet, where Zallet (and Zaino under the indexer profile) reads the shared cookie volume directly. See docs/regtest.md for the full regtest workflow and the curl/grpcurl examples that use the regtest credentials.
If you hit a behavior that looks wrong and the FAQ doesn't cover it, the fastest path to triage is:
- Check the container's actual runtime state (the
uname -m/docker top/ ELF-header trio in the architecture-detection question above). Most "why is this slow?" reports trace back to a platform or resource-limit assumption that wasn't true. - Look at the compose-resolved config, not the source YAML:
docker compose --env-file .env.<network> config <service>shows you the variables after interpolation, which is what Docker actually receives. - Open an issue with the resolved config, the runtime signals, and the symptom. Without the resolved config, every triage round restarts from "is your env var actually set."