Merge remote-tracking branch 'origin/main' into fix-any-chain-treestate

Author: claude

.github/workflows/README.md

@@ -31,6 +31,8 @@ graph TB
   %% Reusable build
   subgraph Build
     BuildDocker[zfnd-build-docker-image.yml]
+    PrepareBinaries[zfnd-release-binaries.yml]
+    AttachBinaries[zfnd-attach-release-binaries.yml]
   end
 
   %% Release automation
@@ -69,14 +71,16 @@ graph TB
   Manual --> IT & DeployNodes & Cleanup
 
   %% Build dependency
+  ReleaseBinaries --> BuildDocker & PrepareBinaries
+  PrepareBinaries --> AttachBinaries
   BuildDocker --> IT
   IT --> FindDisks --> Deploy
 
   %% Styling
   classDef primary fill:#2374ab,stroke:#2374ab,color:white
   classDef secondary fill:#48a9a6,stroke:#48a9a6,color:white
   classDef trigger fill:#95a5a6,stroke:#95a5a6,color:white
-  class BuildDocker primary
+  class BuildDocker,PrepareBinaries,AttachBinaries primary
   class ReleaseWorkflow,ReleaseBinaries primary
   class Unit,Lint,Coverage,DockerCfg,CrateBuild,PRGate,Docs,Security secondary
   class IT,FindDisks,Deploy,DeployNodes,Cleanup secondary
@@ -138,7 +142,40 @@ _The diagram above illustrates the parallel execution patterns in our CI/CD syst
 
 **Note**: Self-hosted Runners are just used to keep the logs running in the GitHub Actions UI for over 6 hours, the Integration Tests are not run in the Self-hosted Runner itself, but in the deployed VMs in GCP through GitHub Actions.
 
-### 5. Queue Management
+### 5. Rust build caching
+
+Rust jobs cache `~/.cargo` and dependency artifacts in `target/` through
+[`actions-rust-lang/setup-rust-toolchain`](https://github.com/actions-rust-lang/setup-rust-toolchain),
+which wraps [`Swatinem/rust-cache`](https://github.com/Swatinem/rust-cache). Caching is on by
+default, so a job opts _out_ with `cache: false` rather than opting in.
+
+GitHub gives each repository 10 GB of cache storage by default; administrators can configure a
+higher paid limit. Least-recently-used entries are evicted when the configured limit is exceeded.
+Caches are also branch-scoped: a branch can read its own caches and the default branch's,
+caches written by PRs can never be read by anyone else, but they still evict main's caches, which
+are the only ones every PR does restore from.
+
+Three rules keep the quota usable:
+
+1. **Only main writes.** The main building jobs set
+   `cache-save-if: ${{ github.ref == 'refs/heads/main' }}`. PRs restore from main and write nothing. (There are some minor exceptions to this rule.)
+2. **Wide matrices share one key.** The `test-crates.yml` matrices use
+   `cache-shared-key` and seed the shared cache from a single build (`zebrad`
+   which has the widest dependency closure; `zebra-rpc` for the MSRV build since
+   it does not build `zebrad` and `zebra-rpc` is second widest option).
+3. **Jobs that don't build don't cache.** `fmt`, `no-test-deps`, `deny` (12 jobs wide), and the
+   crate-matrix generator in `test-crates.yml` set `cache: false`.
+
+To inspect the current state:
+
+```bash
+gh api repos/ZcashFoundation/zebra/actions/cache/usage
+gh api 'repos/ZcashFoundation/zebra/actions/caches?per_page=100' \
+  --jq '[.actions_caches[] | {ref, mb: (.size_in_bytes/1048576|round)}]
+        | group_by(.ref)[] | "\(.[0].ref) n=\(length) \(map(.mb)|add)MB"'
+```
+
+### 6. Queue Management
 
 [Mergify](https://mergify.com)
 
@@ -160,12 +197,14 @@ _The diagram above illustrates the parallel execution patterns in our CI/CD syst
 - **Docs (Book + internal)** (`book.yml`): Builds mdBook and internal rustdoc, publishes to Pages
 - **Security Analysis** (`zizmor.yml`): GitHub Actions security lint (SARIF)
 - **Release** (`release.yml`): Creates or updates Release PRs with release-plz, then uses `ZcashFoundation/cargo-release` and native Cargo to reconcile crates, tags, and one `zebrad` GitHub Release. See the [release process](../../book/src/dev/release-process.md#release-candidate--release-process) for operational instructions.
-- **Release Binaries** (`release-binaries.yml`): Build and publish release artifacts
+- **Release Binaries** (`release-binaries.yml`): Orchestrates release images, prepares and attaches downloadable binaries, and supports manual preparation validation without release attachment
 - **Integration Tests on GCP** (`zfnd-ci-integration-tests-gcp.yml`): Stateful tests, E2E tests, cached disks, lwd flows
 
 ### Supporting/Re-usable Workflows
 
 - **Build docker image** (`zfnd-build-docker-image.yml`): Reusable image build with caching and tagging
+- **Prepare release binaries** (`zfnd-release-binaries.yml`): Builds, attests, checksums, signs, and uploads the immutable binary bundle
+- **Attach release binaries** (`zfnd-attach-release-binaries.yml`): Attaches the prepared binary bundle to an existing GitHub Release
 - **Find cached disks** (`zfnd-find-cached-disks.yml`): Discovers GCP disks for stateful tests
 - **Deploy integration tests** (`zfnd-deploy-integration-tests-gcp.yml`): Orchestrates GCP VMs and test runs
 - **Deploy nodes** (`zfnd-deploy-nodes-gcp.yml`): Provision long-lived nodes

.github/workflows/lint.yml

@@ -70,7 +70,16 @@ jobs:
         with:
           components: clippy
           toolchain: 1.97.0
+          # Both matrix legs build the same workspace, so they share one cache.
+          cache-shared-key: clippy
           cache-on-failure: true
+          # Restore on every ref, but only write from main. Actions caches are
+          # branch-scoped, so a cache written by a PR can never be read by any
+          # other branch, yet it still consumes the repo-wide cache allowance
+          # and can evict main's caches - the only ones PRs can restore from.
+          # The `tests` leg seeds it, because its feature set is a superset.
+          # See "Rust build caching" in .github/workflows/README.md.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.type == 'tests' }}
       - uses: ./.github/actions/setup-zebra-build
       - name: Run clippy
         run: cargo clippy ${{ matrix.args }} --features "${{ matrix.features }}"
@@ -90,6 +99,8 @@ jobs:
       - uses: actions-rust-lang/setup-rust-toolchain@46268bd060767258de96ed93c1251119784f2ab6 #v1.16.1
         with:
           cache-on-failure: true
+          # Only main writes caches, see the clippy job above.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
       - uses: taiki-e/install-action@65851e10cd6c377f11a60e600abc07cb08643468 #v2.79.3
         with:
           tool: cargo-hack
@@ -118,6 +129,8 @@ jobs:
         with:
           toolchain: 1.91.0 # MSRV
           cache-on-failure: true
+          # Only main writes caches, see the clippy job above.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
       - uses: ./.github/actions/setup-zebra-build
       - run: cargo build --bin "${{ matrix.binary }}" --workspace
 
@@ -138,6 +151,8 @@ jobs:
         with:
           toolchain: 1.91.0
           components: rustfmt
+          # `cargo fmt` builds nothing, so a cache here only consumes quota.
+          cache: false
       - name: Run fmt
         run: cargo fmt --all -- --check
 
@@ -154,7 +169,8 @@ jobs:
       - uses: actions-rust-lang/setup-rust-toolchain@46268bd060767258de96ed93c1251119784f2ab6 #v1.16.1
         with:
           toolchain: 1.97.0
-          cache-on-failure: true
+          # `cargo tree` builds nothing, so a cache here only consumes quota.
+          cache: false
       - name: Ensure no arbitrary or proptest dependency on default build
         run: cargo tree --package zebrad -e=features,no-dev | grep -Eq "arbitrary|proptest" && exit 1 || exit 0
 
@@ -175,6 +191,8 @@ jobs:
         with:
           toolchain: 1.97.0
           cache-on-failure: true
+          # Only main writes caches, see the clippy job above.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
       - uses: ./.github/actions/setup-zebra-build
       - run: cargo check --locked --all-features --all-targets
 
@@ -211,7 +229,9 @@ jobs:
           persist-credentials: false
       - uses: actions-rust-lang/setup-rust-toolchain@46268bd060767258de96ed93c1251119784f2ab6 #v1.16.1
         with:
-          cache-on-failure: true
+          # cargo-deny reads the dependency graph without building it, and this
+          # matrix is 12 jobs wide, so caching here is pure quota consumption.
+          cache: false
       - name: Check ${{ matrix.checks }} with features ${{ matrix.features }}
         uses: EmbarkStudios/cargo-deny-action@6c8f9facfa5047ec02d8485b6bf52b587b7777d1 #v2.0.18
         with:

.github/workflows/pr-gate.yml

@@ -137,6 +137,9 @@ jobs:
           feature-group: default-features
           baseline-rev: ${{ steps.baseline.outputs.rev }}
 
+  # This checks for changelog entries depending on the type of the PR (per title).
+  # An entry is required for a crate if any file in the crate changed.
+  # If the title indicates a breaking change, the changelog must also have a `### Breaking Changes` section.
   changelog-gate:
     needs: changes
     if: >-
@@ -192,6 +195,8 @@ jobs:
             exit 1
           fi
           echo "base_sha=${base_sha}" >> "${GITHUB_OUTPUT}"
+      # Checks if the changelogs have entries for the versions being released
+      # (used in the in the release PR).
       - name: Validate release changelogs
         id: changelogs
         continue-on-error: true
@@ -219,7 +224,7 @@ jobs:
       - name: Check desired release state
         id: cargo-release
         continue-on-error: true
-        uses: ZcashFoundation/cargo-release@34a37595755444456ce0e2d2b1258d9a29c14fac
+        uses: ZcashFoundation/cargo-release@0083006dfd267ed560b4cf1aed347ef2f326162e # v0.1.0
         with:
           phase: check
           base-sha: ${{ steps.release-target.outputs.base_sha }}
@@ -284,4 +289,7 @@ jobs:
         uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe # v1.2.2
         with:
           jobs: ${{ toJSON(needs) }}
+          # Leave it for the PR author and reviewer to decide if the changelog
+          # entries are good enough.
+          allowed-failures: changelog-gate
           allowed-skips: semver-checks, changelog-gate, release-readiness

.github/workflows/release-binaries.yml

@@ -7,12 +7,14 @@
 # This workflow is triggered if:
 # - A release is published
 # - A pre-release is changed to a release
+# - A maintainer manually validates binary preparation before a release
 name: Release binaries
 
 on:
   release:
     types:
       - released
+  workflow_dispatch:
 
 permissions:
   contents: read
@@ -51,15 +53,52 @@ jobs:
       DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
       DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
 
-  # Build and attach downloadable zebrad binaries to the GitHub release.
-  binaries:
-    name: Attach release binaries
+  # Resolve the release tag into the version used in binary asset names.
+  binary-version:
+    name: Resolve release binary version
     if: github.repository_owner == 'ZcashFoundation' && startsWith(github.event.release.tag_name, 'v')
+    runs-on: ubuntu-latest
+    timeout-minutes: 1
+    outputs:
+      version: ${{ steps.release.outputs.version }}
+    steps:
+      - name: Resolve release inputs
+        id: release
+        env:
+          RELEASE_TAG: ${{ github.event.release.tag_name }}
+        run: |
+          printf 'version=%s\n' "${RELEASE_TAG#v}" >> "${GITHUB_OUTPUT}"
+
+  prepare-binaries:
+    name: Prepare release binaries
+    needs: binary-version
     permissions:
-      contents: write
-      id-token: write
-      attestations: write
+      contents: read
+      id-token: write # authenticate binary provenance and signatures
+      attestations: write # store binary provenance
     uses: ./.github/workflows/zfnd-release-binaries.yml
+    with:
+      version: ${{ needs.binary-version.outputs.version }}
+      features: default-release-binaries
+
+  validate-binaries:
+    name: Validate release binary preparation
+    if: github.event_name == 'workflow_dispatch' && github.repository_owner == 'ZcashFoundation'
+    permissions:
+      contents: read
+      id-token: write # authenticate test provenance and signatures
+      attestations: write # store test provenance
+    uses: ./.github/workflows/zfnd-release-binaries.yml
+    with:
+      version: 0.0.0-test
+      features: default-release-binaries
+
+  attach-binaries:
+    name: Attach release binaries
+    needs: prepare-binaries
+    permissions:
+      contents: write # attach assets to the existing release
+    uses: ./.github/workflows/zfnd-attach-release-binaries.yml
     with:
       release_tag: ${{ github.event.release.tag_name }}
 
@@ -95,7 +134,9 @@ jobs:
       }}
     needs:
       - build
-      - binaries
+      - binary-version
+      - prepare-binaries
+      - attach-binaries
     timeout-minutes: 1
     steps:
       - name: Decide whether the needed jobs succeeded or failed
@@ -106,7 +147,7 @@ jobs:
   failure-issue:
     name: Open or update issues for release binaries failures
     # When a new job is added to this workflow, add it to this list.
-    needs: [ build, binaries ]
+    needs: [ build, binary-version, prepare-binaries, attach-binaries ]
     # Open tickets for any failed build in this workflow.
     if: failure() || cancelled()
     runs-on: ubuntu-latest

.github/workflows/release.yml

@@ -254,7 +254,7 @@ jobs:
       - name: Check desired release state
         id: cargo-release
         continue-on-error: true
-        uses: ZcashFoundation/cargo-release@34a37595755444456ce0e2d2b1258d9a29c14fac
+        uses: ZcashFoundation/cargo-release@0083006dfd267ed560b4cf1aed347ef2f326162e #v0.1.0
         with:
           phase: check
           source-directory: release-source
@@ -363,7 +363,7 @@ jobs:
         uses: rust-lang/crates-io-auth-action@c6f97d42243bad5fab37ca0427f495c86d5b1a18 #v1.0.5
       - name: Publish and verify missing crates
         id: cargo-release-publish
-        uses: ZcashFoundation/cargo-release@34a37595755444456ce0e2d2b1258d9a29c14fac
+        uses: ZcashFoundation/cargo-release@0083006dfd267ed560b4cf1aed347ef2f326162e #v0.1.0
         with:
           phase: publish
           source-directory: release-source
@@ -409,7 +409,7 @@ jobs:
           exit 1
       - name: Create missing tags and GitHub Release
         id: cargo-release-finalize
-        uses: ZcashFoundation/cargo-release@34a37595755444456ce0e2d2b1258d9a29c14fac
+        uses: ZcashFoundation/cargo-release@0083006dfd267ed560b4cf1aed347ef2f326162e #v0.1.0
         with:
           phase: finalize
           source-directory: release-source

.github/workflows/test-crates.yml

@@ -63,7 +63,9 @@ jobs:
         with:
           toolchain: 1.97.0
           components: clippy
-          cache-on-failure: true
+          # This job only runs `cargo tree` to build the matrix, so a cache here
+          # only consumes quota.
+          cache: false
       - uses: ./.github/actions/setup-zebra-build
 
       # This step dynamically creates a JSON containing the values of each crate
@@ -117,8 +119,16 @@ jobs:
         with:
           toolchain: 1.97.0
           components: clippy
-          cache-key: crate-build-${{ matrix.crate }}
+          # One cache for the whole matrix instead of one per crate. These jobs
+          # compile overlapping subsets of the same dependency graph, so 12
+          # per-crate caches stored ~12 GB of near-duplicate artifacts and blew
+          # through the repo's default 10 GB quota on their own.
+          cache-shared-key: crate-build
           cache-on-failure: true
+          # Only main writes the shared cache, and only from the crate with the
+          # widest dependency closure, so the other legs restore a superset
+          # rather than whichever leg happened to finish first.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.crate == 'zebrad' }}
 
       - uses: ./.github/actions/setup-zebra-build
 
@@ -179,8 +189,13 @@ jobs:
       - uses: actions-rust-lang/setup-rust-toolchain@46268bd060767258de96ed93c1251119784f2ab6 #v1.16.1
         with:
           toolchain: ${{ steps.msrv.outputs.version }}
-          cache-key: crate-build-msrv-${{ matrix.crate }}
+          # One cache for the whole matrix, see the `build` job above.
+          cache-shared-key: crate-build-msrv
           cache-on-failure: true
+          # `zebrad` can't seed this matrix: the build step below skips it, so
+          # that leg compiles nothing. `zebra-rpc` has the widest closure of
+          # the crates that do build here.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.crate == 'zebra-rpc' }}
 
       - uses: ./.github/actions/setup-zebra-build
 

.github/workflows/tests-unit.yml

@@ -69,6 +69,10 @@ jobs:
           toolchain: 1.97.0
           cache-key: unit-tests-1.97.0-default-release-binaries
           cache-on-failure: true
+          # Restore on every ref, but only write from main. This is the longest
+          # job in CI, so it is the one that suffers most when PR-scoped caches
+          # evict main's copy. See "Rust build caching" in README.md.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
       - uses: taiki-e/install-action@65851e10cd6c377f11a60e600abc07cb08643468 #v2.79.3
         with:
           tool: cargo-nextest