--- title: CI Design --- # CI Design The CI/CD trust model splits into two disjoint tiers. Every cache, prebuilt image, or restored artifact is assigned to exactly one tier, and nothing crosses from Tier A into Tier B. This document is the versioned record of that reasoning — it changes only when the tier boundary itself changes. ## The Two Tiers **Tier A — PR-gate, may be cached, optimized for speed.** Everything triggered by `pull_request`, a non-default-branch `push`, or `workflow_dispatch`, whose only output is a pass/fail gate plus disposable artifacts: fmt, clippy, test, docs build, dependency scan, security assurance. **Tier B — release/attestation, must stay hermetic.** `release.yml`'s `build-native` matrix and anything producing a signed, attested, or published artifact: `tar.gz`/`.zip`/`.deb`/`.dmg`/`.msi`, provenance, SBOM attestation, the Pages deploy of release docs. No cache key, restored artifact, or prebuilt image consumed by Tier B may ever be writable by a Tier A trigger. This is the tj-actions/TanStack lesson: the attack needs one cache reachable from both a fork/PR trigger and a release trigger. The tier split structurally denies that by construction, not by a permissions review that has to be re-verified every change. ## What May Be Cached (Tier A Only) 1. The compiled Wolfi builder image, pulled by immutable `@sha256` digest from GHCR, never rebuilt in-job. 2. `cargo`'s `target/` directory and test-binary archive, via `Swatinem/rust-cache`, with `save-if` gated to `main` so fork PRs restore-only and can never write. 3. tox/Sphinx virtualenvs, via `actions/cache` keyed on content hashes. All three are correctness-transparent: a poisoned or stale entry can only make a Tier A gate compile or test wrong code, which that gate's own suite then rejects. It cannot reach a published artifact because Tier B never reads any of them. ## What Must Stay Hermetic (Tier B, Untouched) Wolfi base pinned by `@sha256`; every `apk` pinned to an exact `-rN` version; the RustSec advisory DB pinned to a git rev; vendored crates plus `cargo --locked --frozen --offline`. Release builds compile cold, inside the digest-pinned image, with no `rust-cache`, no restored `target/`, no GHA cache of any kind. Release stays reproducible-by-construction: same image bytes plus same `vendor/` plus same source equals same binary. The prebuilt GHCR image does not weaken this. It is byte-identical to today's per-job Dockerfile build — same Dockerfile, same digest — just built once and pinned instead of rebuilt N times per PR. ## Why the Builder Image Is the Dominant Waste `.github/actions/builder` is a `runs: using: docker, image: Dockerfile` container action. GitHub rebuilds the entire Wolfi image — `apk` installs, `pip install tox`, `cargo install sphinx-rustdocgen`, the advisory-db clone, smoke checks, roughly 2.5–3.2 minutes — from scratch on every job invocation, with zero layer cache. That happens roughly five times per PR across `ci.yml` (rust, docs, dependency-scan), `security-assurance.yml`, and every push to `cd.yml`. Prebuilding once to GHCR and consuming by digest is the single highest-ROI change here, and it is a strict security improvement: a digest pin beats a per-job rebuild-from-Dockerfile that can silently drift between runs if the Dockerfile's own inputs (e.g. upstream mirrors) hiccup. ## Cache Poisoning Is Solved by Posture, Not Permissions Locked, frozen, offline, and vendored already means Tier A has no *dependency* cache to poison — `cargo` never touches a registry. The only new cached surface is the compiled object cache (`target/`) and the builder image, and both are Tier-A-only: fork PRs restore but cannot write (`save-if` plus GitHub's read-only-cache-for-untrusted-triggers backstop). Merge is never gated on cache warmth — a cold cache must still produce a correct, complete build. ## Sequencing Constraint Splitting `fmt` out of `make ci` into its own fail-fastest job only pays off once container-spin-up is cheap. Before the GHCR image lands, splitting fmt adds a second full image rebuild for near-zero benefit. P6.5 is explicitly sequenced after P6.1 for this reason; do not reorder. ## Changes ### P6.1 — Prebuild the Wolfi builder image to GHCR, consume by digest `.github/workflows/builder-image.yml` builds `.github/actions/builder/Dockerfile` via `docker/setup-buildx-action` + `docker/build-push-action`, with `cache-from`/`cache-to type=gha,mode=max`, and pushes to `ghcr.io/jbcom/paranoid-passwd-builder`. Triggers: `push` with `paths: ['.github/actions/builder/**']` on `main`, a weekly `schedule`, and `workflow_dispatch`. The publish job carries `permissions: packages: write` and is unreachable from `pull_request` — a fork PR can never publish an image. `.github/actions/builder/action.yml` moves from `image: Dockerfile` to `image: docker://ghcr.io/jbcom/paranoid-passwd-builder@sha256:`, with the Dockerfile retained in-repo as the build source of truth. **Expected saving:** ~2.5–3.2 min per job × ~5 jobs/PR ≈ 12–16 min of wall-clock removed per PR. Jobs run in parallel, so the PR critical path drops from ~22.5 min toward ~19–20 min immediately; the redundant docs/dep-scan/assurance image builds vanish entirely. Push-to-main sheds the `cd.yml` rebuild too. **Risk (MEDIUM):** the GHCR image must exist and its digest must be pinned before the consuming workflows switch — see Bootstrap Ordering below. A stale digest means CI runs old tooling until the next scheduled or paths-triggered rebuild lands a bump. The image is Tier-A-consumed for PR gates and Tier-B-consumed for release's Linux legs, but because it is byte-identical to today's Dockerfile build and pinned by digest, the hermetic guarantee is preserved, not weakened. **Rollback:** revert `action.yml`'s `image:` field back to `Dockerfile`. The Dockerfile stays in-repo as the GHCR build's source of truth, so this is a one-line revert with zero other changes. ### Bootstrap Ordering The GHCR image does not exist until `builder-image.yml` has run once on `main`, and this PR cannot know that digest in advance. The switch ships **ready-but-gated**: 1. This PR adds `builder-image.yml` and leaves `.github/actions/builder/action.yml` on `image: Dockerfile` (the current, working, digest-pinned-Wolfi-base build). The `docker://` consumption line is present in `action.yml` but commented out, alongside a comment pointing at the bootstrap script. 2. `scripts/bump_builder_digest.sh` is added. It resolves the current `ghcr.io/jbcom/paranoid-passwd-builder:latest` digest via `docker buildx imagetools inspect` (or `skopeo inspect` if present) and rewrites `action.yml`'s commented `docker://...@sha256:` line to the resolved digest, then uncomments it and comments out `image: Dockerfile`. 3. After this PR merges to `main`, `builder-image.yml` runs (path trigger fires on the `.github/actions/builder/**` change) and publishes the first image. 4. The bootstrap script is then run once, in a follow-up commit, to flip `action.yml` onto the digest-pinned `docker://` reference. From that point forward, `builder-image.yml`'s weekly schedule plus `workflow_dispatch` is the update path — bump the digest, open a PR, let CI prove the new image still builds green, merge. This ordering means `action.yml` never references a digest that does not exist yet, and the working tree is never left with a broken builder action mid-PR. ### P6.2 — Deduplicate builder rebuilds and verify-assurance runs After P6.1, `security-assurance.yml` consumes the GHCR image instead of rebuilding it, and is reduced to the base-ref-diffed `security_assurance_gate.py` delta, since `make ci` already runs verify-assurance. The `security-assurance-report.{json,md}` artifact upload is preserved. `fetch-depth: 0` is preserved for the base-ref diff. **Expected saving:** ~3–4.5 min per PR. **Risk:** LOW — Tier A gate, trust-neutral. **Rollback:** restore the standalone rebuilding form, one-file revert. ### P6.3 — Tier-A cargo target-dir caching, fork-PR write protection `ci.yml`'s Rust job gets `Swatinem/rust-cache`, SHA-pinned, with `save-if: github.ref == 'refs/heads/main'`, `CARGO_INCREMENTAL=0` at workflow env, and `CARGO_TARGET_DIR` pointed at a host-mounted path the action can persist (overriding the Dockerfile's ephemeral `/tmp/cargo-target`). `release.yml`'s `build-native` legs carry no `rust-cache` step and restore no `target/` — verified by grep, not by convention. **Expected saving:** ~6–9 min per PR on warm cache. **Risk:** MEDIUM, mitigated by `save-if` main-only, the GitHub read-only-cache backstop, and the fact that a poisoned `target/` can only miscompile a Tier A gate that its own suite then rejects — Tier B never restores it. **Rollback:** delete the cache step and the target-dir override. ### P6.4 — Cache the docs/tox toolchain, remove the double docs build `ci.yml`'s docs job consumes the GHCR image (so `sphinx-rustdocgen`'s `cargo install` cost disappears) and adds `actions/cache`, SHA-pinned, keyed on `tox.ini` plus docs-requirement content hashes. The redundant docs build between `ci.yml` and `cd.yml`'s `deploy-pages` on push-to-main is reduced — but `deploy-pages` carries `id-token:write` and is Tier-B-adjacent, so it must never restore a PR-writable cache. It keeps building fresh from the digest-pinned image; only the `ci.yml` PR-side docs venv is cached. **Expected saving:** docs job ~6–7 min → ~1–1.5 min; `deploy-pages` ~3.5–4 min → ~1.5 min. **Risk:** LOW. **Rollback:** remove the cache step. ### P6.5 — Split fmt into a fastest-first Tier-A job After P6.1, `cargo fmt --check` runs as its own job, separate from clippy+test (which keeps `rust-cache`). Must not precede P6.1 — see Sequencing Constraint above. **Expected saving:** marginal on the happy path, but a formatting-only mistake reports in under a minute instead of after a full compile. **Risk:** LOW. **Rollback:** re-merge into `make ci`. ### P6.6 — Least-privilege and trust-boundary audit Every Tier A workflow keeps `permissions: {}` at top level with per-job least privilege. `packages: write` exists only on the builder-image publish job (push/schedule-only, unreachable from `pull_request`). `id-token:write` exists only on Tier B / deploy jobs and never on a job that restores a PR-writable cache. A written boundary table enumerates every cache, who can write it, who reads it, and asserts no cache is both PR-writable and release-readable. ## Rejected Options **sccache with the GHA cache backend.** sccache's object-level dedup shines across large multi-crate, multi-job matrices sharing objects across many jobs. Neither tokio, ruff, nor rust-analyzer use it — all chose `rust-cache` or plain `CARGO_INCREMENTAL=0` plus nextest. With three crates and an offline/vendored/locked posture, sccache's dedup would not beat `rust-cache`'s simpler `target/` tarball enough to justify a second caching subsystem that itself needs Tier-A/Tier-B boundary reasoning. **`cargo-nextest --partition` matrix sharding.** The test suite is small today; partitioning multiplies container-mount/image-pull overhead and adds an all-green summary job for a suite that is not the bottleneck. Revisit only if the statistical/property suite grows slow — then hashed partitioning stable across vendored-set changes, plus nextest archive build-reuse, is the right shape. **Registry cache exporter (`type=registry`) instead of `type=gha` for the image build.** The registry exporter earns its keep when cache needs to span repos/branches or exceed the 10 GB GHA cache cap. A single-repo builder image with a handful of layers fits GHA cache comfortably; `type=gha, mode=max` is simpler and sufficient. A registry cache would add another cross-boundary shared surface for no benefit at this scale. **Sharing a warm `rust-cache`/target-dir between PR gates and `release.yml`'s Linux `build-native` legs.** Hard no. This is exactly the tj-actions/TanStack cache-poisoning vector: a cache reachable from both a fork/PR trigger and the release/attestation trigger. Release stays cold-compile, locked/frozen/offline, inside the digest-pinned image. The speed cost on release is accepted and non-negotiable — release runs on tag-publish, not the PR feedback loop, so it is not the optimization target. **Caching `vendor/` or a cargo registry cache.** Pointless and boundary-risky. `vendor/` is committed and builds are `--offline --frozen`, so `cargo` never touches a registry — there is no dependency cache to warm. This *is* the mitigation the vendored posture exists to provide; adding a cache here would manufacture a poisoning surface the vendored posture was specifically built to eliminate. **Moving CodeQL / OpenSSF Scorecard / the fleet-managed JS-TS CodeQL into the builder, or consolidating them.** Out of scope and structurally constrained. The JS-TS CodeQL workflow is fleet-managed (`jbdevprimary/gh-fleet-sync`), do-not-edit-in-place, and exists to satisfy an Enterprise branch-protection rule. Scorecard is deliberately isolated because its webapp refuses results if any co-workflow job carries `id-token:write` (conflicts with `cd.yml`'s Pages OIDC). `ci.yml` vs `cd.yml` CodeQL are complementary (non-push vs. push gating), not truly redundant. Touching these risks branch-protection/OIDC breakage for ~1–2 min/lang of native-runner time that is not the bottleneck. **Switching the builder from a Dockerfile container action to an apko/melange-defined image.** Tempting for Wolfi-native reproducibility, but it is a large migration that changes the trust root's tooling and every pinned-package mechanism at once, for no additional PR-speed win beyond what GHCR-prebuild-by-digest already delivers. The current Dockerfile already achieves a digest-pinned Wolfi base plus pinned `apk` versions. Deferred as possible future hardening, not part of the speed-maximization work. ## Supply-Chain Gate Interaction `scripts/supply_chain_verify.sh` asserts the builder image is Wolfi-based and digest-pinned by grepping fixed strings out of `.github/actions/builder/Dockerfile` itself (the `FROM cgr.dev/chainguard/wolfi-base@sha256:...` line and pinned `ARG` values), not out of `action.yml`'s `image:` field. Keeping the Dockerfile as the GHCR build's source of truth during the bootstrap window (see Bootstrap Ordering) means this gate's existing assertions keep passing unchanged whether `action.yml` points at `Dockerfile` or at the pinned `docker://...@sha256:` reference — no allowlist changes were needed for P6.1. Once `action.yml` is flipped to consume the GHCR digest, a future item should extend `scripts/supply_chain_verify.sh` to also assert the `docker://ghcr.io/jbcom/paranoid-passwd-builder@sha256:` reference in `action.yml` matches an image whose provenance traces back to a `builder-image.yml` run against the current `Dockerfile` — so the gate cannot be satisfied by a digest pointing at stale or hand-pushed image content. That assertion is out of scope for P6.1: it depends on the digest that only exists after this PR's bootstrap step runs (see Bootstrap Ordering), so the check would be untestable at PR-review time. ## P6.6 — Least-Privilege and Trust-Boundary Audit Audited every workflow under `.github/workflows/` directly (`grep`/read, not sampling). Two gaps were found and fixed in the same commit that added this table; everything else already matched the design. ### Top-level `permissions` and per-job least privilege | Workflow | Top-level `permissions` | Elevated job grants | Notes | |---|---|---|---| | `ci.yml` | `{}` | none beyond `contents: read`; `codeql` job adds `security-events: write` | all Tier A | | `cd.yml` | `{}` | `release-please`: `contents: write`, `pull-requests: write`; `deploy-pages`: `pages: write`, `id-token: write` | push/schedule/dispatch-only, no `pull_request` trigger | | `release.yml` | `{}` | `attest-and-publish`: `contents: write`, `id-token: write`, `attestations: write`; `release-surface-verify`/`release-download-verify`: `id-token: write` | Tier B; `release`/`workflow_dispatch` only | | `builder-image.yml` | `{}` | `publish`: `packages: write` | push(main, paths-scoped)/schedule/dispatch only — unreachable from `pull_request` | | `security-assurance.yml` | `{}` | none beyond `contents: read` | Tier A | | `codeql.yml` | none set (fleet-managed, do-not-edit-in-place — see file header) | `security-events: write`, `actions: read` | out of scope per design's Rejected Options | | `scorecard.yml` | **was `read-all`, fixed to `{}`** | `security-events: write`, `id-token: write` | isolated on purpose (Scorecard webapp rejects results if a co-workflow job holds `id-token:write`); left untouched otherwise per design | **Fix applied:** `scorecard.yml`'s top-level `permissions: read-all` was broader than needed — the `analysis` job already declares every scope it uses (`contents: read`, `security-events: write`, `id-token: write`). Tightened to `permissions: {}` at top level; job-level grants are unchanged and still sufficient. ### `packages:write` and `id-token:write` reachability | Scope | Job(s) that hold it | Trigger(s) that can reach the job | Fork-`pull_request`-reachable? | |---|---|---|---| | `packages: write` | `builder-image.yml` → `publish` | `push` (main, paths-scoped), `schedule`, `workflow_dispatch` | No | | `id-token: write` | `cd.yml` → `deploy-pages` | `push` (main) | No | | `id-token: write` | `release.yml` → `attest-and-publish`, `release-surface-verify`, `release-download-verify` | `release: published`, `workflow_dispatch` | No | | `id-token: write` | `scorecard.yml` → `analysis` | `push` (main), `schedule` | No | No job carrying `packages: write` or `id-token: write` is reachable from `pull_request`, and none of them restores a cache (see below) — confirmed by `grep -n "id-token\|packages:" .github/workflows/*.yml` cross-referenced against each workflow's `on:` block. ### Concurrency groups on `pull_request`-triggered workflows | Workflow | `pull_request` trigger? | `concurrency` group | `cancel-in-progress` | |---|---|---|---| | `ci.yml` | yes | `ci-${{ pr.number \|\| ref }}` | `true` | | `security-assurance.yml` | yes | `security-assurance-${{ pr.number \|\| ref }}` | `true` | | `codeql.yml` | yes | `codeql-${{ ref }}` | `true` | | `cd.yml` | no (push/schedule/dispatch) | `cd-${{ ref }}` | `true` (branch-scoped; latest push wins, not a trust issue) | | `release.yml` | no (release/dispatch) | none | n/a — cancelling a mid-publish attest/upload run is unsafe, correctly omitted | | `builder-image.yml` | no (push/schedule/dispatch) | `builder-image-${{ ref }}` | `false` (cancelling a mid-push GHCR publish could leave a partial tag; correctly non-cancelling) | | `scorecard.yml` | no (push/schedule) | none | n/a — low-frequency, non-blocking | All three `pull_request`-triggered workflows have a concurrency group with `cancel-in-progress: true`. No fix needed. ### SHA-pinning of external actions `grep -rn "uses:" .github/workflows .github/actions | grep -v "uses: \./" | grep -vE '@[a-f0-9]{40}'` returns zero matches — every external `uses:` (not a local `./`-relative action) is pinned to a full 40-character commit SHA with a trailing `# vX.Y.Z` version comment, matching the repo's existing pinning style. No fix needed. ### Cache inventory — write/read reachability | Cache | Written by | Write gate | Read by | Ever read by a Tier B job? | |---|---|---|---|---| | Docker layer cache (`type=gha`) for the builder image build | `builder-image.yml` → `publish` | push(main)/schedule/dispatch only | same job, same run | No — Tier B consumes the *published, digest-pinned image*, never this build-time layer cache | | `target/` via `Swatinem/rust-cache` | `ci.yml` → `rust` | `save-if: github.ref == 'refs/heads/main'` | `ci.yml` → `rust` (any ref, restore-only off main) | No | | `.tox` via `actions/cache` | `ci.yml` → `docs` | split into `actions/cache/restore` (every run) + `actions/cache/save` gated `github.ref == 'refs/heads/main'` (fixed by this item — previously a single ungated `actions/cache@v6` step) | `ci.yml` → `docs` (any ref, restore-only off main) | No | **Fix applied:** the `docs` job's `.tox` cache previously used the combined `actions/cache` action, which has no `save-if` equivalent — every run, including a same-repo PR branch, would attempt to write the cache key. Split into `actions/cache/restore` (unconditional) and `actions/cache/save` (gated to `github.ref == 'refs/heads/main'`, mirroring the `rust` job's `save-if`), so only a main-branch run can populate or refresh the entry. GitHub's own cache-scoping already prevented a *fork* PR from writing into the base repository's cache namespace; this closes the same-repo-branch gap explicitly rather than relying only on that platform backstop. `grep -rn "actions/cache\|rust-cache" .github/workflows/*.yml` confirms `release.yml` (Tier B) has zero cache references of any kind — it never restores `target/`, `.tox`, or any GHA cache. `cd.yml`'s `deploy-pages` job also has no `actions/cache` step (see its inline comment) and always builds docs fresh from the digest-pinned image. ### Scope confirmations left untouched Per the design's Rejected Options, `codeql.yml` (fleet-managed, synced from `jbdevprimary/gh-fleet-sync`, do-not-edit-in-place) and `scorecard.yml`'s `id-token` isolation (required by `ossf/scorecard-action`'s own workflow-restriction: the Scorecard webapp refuses results if any co-workflow job holds `id-token:write`) are intentionally out of scope for restructuring. Only `scorecard.yml`'s redundant top-level `permissions: read-all` was tightened; its isolation from `cd.yml` and its job-level grants are unchanged.