Infrastructure, CI/CD, incident response
Cutting the cord from GHCR, a self-hosted registry and CI pipeline
zero third-party image dependency, one root-caused GC bug fixed for good
- Docker Registry
- GitHub Actions
- self-hosted runners
- Cloudflare TLS
Problem
Every deploy depended on GHCR being reachable and rate limits staying out of the way: a dependency with no upside for a fleet that already runs its own everything else. The fix was to run the registry at home too, on the network that already had a private CA and internal DNS.
Approach
Stood up registry:2 behind the wildcard-LAN TLS cert, added it to node
trust across the fleet, and migrated every active repo’s publish pipeline
to push there instead of GHCR. Builds run on the Mac first when it’s
available (no Docker-in-Docker hop, much faster) and fall back to a
per-repo self-hosted GitHub Actions runner otherwise. Version tags follow a
plain YYYY.MM.DD-NN convention, pinned in every manifest, never :latest
unless the app is Keel-paired for auto-rollout.
Weeks in, a tag that had pushed clean started 404-ing. Root cause: the
registry’s garbage-collector strands the amd64 child manifest of an OCI
image index during GC, and a later pull just finds nothing. The fix was at
the build step, not the registry: --provenance=false --sbom=false on every
build, so the push is a plain single-manifest image instead of an index with
a detachable child. No index, nothing for GC to strand.
Outcome
Every active project builds and deploys from the self-hosted registry with no third-party image host in the critical path, and the GC/index interaction that caused the original outage is now a one-line build flag baked into the convention for every future repo.