Standardizing Container Images Part 2: Hardened Runtimes and Agentic CI/CD

A year ago, I published a blog post on Medium titled Standardizing Base OS Container Images for Platform Engineering: A Security-First Pipeline. In that article, I explained why platform engineering teams must establish trusted base operating system images across minimal distributions like Alpine, AlmaLinux, Amazon Linux, and Wolfi, enforcing non-root execution and reproducible build pipelines.

Since then, our platform needs have grown. Providing hardened operating system layers is an essential first step, but application developers and AI engineers need ready-to-use language runtimes, such as Python and Node.js, that inherit those exact security guarantees.

To support this scale, I recently overhauled my open-source container-images repository. I introduced a layered container supply chain, split runtimes into lean production and development builder variants, streamlined BaseOS images, added dual-registry mirroring, and structured the repository for deterministic Agentic AI pair programming.

This post is Part 2, walking through how my container supply chain evolved and how you can apply these patterns to your own platform.

The Problem: The Runtime and Build Tool Trap

Standardizing BaseOS images solves only half the problem. When application teams take a hardened base image and add language runtimes casually, several common anti-patterns emerge:

• Privilege Escalation: Developers switch to USER root to install packages and forget to drop back to non-root execution.

• Leaked Build Tools in Production: Compilers, package managers (pip, npm), and development headers remain inside production images, unnecessarily inflating image size and expanding the attack surface.

• Inconsistent Base Tooling: Teams build Python or Node.js on disparate base distributions, creating fragmented patching and debugging workflows.

• Unpinned Upgrades: Running unpinned upgrade commands during container builds introduces unexpected library versions and CVE regressions.

To solve these challenges, I established a strict two-tier architecture: a minimal BaseOS foundation and purpose-built language runtime variants.

Layer 1: Overhauling the Wolfi BaseOS Foundation

The container hierarchy begins with a hardened Wolfi operating system layer (ghcr.io/cybergavin/wolfi:latest). I recently streamlined this image to its absolute bare essentials:

• Minimal Footprint: Removed auxiliary utilities like curl, jq, and openssl, retaining only ca-certificates-bundle on top of Chainguard's wolfi-base.

• Numeric Non-Root Execution: Standardized on numeric user 1001:1001 (appuser:appgroup).

• Tamper-Resistant Permissions: Applied umask 022, configured root-owned application directories (/app and /home/appuser at 0755), and restricted state directories (/var/lib/app and /var/cache/app at 0750).

• Natural Upgrade Prevention: By executing as non-root user 1001:1001, package manager upgrades (apk upgrade) are naturally blocked by filesystem permission boundaries without requiring custom wrapper scripts.

• Standardized OCI Metadata: Standard OCI labels (org.opencontainers.image.*) track versions, revisions, source URLs, and build timestamps.

Here is how concise and hardened the Wolfi BaseOS Dockerfile is:

ARG BASE_IMAGE=cgr.dev/chainguard/wolfi-base:latest
FROM ${BASE_IMAGE}

USER 0:0

ENV HOME=/home/appuser \
    XDG_CACHE_HOME=/var/cache/app \
    PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

RUN set -eu; \
    umask 022; \
    apk upgrade --no-cache; \
    apk add --no-cache ca-certificates-bundle; \
    addgroup -g 1001 appgroup; \
    adduser -D -H -u 1001 -G appgroup \
        -h /home/appuser -s /bin/sh appuser; \
    mkdir -p /app /home/appuser /var/lib/app /var/cache/app; \
    chown 0:0 /app /home/appuser; \
    chmod 0755 /app /home/appuser; \
    chown 1001:1001 /var/lib/app /var/cache/app; \
    chmod 0750 /var/lib/app /var/cache/app

ARG IMAGE_VERSION=dev
ARG VCS_REF=local
ARG BUILD_LABEL=local
ARG SOURCE_URL=https://github.com/cybergavin/container-images

LABEL org.opencontainers.image.title="cg-wolfi-runtime" \
      org.opencontainers.image.description="Minimal Wolfi OS base for downstream language runtimes" \
      org.opencontainers.image.version="${IMAGE_VERSION}" \
      org.opencontainers.image.revision="${VCS_REF}" \
      org.opencontainers.image.source="${SOURCE_URL}" \
      cg.image.maintainer="cybergavin" \
      cg.image.os="wolfi" \
      cg.image.variant="runtime" \
      build_label="${BUILD_LABEL}"

WORKDIR /app
USER 1001:1001

CMD ["/bin/sh"]

Layer 2: The Two-Tier Runtime Pattern (Lean Runtime vs. Dev Builder)

A common dilemma in platform engineering is whether to include package managers in runtime images. Developers need tools like pip and npm to install dependencies, yet production containers should never contain package managers because attackers can use them to download malicious binaries at runtime.

I solved this by publishing every language runtime (Python 3.12, Python 3.13, and Node.js 24) in two distinct variants:

1. Lean Runtime (wolfi): An ultra-minimal production image. For Python, it creates a clean virtual environment without pip using python -m venv --without-pip /opt/venv, and removes write access (chmod -R go-w /opt/venv). For Node.js, it installs only nodejs-24 and omits npm entirely.

2. Dev / Builder (wolfi-dev): A development image containing Python or Node along with complete tooling (pip, setuptools, wheel, or npm). The working directory and virtual environment paths are pre-configured with non-root ownership so multi-stage Dockerfiles can build dependencies without ever switching to root.

Both variants inherit directly FROM ghcr.io/cybergavin/wolfi:latest, completely eliminating duplicate user creation, directory permission scripts, and OS package updates across downstream Dockerfiles.

Here is the resulting Python 3.12 lean production Dockerfile:

ARG BASE_IMAGE=ghcr.io/cybergavin/wolfi:latest
FROM ${BASE_IMAGE}

USER 0:0

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONNOUSERSITE=1 \
    PATH="/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

RUN set -eu; \
    umask 022; \
    apk add --no-cache python-3.12; \
    python3.12 -m venv --without-pip /opt/venv; \
    chmod -R go-w /opt/venv

ARG IMAGE_VERSION=dev
ARG VCS_REF=local
ARG SOURCE_URL=https://github.com/cybergavin/container-images
ARG BUILD_LABEL=local

LABEL org.opencontainers.image.title="cg-python-runtime" \
      org.opencontainers.image.description="Non-root Python 3.12 runtime base" \
      org.opencontainers.image.version="${IMAGE_VERSION}" \
      org.opencontainers.image.revision="${VCS_REF}" \
      org.opencontainers.image.source="${SOURCE_URL}" \
      cg.image.runtime="python" \
      cg.image.runtime.version="3.12" \
      cg.image.variant="runtime" \
      build_label="${BUILD_LABEL}"

WORKDIR /app
USER 1001:1001

CMD ["/opt/venv/bin/python", "--version"]

How It Works in Practice: Multi-Stage Builds

With this architecture, downstream developers use standard multi-stage builds to produce minimal, secure application containers without needing root privileges at any stage:

# Stage 1: Build virtual environment using the dev image (includes pip)
FROM ghcr.io/cybergavin/python-wolfi:3.12-dev AS builder

WORKDIR /app
COPY requirements.txt .
RUN python -m venv /opt/venv && \
    /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

# Stage 2: Production runtime image (zero package managers, minimal attack surface)
FROM ghcr.io/cybergavin/python-wolfi:3.12

# Copy isolated virtualenv from builder
COPY --from=builder --chown=0:0 /opt/venv /opt/venv

# Copy application source
WORKDIR /app
COPY --chown=0:0 . .

USER 1001:1001
CMD ["/opt/venv/bin/python", "main.py"]

Node.js applications use the exact same model: build and bundle dependencies with ghcr.io/cybergavin/node-wolfi:24-dev, then copy the compiled assets into ghcr.io/cybergavin/node-wolfi:24 for production.

Separating CI Concerns: PR Validation vs. Release Pipelines

A common mistake in container CI/CD is using the same pipeline for pull requests and releases. Release pipelines require write secrets, push tags, sign digests, and publish attestations. Running release logic on PRs introduces security risks and pollutes container registries with test tags.

I split the pipelines into three specialized workflows:

1. Read-Only PR Validation (validate.yml)

Runs on pull requests targeting main with permissions: contents: read. It requires zero secrets, making it safe for open-source contributors and forks.

• Static Checks: Validates GitHub Actions YAML syntax, verifies that all third-party actions are pinned to 40-character commit SHAs, and validates agent skill headers.

• Smart Matrix Discovery: Inspects git diff origin/main...HEAD to build and test only the Dockerfiles modified in that specific PR.

• Local Functional Tests: Verifies non-root execution (1001:1001), upgrade prevention guardrails, and tooling presence/absence assertions (pip and npm must exist on dev variants and must be absent on production runtimes).

• Full-Spectrum Advisory CVE Scan: Runs Trivy with ignore-unfixed: false across all severity levels (UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL) to surface vulnerability findings directly in the PR summary.

2. BaseOS and Runtime Release Pipelines

The release workflows (baseos-build-publish.yml and runtime-build-publish.yml) trigger on monthly schedules (configured for Pacific Time) and manual dispatch:

• Local Build and Gate Verification: Builds the image locally and runs functional checks. Any High or Critical CVE immediately aborts the build.

• Dual-Registry Distribution: Images are pushed to GitHub Container Registry (GHCR) as the primary store, and automatically mirrored to Docker Hub using skopeo copy.

• Supply Chain Security: Images are signed using Sigstore Cosign with GitHub Actions OIDC (keyless signing). In-toto provenance attestations (mode=max) and SPDX SBOMs are attached directly to the images and saved as workflow artifacts.

[PR Opened] ---> validate.yml (Hadolint + Local Test Build + Smoke Tests + Trivy Advisory) ---> [Merge to main]

[Scheduled / Dispatch] ---> build-publish.yml ---> Trivy Gate (Block on High/Crit) ---> Push GHCR ---> Mirror Docker Hub ---> Cosign Sign & Verify ---> Upload SBOM

Making the Repository Agentic AI-Driven

A key architectural focus in this overhaul was optimizing the repository for autonomous AI pair programming (using Google Antigravity, OpenAI Codex, or similar agents).

Without guardrails, AI coding agents often make speculative changes, hallucinate package names, or break subtle security contracts. To prevent this, I implemented progressive context disclosure:

1. Lean AGENTS.md as an Architectural Index

Instead of stuffing every instruction into a single massive prompt, AGENTS.md acts as a high-signal index. It establishes operating principles, defines the hierarchy of truth, states immutable contracts (UID 1001, upgrade prevention, action SHA pinning), and routes the agent to modular skills.

2. Modular Skills (.agents/skills/)

I created focused, self-contained skills that agents load on demand:

• branch-commit-push-pr: Enforces branch safety (never push directly to main), explicit file staging, and Conventional Commit messages (feat, fix, docs, ci).

• container-image-change: Instructions for modifying existing Dockerfiles while preserving non-root contracts and running Hadolint.

• add-runtime-image: Step-by-step checklist for introducing new runtime families and registering them in the CI test matrix.

• container-release-workflow: Guidelines for editing GitHub Actions workflows, maintaining action SHA pins, and preserving release gates.

This approach keeps token consumption low, increases determinism, and ensures AI agents operate as disciplined pair programmers rather than unpredictable code generators.

Summary

Standardizing container images is a progressive journey. Starting with minimal BaseOS layers establishes your security baseline, but extending those contracts into purpose-built lean runtimes and dev builders gives developers images that are secure by default.

By combining dual-registry distribution, strict separation of CI concerns, keyless Cosign signing, and Agentic AI guidelines, you can operate a modern container supply chain with high confidence and minimal manual maintenance.

All of the Dockerfiles, GitHub Actions workflows, and agent skills are open source and available in the cybergavin/container-images repository. Feel free to explore, fork, and use them in your own infrastructure!

What is next? While Dockerfiles have served us well to enforce runtime guardrails, the next step is eliminating the Dockerfile entirely. In Part 3, we will explore declarative, daemonless distroless image generation.