<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[The Cybergavin Stack]]></title><description><![CDATA[Architecture, automation, and lessons from building real systems.]]></description><link>https://blog.cybergav.in</link><image><url>https://cdn.hashnode.com/uploads/logos/6aa73ede13ae672728e4d165/08be4c7b-5ea7-4433-81e3-da6b8b856092.png</url><title>The Cybergavin Stack</title><link>https://blog.cybergav.in</link></image><generator>RSS for Node</generator><lastBuildDate>Tue, 17 Nov 2026 04:25:43 GMT</lastBuildDate><atom:link href="https://blog.cybergav.in/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Standardizing Container Images Part 2: Hardened Runtimes and Agentic CI/CD]]></title><description><![CDATA[Learn how to expand minimal BaseOS images to two-tier language runtimes, secure multi-stage builds, dual-registry distribution, and Agentic AI workflows.]]></description><link>https://blog.cybergav.in/container-supply-chain-part-2-runtimes-and-agentic-ci</link><guid isPermaLink="true">https://blog.cybergav.in/container-supply-chain-part-2-runtimes-and-agentic-ci</guid><category><![CDATA[Devops]]></category><category><![CDATA[containers]]></category><category><![CDATA[Security]]></category><category><![CDATA[GitHub Actions]]></category><category><![CDATA[AI]]></category><dc:creator><![CDATA[Gavin Satur]]></dc:creator><pubDate>Thu, 15 Oct 2026 19:00:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/8f4be6bb-6b61-44ec-a684-acda425eb3a8.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A year ago, I published a blog post on Medium titled <a href="https://medium.com/@cybergavin/standardizing-base-os-container-images-for-platform-engineering-a-security-first-pipeline-1f338cca3db7">Standardizing Base OS Container Images for Platform Engineering: A Security-First Pipeline</a>. In that article, I explained why platform engineering teams must establish trusted base operating system images across minimal distributions like Alpine, AlmaLinux, Amazon Linux, and Wolfi, enforcing non-root execution and reproducible build pipelines.</p>
<p>Since then, our platform needs have grown. Providing hardened operating system layers is an essential first step, but application developers and AI engineers need ready-to-use language runtimes, such as Python and Node.js, that inherit those exact security guarantees.</p>
<p>To support this scale, I recently overhauled my open-source <a href="https://github.com/cybergavin/container-images">container-images repository</a>. I introduced a layered container supply chain, split runtimes into lean production and development builder variants, streamlined BaseOS images, added dual-registry mirroring, and structured the repository for deterministic Agentic AI pair programming.</p>
<p>This post is Part 2, walking through how my container supply chain evolved and how you can apply these patterns to your own platform.</p>
<h2>The Problem: The Runtime and Build Tool Trap</h2>
<p>Standardizing BaseOS images solves only half the problem. When application teams take a hardened base image and add language runtimes casually, several common anti-patterns emerge:</p>
<ul>
<li><p><strong>Privilege Escalation</strong>: Developers switch to <code>USER root</code> to install packages and forget to drop back to non-root execution.</p>
</li>
<li><p><strong>Leaked Build Tools in Production</strong>: Compilers, package managers (<code>pip</code>, <code>npm</code>), and development headers remain inside production images, unnecessarily inflating image size and expanding the attack surface.</p>
</li>
<li><p><strong>Inconsistent Base Tooling</strong>: Teams build Python or Node.js on disparate base distributions, creating fragmented patching and debugging workflows.</p>
</li>
<li><p><strong>Unpinned Upgrades</strong>: Running unpinned upgrade commands during container builds introduces unexpected library versions and CVE regressions.</p>
</li>
</ul>
<p>To solve these challenges, I established a strict two-tier architecture: a minimal BaseOS foundation and purpose-built language runtime variants.</p>
<h2>Layer 1: Overhauling the Wolfi BaseOS Foundation</h2>
<p>The container hierarchy begins with a hardened Wolfi operating system layer (<code>ghcr.io/cybergavin/wolfi:latest</code>). I recently streamlined this image to its absolute bare essentials:</p>
<ul>
<li><p><strong>Minimal Footprint</strong>: Removed auxiliary utilities like <code>curl</code>, <code>jq</code>, and <code>openssl</code>, retaining only <code>ca-certificates-bundle</code> on top of Chainguard's <code>wolfi-base</code>.</p>
</li>
<li><p><strong>Numeric Non-Root Execution</strong>: Standardized on numeric user <code>1001:1001</code> (<code>appuser:appgroup</code>).</p>
</li>
<li><p><strong>Tamper-Resistant Permissions</strong>: Applied <code>umask 022</code>, configured root-owned application directories (<code>/app</code> and <code>/home/appuser</code> at <code>0755</code>), and restricted state directories (<code>/var/lib/app</code> and <code>/var/cache/app</code> at <code>0750</code>).</p>
</li>
<li><p><strong>Natural Upgrade Prevention</strong>: By executing as non-root user <code>1001:1001</code>, package manager upgrades (<code>apk upgrade</code>) are naturally blocked by filesystem permission boundaries without requiring custom wrapper scripts.</p>
</li>
<li><p><strong>Standardized OCI Metadata</strong>: Standard OCI labels (<code>org.opencontainers.image.*</code>) track versions, revisions, source URLs, and build timestamps.</p>
</li>
</ul>
<p>Here is how concise and hardened the Wolfi BaseOS Dockerfile is:</p>
<pre><code class="language-dockerfile">ARG BASE_IMAGE=cgr.dev/chainguard/wolfi-base:latest
FROM ${BASE_IMAGE}

USER 0:0

ENV HOME=/home/appuser \
    XDG_CACHE_HOME=/var/cache/app \
    PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

RUN set -eu; \
    umask 022; \
    apk upgrade --no-cache; \
    apk add --no-cache ca-certificates-bundle; \
    addgroup -g 1001 appgroup; \
    adduser -D -H -u 1001 -G appgroup \
        -h /home/appuser -s /bin/sh appuser; \
    mkdir -p /app /home/appuser /var/lib/app /var/cache/app; \
    chown 0:0 /app /home/appuser; \
    chmod 0755 /app /home/appuser; \
    chown 1001:1001 /var/lib/app /var/cache/app; \
    chmod 0750 /var/lib/app /var/cache/app

ARG IMAGE_VERSION=dev
ARG VCS_REF=local
ARG BUILD_LABEL=local
ARG SOURCE_URL=https://github.com/cybergavin/container-images

LABEL org.opencontainers.image.title="cg-wolfi-runtime" \
      org.opencontainers.image.description="Minimal Wolfi OS base for downstream language runtimes" \
      org.opencontainers.image.version="${IMAGE_VERSION}" \
      org.opencontainers.image.revision="${VCS_REF}" \
      org.opencontainers.image.source="${SOURCE_URL}" \
      cg.image.maintainer="cybergavin" \
      cg.image.os="wolfi" \
      cg.image.variant="runtime" \
      build_label="${BUILD_LABEL}"

WORKDIR /app
USER 1001:1001

CMD ["/bin/sh"]
</code></pre>
<h2>Layer 2: The Two-Tier Runtime Pattern (Lean Runtime vs. Dev Builder)</h2>
<p>A common dilemma in platform engineering is whether to include package managers in runtime images. Developers need tools like <code>pip</code> and <code>npm</code> to install dependencies, yet production containers should never contain package managers because attackers can use them to download malicious binaries at runtime.</p>
<p>I solved this by publishing every language runtime (Python 3.12, Python 3.13, and Node.js 24) in two distinct variants:</p>
<ol>
<li><p><strong>Lean Runtime (</strong><code>wolfi</code><strong>)</strong>: An ultra-minimal production image. For Python, it creates a clean virtual environment without <code>pip</code> using <code>python -m venv --without-pip /opt/venv</code>, and removes write access (<code>chmod -R go-w /opt/venv</code>). For Node.js, it installs only <code>nodejs-24</code> and omits <code>npm</code> entirely.</p>
</li>
<li><p><strong>Dev / Builder (</strong><code>wolfi-dev</code><strong>)</strong>: A development image containing Python or Node along with complete tooling (<code>pip</code>, <code>setuptools</code>, <code>wheel</code>, or <code>npm</code>). The working directory and virtual environment paths are pre-configured with non-root ownership so multi-stage Dockerfiles can build dependencies without ever switching to root.</p>
</li>
</ol>
<p>Both variants inherit directly <code>FROM ghcr.io/cybergavin/wolfi:latest</code>, completely eliminating duplicate user creation, directory permission scripts, and OS package updates across downstream Dockerfiles.</p>
<p>Here is the resulting Python 3.12 lean production Dockerfile:</p>
<pre><code class="language-dockerfile">ARG BASE_IMAGE=ghcr.io/cybergavin/wolfi:latest
FROM ${BASE_IMAGE}

USER 0:0

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONNOUSERSITE=1 \
    PATH="/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

RUN set -eu; \
    umask 022; \
    apk add --no-cache python-3.12; \
    python3.12 -m venv --without-pip /opt/venv; \
    chmod -R go-w /opt/venv

ARG IMAGE_VERSION=dev
ARG VCS_REF=local
ARG SOURCE_URL=https://github.com/cybergavin/container-images
ARG BUILD_LABEL=local

LABEL org.opencontainers.image.title="cg-python-runtime" \
      org.opencontainers.image.description="Non-root Python 3.12 runtime base" \
      org.opencontainers.image.version="${IMAGE_VERSION}" \
      org.opencontainers.image.revision="${VCS_REF}" \
      org.opencontainers.image.source="${SOURCE_URL}" \
      cg.image.runtime="python" \
      cg.image.runtime.version="3.12" \
      cg.image.variant="runtime" \
      build_label="${BUILD_LABEL}"

WORKDIR /app
USER 1001:1001

CMD ["/opt/venv/bin/python", "--version"]
</code></pre>
<h2>How It Works in Practice: Multi-Stage Builds</h2>
<p>With this architecture, downstream developers use standard multi-stage builds to produce minimal, secure application containers without needing root privileges at any stage:</p>
<pre><code class="language-dockerfile"># Stage 1: Build virtual environment using the dev image (includes pip)
FROM ghcr.io/cybergavin/python-wolfi:3.12-dev AS builder

WORKDIR /app
COPY requirements.txt .
RUN python -m venv /opt/venv &amp;&amp; \
    /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

# Stage 2: Production runtime image (zero package managers, minimal attack surface)
FROM ghcr.io/cybergavin/python-wolfi:3.12

# Copy isolated virtualenv from builder
COPY --from=builder --chown=0:0 /opt/venv /opt/venv

# Copy application source
WORKDIR /app
COPY --chown=0:0 . .

USER 1001:1001
CMD ["/opt/venv/bin/python", "main.py"]
</code></pre>
<p>Node.js applications use the exact same model: build and bundle dependencies with <code>ghcr.io/cybergavin/node-wolfi:24-dev</code>, then copy the compiled assets into <code>ghcr.io/cybergavin/node-wolfi:24</code> for production.</p>
<h2>Separating CI Concerns: PR Validation vs. Release Pipelines</h2>
<p>A common mistake in container CI/CD is using the same pipeline for pull requests and releases. Release pipelines require write secrets, push tags, sign digests, and publish attestations. Running release logic on PRs introduces security risks and pollutes container registries with test tags.</p>
<p>I split the pipelines into three specialized workflows:</p>
<h3>1. Read-Only PR Validation (<code>validate.yml</code>)</h3>
<p>Runs on pull requests targeting <code>main</code> with <code>permissions: contents: read</code>. It requires zero secrets, making it safe for open-source contributors and forks.</p>
<ul>
<li><p><strong>Static Checks</strong>: Validates GitHub Actions YAML syntax, verifies that all third-party actions are pinned to 40-character commit SHAs, and validates agent skill headers.</p>
</li>
<li><p><strong>Smart Matrix Discovery</strong>: Inspects <code>git diff origin/main...HEAD</code> to build and test only the Dockerfiles modified in that specific PR.</p>
</li>
<li><p><strong>Local Functional Tests</strong>: Verifies non-root execution (<code>1001:1001</code>), upgrade prevention guardrails, and tooling presence/absence assertions (<code>pip</code> and <code>npm</code> must exist on dev variants and must be absent on production runtimes).</p>
</li>
<li><p><strong>Full-Spectrum Advisory CVE Scan</strong>: Runs Trivy with <code>ignore-unfixed: false</code> across all severity levels (UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL) to surface vulnerability findings directly in the PR summary.</p>
</li>
</ul>
<h3>2. BaseOS and Runtime Release Pipelines</h3>
<p>The release workflows (<code>baseos-build-publish.yml</code> and <code>runtime-build-publish.yml</code>) trigger on monthly schedules (configured for Pacific Time) and manual dispatch:</p>
<ul>
<li><p><strong>Local Build and Gate Verification</strong>: Builds the image locally and runs functional checks. Any High or Critical CVE immediately aborts the build.</p>
</li>
<li><p><strong>Dual-Registry Distribution</strong>: Images are pushed to GitHub Container Registry (GHCR) as the primary store, and automatically mirrored to Docker Hub using <code>skopeo copy</code>.</p>
</li>
<li><p><strong>Supply Chain Security</strong>: Images are signed using Sigstore Cosign with GitHub Actions OIDC (keyless signing). In-toto provenance attestations (<code>mode=max</code>) and SPDX SBOMs are attached directly to the images and saved as workflow artifacts.</p>
</li>
</ul>
<pre><code class="language-text">[PR Opened] ---&gt; validate.yml (Hadolint + Local Test Build + Smoke Tests + Trivy Advisory) ---&gt; [Merge to main]

[Scheduled / Dispatch] ---&gt; build-publish.yml ---&gt; Trivy Gate (Block on High/Crit) ---&gt; Push GHCR ---&gt; Mirror Docker Hub ---&gt; Cosign Sign &amp; Verify ---&gt; Upload SBOM
</code></pre>
<h2>Making the Repository Agentic AI-Driven</h2>
<p>A key architectural focus in this overhaul was optimizing the repository for autonomous AI pair programming (using Google Antigravity, OpenAI Codex, or similar agents).</p>
<p>Without guardrails, AI coding agents often make speculative changes, hallucinate package names, or break subtle security contracts. To prevent this, I implemented progressive context disclosure:</p>
<h3>1. Lean <code>AGENTS.md</code> as an Architectural Index</h3>
<p>Instead of stuffing every instruction into a single massive prompt, <code>AGENTS.md</code> acts as a high-signal index. It establishes operating principles, defines the hierarchy of truth, states immutable contracts (UID 1001, upgrade prevention, action SHA pinning), and routes the agent to modular skills.</p>
<h3>2. Modular Skills (<code>.agents/skills/</code>)</h3>
<p>I created focused, self-contained skills that agents load on demand:</p>
<ul>
<li><p><code>branch-commit-push-pr</code>: Enforces branch safety (never push directly to main), explicit file staging, and Conventional Commit messages (<code>feat</code>, <code>fix</code>, <code>docs</code>, <code>ci</code>).</p>
</li>
<li><p><code>container-image-change</code>: Instructions for modifying existing Dockerfiles while preserving non-root contracts and running Hadolint.</p>
</li>
<li><p><code>add-runtime-image</code>: Step-by-step checklist for introducing new runtime families and registering them in the CI test matrix.</p>
</li>
<li><p><code>container-release-workflow</code>: Guidelines for editing GitHub Actions workflows, maintaining action SHA pins, and preserving release gates.</p>
</li>
</ul>
<p>This approach keeps token consumption low, increases determinism, and ensures AI agents operate as disciplined pair programmers rather than unpredictable code generators.</p>
<h2>Summary</h2>
<p>Standardizing container images is a progressive journey. Starting with minimal BaseOS layers establishes your security baseline, but extending those contracts into purpose-built lean runtimes and dev builders gives developers images that are secure by default.</p>
<p>By combining dual-registry distribution, strict separation of CI concerns, keyless Cosign signing, and Agentic AI guidelines, you can operate a modern container supply chain with high confidence and minimal manual maintenance.</p>
<p>All of the Dockerfiles, GitHub Actions workflows, and agent skills are open source and available in the <a href="https://github.com/cybergavin/container-images">cybergavin/container-images repository</a>. Feel free to explore, fork, and use them in your own infrastructure!</p>
<blockquote>
<p><strong>What is next?</strong> While Dockerfiles have served us well to enforce runtime guardrails, the next step is eliminating the Dockerfile entirely. In Part 3, we will explore declarative, daemonless distroless image generation.</p>
</blockquote>
]]></content:encoded></item><item><title><![CDATA[When "Healthy" Isn't Healthy: Debugging a 40-Second Cloudflare-to-AWS NLB Delay]]></title><description><![CDATA[How we debugged a 40-second Cloudflare-to-AWS NLB latency delay caused by asymmetric routing, using ARC zonal shift, curl, and AI-assisted troubleshooting.]]></description><link>https://blog.cybergav.in/when-healthy-isnt-healthy-debugging-cloudflare-aws-nlb-latency</link><guid isPermaLink="true">https://blog.cybergav.in/when-healthy-isnt-healthy-debugging-cloudflare-aws-nlb-latency</guid><category><![CDATA[AWS]]></category><category><![CDATA[cloudflare]]></category><category><![CDATA[networking]]></category><category><![CDATA[Devops]]></category><category><![CDATA[troubleshooting]]></category><dc:creator><![CDATA[Gavin Satur]]></dc:creator><pubDate>Wed, 14 Oct 2026 18:00:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/46a634b0-b771-401e-b2c5-cfa4bfc3aab1.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<blockquote>
<p><em>Note: All domains, IP addresses, CIDRs, and resource names in this article are sanitized. The technical relationships reflect the actual investigation.</em></p>
</blockquote>
<div style="text-align:justify">
We had a test API behind Cloudflare, an internet-facing AWS Network Load Balancer (NLB), and an EKS ingress controller. Reports started coming in of agonizing latency: requests were taking almost exactly 40 seconds to return a response.
</div><div style="text-align:justify">
Curiously, the problem was intermittent across the team. For some engineers, requests succeeded in milliseconds without issue. For others, including my own tests, every single request consistently stalled for 40 seconds. That consistency on affected sessions was far more interesting than a hard failure. DNS was quick. The TCP connection was quick. TLS was quick. Then the request waited.
</div><pre><code class="language-text">dns=0.03 tcp=0.05 tls=0.12 ttfb=40.37 total=40.37
dns=0.03 tcp=0.06 tls=0.14 ttfb=40.71 total=40.71
dns=0.11 tcp=0.14 tls=0.19 ttfb=40.72 total=40.72
</code></pre>
<div style="text-align:justify">
The response was an HTTP <code>404</code>, which was expected for the root path we were testing. The problem was not the response code. The problem was that it took 40 seconds to arrive.
</div><div style="text-align:justify">
The intended path looked simple at first glance:
</div><pre><code class="language-text">Client → Cloudflare Proxy → Hub Ingress Account (NLB → GWLB ⇄ PAN-OS)
  → AWS Transit Gateway (TGW) → Spoke App Account (EKS Ingress)
</code></pre>
<div style="text-align:justify">
In enterprise cloud environments, this follows the standard AWS centralized ingress inspection architecture:
</div><ul>
<li><strong>Centralized Ingress/Security Hub Account &amp; VPC:</strong> Houses the internet-facing NLB, an AWS Gateway Load Balancer (GWLB), and a fleet of inline Next-Generation Firewalls (Palo Alto PAN-OS VM-Series). Inbound client traffic from Cloudflare hits the NLB, is routed transparently through GWLB to PAN-OS for deep packet inspection, and forwards to the Transit Gateway (TGW).</li>
<li><strong>AWS Transit Gateway (TGW):</strong> Serves as the high-speed regional network hub interconnecting VPCs.</li>
<li><strong>Application Spoke Account &amp; VPC:</strong> Hosts the EKS cluster and ingress controllers running the workloads.</li>
<li><strong>The Symmetric Return Flow:</strong> Responses from EKS must travel back across the TGW into the Hub VPC, re-enter the GWLB / PAN-OS firewall to match existing stateful sessions, return to the NLB, and head back out through Cloudflare to the client.</li>
</ul>
<div style="text-align:justify">
As usual, this enterprise architecture concealed most of the places where traffic asymmetry could strike.
</div><h2>AI found the first important clue</h2>
<div style="text-align:justify">
A member of my team did something I want to encourage. They took our end-to-end ingress path architecture, gathered PAN-OS firewall traffic logs and AWS routing details from the network team, and fed the combined context into an AI model.
</div><div style="text-align:justify">
They were still ramping up on the networking concepts involved: CIDRs, longest-prefix matching, Transit Gateway route tables, attachments and stateful firewall paths. AI helped them connect the dots between the intended ingress topology and the raw telemetry in the firewall logs, identifying patterns that would otherwise have taken much longer to interpret.
</div><div style="text-align:justify">
The analysis highlighted two overlapping routes in a Transit Gateway route table:
</div><pre><code class="language-text">10.42.48.0/21   → firewall VPC attachment
10.42.48.160/27 → services VPC attachment directly
</code></pre>
<div style="text-align:justify">
The test NLB had two private frontend addresses:
</div><pre><code class="language-text">10.42.48.180
10.42.49.175
</code></pre>
<div style="text-align:justify">
The first address falls inside <code>10.42.48.160/27</code>. The second does not. Both fall within the broader <code>10.42.48.0/21</code>.
</div><div style="text-align:justify">
Because AWS routing uses longest-prefix matching, traffic involving <code>10.42.48.180</code> would select the <code>/27</code>. Traffic involving <code>10.42.49.175</code> would select the <code>/21</code>.
</div><div style="text-align:justify">
The AI analysis proposed that this created an asymmetric path around our stateful firewall. One direction traversed the firewall, while the return traffic for the <code>.48</code> NLB address matched the more-specific route and bypassed it. The firewall logs appeared consistent with that explanation: incomplete sessions aged out on one path, while successful sessions on the other path closed normally.
</div><div style="text-align:justify">
It was a very good find. It was also presented with more certainty than the evidence supported. The response declared that it had found the exact problem and even supplied an AWS CLI command to replace the route.
</div><div style="text-align:justify">
At that point, we had a strong hypothesis. We did not yet have proof.
</div><div style="text-align:justify">
I took the analysis further with two goals:
</div><ol>
<li>Confirm that the route actually explained the observed 40-second behavior.</li>
<li>Look beyond the immediate route correction and determine how the application should survive a similar zonal failure in the future.</li>
</ol>
<div style="text-align:justify">
This was not about proving the AI analysis wrong. It was about separating a valuable discovery from a verified conclusion.
</div><h2>First, establish what Cloudflare was doing</h2>
<div style="text-align:justify">
The response headers confirmed that the test hostname was proxied through Cloudflare:
</div><pre><code class="language-text">server: cloudflare
cf-cache-status: DYNAMIC
cf-ray: &lt;identifier&gt;
</code></pre>
<div style="text-align:justify">
The <code>CF-Ray</code> header established that the request had traversed Cloudflare. <code>CF-Cache-Status: DYNAMIC</code> told us that Cloudflare was forwarding the request to an origin rather than serving a cached response.
</div><div style="text-align:justify">
This mattered because “Cloudflare is slow” was still far too broad a description. Cloudflare could be waiting for an origin connection, retrying another origin address, or simply relaying a slow application response.
</div><div style="text-align:justify">
The timing strongly suggested a timeout-and-retry sequence. While client-side <code>curl</code> cannot reveal Cloudflare’s internal edge decisions without enterprise logging, Cloudflare’s proxy behavior provides a compelling explanation:
</div><ul>
<li>When connecting to an origin, Cloudflare initiates a TCP handshake. If it receives no <code>SYN+ACK</code>, it waits out an origin connection timeout of roughly 19 seconds while attempting retransmissions.</li>
<li>If that first attempt fails against an unresponsive origin IP, Cloudflare retries an alternate origin IP.</li>
<li>Two stalled connection attempts at ~19 seconds each equal ~38 seconds of silence. When the third attempt hits the working zonal NLB IP, the connection completes in milliseconds, yielding the consistent ~40.3-second TTFB we observed.</li>
</ul>
<div style="text-align:justify">
We recorded this retry behavior as a highly consistent and plausible model, keeping it clearly labeled as an inference from documented proxy behavior rather than origin-side packet capture.
</div><div style="text-align:justify">
There was also an early distraction involving a production hostname that responded quickly. We later confirmed that it pointed to a different group of servers and had not yet been connected to the production EKS environment. It was not a valid comparison, so we removed it from the evidence. This incident occurred only on the configured test path.
</div><h2>One NLB hostname concealed two different outcomes</h2>
<div style="text-align:justify">
The generic NLB hostname resolved to two public addresses, one for each enabled Availability Zone:
</div><pre><code class="language-text">198.51.100.20
203.0.113.20
</code></pre>
<div style="text-align:justify">
Testing the hostname alone would always allow DNS and the upstream client to choose an address for us. We needed to test each NLB frontend independently while preserving the real TLS server name and HTTP Host header.
</div><div style="text-align:justify">
The useful <code>curl</code> feature for that is <code>--resolve</code>:
</div><pre><code class="language-bash">curl --resolve api-test.example.edu:443:198.51.100.20 \
  -o /dev/null -sS \
  -w 'code=%{http_code} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  https://api-test.example.edu/
</code></pre>
<div style="text-align:justify">
The URL remains <code>https://api-test.example.edu</code>, so TLS and ingress host routing still receive the correct hostname. Only the destination address is pinned.
</div><div style="text-align:justify">
Our workstations could not reach the public or private NLB addresses directly because of network controls. Rather than infer what would happen, I deployed a small EC2 test instance in an appropriate AWS network and repeated the test against each private NLB address.
</div><div style="text-align:justify">
The result was unambiguous:
</div><pre><code class="language-text">10.42.49.175 → HTTP 404 in approximately 35 milliseconds
10.42.48.180 → no response before the client timeout
</code></pre>
<div style="text-align:justify">
Again, the <code>404</code> was not a failure for this test. It showed that TCP, TLS and the ingress path completed quickly enough to return an expected application-layer response.
</div><div style="text-align:justify">
The other NLB address did not complete the same transaction.
</div><div style="text-align:justify">
We could now replace the vague statement “the test API is slow” with something much more useful:
</div><blockquote>
<p>Connections through one zonal NLB frontend worked normally. Connections through the other zonal frontend did not.</p>
</blockquote>
<h2>The route explained the difference</h2>
<div style="text-align:justify">
The isolated result matched the route analysis precisely:
</div><ul>
<li><code>10.42.48.180</code> was inside the more-specific <code>10.42.48.160/27</code> route.</li>
<li><code>10.42.49.175</code> used the broader <code>10.42.48.0/21</code> route through the firewall attachment.</li>
<li>The firewall logs showed incomplete sessions for the failing path and normally closed sessions for the working path.</li>
</ul>
<div style="text-align:justify">
The working hypothesis now looked like this:
</div><p><img src="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/1082eb15-c214-4fe1-8d07-55e976df8749.png" alt="Asymmetric Routing Diagram" /></p>
<details>
<summary>View raw Mermaid text diagram definition</summary><pre><code class="language-mermaid">flowchart TD
    subgraph Edge ["Public Edge"]
        CF[Cloudflare Proxy]
    end

    subgraph Hub ["Hub Ingress Account (VPC)"]
        NLB_DNS[Generic NLB DNS]
        NLB_BAD[AZ-b NLB Frontend&lt;br/&gt;10.42.48.180]
        NLB_GOOD[Other-AZ NLB Frontend&lt;br/&gt;10.42.49.175]
        GWLB[AWS GWLB]
        FW[PAN-OS Firewalls]
    end

    subgraph Transit ["Interconnect"]
        TGW[AWS Transit Gateway]
        R27[TGW Route: /27 Override&lt;br/&gt;Direct Spoke Attachment]
        R21[TGW Route: /21&lt;br/&gt;Firewall Attachment]
    end

    subgraph Spoke ["Spoke App Account (VPC)"]
        EKS[AWS EKS Ingress]
    end

    CF --&gt; NLB_DNS
    NLB_DNS --&gt; NLB_BAD
    NLB_DNS --&gt; NLB_GOOD

    NLB_GOOD --&gt; GWLB &lt;--&gt; FW
    GWLB --&gt; TGW --&gt; R21 --&gt; EKS
    EKS -.-&gt;|Symmetric Return| R21 -.-&gt; GWLB -.-&gt; NLB_GOOD -.-&gt; CF
    
    NLB_BAD --&gt; GWLB &lt;--&gt; FW
    GWLB --&gt; TGW --&gt; R27 --&gt; EKS
    EKS -.-&gt;|Bypasses Firewall via /27| R27 -.-&gt;|Asymmetric drop / timeout| SLOW[~19s SYN Timeout &amp; Retry]
</code></pre>
</details><div style="text-align:justify">
The remaining question was whether removing the affected Availability Zone from normal NLB routing would remove the 40-second delay without changing anything else.
</div><h2>Why the healthy target status did not protect us</h2>
<div style="text-align:justify">
The NLB target group reported its targets as healthy throughout the incident.
</div><div style="text-align:justify">
That was not evidence that our per-IP test was wrong. It demonstrated that the target health check and the real client path were answering different questions.
</div><div style="text-align:justify">
NLB target health checks determine whether registered backend targets respond to the configured health-check protocol, port and path. They do not provide a general guarantee that every real client flow through every NLB frontend, Transit Gateway route and stateful inspection path is symmetric and usable.
</div><div style="text-align:justify">
The backend targets could pass their configured checks while a connection through one zonal NLB address still failed because of the route taken by the complete flow.
</div><div style="text-align:justify">
The green indicator was correct within its scope. We had expected it to certify more of the path than it actually tested.
</div><div style="text-align:justify">
This became one of the most useful lessons from the investigation:
</div><blockquote>
<p>A healthy component does not imply a healthy end-to-end path.</p>
</blockquote>
<div style="text-align:justify">
A healthy pod does not prove the ingress path is healthy. A healthy ingress target does not prove every NLB frontend is usable. A healthy NLB target group does not prove that a return path through a stateful firewall is symmetric.
</div><h2>Using ARC zonal shift as a controlled experiment</h2>
<div style="text-align:justify">
AWS Application Recovery Controller (ARC) zonal shift is intended to move traffic away from an impaired Availability Zone. In this case, it also gave us an unusually clean diagnostic experiment.
</div><div style="text-align:justify">
We enabled ARC zonal shift integration only for the test NLB. Because cross-zone load balancing was enabled, AWS required the attached target groups to disable immediate connection termination when targets become unhealthy. Existing connections could then close naturally while new traffic shifted away from the selected zone.
</div><div style="text-align:justify">
We started a short customer-initiated zonal shift away from the Availability Zone containing <code>10.42.48.180</code>.
</div><div style="text-align:justify">
Before the shift, the generic NLB hostname returned both public addresses:
</div><pre><code class="language-text">198.51.100.20
203.0.113.20
</code></pre>
<div style="text-align:justify">
After the shift, the affected zonal address disappeared from the generic NLB DNS response:
</div><pre><code class="language-text">198.51.100.20
</code></pre>
<div style="text-align:justify">
Then we repeated the original request through Cloudflare:
</div><pre><code class="language-text">ttfb=0.35 total=0.35
ttfb=0.32 total=0.33
ttfb=0.31 total=0.31
ttfb=0.16 total=0.16
ttfb=0.29 total=0.29
</code></pre>
<div style="text-align:justify">
Nothing else had changed. We had not modified the application, EKS ingress, firewall policy, Cloudflare proxy configuration or Transit Gateway routes. We had changed only whether the impaired NLB Availability Zone was offered for new connections.
</div><div style="text-align:justify">
The result was immediate and repeatable:
</div><ul>
<li>With both zones active in DNS, requests averaged ~40.5 seconds.</li>
<li>With ARC zonal shift redirecting traffic away from the impaired zone, requests completed in 0.16–0.35 seconds.</li>
</ul>
<div style="text-align:justify">
To close the loop, we performed a full A/B reproduction:
</div><ol>
<li><strong>Cancel the zonal shift:</strong> We waited for the NLB’s ~60-second DNS TTL to expire. The generic NLB hostname resumed advertising both public IPs.</li>
<li><strong>Observe regression:</strong> Direct requests through Cloudflare immediately reverted to the ~40-second delay. Meanwhile, a pinned <code>curl --resolve</code> against the working zonal NLB IP still returned responses in ~35ms, proving the application itself was unaffected and only the specific zonal transit was broken.</li>
</ol>
<table>
<thead>
<tr>
<th>Test state</th>
<th>NLB addresses advertised</th>
<th>Observed result</th>
</tr>
</thead>
<tbody><tr>
<td>Broken route + both NLB zones available</td>
<td>Both IPs advertised</td>
<td>~40.3–40.7 seconds</td>
</tr>
<tr>
<td>Broken route + ARC zonal shift away from impaired zone</td>
<td>Healthy zonal IP only</td>
<td>0.16–0.35 seconds</td>
</tr>
<tr>
<td>Broken route + ARC shift cancelled (A/B reproduction)</td>
<td>Both IPs advertised</td>
<td>~40.5 seconds</td>
</tr>
<tr>
<td>Corrected TGW route + both zones available</td>
<td>Both IPs advertised</td>
<td>0.18–0.32 seconds</td>
</tr>
</tbody></table>
<div style="text-align:justify">
The zonal shift proved beyond doubt that the latency was tied exclusively to the affected NLB zonal path. The direct-address tests had already identified which path failed. The route table and PAN-OS firewall sessions (which showed aged-out, incomplete sessions for the <code>.48</code> subnet) explained why.
</div><div style="text-align:justify">
When the network team corrected the Transit Gateway route table by eliminating the overriding <code>/27</code> prefix so traffic flowed symmetrically through the firewall attachment, we reran the isolation tests. Both zonal NLB IPs responded in milliseconds, and end-to-end client requests remained consistently fast. The causal chain was complete.
</div><h2>Fixing the route is necessary, but it is not enough</h2>
<div style="text-align:justify">
The route must be corrected. Depending on its original intent, that may mean redirecting the <code>/27</code> through the firewall attachment or removing it so that the broader <code>/21</code> applies.
</div><div style="text-align:justify">
That decision belongs to the network owners. A syntactically valid route change is not automatically the right architectural change, which is why copying an AI-generated <code>replace-transit-gateway-route</code> command into AWS would have been premature.
</div><div style="text-align:justify">
But I did not want to stop at fixing the route.
</div><div style="text-align:justify">
The incident exposed a failure mode that could happen again for a different reason. A future problem might involve AWS infrastructure, a subnet route, a Network ACL, a firewall path or another zonal dependency. If one NLB frontend becomes unusable while backend health checks remain green, the application could again suffer long delays without a complete outage.
</div><div style="text-align:justify">
We therefore looked at two additional controls.
</div><h3>ARC zonal shift as the AWS-side recovery control</h3>
<div style="text-align:justify">
ARC now gives us a tested way to withdraw an impaired NLB Availability Zone. An operator can start a short zonal shift, observe the result and cancel it when the underlying problem is corrected.
</div><div style="text-align:justify">
That is valuable, but it still requires detection and an operational decision. We need monitoring that tests each externally reachable zonal path independently, not merely the NLB hostname or backend target group.
</div><h3>Architectural Recommendation: Cloudflare Load Balancing with separate zonal endpoints</h3>
<div style="text-align:justify">
The ordinary Cloudflare-proxied DNS record used the generic public NLB hostname. That hostname concealed both NLB addresses behind one origin name.
</div><div style="text-align:justify">
Configuring the generic NLB hostname as a single origin endpoint does not provide independent health state for each zonal frontend.
</div><div style="text-align:justify">
AWS provides a zonal DNS name for each NLB frontend:
</div><pre><code class="language-text">us-west-2a.example-nlb.elb.us-west-2.amazonaws.com
us-west-2b.example-nlb.elb.us-west-2.amazonaws.com
</code></pre>
<div style="text-align:justify">
Alternatively, an internet-facing NLB's static public IP address for each zone can be configured directly.
</div><div style="text-align:justify">
In an enterprise setup with Cloudflare Load Balancing (a paid add-on), an ideal pool structure would contain two independent origin endpoints, one for each NLB Availability Zone:
</div><ul>
<li><strong>Monitored Endpoint 1:</strong> <code>us-west-2a</code> zonal NLB address / DNS</li>
<li><strong>Monitored Endpoint 2:</strong> <code>us-west-2b</code> zonal NLB address / DNS</li>
<li><strong>Health Check Monitor:</strong> HTTPS / port 443 with Host header <code>api-test.example.edu</code> targeting <code>/healthz</code></li>
</ul>
<div style="text-align:justify">
In this model, when a single zonal transit path fails, Cloudflare detects the unhealthy zonal endpoint, isolates it, and routes all traffic to the healthy zone automatically, preventing proxy retry delays without requiring manual intervention.
</div><h3>The Low-Cost Alternative: External Synthetic Probing</h3>
<div style="text-align:justify">
If Cloudflare Load Balancing is not licensed or available in your tier, you cannot rely on Cloudflare to steer traffic away from a single degraded origin IP automatically.
</div><div style="text-align:justify">
Instead, the operational pattern is <b>external synthetic monitoring coupled with manual or automated ARC zonal shift</b>:
</div><ol>
<li>Run continuous synthetic health checks (via Datadog, Prometheus Blackbox, CloudWatch Synthetics, or custom probes) directly against each zonal NLB IP using the <code>curl --resolve</code> technique.</li>
<li>If probes against a specific zonal IP consistently fail or timeout while backend target groups report healthy, raise a high-severity zonal degradation alert.</li>
<li>Use the AWS CLI or Console to initiate an <strong>ARC Zonal Shift</strong> to immediately evacuate the impaired AZ until the network team addresses the root cause.</li>
</ol>
<div style="text-align:justify">
The three controls solve different parts of the problem:
</div><table>
<thead>
<tr>
<th>Control</th>
<th>Type</th>
<th>What it provides</th>
</tr>
</thead>
<tbody><tr>
<td>Correct TGW routing</td>
<td>Prevention / Root Cause</td>
<td>Ensures symmetric, clean routing across all paths</td>
</tr>
<tr>
<td>Per-zone origin health checks</td>
<td>Detection</td>
<td>Catches gray failures hidden by generic NLB hostnames</td>
</tr>
<tr>
<td>ARC zonal shift</td>
<td>Mitigation / Recovery</td>
<td>Allows operators to evacuate an impaired AZ in seconds</td>
</tr>
</tbody></table>
<h2>The role AI actually played</h2>
<div style="text-align:justify">
This was not a story in which AI produced a bad answer and experienced engineers rescued the situation. That would be both inaccurate and unhelpful.
</div><div style="text-align:justify">
AI found the most important technical clue early. When provided with our intended ingress architecture alongside PAN-OS traffic logs, it connected the dots: identifying the more-specific route, correlating it with the failing zonal NLB address, and spotting the incomplete/aged-out firewall sessions. That contributed directly to resolving the incident.
</div><div style="text-align:justify">
The human work was different:
</div><ul>
<li>Recognizing that the initial evidence supported a hypothesis rather than certainty.</li>
<li>Obtaining missing context from the relevant teams.</li>
<li>Discarding an invalid production comparison.</li>
<li>Building an EC2-based isolation test when workstation access was blocked.</li>
<li>Testing each NLB frontend independently.</li>
<li>Using a reversible zonal shift to change one variable at a time.</li>
<li>Looking beyond the immediate fix to the application's future resilience.</li>
</ul>
<div style="text-align:justify">
Those are not reasons to use AI less. They are the practices that make AI useful in operational engineering.
</div><div style="text-align:justify">
The lesson for my team is not “do not trust AI.” It is more precise:
</div><blockquote>
<p>Use AI aggressively to find patterns, interpret unfamiliar material and generate hypotheses. Do not quietly convert its confidence into your own certainty. Own the evidence, the recommendation and the consequences of any change.</p>
</blockquote>
<div style="text-align:justify">
An AI-generated command may be technically correct and still be the wrong action. An AI-generated diagnosis may identify a real defect without fully explaining the incident. The engineer communicating the result must know which parts were observed, which were inferred and which still need to be tested.
</div><h2>Incident Resolution &amp; Key Takeaways</h2>
<div style="text-align:justify">
The complete test sequence definitively proved the root cause and verified the permanent fix:
</div><ol>
<li><strong>A/B Confirmation:</strong> Reverting the ARC zonal shift brought back the ~40.5s delay on generic traffic while the isolated healthy IP remained sub-50ms.</li>
<li><strong>Permanent Route Remediation:</strong> The network team removed the overriding <code>/27</code> Transit Gateway route, ensuring all traffic to the services VPC traversed the stateful firewall attachment symmetrically.</li>
<li><strong>Post-Fix Verification:</strong> Both <code>curl --resolve</code> zonal tests and repeated end-to-end client requests through Cloudflare returned in ~200–300ms. PAN-OS session monitors confirmed clean bidirectional sessions with zero aged-out drops.</li>
<li><strong>Resilience Established:</strong> The team verified ARC zonal shift integration as an active runbook control for any future single-zone impairment.</li>
</ol>
<h2>Conclusion</h2>
<div style="text-align:justify">
The visible symptom was a Cloudflare-proxied API that consistently took about 40 seconds to return a response. The application did not fail. The NLB targets remained healthy. No individual dashboard identified the actual problem.
</div><div style="text-align:justify">
AI gave the team an important lead by identifying a suspicious, more-specific TGW route. We then reduced the problem one layer at a time:
</div><ul>
<li>Cloudflare headers established the public path, and documented proxy retry timeouts (~19s SYN timer) explained the exact ~40s duration.</li>
<li>Direct address testing (<code>curl --resolve</code>) isolated one NLB frontend as dead while the other was healthy.</li>
<li>The route table and firewall logs explained the zonal asymmetry.</li>
<li>ARC zonal shift removed the impaired path and immediately changed the request duration from ~40 seconds to ~0.3 seconds.</li>
<li>Removing the conflicting route resolved the issue permanently across all zones.</li>
</ul>
<div style="text-align:justify">
The larger lesson is that “healthy” is always qualified. Healthy according to which check, from which source, through which path and at which layer?
</div><div style="text-align:justify">
A backend health check cannot prove an entire ingress architecture. A generic hostname can hide materially different zonal paths. A proxy can turn a partial origin failure into consistent latency rather than a visible outage. A plausible diagnosis can be valuable without yet being proven.
</div><div style="text-align:justify">
The route fix permanently removed this defect. Establishing external synthetic per-zone probing and a tested ARC zonal-shift procedure ensures the application is prepared for the next zonal gray failure, regardless of its cause.
</div><div style="text-align:justify">
We did not reach that point by rejecting AI or accepting it blindly. AI helped us find the clue. Engineering discipline turned the clue into a testable explanation, and the testing exposed how the system should be made more resilient.
</div><div style="text-align:justify">
That is the balance I want my teams to learn.
</div><h2>References</h2>
<ul>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html">AWS: Health checks for Network Load Balancer target groups</a></li>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/zonal-shift.html">AWS: Zonal shift for Network Load Balancers</a></li>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#availability-zones">AWS: Network Load Balancer DNS names and zonal addresses</a></li>
<li><a href="https://developers.cloudflare.com/load-balancing/monitors/">Cloudflare: Load Balancing monitors</a></li>
<li><a href="https://developers.cloudflare.com/load-balancing/additional-options/override-http-host-headers/">Cloudflare: Override Host headers for load-balancing endpoints</a></li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Managing the LiteLLM Control Plane at Scale with Git-Driven Automation]]></title><description><![CDATA[Learn how to use OpenTofu and Git-driven automation to manage the LiteLLM control plane at scale, covering configuration, budgets, and user provisioning.]]></description><link>https://blog.cybergav.in/managing-the-litellm-control-plane-at-scale-with-gitops</link><guid isPermaLink="true">https://blog.cybergav.in/managing-the-litellm-control-plane-at-scale-with-gitops</guid><category><![CDATA[gitops]]></category><category><![CDATA[AI]]></category><category><![CDATA[opentofu]]></category><category><![CDATA[litellm]]></category><category><![CDATA[architecture]]></category><dc:creator><![CDATA[Gavin Satur]]></dc:creator><pubDate>Sun, 11 Oct 2026 05:00:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/410e9c23-a5e8-4343-8e12-09c8fb359e89.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div style="text-align:justify">
If your engineering teams are adopting generative AI, you are likely relying on proxies like LiteLLM to sit between your internal systems and the dozens of large language models (LLMs) your teams want to use. LiteLLM handles routing, API key management, and billing limits, among other features.
</div><div style="text-align:justify">
Using UI-driven configurations for this works initially, but at an enterprise scale, it quickly becomes unmanageable.
</div><div style="text-align:justify">
In this post, I will share the Git-driven architecture I designed and implemented for my organization's AI Gateway, which is built on LiteLLM. While we manage our <i>entire</i> AI Gateway infrastructure (AWS accounts, AWS resources, and the LiteLLM configuration) with infrastructure-as-code (IaC) to ensure it is auditable, secure, and developer-friendly, this blog post focuses exclusively on the <b>LiteLLM control plane</b> using OpenTofu and YAML.
</div><div style="text-align:justify">
To help others adopt this, I've open-sourced a generalized starter kit for the community based on this exact architecture!
</div><h2>The Problem: UI ClickOps Does Not Scale</h2>
<div style="text-align:justify">
Tools like <a href="https://github.com/BerriAI/litellm" target="_blank" rel="noopener noreferrer">LiteLLM</a> are excellent for proxying AI requests. However, managing hundreds of API keys, model routing rules, and strict budget limits via a web console inevitably leads to:
</div><ul>
<li><strong>Lack of auditability</strong>: "Who increased the budget for marketing-analysts by $5,000?"</li>
<li><strong>Configuration drift</strong>: "Why does staging behave differently from production?"</li>
<li><strong>Security risks</strong>: Manual management of API keys makes offboarding complicated.</li>
</ul>
<div style="text-align:justify">
I needed to architect a system where every change was reviewed as code, checked against organizational policies automatically, and deployed reliably.
</div><h2>The Git-Driven Solution</h2>
<div style="text-align:justify">
I mapped the AI proxy infrastructure configuration strictly to a Git repository. Using OpenTofu (the open-source Terraform fork) and the official LiteLLM Terraform provider, I abstracted the configuration behind simple, readable, schema-validated YAML files.
</div><div style="text-align:justify">
<i>Note: While purists argue true "GitOps" requires an in-cluster agent (like ArgoCD) for constant drift reconciliation, invoking OpenTofu via CI/CD acts as a highly effective "GitOps-style" or push-based automation that solves these auditability issues.</i>
</div><h3>1. Environment-Aware Configuration</h3>
<div style="text-align:justify">
I split the configurations cleanly by environment. <i>(Note: While the Trojan AI Gateway officially runs <code>test</code> and <code>prod</code> environments, the open-source starter repo generalizes these to <code>staging</code> and <code>prod</code>.)</i> For instance, <code>staging</code> is loosely governed and allows developers to freely test models and create their own virtual keys, subject to a hard monthly budget and personal key expiry. Meanwhile, <code>prod</code> has strict governance enforcing service accounts for workloads and hard team budgets.
</div><div style="text-align:justify">
Inside a <code>prod/teams.yaml</code>, declaring team budgets looks exactly like this:
</div><pre><code class="language-yaml">teams:
  engineering-platform:
    team_alias: "Platform Engineering"
    organization: "engineering"
    models:
      - "openai/gpt-4o"
      - "openai/gpt-4o-mini"
    max_budget: 200
    budget_duration: "1mo"
</code></pre>
<div style="text-align:justify">
Every PR that modifies this file is automatically planned and reviewed. Destructive changes (like entirely removing a team) are guarded by automated CI/CD checks before a human approval gate.
</div><h3>2. Why YAML Instead of tfvars?</h3>
<div style="text-align:justify">
OpenTofu's native input format, <code>.tfvars</code>, is workable for a handful of scalar values but awkward for deeply nested data such as teams, model lists, and budgets. It is noisy, unfamiliar to reviewers outside the infrastructure team, and hard to validate before <code>tofu plan</code>.
</div><div style="text-align:justify">
Keeping policy in YAML and logic in HCL (via <code>yamldecode(file(...))</code> and <code>for_each</code>) gave us several benefits:
</div><ul>
<li><strong>Readable by everyone</strong>: Security, finance, and team leads can review a budget change in a PR without knowing HCL.</li>
<li><strong>Schema-validated in CI</strong>: JSON Schema catches missing fields, invalid model names, and out-of-policy budgets before OpenTofu runs.</li>
<li><strong>Clean, auditable diffs</strong>: A one-line budget change is a one-line diff.</li>
<li><strong>Data-only changes</strong>: Onboarding a team means adding a YAML block, with no changes to module code.</li>
<li><strong>Self-documenting</strong>: Comments can capture why a limit exists.</li>
</ul>
<h3>3. Zero Credentials in Code</h3>
<div style="text-align:justify">
With this setup, the master LLM API keys and proxy credentials never touch the repository. OpenTofu retrieves the necessary secrets directly from AWS Secrets Manager at runtime and maps them to the respective LiteLLM configurations securely.
</div><h2>Identity vs. Governance</h2>
<div style="text-align:justify">
When building the Trojan AI Gateway, I had to solve a major problem: <i>How do we handle human accounts?</i>
</div><div style="text-align:justify">
Initially, I added about 20 early-adopter users directly into our IaC configuration. I quickly became uncomfortable with this approach, and a team member rightly pointed out that user provisioning should remain outside IaC. So, I came to a firm conclusion: <b>human user provisioning does not belong in IaC.</b>
</div><div style="text-align:justify">
Maintaining several individual YAML blocks for human users creates significant operational overhead and duplicates identity information already managed by the Identity Provider (IdP).
</div><div style="text-align:justify">
I considered three complementary mechanisms for user provisioning and access management:
</div><ol>
<li><p><strong>SCIM: IdP-driven lifecycle management</strong><br />With the appropriate LiteLLM Enterprise license and connectivity, <b><a href="https://scim.cloud/" target="_blank" rel="noopener noreferrer">SCIM</a></b> allows an IdP such as Entra ID or Okta to automate user and team provisioning, updates, and deprovisioning. This is the preferred approach when the environment supports it.</p>
</li>
<li><p><strong>SSO with Just-in-Time Provisioning: Authentication and initial access</strong><br />Users can be created in LiteLLM when they first sign in through SSO, receiving configured defaults for their role and model access. This simplifies onboarding, but those defaults do not necessarily satisfy each approved access request. JIT provisioning alone also does not provide ongoing lifecycle synchronization or automated offboarding.</p>
</li>
<li><p><strong>Automated Workflows: Applying approved access</strong><br />A GitHub Actions workflow, backed by Python scripts calling LiteLLM’s administration APIs, creates or updates users and applies the model access specified in their approved onboarding requests. This complements SSO and supports requirements beyond the configured defaults.</p>
</li>
</ol>
<div style="text-align:justify">
Our implementation combines <b>SSO with workflow-based provisioning</b>. After an onboarding request is approved, we add the user to the appropriate Entra ID group and run the GitHub Actions workflow with the required inputs. The workflow provisions or updates the LiteLLM user, applies the approved model access, and sends a welcome email with SSO login instructions.
</div><div style="text-align:justify">
SSO remains the authentication mechanism. JIT creation remains available for users who do not yet have a LiteLLM record, while the onboarding workflow establishes requested access before their first login.
</div><div style="text-align:justify">
The separation of concerns is clear: <b>the IdP owns identity and authentication; IaC defines platform policy; workflows apply approved user access.</b>
</div><h2>Get the Open-Source Starter Kit</h2>
<div style="text-align:justify">
The lessons from building USC’s Trojan AI Gateway can help other organizations bring their LiteLLM control plane under version control. To share that experience, I’ve published an open-source starter repository:
</div><p><b><a href="https://github.com/cybergavin/litellm-control-plane" target="_blank" rel="noopener noreferrer">cybergavin/litellm-control-plane</a></b></p>
<div style="text-align:justify">
It includes OpenTofu code, JSON Schema validation, separate staging and production configurations, and GitHub Actions workflows for planning, applying, and validating changes.
</div><div style="text-align:justify">
The repository provides a foundation for managing models, organizations, teams, and service-account keys. You can adapt it to your environment and extend it to additional resources supported by the LiteLLM Terraform provider, including access groups and MCP servers.
</div><div style="text-align:justify">
<i>Two caveats worth knowing. First, the LiteLLM Terraform provider sometimes lags behind the UI and API. For models it cannot fully represent, the starter kit supports <code>preserve_api_configuration: true</code>: OpenTofu tracks the model's existence while LiteLLM remains the owner of its attributes, so changes to those attributes are deliberately not enforced. It is a pragmatic escape hatch, not the default. Second, for brownfield rollouts or models created through the UI or API, a temporary <code>model_imports</code> map adopts existing models into state without creating duplicates. The README walks through the full procedure.</i>
</div><div style="text-align:justify">
Give Git-driven automation a try. Fewer manual changes. Clearer governance. A LiteLLM control plane you can review, version, and maintain.
</div>]]></content:encoded></item><item><title><![CDATA[Bypassing the Bottleneck: Architecting an AWS Express Lane for Heavy On-Premises Workloads]]></title><description><![CDATA[Design a controlled bypass for AWS centralized firewall inspection using Large Bandwidth VPN, PAN-OS PBF, SNAT, and an isolated TGW routing domain.]]></description><link>https://blog.cybergav.in/aws-vpn-express-lane-large-bandwidth-bypass</link><guid isPermaLink="true">https://blog.cybergav.in/aws-vpn-express-lane-large-bandwidth-bypass</guid><category><![CDATA[AWS]]></category><category><![CDATA[networking]]></category><category><![CDATA[vpn]]></category><category><![CDATA[Security]]></category><category><![CDATA[Hybrid Cloud]]></category><dc:creator><![CDATA[Gavin Satur]]></dc:creator><pubDate>Tue, 22 Sep 2026 02:00:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/b9565bc6-3140-4709-82f3-83e05a5475f8.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div style="text-align:justify">
If you are migrating or replicating large databases to AWS, you may eventually run into an unexpected bottleneck: not the VPN itself, but everything sitting in the traffic path.
For months, our Database Administrators (DBAs) battled timeouts and failures while using Oracle RMAN DUPLICATE to seed an approximately 1 TB database from our on-premises datacenter into AWS over Site-to-Site VPN.
One obvious answer was AWS Direct Connect. But in our environment, our estimated monthly connectivity cost for two Site-to-Site VPN connections was less than half that of comparable redundant Direct Connect connectivity. These are environment-specific estimates rather than generally applicable AWS prices, but they gave us a good reason to first determine whether we could solve the problem using VPN. It turned out that the VPN wasn't the primary bottleneck.</div><div style="text-align:justify">
Our default hybrid-cloud security architecture routes traffic through centralized Palo Alto Networks firewalls running in an AWS Security VPC. That's an appropriate default security posture, but forcing a known, tightly scoped, sustained database-transfer flow through a stateful inspection tier imposed a significant reliability and performance penalty.
For communication between explicitly approved database systems, we concluded that the incremental security benefit of centralized inspection did not justify that reliability and performance impact. Instead, we could constrain the exception through IPsec encryption, tightly scoped source and destination addresses, PAN-OS policy, AWS routing, VPC security controls, and database-level controls.</div><div style="text-align:justify">
What we needed was therefore an explicitly controlled exception to an otherwise secure-by-default network.
AWS's introduction of <b>Large Bandwidth Site-to-Site VPN</b> tunnels, supporting up to 5 Gbps per tunnel, made that much more practical. Standard AWS Site-to-Site VPN tunnels support up to 1.25 Gbps per tunnel; when using Transit Gateway and ECMP, multiple tunnels can provide higher aggregate capacity. Large Bandwidth Tunnels raise the individual tunnel capacity to as much as 5 Gbps and are specifically positioned by AWS for bandwidth-intensive hybrid applications and large data migrations.</div><p>Here's how we built our AWS "express lane" using Palo Alto Networks PAN-OS firewalls, AWS Transit Gateway, an isolated TGW routing domain, and a dedicated Large Bandwidth Site-to-Site VPN.</p>
<h2>The Baseline: Inspection by Default</h2>
<p>Our AWS network follows a hub-and-spoke model with centralized firewall inspection.
Two Transit Gateway route tables provide the core of the inspection architecture:</p>
<p><strong>TGW-RT-TO-FW</strong> - Pre-Inspection: Standard spoke VPC and hybrid-connectivity attachments enter this routing domain and are directed toward the Security VPC.</p>
<p><strong>TGW-RT-FROM-FW</strong> - Post-Inspection: The Security VPC attachment uses this table to reach the appropriate destination after inspection.</p>
<p>AWS Transit Gateway associates each attachment with one TGW route table, while an attachment can propagate routes into multiple route tables. That distinction is important: the associated table determines the routing policy applied to traffic entering the Transit Gateway from that attachment.</p>
<p>Our standard Site-to-Site VPN, VPN-A, remains the normal hybrid connectivity path. It uses standard-bandwidth AWS VPN tunnels — up to 1.25 Gbps per tunnel, with higher aggregate capacity possible through ECMP, while our Security VPC remains the default inspection point.</p>
<p><img src="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/c2be862e-6c45-4491-b055-dcd2d8a03722.png" alt="Fig. 1: Standard on-premises-to-AWS traffic traverses the Security VPC for inspection." /></p>
<h2>The Exception: A Dedicated High-Bandwidth VPN</h2>
<div style="text-align:justify">
For high-volume trusted database transfers, we introduced a <b>second Site-to-Site VPN</b>, VPN-B, using AWS Large Bandwidth Tunnels with up to 5 Gbps of capacity per tunnel. The 5 Gbps VPN alone wasn't the solution.
The more important design decision was <i>selectively steering</i> only approved traffic onto it while keeping centralized inspection as the default for everything else.
</div><p><img src="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/5f8c6059-f2f1-411d-8ef6-371a3f1baed4.png" alt="Fig. 2: Approved high-volume database traffic enters through VPN-B and bypasses the AWS Security VPC" /></p>
<p>In our implementation, both the VPN-B attachment and Database VPC attachment are associated with TGW-RT-BYPASS-FW.
This is an isolated TGW routing domain: the Database VPC is not generally exempt from inspection simply because it participates in a bypass path.</p>
<p>The governing principle is:</p>
<blockquote>
<p>Bypass must be explicit. Inspection remains the default.</p>
</blockquote>
<h2>The Solution: PBF, SNAT and Isolated TGW Routing Domain</h2>
<h3>1. Steering the Traffic On-Premises</h3>
<div style="text-align:justify">
The first routing decision happens before the packets leave our datacenter.
Our Palo Alto border firewalls use <b>Policy-Based Forwarding (PBF)</b> to identify approved on-premises-to-AWS database traffic and direct it through VPN-B rather than the standard path.
PAN-OS PBF can match attributes including source and destination addresses and then override the normal routing decision.
Separately, our NAT policy source-NATs selected on-premises database traffic into a dedicated bypass prefix, for example: <code>192.168.250.0/24</code>
This gives bypassed traffic a unique routing identity.
Instead of AWS having to distinguish whether the original on-premises networks should return through VPN-A or VPN-B, it can simply know: <code>192.168.250.0/24 → VPN-B</code>
SNAT isn't an inherent requirement of AWS Site-to-Site VPN. In our architecture it acts as a <i>route-disambiguation</i> mechanism, allowing us to introduce the bypass without changing the routing behavior of the original shared on-premises CIDRs.
PBF determines the path; NAT determines the translated identity. They are separate PAN-OS policy mechanisms.
</div><h3>2. Creating an AWS Bypass Routing Domain</h3>
<p>We created a dedicated Transit Gateway route table: <code>TGW-RT-BYPASS-FW</code>
and associated both the VPN-B attachment and the Database VPC attachment with it.
The table is intentionally sparse. Conceptually:</p>
<pre><code class="language-text">Approved on-prem DB prefixes   → VPN-B
On-prem bypass SNAT prefix     → VPN-B
Database VPC CIDR              → Database VPC attachment
0.0.0.0/0                      → Security VPC
</code></pre>
<ul>
<li>The first routes implement explicit exceptions.</li>
<li>The final route is the critical guardrail: <code>0.0.0.0/0 → Security VPC</code></li>
<li>Anything that does not match an approved bypass route goes back through our normal PAN-OS inspection architecture.</li>
<li>That means a database EC2 instance attempting to communicate with an ordinary VPC, an unapproved on-premises network, or an internet destination continues through the centralized firewall path.</li>
</ul>
<h3>3. Keep the Bypass Route Table Sparse</h3>
<div style="text-align:justify">
There is an important operational detail here.
The security property above works only if <code>TGW-RT-BYPASS-FW</code> does not acquire more-specific routes that unintentionally bypass the Security VPC.
Transit Gateway propagation can automatically install attachment routes into selected route tables. Because more-specific routes beat the <code>0.0.0.0/0</code> fail-safe, blindly propagating every spoke VPC into the bypass table could silently turn: <code>Database VPC → Application VPC</code> into direct, uninspected communication.
AWS supports independently controlling route-table association and propagation, so we deliberately control propagation into the bypass routing domain.</div><p>The rule is simple:</p>
<blockquote>
<p>Only destinations explicitly approved for bypass should have more-specific routes in the bypass table. Everything else falls through to the Security VPC.</p>
</blockquote>
<p>That guardrail is arguably more important than the VPN itself.</p>
<h2>Bidirectional Traffic Is Not the Same as Bidirectional Session Initiation</h2>
<p>Our first use case was straightforward: an on-premises Oracle system initiated a high-volume RMAN operation toward AWS.
PBF selected VPN-B, SNAT assigned the dedicated bypass identity, and AWS returned traffic for that SNAT prefix through VPN-B.
Conceptually:</p>
<pre><code class="language-text">---------------------
ON-PREMISES INITIATED
---------------------

On-prem Oracle
      |
      | PBF → VPN-B
      | SNAT → bypass prefix
      v
    VPN-B
      |
      v
AWS Oracle

---------
RETURN
---------

AWS Oracle
      |
      | destination = bypass SNAT prefix
      v
    VPN-B
      |
      v
On-prem Oracle
</code></pre>
<div style="text-align:justify">
That is symmetric and deterministic. Later, the requirement evolved.
The DBAs needed the Oracle systems to establish sessions in either direction, and Oracle Enterprise Manager also needed to communicate with the database systems. That exposed an important distinction.
A VPN is inherently bidirectional. There is nothing about AWS Site-to-Site VPN that prevents AWS from initiating a TCP session toward on-premises. The challenge is routing the response back through the same VPN.</div><p>Consider an AWS-initiated Oracle connection:</p>
<pre><code class="language-text">AWS Oracle:ephemeral-port
        |
        | SYN
        v
      VPN-B
        |
        v
On-prem Oracle:1521
</code></pre>
<p>The response looks like:</p>
<pre><code class="language-text">On-prem Oracle:1521
        |
        | SYN/ACK
        v
AWS Oracle:ephemeral-port
</code></pre>
<p>If the on-premises PBF policy only identifies traffic by destination service — for example TCP/1521 — the SYN/ACK no longer matches that rule because its destination is an ephemeral client port.
The firewall can therefore select the normal VPN-A route for the response:</p>
<pre><code class="language-text">AWS ---- SYN ----&gt; VPN-B ----&gt; On-prem

AWS &lt;--- SYN/ACK -- VPN-A &lt;---- On-prem
</code></pre>
<p>That's asymmetric routing, exactly what stateful firewalls don't want. PAN-OS documents that PBF can be applied to the initial SYN or the first SYN/ACK of a session, which is particularly relevant to this scenario.</p>
<h2>Make PBF Endpoint-Based, Not Oracle-Port-Based</h2>
<blockquote>
<p>Traffic between these explicitly approved systems belongs to the high-bandwidth hybrid path.</p>
</blockquote>
<p>PAN-OS supports using addresses as PBF match criteria, and Palo Alto explicitly documents that setting Service to Any causes all otherwise matching traffic to use the PBF rule.</p>
<p>So the cleaner PBF is conceptually:</p>
<pre><code class="language-text">Source:
    approved on-prem DB/OEM endpoints

Destination:
    approved AWS DB endpoints

Service:
    Any

Forwarding:
    VPN-B
</code></pre>
<div style="text-align:justify">
Now the policy doesn't care whether Oracle, RMAN, OEM, SSH for troubleshooting, or a TCP return packet happens to use 1521, 3872, or an ephemeral port. It cares about <i>which systems are communicating</i>.
This also means an AWS-initiated SYN/ACK returning from an approved on-premises endpoint toward an approved AWS endpoint remains eligible for VPN-B rather than falling back to VPN-A simply because its destination port is ephemeral.</div><blockquote>
<p><code>Service=Any</code> in PBF does not mean <code>Allow Any</code>. PAN-OS Security Policy can still independently allow only the Oracle, RMAN, OEM, administrative, or other services actually required.</p>
</blockquote>
<p>In other words:</p>
<pre><code class="language-text">PBF       = path
NAT       = identity
Security  = permission
</code></pre>
<p>This separation makes the design both simpler and easier to govern.</p>
<h3>Completing Bidirectional Initiation</h3>
<div style="text-align:justify">
Endpoint-based PBF solves only the on-premises path-selection half of the problem.
AWS must also know that connections initiated in AWS toward the real on-premises database addresses should enter VPN-B.
So TGW-RT-BYPASS-FW also needs approved real on-premises endpoint routes pointing to VPN-B, in addition to the existing SNAT return-prefix route:</div><pre><code class="language-text">Approved real on-prem DB CIDRs → VPN-B
Bypass SNAT prefix             → VPN-B
0.0.0.0/0                      → Security VPC
</code></pre>
<p>This produces two valid traffic models.</p>
<p>For <strong>on-premises-initiated</strong> sessions:</p>
<pre><code class="language-text">On-premises
  → PBF
  → SNAT
  → VPN-B
  → AWS

AWS
  → bypass SNAT prefix
  → VPN-B
  → On-premises
</code></pre>
<p>For <strong>AWS-initiated</strong> sessions:</p>
<pre><code class="language-text">AWS
  → real on-prem DB prefix
  → VPN-B
  → On-premises

On-premises
  → endpoint-based PBF
  → VPN-B
  → AWS
</code></pre>
<p>Both sides can initiate, and both directions remain on VPN-B.</p>
<h2>The Result</h2>
<div style="text-align:justify">
By combining PAN-OS Policy-Based Forwarding, selective SNAT, a dedicated Large Bandwidth Site-to-Site VPN and an intentionally isolated Transit Gateway routing domain, we decoupled a high-volume database workload from our standard centralized inspection path without deploying Direct Connect solely to solve this problem.
The DBAs can perform RMAN database seeding over a path with <i>up to 5 Gbps</i> of tunnel capacity, without forcing those sustained transfers through the AWS PAN-OS inspection tier. At the same time, we haven't made the Database VPC generally uninspected. The default route still sends non-bypass traffic to the Security VPC, and only explicitly approved prefixes escape that path. Our default inspection posture therefore remains intact; we've created a controlled exception rather than removing the control. And as the requirement evolved from one-way database seeding to Oracle, RMAN and OEM sessions that could be initiated from either AWS or on-premises, another lesson became clear:</div><blockquote>
<p>Don't encode a workload-routing policy as a growing list of application ports when the real policy is about trusted endpoints.</p>
</blockquote>
<div style="text-align:justify">Routing should determine the path. NAT should solve address-identity problems. Security Policy should determine what's allowed. When it comes to hybrid cloud networking, you don't always have to choose between strict security, high performance and cost control. Sometimes you just need a better toll lane.</div>]]></content:encoded></item><item><title><![CDATA[One Storage Service, Two Operating Models: Bringing NFS, SMB, and S3 Under Automation]]></title><description><![CDATA[Build a GitOps storage platform for DevOps and sysadmins using Ansible, Git, and a constrained CLI on an Everpure FlashBlade.]]></description><link>https://blog.cybergav.in/gitops-storage-automation-nfs-smb-s3</link><guid isPermaLink="true">https://blog.cybergav.in/gitops-storage-automation-nfs-smb-s3</guid><category><![CDATA[Devops]]></category><category><![CDATA[Platform Engineering ]]></category><category><![CDATA[ansible]]></category><category><![CDATA[gitops]]></category><category><![CDATA[storage]]></category><dc:creator><![CDATA[Gavin Satur]]></dc:creator><pubDate>Mon, 21 Sep 2026 02:00:00 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/553f7f3d-0831-4d52-bf06-20f889b21df7.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>For years, enterprise storage was delivered through a familiar process: an administrator opened a management interface, created a filesystem or bucket, configured access, and handed the result to an application team.</p>
<p>That approach worked, but it also allowed every request to become a small design exercise. Two administrators could receive similar requirements and produce different configurations. Naming, access rules, capacity settings, encryption, lifecycle behavior, and documentation depended too much on who performed the work. The storage platform was centrally managed, but the service was not consistently expressed.</p>
<p>When we introduced a modern on-premises storage system, an <strong><a href="https://www.everpuredata.com/content/dam/pdf/en/datasheets/ds-pure-storage-flashblade-s.pdf">Everpure FlashBlade//S200</a></strong> capable of serving NFS, SMB, and S3-compatible object storage from the same platform, I saw an opportunity to do more than reproduce that operating model on newer hardware. I wanted to define an institution-wide storage service that could be delivered consistently by two groups with very different working styles:</p>
<ul>
<li>Platform engineers who are comfortable with Git, code review, CI/CD, and AI-assisted development.</li>
<li>Traditional systems administrators who understand storage operations but would rather use a form or an operations interface than work directly in a repository.</li>
</ul>
<p>Before writing a line of automation, I spent time with the Everpure documentation and worked through manual configurations in the FlashBlade UI. Understanding the product's resource model, its boundaries, and its defaults proved essential before encoding any of that into manifests and Ansible tasks. I then designed and built the initial automation, repository structure, validation, delivery workflows, and constrained operations interface with AI assistance (Codex), which made the implementation considerably faster, especially when navigating the API surface, generating tests, and examining edge cases. The architecture and operating model, however, came from understanding the institutional problem and deciding where automation, human review, and team responsibilities should meet.</p>
<p>The most important design lesson was that storage administrators and the platform engineers did not need separate automation systems. They needed separate interfaces to the same controlled delivery process.</p>
<h2>The difficult part was defining the service</h2>
<p>Writing an Ansible task to create an NFS export is relatively straightforward. Defining what every supported NFS export should look like is more important.</p>
<p>Before building the automation, I had to translate operational knowledge and agreed service standards into an explicit service contract. That included decisions such as:</p>
<ul>
<li>Which protocols and versions are supported?</li>
<li>What are the standard filesystem and capacity settings?</li>
<li>Which access modes are permitted?</li>
<li>How are NFS client networks represented and validated?</li>
<li>Which SMB administrative access must exist on every share?</li>
<li>How are object storage accounts, buckets, users, and policies related?</li>
<li>Where are generated S3-compatible object storage credentials stored?</li>
<li>What does removal mean, and which deletions must be recoverable?</li>
<li>Which requests are standard, and which require engineering review?</li>
</ul>
<p>These standards became defaults and validation rules rather than instructions in a runbook. An operator should not have to remember twenty settings for every routine request. The interface should ask for the few pieces of information that genuinely vary and derive the rest from the approved service definition.</p>
<p>An illustrative request might be as small as:</p>
<pre><code class="language-yaml">request:
  type: nfs
  name: application-data
  export: /application-data
  clients:
    - 192.0.2.40/32
    - 192.0.2.64/27
  change_reference: CHG0001234
</code></pre>
<p>This is deliberately not a complete storage configuration. The automation supplies the approved protocol, security, capacity, naming, and lifecycle defaults. A request that needs to override those defaults is classified as nonstandard and moves to the engineering path.</p>
<p>That boundary matters. A self-service interface becomes dangerous when it exposes every underlying product option. Good platform engineering reduces the number of decisions a consumer must make while preserving an explicit route for legitimate exceptions.</p>
<h2>I chose Git as the control plane</h2>
<p>I made the repository the source of truth for the desired state of the service: manifests, automation, validation, tests, and deployment workflows. Changes to storage begin as changes to that desired state.</p>
<p>The basic flow is:</p>
<ol>
<li>A user submits a standard request or an engineer proposes a broader change.</li>
<li>The request is converted into a small manifest change on a new branch.</li>
<li>Validation runs without production credentials.</li>
<li>A pull request shows the exact proposed state change.</li>
<li>The appropriate service owner or engineer approves it.</li>
<li>After merge, a protected deployment workflow reconciles the storage platform.</li>
<li>The deploy pipeline runs a <strong>second apply immediately after the first and asserts zero changes</strong>—idempotency is verified automatically, not assumed.</li>
<li>Operations verifies the live service outcome.</li>
</ol>
<p>The pull request is not ceremony added for developers. It is the human-readable change record. It answers who requested the change, what will be different, which checks passed, who approved it, and which revision was deployed.</p>
<p>It also separates intent from execution. Pull-request validation does not need access to the storage array or credential store. Production credentials are available only to the protected deployment job after the change has passed review and been merged.</p>
<p>This is safer than allowing a script, chatbot, or web form to call the storage API directly. The interface can prepare a change, but it cannot silently bypass review or mutate production.</p>
<h2>The engineering interface: Git and an AI coding agent</h2>
<p>Platform engineers need access to the full repository because their work is open-ended. They may add a feature, change validation, fix an Ansible module workaround, extend a lifecycle operation, update tests, or troubleshoot a failed deployment.</p>
<p>For that audience, Git and an AI coding agent work well together. I have used AI extensively while building the platform, but as an engineering accelerator rather than an autonomous operator. I can describe an intended change in plain English, allow the agent to inspect the repository's instructions and tests, and then review the resulting diff. It shortens the path from an idea to a tested implementation, but it does not become the production authority.</p>
<p>That distinction also matters when describing the work honestly. AI helped me implement and test faster; it did not identify the organizational problem, negotiate the boundaries between teams, or decide what a safe operating model should be. Those remained engineering decisions.</p>
<p>The deterministic controls remain in the pipeline:</p>
<ul>
<li>Schema and semantic validation.</li>
<li>YAML and Ansible linting.</li>
<li>Syntax checks.</li>
<li>Unit tests for request generation.</li>
<li>Offline check-mode tests where practical.</li>
<li>Secret scanning and focused diff review.</li>
<li>Code ownership and protected branches.</li>
<li>Environment approval before deployment.</li>
<li>Post-apply idempotency check: a second reconciliation run that must report zero changes.</li>
</ul>
<p>This distinction is important. AI is useful for interpreting an engineering request and proposing code. It should not replace predictable validation, peer review, or the deployment boundary.</p>
<h2>The operations interface: a form, a CLI, and a consistent contract</h2>
<p>The systems administrators did not need the repository's full flexibility. Most of their work involved standard additions: create an NFS share, create an SMB share, or create an S3 account and bucket using established defaults.</p>
<p>For them, I built two things that express the same constrained operation:</p>
<p><strong>A GitHub Actions form.</strong> Operators open the repository's <em>Request standard Storage</em> workflow in the Actions UI, select a storage type, fill in only the fields specific to that request—filesystem name, export path, client IPs—and run it. The workflow validates the inputs, calls <code>storagectl</code> internally, and opens a draft pull request. The operator never touches a terminal or a YAML file. We are about to migrate this interface to our <a href="https://semaphoreui.com/">Semaphore automation platform</a>, which will present the same fields in a more purpose-built UI. The underlying contract — <code>storagectl</code>, the manifest schema, the validation pipeline, and the deployment workflow require no changes at all. That is the intended property of the design.</p>
<p><img src="https://cdn.hashnode.com/uploads/gql/6aa73ede13ae672728e4d165/5d2ae281-6f04-44c9-935d-219731667af9.png" alt="The GitHub Actions form interface for requesting storage" /></p>
<p><strong>A <code>storagectl</code> CLI</strong> for engineers or scripts that need a programmatic path. Its purpose is intentionally narrow: validate a supported request, update the appropriate manifest, and open a draft pull request.</p>
<p>A CLI invocation resembles:</p>
<pre><code class="language-shell">storagectl request nfs \
  --name application-data \
  --export /application-data \
  --client 192.0.2.40/32 \
  --change CHG0001234
</code></pre>
<p>Both the form and the CLI produce the same manifest change. Neither connects to the storage system, and neither deploys anything.</p>
<p>The same contract can later sit behind an automation portal such as Semaphore or a tightly constrained AI tool. Those are presentation layers. They should all invoke the same tested operation rather than reimplement storage logic independently.</p>
<p>This led to a simple design rule:</p>
<blockquote>
<p>Build one deterministic operations contract, then provide interfaces suited to different users.</p>
</blockquote>
<p>The platform engineer can work directly with Git. The administrator can use a web form. Both produce the same manifest change, validation results, pull request, approval trail, and deployment behavior.</p>
<h2>Approval follows the change, not the interface</h2>
<p>Introducing two interfaces raised an organizational question: who should approve a pull request created by an operations administrator?</p>
<p>The answer depends on what changed.</p>
<p>A routine manifest addition should be reviewed by another storage operator or service owner who can validate the requested service outcome. A change to automation code, tests, workflows, defaults, or security behavior should be reviewed by the platform engineering team. A change that affects both service intent and platform behavior should receive both perspectives.</p>
<p>This avoids two unhelpful extremes. Platform engineers do not become a ticket queue for every standard storage request, and operations staff are not expected to approve changes to automation internals they do not maintain.</p>
<p>The submitter's team and chosen interface do not determine governance. The content and risk of the change do.</p>
<h2>NFS, SMB, and S3 are not the same resource with different labels</h2>
<p>One temptation was to create a highly abstract storage model that treated every protocol identically. That would have produced elegant-looking code and a confusing service.</p>
<p>The three offerings share a delivery pipeline, but their operational models are different:</p>
<ul>
<li><strong>NFS</strong> centers on filesystems, export policies, client networks, protocol compatibility, and squash behavior.</li>
<li><strong>SMB</strong> adds directory-service identities, share permissions, administrative access, and SMB-specific security behavior.</li>
<li><strong>S3-compatible object storage</strong> on the FlashBlade involves accounts, buckets, users, access policies, generated keys, and a separate secrets-management lifecycle. This is <em>not</em> AWS S3 — it is the FlashBlade's own S3-compatible endpoint. The only AWS involvement is storing the generated credentials in AWS Secrets Manager, accessed via GitHub OIDC with no long-lived cloud key.</li>
</ul>
<p>I reused common mechanics where they were genuinely common: request validation, Git operations, pull requests, approval, protected deployment, and audit history. I kept protocol-specific logic explicit and readable.</p>
<p>That balance has kept the implementation understandable. Modularity is useful when it reflects a real boundary. Abstraction for its own sake merely makes the next administrator reverse-engineer the framework before changing it.</p>
<h2>Credential handling shaped the architecture</h2>
<p>Object storage made credential handling particularly important. I designed the workflow around a simple rule: an automation run may need to generate an access key, but that credential should never appear in a manifest, pull-request comment, workflow output, or AI prompt.</p>
<p>The safer pattern is to pass a generated credential directly from the storage API response into an enterprise secrets manager while suppressing sensitive task output. The manifest records only the desired identity and the approved secret location—not the credential value.</p>
<p>The deployment workflow also uses short-lived credentials obtained through <strong>GitHub OIDC workload identity federation</strong> to assume a scoped AWS role, rather than maintaining a long-lived cloud key in the repository. The FlashBlade API token and the AWS role are available only within the protected <code>prd</code> deployment environment, after environment protections pass. Storage API credentials remain limited to that protected environment and never appear in logs or pull-request output.</p>
<p>This produces three useful trust zones:</p>
<ul>
<li><strong>Request and pull request:</strong> no production credentials and no production connectivity.</li>
<li><strong>Validation:</strong> enough context to reject an invalid or unsafe desired state, but no ability to apply it.</li>
<li><strong>Deployment:</strong> protected credentials, restricted network access, and only the permissions required to reconcile approved resources.</li>
</ul>
<p>The same separation should survive whichever interface is added later. A friendly UI must not become a reason to flatten the security model.</p>
<h2>Deletion deserved a different workflow</h2>
<p>Creating a storage resource is usually reversible. Deleting one may destroy data or credentials. I therefore avoided presenting deletion as just another self-service checkbox.</p>
<p>Routine creation can use the constrained operations interface. Changes to existing resources, migrations, and removals follow the engineering path and require broader review. Resources marked absent remain visible in desired state until operators verify the outcome; cleanup of the manifest happens separately.</p>
<p>This is less convenient than one-click deletion, and intentionally so. A good platform makes the safe path easy without pretending every operation has the same risk.</p>
<h2>Could this have been just Ansible and Semaphore?</h2>
<p>I repeatedly asked myself whether this should have been just a set of Ansible playbooks launched from <a href="https://semaphoreui.com/">Semaphore</a>. For a small team managing a limited number of resources, that could be entirely sufficient. Not every automation effort needs a GitOps repository, a custom CLI, AI instructions, and multiple workflows.</p>
<p>In this case, the additional structure became worthwhile because the service crossed several boundaries:</p>
<ul>
<li>Three protocols with different resource and security models.</li>
<li>Multiple teams with different technical preferences.</li>
<li>Institution-wide standards that needed to remain consistent.</li>
<li>Production changes requiring peer review and an audit trail.</li>
<li>Credentials spanning the storage system and a separate secrets platform.</li>
<li>A need to support both routine operations and continued engineering.</li>
</ul>
<p>An MCP server could eventually expose those same operations to an internal assistant. Neither needs to replace the repository or deployment pipeline.</p>
<p>The mistake would be implementing storage behavior separately in every interface. The command, portal, and AI tool should all converge on one versioned contract.</p>
<h2>The role of platform engineering</h2>
<p>My contribution was not learning how to operate the storage interface faster. It was recognizing that the institution needed a repeatable service, designing the operating model, and building the initial platform that could deliver it.</p>
<p>That meant encoding standards, creating safe defaults, building validation, separating credentials, defining lifecycle behavior, establishing review boundaries, and providing more than one way to consume the result.</p>
<p>The infrastructure operations team still owns essential parts of the service. Its administrators understand client behavior, access requirements, operational risk, live verification, and customer impact. Their knowledge informs the standards, and their verification establishes whether a deployed service actually works. Platform engineering packages that shared operational knowledge into a paved path and maintains the machinery beneath it.</p>
<p>This is the partnership I want platform engineering to create:</p>
<ul>
<li>Operations defines and validates the service outcome.</li>
<li>Platform engineering turns recurring delivery into a reliable product.</li>
<li>Consumers choose an interface appropriate to their skills.</li>
<li>Governance remains consistent regardless of interface.</li>
</ul>
<p>The result is not "developers replacing sysadmins." It is fewer handcrafted changes, clearer ownership, and more time for both groups to work on problems that actually require their judgment.</p>
<h2>Core Principles Behind the Architecture</h2>
<ol>
<li><strong>Define the service before automating the product.</strong> A playbook without standards only reproduces inconsistency more quickly.</li>
<li><strong>Identify values that truly vary.</strong> Everything else should come from reviewed defaults.</li>
<li><strong>Keep the operational interface constrained.</strong> Standard requests and engineering changes should not pretend to be the same thing.</li>
<li><strong>Use Git as an audit and approval boundary, not as a user-interface requirement.</strong> Consumers should not need to understand Git to benefit from Git-based controls.</li>
<li><strong>Let approval follow risk and change content.</strong> Service owners review intent; platform engineers review platform behavior.</li>
<li><strong>Keep AI outside the trust boundary.</strong> Let it propose changes while deterministic checks and humans retain authority.</li>
<li><strong>Design secrets handling before adding object storage.</strong> A credential that reaches a log or pull request is already mishandled.</li>
<li><strong>Make destructive operations intentionally harder.</strong> Convenience is not the primary design goal for deletion.</li>
<li><strong>Abstract shared delivery mechanics, not protocol differences.</strong> Readable duplication is sometimes better than a clever universal model.</li>
<li><strong>Build the reusable operation once.</strong> CLIs, portals, and AI tools should be adapters around the same contract.</li>
</ol>
<h2>Conclusion</h2>
<p>For me, modernizing storage was never primarily about replacing a graphical interface with YAML. It was about replacing individual interpretation with a service contract while preserving the operational judgment of the people responsible for the platform.</p>
<p>The important outcome is that a DevOps engineer and a traditional systems administrator can approach the same enterprise service through interfaces that make sense to each of them and still arrive at the same reviewed, secure, and supportable result.</p>
<p>That is where I believe platform engineering earns its place: not by adding another layer between teams, but by making good engineering reusable beyond the person or team that first created it.</p>
]]></content:encoded></item></channel></rss>