# Managing the LiteLLM Control Plane at Scale with Git-Driven Automation

<div style="text-align: justify;">
If your engineering teams are adopting generative AI, you are likely relying on proxies like LiteLLM to sit between your internal systems and the dozens of large language models (LLMs) your teams want to use. LiteLLM handles routing, API key management, and billing limits, among other features.
</div>

<div style="text-align: justify;">
Using UI-driven configurations for this works initially, but at an enterprise scale, it quickly becomes unmanageable.
</div>

<div style="text-align: justify;">
In this post, I will share the Git-driven architecture I designed and implemented for my organization's AI Gateway, which is built on LiteLLM. While we manage our <i>entire</i> AI Gateway infrastructure (AWS accounts, AWS resources, and the LiteLLM configuration) with infrastructure-as-code (IaC) to ensure it is auditable, secure, and developer-friendly, this blog post focuses exclusively on the <b>LiteLLM control plane</b> using OpenTofu and YAML.
</div>

<div style="text-align: justify;">
To help others adopt this, I've open-sourced a generalized starter kit for the community based on this exact architecture!
</div>

## The Problem: UI ClickOps Does Not Scale

<div style="text-align: justify;">
Tools like <a href="https://github.com/BerriAI/litellm" target="_blank" rel="noopener noreferrer">LiteLLM</a> are excellent for proxying AI requests. However, managing hundreds of API keys, model routing rules, and strict budget limits via a web console inevitably leads to:
</div>

- **Lack of auditability**: "Who increased the budget for marketing-analysts by \$5,000?"
- **Configuration drift**: "Why does staging behave differently from production?"
- **Security risks**: Manual management of API keys makes offboarding complicated.

<div style="text-align: justify;">
I needed to architect a system where every change was reviewed as code, checked against organizational policies automatically, and deployed reliably.
</div>

## The Git-Driven Solution

<div style="text-align: justify;">
I mapped the AI proxy infrastructure configuration strictly to a Git repository. Using OpenTofu (the open-source Terraform fork) and the official LiteLLM Terraform provider, I abstracted the configuration behind simple, readable, schema-validated YAML files.
</div>

<div style="text-align: justify;">
<i>Note: While purists argue true "GitOps" requires an in-cluster agent (like ArgoCD) for constant drift reconciliation, invoking OpenTofu via CI/CD acts as a highly effective "GitOps-style" or push-based automation that solves these auditability issues.</i>
</div>

### 1. Environment-Aware Configuration

<div style="text-align: justify;">
I split the configurations cleanly by environment. <i>(Note: While the Trojan AI Gateway officially runs <code>test</code> and <code>prod</code> environments, the open-source starter repo generalizes these to <code>staging</code> and <code>prod</code>.)</i> For instance, <code>staging</code> is loosely governed and allows developers to freely test models and create their own virtual keys, subject to a hard monthly budget and personal key expiry. Meanwhile, <code>prod</code> has strict governance enforcing service accounts for workloads and hard team budgets.
</div>

<div style="text-align: justify;">
Inside a <code>prod/teams.yaml</code>, declaring team budgets looks exactly like this:
</div>

```yaml
teams:
  engineering-platform:
    team_alias: "Platform Engineering"
    organization: "engineering"
    models:
      - "openai/gpt-4o"
      - "openai/gpt-4o-mini"
    max_budget: 200
    budget_duration: "1mo"
```

<div style="text-align: justify;">
Every PR that modifies this file is automatically planned and reviewed. Destructive changes (like entirely removing a team) are guarded by automated CI/CD checks before a human approval gate.
</div>

### 2. Why YAML Instead of tfvars?

<div style="text-align: justify;">
OpenTofu's native input format, <code>.tfvars</code>, is workable for a handful of scalar values but awkward for deeply nested data such as teams, model lists, and budgets. It is noisy, unfamiliar to reviewers outside the infrastructure team, and hard to validate before <code>tofu plan</code>.
</div>

<div style="text-align: justify;">
Keeping policy in YAML and logic in HCL (via <code>yamldecode(file(...))</code> and <code>for_each</code>) gave us several benefits:
</div>

- **Readable by everyone**: Security, finance, and team leads can review a budget change in a PR without knowing HCL.
- **Schema-validated in CI**: JSON Schema catches missing fields, invalid model names, and out-of-policy budgets before OpenTofu runs.
- **Clean, auditable diffs**: A one-line budget change is a one-line diff.
- **Data-only changes**: Onboarding a team means adding a YAML block, with no changes to module code.
- **Self-documenting**: Comments can capture why a limit exists.

### 3. Zero Credentials in Code

<div style="text-align: justify;">
With this setup, the master LLM API keys and proxy credentials never touch the repository. OpenTofu retrieves the necessary secrets directly from AWS Secrets Manager at runtime and maps them to the respective LiteLLM configurations securely.
</div>

## Identity vs. Governance

<div style="text-align: justify;">
When building the Trojan AI Gateway, I had to solve a major problem: <i>How do we handle human accounts?</i>
</div>

<div style="text-align: justify;">
Initially, I added about 20 early-adopter users directly into our IaC configuration. I quickly became uncomfortable with this approach, and a team member rightly pointed out that user provisioning should remain outside IaC. So, I came to a firm conclusion: <b>human user provisioning does not belong in IaC.</b>
</div>

<div style="text-align: justify;">
Maintaining several individual YAML blocks for human users creates significant operational overhead and duplicates identity information already managed by the Identity Provider (IdP).
</div>

<div style="text-align: justify;">
I considered three complementary mechanisms for user provisioning and access management:
</div>

1. **SCIM: IdP-driven lifecycle management**  
   With the appropriate LiteLLM Enterprise license and connectivity, <b><a href="https://scim.cloud/" target="_blank" rel="noopener noreferrer">SCIM</a></b> allows an IdP such as Entra ID or Okta to automate user and team provisioning, updates, and deprovisioning. This is the preferred approach when the environment supports it.

2. **SSO with Just-in-Time Provisioning: Authentication and initial access**  
   Users can be created in LiteLLM when they first sign in through SSO, receiving configured defaults for their role and model access. This simplifies onboarding, but those defaults do not necessarily satisfy each approved access request. JIT provisioning alone also does not provide ongoing lifecycle synchronization or automated offboarding.

3. **Automated Workflows: Applying approved access**  
   A GitHub Actions workflow, backed by Python scripts calling LiteLLM’s administration APIs, creates or updates users and applies the model access specified in their approved onboarding requests. This complements SSO and supports requirements beyond the configured defaults.

<div style="text-align: justify;">
Our implementation combines <b>SSO with workflow-based provisioning</b>. After an onboarding request is approved, we add the user to the appropriate Entra ID group and run the GitHub Actions workflow with the required inputs. The workflow provisions or updates the LiteLLM user, applies the approved model access, and sends a welcome email with SSO login instructions.
</div>

<div style="text-align: justify;">
SSO remains the authentication mechanism. JIT creation remains available for users who do not yet have a LiteLLM record, while the onboarding workflow establishes requested access before their first login.
</div>

<div style="text-align: justify;">
The separation of concerns is clear: <b>the IdP owns identity and authentication; IaC defines platform policy; workflows apply approved user access.</b>
</div>

## Get the Open-Source Starter Kit

<div style="text-align: justify;">
The lessons from building USC’s Trojan AI Gateway can help other organizations bring their LiteLLM control plane under version control. To share that experience, I’ve published an open-source starter repository:
</div>

<b><a href="https://github.com/cybergavin/litellm-control-plane" target="_blank" rel="noopener noreferrer">cybergavin/litellm-control-plane</a></b>

<div style="text-align: justify;">
It includes OpenTofu code, JSON Schema validation, separate staging and production configurations, and GitHub Actions workflows for planning, applying, and validating changes.
</div>

<div style="text-align: justify;">
The repository provides a foundation for managing models, organizations, teams, and service-account keys. You can adapt it to your environment and extend it to additional resources supported by the LiteLLM Terraform provider, including access groups and MCP servers.
</div>

<div style="text-align: justify;">
<i>Two caveats worth knowing. First, the LiteLLM Terraform provider sometimes lags behind the UI and API. For models it cannot fully represent, the starter kit supports <code>preserve_api_configuration: true</code>: OpenTofu tracks the model's existence while LiteLLM remains the owner of its attributes, so changes to those attributes are deliberately not enforced. It is a pragmatic escape hatch, not the default. Second, for brownfield rollouts or models created through the UI or API, a temporary <code>model_imports</code> map adopts existing models into state without creating duplicates. The README walks through the full procedure.</i>
</div>

<div style="text-align: justify;">
Give Git-driven automation a try. Fewer manual changes. Clearer governance. A LiteLLM control plane you can review, version, and maintain.
</div>
