A shared AI gateway with fleet-wide cost control

Problem

A dozen small projects (chat bots, dashboards, automated reports) all needed LLM access. Left alone, that becomes a dozen scattered API keys, a dozen places pricing has to be updated by hand, and no way to tell “we’re about to get billed” from “everything’s fine” until the invoice arrives.

Approach

Every internal caller talks to one in-cluster gateway, passing an effort alias (best, balanced, fast) rather than a concrete model ID, so a new model release is a single edit to one pricing table. The gateway routes across a small chain of backends for resilience and cost control, and falls back automatically if the primary is unavailable. A Postgres ledger records every call, a live SSE dashboard shows usage in real time, and an alerter pages immediately if spend on any caller looks unexpected, catching a runaway loop within minutes.

Outcome

Every caller in the fleet routes through one chokepoint, with the ledger and alerting to prove usage rather than assume it. When the provider ships a new model, updating the whole fleet is a one-line change in one file.