Why we built the AI gateway before the AI features
A brief on the architectural decision that shaped every product on the platform — and why we'd do it again.
When we started Flaxxa, we faced an architectural fork: ship an AI feature in one product first (probably the helpdesk's auto-resolve bot) and refactor later, or build the gateway first and let products read from it.
We chose the gateway. Here's why, and what we'd change if we did it again.
The temptation of "ship first"
The auto-resolve bot was a closer shot to revenue. A working demo of "AI answers your support tickets" sells a Pro tier in a way "we have a multi-provider AI gateway with caching and failover" does not. Most teams in our position pick the demo.
The cost of that path is hidden. Six months later, when you ship the second AI feature, you have a copy of the same wrapper code. A year later, when you ship the fifth, you have five wrappers and zero observability. By the time finance asks the per-feature cost question, the answer is "we'd need a quarter to refactor". So nobody asks.
What the gateway-first decision bought us
- Per-request cost ledger from day one. Every product squad's AI work writes to one ledger. Per-feature attribution is automatic.
- Caching was free across products. Cache hit rates compound when multiple products see the same prompts (which happens more than you'd think — same FAQ asked by two products' bots).
- Provider migrations are policy edits, not refactors. When OpenAI's pricing shifted in April, we moved the chat workload to Anthropic via a one-line policy change. No squad had to deploy.
- Procurement passes once. The AI gateway inherits the platform's SOC 2 posture. Each new AI feature doesn't trigger its own security review.
What we'd do differently
Two things, both about pacing:
- We over-engineered the routing policy DSL early. We wanted it to be declarative, hot-reloadable, with a UI editor on day one. We needed it to be a YAML file in version control. The DSL came later when there were enough policies to justify it.
- We under-invested in the ledger UI. The data was right; the dashboard was a Looker query for the first 4 months. PMs couldn't self-serve. When we built the per-tag dashboard in product, AI cost decisions accelerated 5×.
What this means for your team
If you're shipping more than one AI-powered feature, build a gateway. The ROI breakeven is around feature #2; by feature #5 it's saving you a quarter of work each year. The "ship first, refactor later" path doesn't refactor later; it ships forever.
If you're shipping exactly one AI feature, you can defer. But don't pretend you're going to stay at one.
Keep reading
