AI Tools
The LLM Gateway is our in-house control plane for LLM traffic. We deploy it inside your network as one self-contained system, and with in-network model backends its traffic stays inside your network.
Six capabilities the gateway carries into a Private Vector delivery. Every one of them ships in the software today.
Your applications call familiar OpenAI-compatible endpoint paths for completions, embeddings, and image generation. Each family routes to the backend you choose, and the gateway proxies every call to its configured upstream.
The whole gateway is one binary with its own embedded database file. There is no external database server to install, secure, or operate, and the schema sets itself up on startup.
Every person gets their own gateway key with per-person endpoint permissions, and you can revoke any key at once. One person can hold chat and embedding access while another also holds image generation.
You set a spend limit per person, and the gateway automatically blocks further calls once usage reaches it. A background checker sets the block and clears it when it expires, so the limit holds without anyone policing it.
Token usage and image costs are recorded per person and per key in fixed time windows. The history is queryable, and you set how long it is kept.
Under pressure the gateway holds requests in bounded queues, sheds load when a backend slows down, and stops routing to a backend that fails its health checks. When a backend recovers, routing resumes on its own.
The gateway runs as one system on infrastructure inside your network. The rows below record where its keys, its counters, and its audit records live, and how it governs the traffic it carries.
The records and rules that stay inside your network.
Every proxied response is captured as audit records in tiers, and each tier keeps its records for its own retention window.
An optional moderation gate screens request content before it is forwarded, using a fast local check first and a classifier service second.
The gateway periodically copies its live database to a backup file while it keeps serving, and publishes each copy atomically.
On shutdown the gateway stops its background work first, then drains in-flight requests within a fixed time budget.
Every request carries a correlation ID that is stored and sent upstream, so any call can be traced end to end.
The gateway detects the CPU set and memory envelope it runs in and adapts to both, under a soft memory ceiling you set.
The LLM Gateway is one part of a Private Vector delivery, not the delivery itself. The pages below carry the rest of the story: the stage where metered usage turns into measured improvement, and the capability that deploys AI infrastructure inside your network.