> For the complete documentation index, see [llms.txt](https://docs.visdom.virtuslab.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.visdom.virtuslab.com/visdom-ai-gateway/architecture.md). # Architecture The components, the path of a call, and where each Visdom layer plugs in. ## Components | Component | Origin | Role | | ----------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | **LLM gateway** | LiteLLM proxy (MIT) | OpenAI- and Anthropic-compatible endpoint, provider connectors, virtual keys, teams, budgets, rate limits, routing and fallback, spend log | | **MCP gateway** | IBM MCP Context Forge (Apache-2.0) | Registry of approved MCP servers, federation behind one address, virtual servers, encrypted per-server credentials, plugins, admin UI | | **PostgreSQL** | | State for both gateways, in separate databases | | **Control Plane** | Visdom | People, teams and projects; issues keys and budgets; kill switch (roadmap) | | **Policy Gate** | Visdom | The decision before every model and tool call (roadmap) | | **AI Tracing** | Visdom | Signed evidence chain and cost ledger | | **Keycloak** | Apache-2.0 | Single sign-on, federation to the organisation's directory, OAuth tokens for MCP | | **Local models** | vLLM or any OpenAI-compatible server | Open-weight models on the organisation's GPUs, behind the same aliases | Each component has one role and a documented interface. Clients talk to the gateway's address and its model aliases, never to the upstream product, so either gateway can be replaced without changing a client. ## The path of a model call 1. **Identity.** The virtual key or the OIDC token resolves to a person or an application, and to a team. A call with no owner is refused. 2. **Authorisation.** The model is checked against what the team may use. Deny by default. *Today:* the team's model allow-list in the LLM gateway. *Next:* Policy Gate. 3. **Budget and limits.** An alert at the soft budget, a hard stop at the limit, rate limits per key and team. *Today:* in the LLM gateway. 4. **Input guardrails.** Secrets blocked; PII masked; prompt injection screened. *Next and later.* 5. **Execution.** The gateway routes to an approved deployment with the organisation's own credentials and fails over to another deployment of the same model. 6. **Output guardrails.** Checked on the stream or after buffering, per team policy. *Later.* 7. **Evidence.** Identity, team, model, tokens, cost, timing, outcome and the policy decision are written to AI Tracing's signed chain. A refusal is written the same way as a completed call. *Next.* ## The path of a tool call The same seven steps, with three differences: * Authorisation covers the MCP server, the tool and the data it exposes, independently of the tool server's own permissions. * The call is made with a credential the user never sees – the server's stored credential today, the user's own OAuth token later. * A high-risk tool can require a human approval before step 5. ## Deployment **Reference deployment (today).** One host, Docker Compose: the two gateways and PostgreSQL. It is the fastest way to evaluate and the basis of the smoke test. **Production (next).** The organisation's Kubernetes, installed from a Helm chart through GitOps: * Both gateways stateless, three or more replicas each, with Redis for shared limit counters. * PostgreSQL in high availability, or the organisation's managed instance. * Default-deny network policies. The only outbound traffic is from the LLM gateway to allow-listed model hosts, through the corporate proxy. * Secrets from the organisation's store through External Secrets Operator. * Single tenant: one installation per organisation. **Air-gapped.** Everything above works without internet access with local models and local tool servers. See [Offline and air-gapped](/visdom-ai-gateway/user-guide/offline-and-air-gapped.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.visdom.virtuslab.com/visdom-ai-gateway/architecture.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.