# Architecture This page explains how a request moves through LMR and where to add things. Read the [README](../README.md) first for the feature list and the repo layout. ## One request, end to end A client sends `POST /v1/chat/completions` with `Authorization: Bearer sk-...` and `"model": "openrouter/anthropic/claude-sonnet-4.5"`. 1. **Key auth** (`internal/api/middleware.go`). The middleware hashes the bearer token with SHA-256 and loads the matching virtual key with its allowlist rules. Revoked and expired keys stop here. 2. **Resolve** (`internal/catalog`). The model address is split into `/` and looked up in the in-memory catalog cache. The reserved `any/` prefix picks one provider by the routing strategy. 3. **Allowlist and limits** (`internal/api`). The key's rules are matched against the resolved address. Then the per-model RPM and TPM limits from the catalog entry are checked in `internal/ratelimit`. 4. **Dispatch** (`internal/providermgr`, `internal/providers`). The provider instance is fetched by name and type-asserted for the capability the route needs, for example `ChatCompleter`. The request payload is copied, the model is swapped for the upstream slug, and admin parameter overrides are merged in. 5. **Relay**. The provider sends the upstream call and returns either a body or an SSE stream. The handler writes it to the client unchanged, apart from adding a `created` timestamp when the upstream omits it. 6. **Account** (`internal/billing`, `internal/store`). Token usage is parsed from the response or the final stream chunk. Cost is the upstream-reported dollar amount when the provider gives one, else tokens times the catalog prices. The log row and the prompt and completion text are written in a goroutine so they never delay the response. `/v1/embeddings` and `/v1/images/generations` follow the same six steps with their own capability interfaces. ## Components ``` cmd/server/main.go config → db → migrate → providers → catalog → HTTP internal/config viper, GATEWAY_* environment variables internal/db pgxpool and the golang-migrate runner internal/store hand-written SQL, one file per table group internal/crypto AES-256-GCM seal and open for provider credentials internal/providermgr builds live provider clients from DB rows internal/providers provider abstraction and the per-type packages internal/catalog model library: sync, cache, address resolution internal/ratelimit per-model fixed-window RPM and TPM counters internal/billing cost calculation internal/influx InfluxDB 3 exporter: one point per request, batched internal/api chi routes: /v1 proxy, /api/admin, /auth internal/oidcauth OIDC discovery, ID token verification, group check internal/web go:embed of the built dashboard, served as a SPA dashboard/ Vite + React + TypeScript management UI migrations/ numbered SQL pairs, applied at startup ``` ## The provider abstraction `providers.Provider` is small on purpose. It has a name, a type, and a list of capabilities. Everything a route needs lives in a capability interface: | Interface | Route | Method | |---|---|---| | `ChatCompleter` | `/v1/chat/completions` | `ChatCompletion`, `ChatCompletionStream` | | `Embedder` | `/v1/embeddings` | `Embeddings` | | `ImageGenerator` | `/v1/images/generations` | `ImageGeneration` | | `ModelLister` | catalog sync | `ListModels` | A handler type-asserts for the interface it needs and answers 501 when the provider lacks it. New modalities add a new interface and a new route. Existing providers do not change. Each provider type is a package under `internal/providers/` that registers a factory in `init()`. `internal/providers/all` blank-imports every package so the server binary links them all. The registry maps the `type` column of the providers table to the factory. The OpenAI-compatible types (OpenAI, OpenRouter, llama.cpp, Ollama, Copilot) embed `openaicompat.Transport`, which holds the HTTP client, base URL, auth hook, static headers, and the flags for usage and cost reporting. A type only adds what differs. OpenRouter adds attribution headers and the image translation. Copilot adds a token exchange in its auth hook. Anthropic is the one type that does not embed the transport. It translates between the OpenAI and Messages schemas in both directions, including streamed tool calls. ### Why the image route translates OpenRouter has no `/images/generations` endpoint. Its image models generate through chat completions with `modalities: ["image", "text"]` and return the image in `message.images`. The OpenRouter provider's `ImageGeneration` builds that chat request, maps OpenAI `size` and `quality` to OpenRouter's `image_config`, and renders the reply in the OpenAI images schema. Clients written for the OpenAI Images API work unchanged. ## The catalog Admins configure providers, not models. On a timer, and after every admin change, the syncer asks each provider for its model list and upserts the rows into `provider_models`. The syncer extracts what the provider reports: display name, context length, pricing, and output modalities. Output modalities become the row's capabilities, which is how the image route knows which models accept an image request. The llama.cpp provider goes one step further. The OpenAI listing only carries the training context, so the provider asks the server for the running size: `/props` on a single server, `/props?model=` or the launch arguments on the router, and `/upstream//props` for models llama-swap has loaded. It never asks llama-swap about a stopped model, because that would start it. Admin overrides live in separate columns on the same row and survive syncs: enabled, blocked, parameter overrides, prices, context window, rate limits, and a backup model. A model that disappears from the upstream list is marked unavailable, not deleted. The cache holds every servable row in memory, keyed by address. It is rebuilt after each sync, so the request path never queries the database for routing. ### Backup models A model's `fallback_model_id` names another catalog row. The cache turns an address into a route: the requested model first, then its backups in chain order, with models that are not servable skipped. Two models may back each other up; the walk tracks visited ids and stops after a fixed number of hops. A model that is enabled but unavailable still routes, straight to its backups, and still appears in `/v1/models`. The proxy handlers walk the route. A target is skipped when the rate limiter refuses it, its provider is missing, the transport fails, or the upstream answers 408, 429, or 5xx. Client errors (4xx) are relayed, because the backup would most likely answer the same. For a stream, the first chunk is read before the status line goes to the client, so a stream that dies at once can still move to a backup. After the first byte is sent, a failure can only be logged. The switch is invisible to the client and visible in the log. A backup's response reports the requested model's id. The key allowlist is checked against the requested address. Every failed attempt gets its own log row, with the error and a note that names the backup. The row for the backup keeps the requested address in `model_name`, puts the backup in `served_model`, and uses the backup's prices for the cost. ## Authentication Two systems that do not mix: - **Proxy routes** under `/v1` take a virtual key. Keys are stored as hashes. Each key is either allow-all or carries a list of address patterns, where `openrouter/*` matches a whole provider. - **Admin and user routes** under `/api` take an OIDC ID token. The login flow is the authorization code flow. The server does the code exchange and hands the ID token to the dashboard. Admin access needs the configured group in the `groups` claim. The dashboard's test chat uses a synthetic allow-all key bound to the signed-in user, so it needs no real API key. Provider credentials are sealed with AES-256-GCM using the 32-byte key from `GATEWAY_SECURITY_ENCRYPTION_KEY`. They are opened in memory when the provider manager builds a client and are never returned by the admin API. ## Logging and retention Every proxy request writes one row to `request_logs` with the key prefix, model, provider, status, token counts, cost, latency, and time to first token for streams. The prompt and completion text go to a second table. Inline image data in prompts is replaced by a size marker before the write. A daily job deletes rows older than the `log.retention_days` setting. Zero keeps everything. Settings are rows in a key-value table and are edited from the dashboard. ## Dashboard The dashboard is a Vite project in `backend/dashboard`. `npm run build` writes to `backend/internal/web/dist`, and the Go binary embeds that folder. In development, Vite proxies `/api`, `/v1`, and `/auth` to the Go server on port 8080, so the UI hot-reloads against real data. Pages map to admin routes one to one: Providers, Models, Routing, API Keys, Logs, Analytics, Settings, and a Chat page for testing a model through the same proxy path that clients use.