localmodelroutex/docs/ARCHITECTURE.md

Architecture

This page explains how a request moves through LMR and where to add things. Read the README first for the feature list and the repo layout.

One request, end to end

A client sends POST /v1/chat/completions with Authorization: Bearer sk-... and "model": "openrouter/anthropic/claude-sonnet-4.5".

1. Key auth (internal/api/middleware.go). The middleware hashes the bearer token with SHA-256 and loads the matching virtual key with its allowlist rules. Revoked and expired keys stop here.
2. Resolve (internal/catalog). The model address is split into <provider>/<slug> and looked up in the in-memory catalog cache. The reserved any/<slug> prefix picks one provider by the routing strategy.
3. Allowlist and limits (internal/api). The key's rules are matched against the resolved address. Then the per-model RPM and TPM limits from the catalog entry are checked in internal/ratelimit.
4. Dispatch (internal/providermgr, internal/providers). The provider instance is fetched by name and type-asserted for the capability the route needs, for example ChatCompleter. The request payload is copied, the model is swapped for the upstream slug, and admin parameter overrides are merged in.
5. Relay. The provider sends the upstream call and returns either a body or an SSE stream. The handler writes it to the client unchanged, apart from adding a created timestamp when the upstream omits it.
6. Account (internal/billing, internal/store). Token usage is parsed from the response or the final stream chunk. Cost is the upstream-reported dollar amount when the provider gives one, else tokens times the catalog prices. The log row and the prompt and completion text are written in a goroutine so they never delay the response.

/v1/embeddings and /v1/images/generations follow the same six steps with their own capability interfaces.

Components

cmd/server/main.go        config → db → migrate → providers → catalog → HTTP
internal/config           viper, GATEWAY_* environment variables
internal/db               pgxpool and the golang-migrate runner
internal/store            hand-written SQL, one file per table group
internal/crypto           AES-256-GCM seal and open for provider credentials
internal/providermgr      builds live provider clients from DB rows
internal/providers        provider abstraction and the per-type packages
internal/catalog          model library: sync, cache, address resolution
internal/ratelimit        per-model fixed-window RPM and TPM counters
internal/billing          cost calculation
internal/influx           InfluxDB 3 exporter: one point per request, batched
internal/api              chi routes: /v1 proxy, /api/admin, /auth
internal/oidcauth         OIDC discovery, ID token verification, group check
internal/web              go:embed of the built dashboard, served as a SPA
dashboard/                Vite + React + TypeScript management UI
migrations/               numbered SQL pairs, applied at startup

The provider abstraction

providers.Provider is small on purpose. It has a name, a type, and a list of capabilities. Everything a route needs lives in a capability interface:

Interface	Route	Method
ChatCompleter	/v1/chat/completions	ChatCompletion, ChatCompletionStream
Embedder	/v1/embeddings	Embeddings
ImageGenerator	/v1/images/generations	ImageGeneration
ModelLister	catalog sync	ListModels

A handler type-asserts for the interface it needs and answers 501 when the provider lacks it. New modalities add a new interface and a new route. Existing providers do not change.

Each provider type is a package under internal/providers/ that registers a factory in init(). internal/providers/all blank-imports every package so the server binary links them all. The registry maps the type column of the providers table to the factory.

The OpenAI-compatible types (OpenAI, OpenRouter, llama.cpp, Ollama, Copilot) embed openaicompat.Transport, which holds the HTTP client, base URL, auth hook, static headers, and the flags for usage and cost reporting. A type only adds what differs. OpenRouter adds attribution headers and the image translation. Copilot adds a token exchange in its auth hook. Anthropic is the one type that does not embed the transport. It translates between the OpenAI and Messages schemas in both directions, including streamed tool calls.

Why the image route translates

OpenRouter has no /images/generations endpoint. Its image models generate through chat completions with modalities: ["image", "text"] and return the image in message.images. The OpenRouter provider's ImageGeneration builds that chat request, maps OpenAI size and quality to OpenRouter's image_config, and renders the reply in the OpenAI images schema. Clients written for the OpenAI Images API work unchanged.

The catalog

Admins configure providers, not models. On a timer, and after every admin change, the syncer asks each provider for its model list and upserts the rows into provider_models. The syncer extracts what the provider reports: display name, context length, pricing, and output modalities. Output modalities become the row's capabilities, which is how the image route knows which models accept an image request.

The llama.cpp provider goes one step further. The OpenAI listing only carries the training context, so the provider asks the server for the running size: /props on a single server, /props?model= or the launch arguments on the router, and /upstream/<model>/props for models llama-swap has loaded. It never asks llama-swap about a stopped model, because that would start it.

Admin overrides live in separate columns on the same row and survive syncs: enabled, blocked, parameter overrides, prices, context window, and rate limits. A model that disappears from the upstream list is marked unavailable, not deleted.

The cache holds every servable row in memory, keyed by address. It is rebuilt after each sync, so the request path never queries the database for routing.

Authentication

Two systems that do not mix:

• Proxy routes under /v1 take a virtual key. Keys are stored as hashes. Each key is either allow-all or carries a list of address patterns, where openrouter/* matches a whole provider.
• Admin and user routes under /api take an OIDC ID token. The login flow is the authorization code flow. The server does the code exchange and hands the ID token to the dashboard. Admin access needs the configured group in the groups claim. The dashboard's test chat uses a synthetic allow-all key bound to the signed-in user, so it needs no real API key.

Provider credentials are sealed with AES-256-GCM using the 32-byte key from GATEWAY_SECURITY_ENCRYPTION_KEY. They are opened in memory when the provider manager builds a client and are never returned by the admin API.

Logging and retention

Every proxy request writes one row to request_logs with the key prefix, model, provider, status, token counts, cost, latency, and time to first token for streams. The prompt and completion text go to a second table. Inline image data in prompts is replaced by a size marker before the write.

A daily job deletes rows older than the log.retention_days setting. Zero keeps everything. Settings are rows in a key-value table and are edited from the dashboard.

Dashboard

The dashboard is a Vite project in backend/dashboard. npm run build writes to backend/internal/web/dist, and the Go binary embeds that folder. In development, Vite proxies /api, /v1, and /auth to the Go server on port 8080, so the UI hot-reloads against real data.

Pages map to admin routes one to one: Providers, Models, Routing, API Keys, Logs, Analytics, Settings, and a Chat page for testing a model through the same proxy path that clients use.