localmodelroutex/docs/ARCHITECTURE.md

# Architecture

This page explains how a request moves through LMR and where to add things.
Read the [README](../README.md) first for the feature list and the repo layout.

## One request, end to end

A client sends `POST /v1/chat/completions` with `Authorization: Bearer sk-...`
and `"model": "openrouter/anthropic/claude-sonnet-4.5"`.

1. **Key auth** (`internal/api/middleware.go`). The middleware hashes the
   bearer token with SHA-256 and loads the matching virtual key with its
   allowlist rules. Revoked and expired keys stop here.
2. **Resolve** (`internal/catalog`). The model address is split into
   `<provider>/<slug>` and looked up in the in-memory catalog cache. The
   reserved `any/<slug>` prefix picks one provider by the routing strategy.
3. **Allowlist and limits** (`internal/api`). The key's rules are matched
   against the resolved address. Then the per-model RPM and TPM limits from
   the catalog entry are checked in `internal/ratelimit`.
4. **Dispatch** (`internal/providermgr`, `internal/providers`). The provider
   instance is fetched by name and type-asserted for the capability the route
   needs, for example `ChatCompleter`. The request payload is copied, the
   model is swapped for the upstream slug, and admin parameter overrides are
   merged in.
5. **Relay**. The provider sends the upstream call and returns either a body
   or an SSE stream. The handler writes it to the client unchanged, apart from
   adding a `created` timestamp when the upstream omits it.
6. **Account** (`internal/billing`, `internal/store`). Token usage is parsed
   from the response or the final stream chunk. Cost is the upstream-reported
   dollar amount when the provider gives one, else tokens times the catalog
   prices. The log row and the prompt and completion text are written in a
   goroutine so they never delay the response.

`/v1/embeddings` and `/v1/images/generations` follow the same six steps with
their own capability interfaces.

## Components

```
cmd/server/main.go        config → db → migrate → providers → catalog → HTTP
internal/config           viper, GATEWAY_* environment variables
internal/db               pgxpool and the golang-migrate runner
internal/store            hand-written SQL, one file per table group
internal/crypto           AES-256-GCM seal and open for provider credentials
internal/providermgr      builds live provider clients from DB rows
internal/providers        provider abstraction and the per-type packages
internal/catalog          model library: sync, cache, address resolution
internal/ratelimit        per-model fixed-window RPM and TPM counters
internal/billing          cost calculation
internal/influx           InfluxDB 3 exporter: one point per request, batched
internal/api              chi routes: /v1 proxy, /api/admin, /auth
internal/oidcauth         OIDC discovery, ID token verification, group check
internal/web              go:embed of the built dashboard, served as a SPA
dashboard/                Vite + React + TypeScript management UI
migrations/               numbered SQL pairs, applied at startup
```

## The provider abstraction

`providers.Provider` is small on purpose. It has a name, a type, and a list of
capabilities. Everything a route needs lives in a capability interface:

| Interface | Route | Method |
|---|---|---|
| `ChatCompleter` | `/v1/chat/completions` | `ChatCompletion`, `ChatCompletionStream` |
| `Embedder` | `/v1/embeddings` | `Embeddings` |
| `ImageGenerator` | `/v1/images/generations` | `ImageGeneration` |
| `ModelLister` | catalog sync | `ListModels` |

A handler type-asserts for the interface it needs and answers 501 when the
provider lacks it. New modalities add a new interface and a new route. Existing
providers do not change.

Each provider type is a package under `internal/providers/` that registers a
factory in `init()`. `internal/providers/all` blank-imports every package so
the server binary links them all. The registry maps the `type` column of the
providers table to the factory.

The OpenAI-compatible types (OpenAI, OpenRouter, llama.cpp, Ollama, Copilot)
embed `openaicompat.Transport`, which holds the HTTP client, base URL, auth
hook, static headers, and the flags for usage and cost reporting. A type only
adds what differs. OpenRouter adds attribution headers and the image
translation. Copilot adds a token exchange in its auth hook. Anthropic is the
one type that does not embed the transport. It translates between the OpenAI
and Messages schemas in both directions, including streamed tool calls.

### Why the image route translates

OpenRouter has no `/images/generations` endpoint. Its image models generate
through chat completions with `modalities: ["image", "text"]` and return the
image in `message.images`. The OpenRouter provider's `ImageGeneration` builds
that chat request, maps OpenAI `size` and `quality` to OpenRouter's
`image_config`, and renders the reply in the OpenAI images schema. Clients
written for the OpenAI Images API work unchanged.

## The catalog

Admins configure providers, not models. On a timer, and after every admin
change, the syncer asks each provider for its model list and upserts the rows
into `provider_models`. The syncer extracts what the provider reports: display
name, context length, pricing, and output modalities. Output modalities become
the row's capabilities, which is how the image route knows which models accept
an image request.

The llama.cpp provider goes one step further. The OpenAI listing only carries
the training context, so the provider asks the server for the running size:
`/props` on a single server, `/props?model=` or the launch arguments on the
router, and `/upstream/<model>/props` for models llama-swap has loaded. It
never asks llama-swap about a stopped model, because that would start it.

Admin overrides live in separate columns on the same row and survive syncs:
enabled, blocked, parameter overrides, prices, context window, rate limits,
and a backup model. A model that disappears from the upstream list is marked
unavailable, not deleted.

The cache holds every servable row in memory, keyed by address. It is rebuilt
after each sync, so the request path never queries the database for routing.

### Backup models

A model's `fallback_model_id` names another catalog row. The cache turns an
address into a route: the requested model first, then its backups in chain
order, with models that are not servable skipped. Two models may back each
other up; the walk tracks visited ids and stops after a fixed number of hops.
A model that is enabled but unavailable still routes, straight to its backups,
and still appears in `/v1/models`.

The proxy handlers walk the route. A target is skipped when the rate limiter
refuses it, its provider is missing, the transport fails, or the upstream
answers 408, 429, or 5xx. Client errors (4xx) are relayed, because the backup
would most likely answer the same. For a stream, the first chunk is read
before the status line goes to the client, so a stream that dies at once can
still move to a backup. After the first byte is sent, a failure can only be
logged.

The switch is invisible to the client and visible in the log. A backup's
response reports the requested model's id. The key allowlist is checked
against the requested address. Every failed attempt gets its own log row, with
the error and a note that names the backup. The row for the backup keeps the
requested address in `model_name`, puts the backup in `served_model`, and
uses the backup's prices for the cost.

## Authentication

Two systems that do not mix:

- **Proxy routes** under `/v1` take a virtual key. Keys are stored as hashes.
  Each key is either allow-all or carries a list of address patterns, where
  `openrouter/*` matches a whole provider.
- **Admin and user routes** under `/api` take an OIDC ID token. The login flow
  is the authorization code flow. The server does the code exchange and hands
  the ID token to the dashboard. Admin access needs the configured group in
  the `groups` claim. The dashboard's test chat uses a synthetic allow-all key
  bound to the signed-in user, so it needs no real API key.

Provider credentials are sealed with AES-256-GCM using the 32-byte key from
`GATEWAY_SECURITY_ENCRYPTION_KEY`. They are opened in memory when the provider
manager builds a client and are never returned by the admin API.

## Logging and retention

Every proxy request writes one row to `request_logs` with the key prefix,
model, provider, status, token counts, cost, latency, and time to first token
for streams. The prompt and completion text go to a second table. Inline
image data in prompts is replaced by a size marker before the write.

A daily job deletes rows older than the `log.retention_days` setting. Zero
keeps everything. Settings are rows in a key-value table and are edited from
the dashboard.

## Dashboard

The dashboard is a Vite project in `backend/dashboard`. `npm run build` writes
to `backend/internal/web/dist`, and the Go binary embeds that folder. In
development, Vite proxies `/api`, `/v1`, and `/auth` to the Go server on port
8080, so the UI hot-reloads against real data.

Pages map to admin routes one to one: Providers, Models, Routing, API Keys,
Logs, Analytics, Settings, and a Chat page for testing a model through the
same proxy path that clients use.