# LMR — Local Model Route

An OpenAI-compatible **LLM gateway**. It puts a single, authenticated, observable
endpoint in front of any number of model backends — local servers and commercial
APIs alike — and adds virtual-key auth, provider routing, request logging,
usage-based billing, and a management UI.

Meant to run locally in a homelab environment as a way to securely-ish expose your local models and back them up with the big boys.

Clients talk to it exactly as they would to OpenAI (`POST /v1/chat/completions`,
SSE streaming included); the gateway handles credentials, routing, and accounting
behind the scenes.

![LMR dashboard overview](docs/screenshots/overview.png)

<details>
<summary>More screenshots</summary>

Models page, with discovered models, pricing and per-model overrides:

![Models page](docs/screenshots/models.png)

Request log, with per-request tokens, cost and latency:

![Logs page](docs/screenshots/logs.png)

</details>

## Features

- **OpenAI-compatible proxy** — drop-in `/v1/chat/completions` with streaming,
  `/v1/embeddings`, and `/v1/images/generations`.
- **Image generation** — OpenAI Images API clients (such as Open WebUI's image
  setting) work against OpenRouter image models. The gateway translates the
  call to a chat completion with image output and back. Only models whose
  provider reports image output accept the call.
- **Virtual keys** — issue, scope, revoke, and expire `sk-…` keys per user/group,
  each with its own model allowlist.
- **Multi-provider routing** — local (llama.cpp / Ollama) and API providers
  (OpenAI, Anthropic, GitHub Copilot) behind one interface.
- **Provider abstraction** — per-provider quirks handled internally (Anthropic
  `cache_control` injection, OpenAI `cached_tokens`, Copilot token exchange).
- **Usage & billing** — per-request cost from a pricing table, written
  asynchronously so it never blocks the request path.
- **OIDC admin auth** — Authentik / any OIDC provider, group-gated admin access.
- **Encrypted secrets** — provider credentials stored AES-256-GCM encrypted at rest.
- **Single binary** — the React management UI is embedded via `go:embed`.

## Stack

- **Go** (chi, pgx/v5, viper, go-oidc, golang-migrate) — see [`backend/go.mod`](backend/go.mod)
- **PostgreSQL 16** — keys, request logs, pricing
- **React + TypeScript** (Vite) — management UI, built into the server binary

## Repo layout

```
.
├── backend/                 # Go module: git.govekar.net/LovroG05/localmodelroutex
│   ├── cmd/server/          # entrypoint: config → db → migrate → HTTP server
│   ├── internal/
│   │   ├── api/             # HTTP handlers + middleware (proxy, admin, auth)
│   │   ├── providers/       # modular provider layer (registry + capability ifaces)
│   │   │   ├── openaicompat/ #   shared OpenAI-compatible transport
│   │   │   ├── openai|openrouter|llamacpp|ollama|copilot/  # OpenAI-compatible
│   │   │   ├── anthropic/   #   Messages API + OpenAI<->Anthropic translation
│   │   │   └── all/         #   blank-import aggregator (self-registration)
│   │   ├── catalog/         # runtime model library: cache, sync, any/<slug> routing
│   │   ├── store/           # hand-written pgx data-access layer
│   │   ├── crypto/          # AES-256-GCM seal/open for provider secrets
│   │   ├── config/          # viper config, GATEWAY_ env prefix
│   │   ├── db/              # pgxpool setup + migration runner
│   │   ├── billing/         # cost calculation + async usage logging
│   │   └── web/             # go:embed of the built UI, served as SPA
│   ├── migrations/          # golang-migrate SQL, applied automatically at startup
│   └── dashboard/           # Vite + React TS project (builds into internal/web/dist)
└── README.md
```

## Authentication

Two auth systems that never mix:

| Surface | Header | Verification |
|---|---|---|
| Proxy routes (`/v1/*`) | `Bearer sk-xxx` | sha256 of the key → lookup in `virtual_keys`; check revoked / expired / model allowlist |
| Admin routes (`/api/admin/*`) | `Bearer <OIDC JWT>` | `go-oidc` `Verify()` → `groups` claim → must contain the admin group |

Dashboard login uses the OIDC authorization code flow: `GET /auth/login`
redirects to the provider, and `GET /auth/callback` exchanges the code server
side and hands the ID token to the SPA. Every signed-in user can create keys,
see their own logs and use the test chat. Members of the admin group also
manage providers, the model catalog, routing and settings.

Provider credentials are encrypted with AES-256-GCM using the 32-byte key in
`GATEWAY_SECURITY_ENCRYPTION_KEY` and decrypted only in memory at request time.

## Configuration

All settings come from environment variables prefixed `GATEWAY_`. Dot-notation
config keys map to underscores in env (e.g. `oidc.client_id` →
`GATEWAY_OIDC_CLIENT_ID`). In development, a `.env` file in the working directory
is loaded automatically; in production, set real environment variables.

| Variable | Default | Required | Notes |
|---|---|---|---|
| `GATEWAY_SERVER_PORT` | `8080` | no | HTTP listen port |
| `GATEWAY_SERVER_ENV` | `development` | no | `development` \| `production` (controls log format/level) |
| `GATEWAY_SERVER_PUBLIC_URL` | — | **yes** | canonical external URL, used for the OIDC redirect URI |
| `GATEWAY_DB_URL` | — | **yes** | `postgres://user:pass@host:5432/db` |
| `GATEWAY_DB_MAX_CONNS` | `20` | no | pool max connections |
| `GATEWAY_DB_MIN_CONNS` | `2` | no | pool min connections |
| `GATEWAY_DB_MAX_CONN_LIFETIME` | `30` | no | minutes |
| `GATEWAY_OIDC_ISSUER` | — | **yes** | OIDC issuer URL |
| `GATEWAY_OIDC_CLIENT_ID` | — | **yes** | OIDC client ID |
| `GATEWAY_OIDC_CLIENT_SECRET` | — | **yes** | OIDC client secret (confidential client) |
| `GATEWAY_OIDC_ADMIN_GROUP` | — | **yes** | group required for admin access |
| `GATEWAY_SECURITY_ENCRYPTION_KEY` | — | **yes** | 64 hex chars (32 bytes) for AES-256-GCM |
| `GATEWAY_CATALOG_SYNC_INTERVAL` | `30m` | no | how often provider model lists are refreshed |
| `GATEWAY_INFLUX_URL` | — | no | InfluxDB 3 base URL, e.g. `http://influxdb:8181`. Empty = exporter off |
| `GATEWAY_INFLUX_TOKEN` | — | no | InfluxDB 3 token, sent as a Bearer token |
| `GATEWAY_INFLUX_DATABASE` | — | with URL | target database |
| `GATEWAY_INFLUX_MEASUREMENT` | `lmr_request` | no | measurement name for request points |
| `GATEWAY_INFLUX_FLUSH_INTERVAL` | `10s` | no | max time a point waits in the buffer |
| `GATEWAY_INFLUX_BATCH_SIZE` | `500` | no | points per write request |

Config is validated on startup; missing required values or a wrong-length
encryption key cause the server to exit immediately.

### InfluxDB 3 export

Set `GATEWAY_INFLUX_URL` and `GATEWAY_INFLUX_DATABASE` to export one point per
proxied request. Points are batched in memory and written with the
`/api/v3/write_lp` endpoint. A full buffer drops points and logs a warning, so
the exporter never delays a response. Each point has these tags and fields:

| Kind | Name | Notes |
|---|---|---|
| tag | `provider`, `model`, `key_prefix` | group by provider, model address, or key. `key_prefix` is `(no key)` for the dashboard chat |
| tag | `user_id`, `provider_id`, `provider_model_id` | UUIDs, set when known |
| tag | `status_code`, `streamed` | HTTP status and `true`/`false` |
| field | `requests`, `errors` | always `1` and `0` or `1`, so `sum()` gives counts |
| field | `prompt_tokens`, `completion_tokens`, `cached_tokens`, `total_tokens` | integers |
| field | `cost` | dollars, float |
| field | `latency_ms`, `ttft_ms` | integers. `ttft_ms` is `0` when the response was not streamed |
| field | `request_id`, `finish_reason`, `error` | strings, empty when not known |

Every field is written on every point. InfluxDB 3 creates a column on its first
write, and a query that names a missing column fails, so optional values are
written as `0` or an empty string.

Generate an encryption key:

```bash
openssl rand -hex 32
```

## Running locally

Requires Go (see the `go` directive in [`backend/go.mod`](backend/go.mod)) and a
PostgreSQL 16 instance.

```bash
# 1. Start Postgres
docker run -d --name lmr-postgres \
  -e POSTGRES_USER=gateway -e POSTGRES_PASSWORD=gateway -e POSTGRES_DB=gateway \
  -p 5432:5432 postgres:16

# 2. Configure
cd backend
cp .env.example .env   # fill in OIDC issuer/client + a 32-byte encryption key

# 3. Run — make loads .env, applies migrations, and starts the server
make run
```

The server listens on `GATEWAY_SERVER_PORT` (default `:8080`).

`.env` is consumed as **environment variables**: the Makefile loads it via
`include .env` / `export`, and viper reads `GATEWAY_*` from the environment. For
Docker/systemd, pass it with `env_file` / `EnvironmentFile` instead.

### Make targets

| Target | Action |
|---|---|
| `make run` | Run the gateway (migrations auto-applied on startup) |
| `make build` | Build the dashboard, then compile `bin/gateway` |
| `make ui` | Build the dashboard into the embedded asset dir |
| `make migrate-up` | Apply all pending migrations |
| `make migrate-down` | Roll back the most recent migration |
| `make test` / `make lint` / `make tidy` | Go tests / lint / module tidy |

### Health checks

| Endpoint | Meaning |
|---|---|
| `GET /healthz` | Liveness — process is up |
| `GET /readyz` | Readiness — database reachable (pings the pool) |

## Deploy

Any OCI image host works. Build the root `Dockerfile` and run the image with the
`GATEWAY_*` variables from [`.env.example`](.env.example). `compose.yml` runs it
next to your own Postgres, and `compose.bundled.yml` runs Postgres inside the
container for a single-host setup.

The files below are the author's own deployment. They are tracked as a worked
example. Replace the registry and hostnames with yours.

- `.woodpecker.yml` runs the tests on every push and pull request, then builds
  the image and pushes it to a private registry on `master`.
- [`deploy/k8s/lmr.yaml`](deploy/k8s/lmr.yaml) runs the gateway on k3s. The
  manifest expects `configmap/lmr-config`, `secret/lmr-secret` and a registry
  pull secret to exist in the cluster. To roll out a new image:

```bash
kubectl apply -f deploy/k8s/lmr.yaml                 # only when the manifest changed
kubectl rollout restart deployment/localmodelroutex
kubectl rollout status deployment/localmodelroutex
```

`compose.yml` and `compose.bundled.yml` build from source for single-host use.
The bundled variant runs Postgres inside the same container.

## Database & migrations

SQL migrations live in [`backend/migrations/`](backend/migrations/) using
golang-migrate's `{version}_{name}.{up,down}.sql` naming. They are **embedded into
the binary** (`go:embed`) and **applied automatically on startup**, so a deployed
binary is self-contained — no migration files needed on the host.

To add a migration, create a new numbered pair, e.g.:

```
migrations/000002_virtual_keys.up.sql
migrations/000002_virtual_keys.down.sql
```

## Providers

| Name | Kind | Notes |
|---|---|---|
| `llamacpp` | local | OpenAI-compatible; works with a single llama-server, the llama.cpp router, or llama-swap; reads the running context size from `/props` during sync |
| `ollama` | local | OpenAI-compatible |
| `openrouter` | api key | OpenAI-compatible; HTTP-Referer/X-Title attribution; reports pricing; serves `/v1/images/generations` through chat completions |
| `openai` | api key | reads `cached_tokens` from the usage response |
| `anthropic` | api key | Messages API ↔ OpenAI translation, including streamed tool calls; injects `cache_control` on the system prompt + penultimate message |
| `github_copilot` | subscription | exchanges a GitHub token for a ~30-min Copilot token, mutex-guarded refresh |

## Model addressing

Models are addressed as **`<provider>/<slug>`**, where `<provider>` is the
configured provider instance name and `<slug>` is the upstream model id (which
may itself contain slashes, e.g. `openrouter/anthropic/claude-3.7-sonnet`). The
**catalog** is built at runtime by syncing each provider's model listing on a
timer and cached in memory.

- **`any/<slug>`** — a reserved prefix that fans out across every provider
  serving that model and picks one by the configured **routing strategy**
  (`round_robin` | `lowest_cost`, set from the dashboard Routing page).
- **Admin overrides** — per-model `enabled`/`blocked`, parameter overrides,
  pricing, context window, rate limits, and a backup model live alongside the
  discovered data and survive syncs. Clearing an override in the dashboard
  falls back to the provider-reported value.
- **Backup models** — any model can name another model as its backup. When
  the model is offline, rate limited, or its upstream fails before the client
  has received anything, the gateway sends the same request to the backup and
  returns the answer under the requested model's name. The client sees no
  difference. The request log records each failed attempt, the address that
  served the request, and the backup's cost.
- **Curated aliases** — the schema has `models` / `model_targets` tables for
  friendly names with priority/weight failover. No code uses them yet.

## Frontend

The management UI is a Vite + React + TypeScript project in
[`backend/dashboard/`](backend/dashboard/). Its production build is written into
`backend/internal/web/dist/` (set via `build.outDir` in
[`vite.config.ts`](backend/dashboard/vite.config.ts)) and embedded into the server
binary, which serves it as a single-page app with history-API fallback (API
routes take precedence).

```bash
cd backend/dashboard
npm install
npm run build      # type-checks (tsc), then outputs to ../internal/web/dist
```

During development, run the Vite dev server alongside the gateway:

```bash
npm run dev        # serves the UI on http://localhost:5173 with hot reload
```

The dev server proxies `/v1`, `/api`, and `/auth` to `http://localhost:8080`
(configured under `server.proxy` in `vite.config.ts`), so the running gateway
backs the UI in development.

## Project status

Implemented end to end and verified against Postgres: configuration, migrations,
the OIDC login flow, virtual keys with model allowlists, the `/v1` proxy
(chat completions with streaming, embeddings, image generation, model listing), provider dispatch
for all six provider types, the runtime catalog with `any/<slug>` routing,
rate limits, request and content logging with retention, billing, and the
management dashboard.

Not built yet: curated aliases (`models` / `model_targets`), group-scoped
session keys, and per-key budgets.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the test and
lint commands, and the pull request checklist. Report security problems as
described in [SECURITY.md](SECURITY.md), not in public issues.

## License

LMR is released under the GNU Affero General Public License, version 3. See
[LICENSE](LICENSE). If you run a modified LMR as a service for others, the AGPL
requires you to offer them the source of your modified version.