LovroG05
8bb62a1c20
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Any model can name another model as its backup. The gateway tries the backup when the model is offline, rate limited, or its upstream fails before the client has received anything. The response reports the requested model, so the client sees no switch. The request log records each failed attempt and the address that served the request, and the cost uses the backup's prices.
323 lines
15 KiB
Markdown
323 lines
15 KiB
Markdown
# LMR — Local Model Route
|
|
|
|
An OpenAI-compatible **LLM gateway**. It puts a single, authenticated, observable
|
|
endpoint in front of any number of model backends — local servers and commercial
|
|
APIs alike — and adds virtual-key auth, provider routing, request logging,
|
|
usage-based billing, and a management UI.
|
|
|
|
Meant to run locally in a homelab environment as a way to securely-ish expose your local models and back them up with the big boys.
|
|
|
|
Clients talk to it exactly as they would to OpenAI (`POST /v1/chat/completions`,
|
|
SSE streaming included); the gateway handles credentials, routing, and accounting
|
|
behind the scenes.
|
|
|
|
|
|
|
|
<details>
|
|
<summary>More screenshots</summary>
|
|
|
|
Models page, with discovered models, pricing and per-model overrides:
|
|
|
|
|
|
|
|
Request log, with per-request tokens, cost and latency:
|
|
|
|
|
|
|
|
</details>
|
|
|
|
## Features
|
|
|
|
- **OpenAI-compatible proxy** — drop-in `/v1/chat/completions` with streaming,
|
|
`/v1/embeddings`, and `/v1/images/generations`.
|
|
- **Image generation** — OpenAI Images API clients (such as Open WebUI's image
|
|
setting) work against OpenRouter image models. The gateway translates the
|
|
call to a chat completion with image output and back. Only models whose
|
|
provider reports image output accept the call.
|
|
- **Virtual keys** — issue, scope, revoke, and expire `sk-…` keys per user/group,
|
|
each with its own model allowlist.
|
|
- **Multi-provider routing** — local (llama.cpp / Ollama) and API providers
|
|
(OpenAI, Anthropic, GitHub Copilot) behind one interface.
|
|
- **Provider abstraction** — per-provider quirks handled internally (Anthropic
|
|
`cache_control` injection, OpenAI `cached_tokens`, Copilot token exchange).
|
|
- **Usage & billing** — per-request cost from a pricing table, written
|
|
asynchronously so it never blocks the request path.
|
|
- **OIDC admin auth** — Authentik / any OIDC provider, group-gated admin access.
|
|
- **Encrypted secrets** — provider credentials stored AES-256-GCM encrypted at rest.
|
|
- **Single binary** — the React management UI is embedded via `go:embed`.
|
|
|
|
## Stack
|
|
|
|
- **Go** (chi, pgx/v5, viper, go-oidc, golang-migrate) — see [`backend/go.mod`](backend/go.mod)
|
|
- **PostgreSQL 16** — keys, request logs, pricing
|
|
- **React + TypeScript** (Vite) — management UI, built into the server binary
|
|
|
|
## Repo layout
|
|
|
|
```
|
|
.
|
|
├── backend/ # Go module: git.govekar.net/LovroG05/localmodelroutex
|
|
│ ├── cmd/server/ # entrypoint: config → db → migrate → HTTP server
|
|
│ ├── internal/
|
|
│ │ ├── api/ # HTTP handlers + middleware (proxy, admin, auth)
|
|
│ │ ├── providers/ # modular provider layer (registry + capability ifaces)
|
|
│ │ │ ├── openaicompat/ # shared OpenAI-compatible transport
|
|
│ │ │ ├── openai|openrouter|llamacpp|ollama|copilot/ # OpenAI-compatible
|
|
│ │ │ ├── anthropic/ # Messages API + OpenAI<->Anthropic translation
|
|
│ │ │ └── all/ # blank-import aggregator (self-registration)
|
|
│ │ ├── catalog/ # runtime model library: cache, sync, any/<slug> routing
|
|
│ │ ├── store/ # hand-written pgx data-access layer
|
|
│ │ ├── crypto/ # AES-256-GCM seal/open for provider secrets
|
|
│ │ ├── config/ # viper config, GATEWAY_ env prefix
|
|
│ │ ├── db/ # pgxpool setup + migration runner
|
|
│ │ ├── billing/ # cost calculation + async usage logging
|
|
│ │ └── web/ # go:embed of the built UI, served as SPA
|
|
│ ├── migrations/ # golang-migrate SQL, applied automatically at startup
|
|
│ └── dashboard/ # Vite + React TS project (builds into internal/web/dist)
|
|
└── README.md
|
|
```
|
|
|
|
## Authentication
|
|
|
|
Two auth systems that never mix:
|
|
|
|
| Surface | Header | Verification |
|
|
|---|---|---|
|
|
| Proxy routes (`/v1/*`) | `Bearer sk-xxx` | sha256 of the key → lookup in `virtual_keys`; check revoked / expired / model allowlist |
|
|
| Admin routes (`/api/admin/*`) | `Bearer <OIDC JWT>` | `go-oidc` `Verify()` → `groups` claim → must contain the admin group |
|
|
|
|
Dashboard login uses the OIDC authorization code flow: `GET /auth/login`
|
|
redirects to the provider, and `GET /auth/callback` exchanges the code server
|
|
side and hands the ID token to the SPA. Every signed-in user can create keys,
|
|
see their own logs and use the test chat. Members of the admin group also
|
|
manage providers, the model catalog, routing and settings.
|
|
|
|
Provider credentials are encrypted with AES-256-GCM using the 32-byte key in
|
|
`GATEWAY_SECURITY_ENCRYPTION_KEY` and decrypted only in memory at request time.
|
|
|
|
## Configuration
|
|
|
|
All settings come from environment variables prefixed `GATEWAY_`. Dot-notation
|
|
config keys map to underscores in env (e.g. `oidc.client_id` →
|
|
`GATEWAY_OIDC_CLIENT_ID`). In development, a `.env` file in the working directory
|
|
is loaded automatically; in production, set real environment variables.
|
|
|
|
| Variable | Default | Required | Notes |
|
|
|---|---|---|---|
|
|
| `GATEWAY_SERVER_PORT` | `8080` | no | HTTP listen port |
|
|
| `GATEWAY_SERVER_ENV` | `development` | no | `development` \| `production` (controls log format/level) |
|
|
| `GATEWAY_SERVER_PUBLIC_URL` | — | **yes** | canonical external URL, used for the OIDC redirect URI |
|
|
| `GATEWAY_DB_URL` | — | **yes** | `postgres://user:pass@host:5432/db` |
|
|
| `GATEWAY_DB_MAX_CONNS` | `20` | no | pool max connections |
|
|
| `GATEWAY_DB_MIN_CONNS` | `2` | no | pool min connections |
|
|
| `GATEWAY_DB_MAX_CONN_LIFETIME` | `30` | no | minutes |
|
|
| `GATEWAY_OIDC_ISSUER` | — | **yes** | OIDC issuer URL |
|
|
| `GATEWAY_OIDC_CLIENT_ID` | — | **yes** | OIDC client ID |
|
|
| `GATEWAY_OIDC_CLIENT_SECRET` | — | **yes** | OIDC client secret (confidential client) |
|
|
| `GATEWAY_OIDC_ADMIN_GROUP` | — | **yes** | group required for admin access |
|
|
| `GATEWAY_SECURITY_ENCRYPTION_KEY` | — | **yes** | 64 hex chars (32 bytes) for AES-256-GCM |
|
|
| `GATEWAY_CATALOG_SYNC_INTERVAL` | `30m` | no | how often provider model lists are refreshed |
|
|
| `GATEWAY_INFLUX_URL` | — | no | InfluxDB 3 base URL, e.g. `http://influxdb:8181`. Empty = exporter off |
|
|
| `GATEWAY_INFLUX_TOKEN` | — | no | InfluxDB 3 token, sent as a Bearer token |
|
|
| `GATEWAY_INFLUX_DATABASE` | — | with URL | target database |
|
|
| `GATEWAY_INFLUX_MEASUREMENT` | `lmr_request` | no | measurement name for request points |
|
|
| `GATEWAY_INFLUX_FLUSH_INTERVAL` | `10s` | no | max time a point waits in the buffer |
|
|
| `GATEWAY_INFLUX_BATCH_SIZE` | `500` | no | points per write request |
|
|
|
|
Config is validated on startup; missing required values or a wrong-length
|
|
encryption key cause the server to exit immediately.
|
|
|
|
### InfluxDB 3 export
|
|
|
|
Set `GATEWAY_INFLUX_URL` and `GATEWAY_INFLUX_DATABASE` to export one point per
|
|
proxied request. Points are batched in memory and written with the
|
|
`/api/v3/write_lp` endpoint. A full buffer drops points and logs a warning, so
|
|
the exporter never delays a response. Each point has these tags and fields:
|
|
|
|
| Kind | Name | Notes |
|
|
|---|---|---|
|
|
| tag | `provider`, `model`, `key_prefix` | group by provider, model address, or key. `key_prefix` is `(no key)` for the dashboard chat |
|
|
| tag | `user_id`, `provider_id`, `provider_model_id` | UUIDs, set when known |
|
|
| tag | `status_code`, `streamed` | HTTP status and `true`/`false` |
|
|
| field | `requests`, `errors` | always `1` and `0` or `1`, so `sum()` gives counts |
|
|
| field | `prompt_tokens`, `completion_tokens`, `cached_tokens`, `total_tokens` | integers |
|
|
| field | `cost` | dollars, float |
|
|
| field | `latency_ms`, `ttft_ms` | integers. `ttft_ms` is `0` when the response was not streamed |
|
|
| field | `request_id`, `finish_reason`, `error` | strings, empty when not known |
|
|
|
|
Every field is written on every point. InfluxDB 3 creates a column on its first
|
|
write, and a query that names a missing column fails, so optional values are
|
|
written as `0` or an empty string.
|
|
|
|
Generate an encryption key:
|
|
|
|
```bash
|
|
openssl rand -hex 32
|
|
```
|
|
|
|
## Running locally
|
|
|
|
Requires Go (see the `go` directive in [`backend/go.mod`](backend/go.mod)) and a
|
|
PostgreSQL 16 instance.
|
|
|
|
```bash
|
|
# 1. Start Postgres
|
|
docker run -d --name lmr-postgres \
|
|
-e POSTGRES_USER=gateway -e POSTGRES_PASSWORD=gateway -e POSTGRES_DB=gateway \
|
|
-p 5432:5432 postgres:16
|
|
|
|
# 2. Configure
|
|
cd backend
|
|
cp .env.example .env # fill in OIDC issuer/client + a 32-byte encryption key
|
|
|
|
# 3. Run — make loads .env, applies migrations, and starts the server
|
|
make run
|
|
```
|
|
|
|
The server listens on `GATEWAY_SERVER_PORT` (default `:8080`).
|
|
|
|
`.env` is consumed as **environment variables**: the Makefile loads it via
|
|
`include .env` / `export`, and viper reads `GATEWAY_*` from the environment. For
|
|
Docker/systemd, pass it with `env_file` / `EnvironmentFile` instead.
|
|
|
|
### Make targets
|
|
|
|
| Target | Action |
|
|
|---|---|
|
|
| `make run` | Run the gateway (migrations auto-applied on startup) |
|
|
| `make build` | Build the dashboard, then compile `bin/gateway` |
|
|
| `make ui` | Build the dashboard into the embedded asset dir |
|
|
| `make migrate-up` | Apply all pending migrations |
|
|
| `make migrate-down` | Roll back the most recent migration |
|
|
| `make test` / `make lint` / `make tidy` | Go tests / lint / module tidy |
|
|
|
|
### Health checks
|
|
|
|
| Endpoint | Meaning |
|
|
|---|---|
|
|
| `GET /healthz` | Liveness — process is up |
|
|
| `GET /readyz` | Readiness — database reachable (pings the pool) |
|
|
|
|
## Deploy
|
|
|
|
Any OCI image host works. Build the root `Dockerfile` and run the image with the
|
|
`GATEWAY_*` variables from [`.env.example`](.env.example). `compose.yml` runs it
|
|
next to your own Postgres, and `compose.bundled.yml` runs Postgres inside the
|
|
container for a single-host setup.
|
|
|
|
The files below are the author's own deployment. They are tracked as a worked
|
|
example. Replace the registry and hostnames with yours.
|
|
|
|
- `.woodpecker.yml` runs the tests on every push and pull request, then builds
|
|
the image and pushes it to a private registry on `master`.
|
|
- [`deploy/k8s/lmr.yaml`](deploy/k8s/lmr.yaml) runs the gateway on k3s. The
|
|
manifest expects `configmap/lmr-config`, `secret/lmr-secret` and a registry
|
|
pull secret to exist in the cluster. To roll out a new image:
|
|
|
|
```bash
|
|
kubectl apply -f deploy/k8s/lmr.yaml # only when the manifest changed
|
|
kubectl rollout restart deployment/localmodelroutex
|
|
kubectl rollout status deployment/localmodelroutex
|
|
```
|
|
|
|
`compose.yml` and `compose.bundled.yml` build from source for single-host use.
|
|
The bundled variant runs Postgres inside the same container.
|
|
|
|
## Database & migrations
|
|
|
|
SQL migrations live in [`backend/migrations/`](backend/migrations/) using
|
|
golang-migrate's `{version}_{name}.{up,down}.sql` naming. They are **embedded into
|
|
the binary** (`go:embed`) and **applied automatically on startup**, so a deployed
|
|
binary is self-contained — no migration files needed on the host.
|
|
|
|
To add a migration, create a new numbered pair, e.g.:
|
|
|
|
```
|
|
migrations/000002_virtual_keys.up.sql
|
|
migrations/000002_virtual_keys.down.sql
|
|
```
|
|
|
|
## Providers
|
|
|
|
| Name | Kind | Notes |
|
|
|---|---|---|
|
|
| `llamacpp` | local | OpenAI-compatible; works with a single llama-server, the llama.cpp router, or llama-swap; reads the running context size from `/props` during sync |
|
|
| `ollama` | local | OpenAI-compatible |
|
|
| `openrouter` | api key | OpenAI-compatible; HTTP-Referer/X-Title attribution; reports pricing; serves `/v1/images/generations` through chat completions |
|
|
| `openai` | api key | reads `cached_tokens` from the usage response |
|
|
| `anthropic` | api key | Messages API ↔ OpenAI translation, including streamed tool calls; injects `cache_control` on the system prompt + penultimate message |
|
|
| `github_copilot` | subscription | exchanges a GitHub token for a ~30-min Copilot token, mutex-guarded refresh |
|
|
|
|
## Model addressing
|
|
|
|
Models are addressed as **`<provider>/<slug>`**, where `<provider>` is the
|
|
configured provider instance name and `<slug>` is the upstream model id (which
|
|
may itself contain slashes, e.g. `openrouter/anthropic/claude-3.7-sonnet`). The
|
|
**catalog** is built at runtime by syncing each provider's model listing on a
|
|
timer and cached in memory.
|
|
|
|
- **`any/<slug>`** — a reserved prefix that fans out across every provider
|
|
serving that model and picks one by the configured **routing strategy**
|
|
(`round_robin` | `lowest_cost`, set from the dashboard Routing page).
|
|
- **Admin overrides** — per-model `enabled`/`blocked`, parameter overrides,
|
|
pricing, context window, rate limits, and a backup model live alongside the
|
|
discovered data and survive syncs. Clearing an override in the dashboard
|
|
falls back to the provider-reported value.
|
|
- **Backup models** — any model can name another model as its backup. When
|
|
the model is offline, rate limited, or its upstream fails before the client
|
|
has received anything, the gateway sends the same request to the backup and
|
|
returns the answer under the requested model's name. The client sees no
|
|
difference. The request log records each failed attempt, the address that
|
|
served the request, and the backup's cost.
|
|
- **Curated aliases** — the schema has `models` / `model_targets` tables for
|
|
friendly names with priority/weight failover. No code uses them yet.
|
|
|
|
## Frontend
|
|
|
|
The management UI is a Vite + React + TypeScript project in
|
|
[`backend/dashboard/`](backend/dashboard/). Its production build is written into
|
|
`backend/internal/web/dist/` (set via `build.outDir` in
|
|
[`vite.config.ts`](backend/dashboard/vite.config.ts)) and embedded into the server
|
|
binary, which serves it as a single-page app with history-API fallback (API
|
|
routes take precedence).
|
|
|
|
```bash
|
|
cd backend/dashboard
|
|
npm install
|
|
npm run build # type-checks (tsc), then outputs to ../internal/web/dist
|
|
```
|
|
|
|
During development, run the Vite dev server alongside the gateway:
|
|
|
|
```bash
|
|
npm run dev # serves the UI on http://localhost:5173 with hot reload
|
|
```
|
|
|
|
The dev server proxies `/v1`, `/api`, and `/auth` to `http://localhost:8080`
|
|
(configured under `server.proxy` in `vite.config.ts`), so the running gateway
|
|
backs the UI in development.
|
|
|
|
## Project status
|
|
|
|
Implemented end to end and verified against Postgres: configuration, migrations,
|
|
the OIDC login flow, virtual keys with model allowlists, the `/v1` proxy
|
|
(chat completions with streaming, embeddings, image generation, model listing), provider dispatch
|
|
for all six provider types, the runtime catalog with `any/<slug>` routing,
|
|
rate limits, request and content logging with retention, billing, and the
|
|
management dashboard.
|
|
|
|
Not built yet: curated aliases (`models` / `model_targets`), group-scoped
|
|
session keys, and per-key budgets.
|
|
|
|
## Contributing
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the test and
|
|
lint commands, and the pull request checklist. Report security problems as
|
|
described in [SECURITY.md](SECURITY.md), not in public issues.
|
|
|
|
## License
|
|
|
|
LMR is released under the GNU Affero General Public License, version 3. See
|
|
[LICENSE](LICENSE). If you run a modified LMR as a service for others, the AGPL
|
|
requires you to offer them the source of your modified version.
|