localmodelroutex/README.md

LMR — Local Model Route

An OpenAI-compatible LLM gateway. It puts a single, authenticated, observable endpoint in front of any number of model backends — local servers and commercial APIs alike — and adds virtual-key auth, provider routing, request logging, usage-based billing, and a management UI.

Meant to run locally in a homelab environment as a way to securely-ish expose your local models and back them up with the big boys.

Clients talk to it exactly as they would to OpenAI (POST /v1/chat/completions, SSE streaming included); the gateway handles credentials, routing, and accounting behind the scenes.

More screenshots

Models page, with discovered models, pricing and per-model overrides:

Request log, with per-request tokens, cost and latency:

Features

• OpenAI-compatible proxy — drop-in /v1/chat/completions with streaming, /v1/embeddings, and /v1/images/generations.
• Image generation — OpenAI Images API clients (such as Open WebUI's image setting) work against OpenRouter image models. The gateway translates the call to a chat completion with image output and back. Only models whose provider reports image output accept the call.
• Virtual keys — issue, scope, revoke, and expire sk-… keys per user/group, each with its own model allowlist.
• Multi-provider routing — local (llama.cpp / Ollama) and API providers (OpenAI, Anthropic, GitHub Copilot) behind one interface.
• Provider abstraction — per-provider quirks handled internally (Anthropic cache_control injection, OpenAI cached_tokens, Copilot token exchange).
• Usage & billing — per-request cost from a pricing table, written asynchronously so it never blocks the request path.
• OIDC admin auth — Authentik / any OIDC provider, group-gated admin access.
• Encrypted secrets — provider credentials stored AES-256-GCM encrypted at rest.
• Single binary — the React management UI is embedded via go:embed.

Stack

• Go (chi, pgx/v5, viper, go-oidc, golang-migrate) — see backend/go.mod
• PostgreSQL 16 — keys, request logs, pricing
• React + TypeScript (Vite) — management UI, built into the server binary

Repo layout

.
├── backend/                 # Go module: git.govekar.net/LovroG05/localmodelroutex
│   ├── cmd/server/          # entrypoint: config → db → migrate → HTTP server
│   ├── internal/
│   │   ├── api/             # HTTP handlers + middleware (proxy, admin, auth)
│   │   ├── providers/       # modular provider layer (registry + capability ifaces)
│   │   │   ├── openaicompat/ #   shared OpenAI-compatible transport
│   │   │   ├── openai|openrouter|llamacpp|ollama|copilot/  # OpenAI-compatible
│   │   │   ├── anthropic/   #   Messages API + OpenAI<->Anthropic translation
│   │   │   └── all/         #   blank-import aggregator (self-registration)
│   │   ├── catalog/         # runtime model library: cache, sync, any/<slug> routing
│   │   ├── store/           # hand-written pgx data-access layer
│   │   ├── crypto/          # AES-256-GCM seal/open for provider secrets
│   │   ├── config/          # viper config, GATEWAY_ env prefix
│   │   ├── db/              # pgxpool setup + migration runner
│   │   ├── billing/         # cost calculation + async usage logging
│   │   └── web/             # go:embed of the built UI, served as SPA
│   ├── migrations/          # golang-migrate SQL, applied automatically at startup
│   └── dashboard/           # Vite + React TS project (builds into internal/web/dist)
└── README.md

Authentication

Two auth systems that never mix:

Surface	Header	Verification
Proxy routes (/v1/*)	Bearer sk-xxx	sha256 of the key → lookup in virtual_keys; check revoked / expired / model allowlist
Admin routes (/api/admin/*)	Bearer <OIDC JWT>	go-oidc Verify() → groups claim → must contain the admin group

Dashboard login uses the OIDC authorization code flow: GET /auth/login redirects to the provider, and GET /auth/callback exchanges the code server side and hands the ID token to the SPA. Every signed-in user can create keys, see their own logs and use the test chat. Members of the admin group also manage providers, the model catalog, routing and settings.

Provider credentials are encrypted with AES-256-GCM using the 32-byte key in GATEWAY_SECURITY_ENCRYPTION_KEY and decrypted only in memory at request time.

Configuration

All settings come from environment variables prefixed GATEWAY_. Dot-notation config keys map to underscores in env (e.g. oidc.client_id → GATEWAY_OIDC_CLIENT_ID). In development, a .env file in the working directory is loaded automatically; in production, set real environment variables.

Variable	Default	Required	Notes
GATEWAY_SERVER_PORT	8080	no	HTTP listen port
GATEWAY_SERVER_ENV	development	no	development | production (controls log format/level)
GATEWAY_SERVER_PUBLIC_URL	—	yes	canonical external URL, used for the OIDC redirect URI
GATEWAY_DB_URL	—	yes	postgres://user:pass@host:5432/db
GATEWAY_DB_MAX_CONNS	20	no	pool max connections
GATEWAY_DB_MIN_CONNS	2	no	pool min connections
GATEWAY_DB_MAX_CONN_LIFETIME	30	no	minutes
GATEWAY_OIDC_ISSUER	—	yes	OIDC issuer URL
GATEWAY_OIDC_CLIENT_ID	—	yes	OIDC client ID
GATEWAY_OIDC_CLIENT_SECRET	—	yes	OIDC client secret (confidential client)
GATEWAY_OIDC_ADMIN_GROUP	—	yes	group required for admin access
GATEWAY_SECURITY_ENCRYPTION_KEY	—	yes	64 hex chars (32 bytes) for AES-256-GCM
GATEWAY_CATALOG_SYNC_INTERVAL	30m	no	how often provider model lists are refreshed
GATEWAY_INFLUX_URL	—	no	InfluxDB 3 base URL, e.g. http://influxdb:8181. Empty = exporter off
GATEWAY_INFLUX_TOKEN	—	no	InfluxDB 3 token, sent as a Bearer token
GATEWAY_INFLUX_DATABASE	—	with URL	target database
GATEWAY_INFLUX_MEASUREMENT	lmr_request	no	measurement name for request points
GATEWAY_INFLUX_FLUSH_INTERVAL	10s	no	max time a point waits in the buffer
GATEWAY_INFLUX_BATCH_SIZE	500	no	points per write request

Config is validated on startup; missing required values or a wrong-length encryption key cause the server to exit immediately.

InfluxDB 3 export

Set GATEWAY_INFLUX_URL and GATEWAY_INFLUX_DATABASE to export one point per proxied request. Points are batched in memory and written with the /api/v3/write_lp endpoint. A full buffer drops points and logs a warning, so the exporter never delays a response. Each point has these tags and fields:

Kind	Name	Notes
tag	provider, model, key_prefix	group by provider, model address, or key
tag	user_id, provider_id, provider_model_id	UUIDs, set when known
tag	status_code, streamed	HTTP status and true/false
field	requests, errors	always 1 and 0 or 1, so sum() gives counts
field	prompt_tokens, completion_tokens, cached_tokens, total_tokens	integers
field	cost	dollars, float
field	latency_ms, ttft_ms	integers, ttft_ms only for streamed chat
field	request_id, finish_reason, error	strings, set when known

Generate an encryption key:

openssl rand -hex 32

Running locally

Requires Go (see the go directive in backend/go.mod) and a PostgreSQL 16 instance.

# 1. Start Postgres
docker run -d --name lmr-postgres \
  -e POSTGRES_USER=gateway -e POSTGRES_PASSWORD=gateway -e POSTGRES_DB=gateway \
  -p 5432:5432 postgres:16

# 2. Configure
cd backend
cp .env.example .env   # fill in OIDC issuer/client + a 32-byte encryption key

# 3. Run — make loads .env, applies migrations, and starts the server
make run

The server listens on GATEWAY_SERVER_PORT (default :8080).

.env is consumed as environment variables: the Makefile loads it via include .env / export, and viper reads GATEWAY_* from the environment. For Docker/systemd, pass it with env_file / EnvironmentFile instead.

Make targets

Target	Action
make run	Run the gateway (migrations auto-applied on startup)
make build	Build the dashboard, then compile bin/gateway
make ui	Build the dashboard into the embedded asset dir
make migrate-up	Apply all pending migrations
make migrate-down	Roll back the most recent migration
make test / make lint / make tidy	Go tests / lint / module tidy

Health checks

Endpoint	Meaning
GET /healthz	Liveness — process is up
GET /readyz	Readiness — database reachable (pings the pool)

Deploy

Any OCI image host works. Build the root Dockerfile and run the image with the GATEWAY_* variables from .env.example. compose.yml runs it next to your own Postgres, and compose.bundled.yml runs Postgres inside the container for a single-host setup.

The files below are the author's own deployment. They are tracked as a worked example. Replace the registry and hostnames with yours.

• .woodpecker.yml runs the tests on every push and pull request, then builds the image and pushes it to a private registry on master.
• deploy/k8s/lmr.yaml runs the gateway on k3s. The manifest expects configmap/lmr-config, secret/lmr-secret and a registry pull secret to exist in the cluster. To roll out a new image:

kubectl apply -f deploy/k8s/lmr.yaml                 # only when the manifest changed
kubectl rollout restart deployment/localmodelroutex
kubectl rollout status deployment/localmodelroutex

compose.yml and compose.bundled.yml build from source for single-host use. The bundled variant runs Postgres inside the same container.

Database & migrations

SQL migrations live in backend/migrations/ using golang-migrate's {version}_{name}.{up,down}.sql naming. They are embedded into the binary (go:embed) and applied automatically on startup, so a deployed binary is self-contained — no migration files needed on the host.

To add a migration, create a new numbered pair, e.g.:

migrations/000002_virtual_keys.up.sql
migrations/000002_virtual_keys.down.sql

Providers

Name	Kind	Notes
llamacpp	local	OpenAI-compatible; works with a single llama-server, the llama.cpp router, or llama-swap; reads the running context size from /props during sync
ollama	local	OpenAI-compatible
openrouter	api key	OpenAI-compatible; HTTP-Referer/X-Title attribution; reports pricing; serves /v1/images/generations through chat completions
openai	api key	reads cached_tokens from the usage response
anthropic	api key	Messages API ↔ OpenAI translation, including streamed tool calls; injects cache_control on the system prompt + penultimate message
github_copilot	subscription	exchanges a GitHub token for a ~30-min Copilot token, mutex-guarded refresh

Model addressing

Models are addressed as <provider>/<slug>, where <provider> is the configured provider instance name and <slug> is the upstream model id (which may itself contain slashes, e.g. openrouter/anthropic/claude-3.7-sonnet). The catalog is built at runtime by syncing each provider's model listing on a timer and cached in memory.

• any/<slug> — a reserved prefix that fans out across every provider serving that model and picks one by the configured routing strategy (round_robin | lowest_cost, set from the dashboard Routing page).
• Admin overrides — per-model enabled/blocked, parameter overrides, pricing, context window, and rate limits live alongside the discovered data and survive syncs. Clearing an override in the dashboard falls back to the provider-reported value.
• Curated aliases — the schema has models / model_targets tables for friendly names with priority/weight failover. No code uses them yet.

Frontend

The management UI is a Vite + React + TypeScript project in backend/dashboard/. Its production build is written into backend/internal/web/dist/ (set via build.outDir in vite.config.ts) and embedded into the server binary, which serves it as a single-page app with history-API fallback (API routes take precedence).

cd backend/dashboard
npm install
npm run build      # type-checks (tsc), then outputs to ../internal/web/dist

During development, run the Vite dev server alongside the gateway:

npm run dev        # serves the UI on http://localhost:5173 with hot reload

The dev server proxies /v1, /api, and /auth to http://localhost:8080 (configured under server.proxy in vite.config.ts), so the running gateway backs the UI in development.

Project status

Implemented end to end and verified against Postgres: configuration, migrations, the OIDC login flow, virtual keys with model allowlists, the /v1 proxy (chat completions with streaming, embeddings, image generation, model listing), provider dispatch for all six provider types, the runtime catalog with any/<slug> routing, rate limits, request and content logging with retention, billing, and the management dashboard.

Not built yet: curated aliases (models / model_targets), group-scoped session keys, and per-key budgets.

Contributing

See CONTRIBUTING.md for the development setup, the test and lint commands, and the pull request checklist. Report security problems as described in SECURITY.md, not in public issues.

License

LMR is released under the GNU Affero General Public License, version 3. See LICENSE. If you run a modified LMR as a service for others, the AGPL requires you to offer them the source of your modified version.