localmodelroutex/README.md

LMR — Local Model Route

An OpenAI-compatible LLM gateway. It puts a single, authenticated, observable endpoint in front of any number of model backends — local servers and commercial APIs alike — and adds virtual-key auth, provider routing, request logging, usage-based billing, and a management UI.

Meant to run locally in a homelab environment as a way to securely-ish expose your local models and back them up with the big boys.

Clients talk to it exactly as they would to OpenAI (POST /v1/chat/completions, SSE streaming included); the gateway handles credentials, routing, and accounting behind the scenes.

Features

• OpenAI-compatible proxy — drop-in /v1/chat/completions with streaming.
• Virtual keys — issue, scope, revoke, and expire sk-… keys per user/group, each with its own model allowlist.
• Multi-provider routing — local (llama.cpp / Ollama) and API providers (OpenAI, Anthropic, GitHub Copilot) behind one interface.
• Provider abstraction — per-provider quirks handled internally (Anthropic cache_control injection, OpenAI cached_tokens, Copilot token exchange).
• Usage & billing — per-request cost from a pricing table, written asynchronously so it never blocks the request path.
• OIDC admin auth — Authentik / any OIDC provider, group-gated admin access.
• Encrypted secrets — provider credentials stored AES-256-GCM encrypted at rest.
• Single binary — the React management UI is embedded via go:embed.

Stack

• Go (chi, pgx/v5, viper, go-oidc, golang-migrate) — see backend/go.mod
• PostgreSQL 16 — keys, request logs, pricing
• React + TypeScript (Vite) — management UI, built into the server binary

Repo layout

.
├── backend/                 # Go module: git.govekar.net/LovroG05/localmodelroutex
│   ├── cmd/server/          # entrypoint: config → db → migrate → HTTP server
│   ├── internal/
│   │   ├── api/             # HTTP handlers + middleware (proxy, admin, auth)
│   │   ├── providers/       # modular provider layer (registry + capability ifaces)
│   │   │   ├── openaicompat/ #   shared OpenAI-compatible transport
│   │   │   ├── openai|openrouter|llamacpp|ollama|copilot/  # OpenAI-compatible
│   │   │   ├── anthropic/   #   Messages API + OpenAI<->Anthropic translation
│   │   │   └── all/         #   blank-import aggregator (self-registration)
│   │   ├── catalog/         # runtime model library: cache, sync, any/<slug> routing
│   │   ├── store/           # hand-written pgx data-access layer
│   │   ├── crypto/          # AES-256-GCM seal/open for provider secrets
│   │   ├── config/          # viper config, GATEWAY_ env prefix
│   │   ├── db/              # pgxpool setup + migration runner
│   │   ├── billing/         # cost calculation + async usage logging
│   │   └── web/             # go:embed of the built UI, served as SPA
│   ├── migrations/          # golang-migrate SQL, applied automatically at startup
│   └── dashboard/           # Vite + React TS project (builds into internal/web/dist)
└── README.md

Authentication

Two auth systems that never mix:

Surface	Header	Verification
Proxy routes (/v1/*)	Bearer sk-xxx	sha256 of the key → lookup in virtual_keys; check revoked / expired / model allowlist
Admin routes (/api/admin/*)	Bearer <OIDC JWT>	go-oidc Verify() → groups claim → must contain the admin group

Dashboard login uses the OIDC authorization code flow: GET /auth/login redirects to the provider, and GET /auth/callback exchanges the code server side and hands the ID token to the SPA. Every signed-in user can create keys, see their own logs and use the test chat. Members of the admin group also manage providers, the model catalog, routing and settings.

Provider credentials are encrypted with AES-256-GCM using the 32-byte key in GATEWAY_SECURITY_ENCRYPTION_KEY and decrypted only in memory at request time.

Configuration

All settings come from environment variables prefixed GATEWAY_. Dot-notation config keys map to underscores in env (e.g. oidc.client_id → GATEWAY_OIDC_CLIENT_ID). In development, a .env file in the working directory is loaded automatically; in production, set real environment variables.

Variable	Default	Required	Notes
GATEWAY_SERVER_PORT	8080	no	HTTP listen port
GATEWAY_SERVER_ENV	development	no	development | production (controls log format/level)
GATEWAY_SERVER_PUBLIC_URL	—	yes	canonical external URL, used for the OIDC redirect URI
GATEWAY_DB_URL	—	yes	postgres://user:pass@host:5432/db
GATEWAY_DB_MAX_CONNS	20	no	pool max connections
GATEWAY_DB_MIN_CONNS	2	no	pool min connections
GATEWAY_DB_MAX_CONN_LIFETIME	30	no	minutes
GATEWAY_OIDC_ISSUER	—	yes	OIDC issuer URL
GATEWAY_OIDC_CLIENT_ID	—	yes	OIDC client ID
GATEWAY_OIDC_CLIENT_SECRET	—	yes	OIDC client secret (confidential client)
GATEWAY_OIDC_ADMIN_GROUP	—	yes	group required for admin access
GATEWAY_SECURITY_ENCRYPTION_KEY	—	yes	64 hex chars (32 bytes) for AES-256-GCM
GATEWAY_CATALOG_SYNC_INTERVAL	30m	no	how often provider model lists are refreshed

Config is validated on startup; missing required values or a wrong-length encryption key cause the server to exit immediately.

Generate an encryption key:

openssl rand -hex 32

Running locally

Requires Go (see the go directive in backend/go.mod) and a PostgreSQL 16 instance.

# 1. Start Postgres
docker run -d --name lmr-postgres \
  -e POSTGRES_USER=gateway -e POSTGRES_PASSWORD=gateway -e POSTGRES_DB=gateway \
  -p 5432:5432 postgres:16

# 2. Configure
cd backend
cp .env.example .env   # fill in OIDC issuer/client + a 32-byte encryption key

# 3. Run — make loads .env, applies migrations, and starts the server
make run

The server listens on GATEWAY_SERVER_PORT (default :8080).

.env is consumed as environment variables: the Makefile loads it via include .env / export, and viper reads GATEWAY_* from the environment. For Docker/systemd, pass it with env_file / EnvironmentFile instead.

Make targets

Target	Action
make run	Run the gateway (migrations auto-applied on startup)
make build	Build the dashboard, then compile bin/gateway
make ui	Build the dashboard into the embedded asset dir
make migrate-up	Apply all pending migrations
make migrate-down	Roll back the most recent migration
make test / make lint / make tidy	Go tests / lint / module tidy

Health checks

Endpoint	Meaning
GET /healthz	Liveness — process is up
GET /readyz	Readiness — database reachable (pings the pool)

Deploy

A push to master runs the Woodpecker pipeline in .woodpecker.yml. It builds the root Dockerfile and pushes registry.govekar.net/lovrog05/localmodelroutex with the tags latest and the commit SHA.

The gateway runs on k3s from deploy/k8s/lmr.yaml. The manifest expects configmap/lmr-config, secret/lmr-secret and the pull secret regcred2 to exist in the cluster. To roll out a new image:

kubectl apply -f deploy/k8s/lmr.yaml                 # only when the manifest changed
kubectl rollout restart deployment/localmodelroutex
kubectl rollout status deployment/localmodelroutex

compose.yml and compose.bundled.yml build from source for single-host use. The bundled variant runs Postgres inside the same container.

Database & migrations

SQL migrations live in backend/migrations/ using golang-migrate's {version}_{name}.{up,down}.sql naming. They are embedded into the binary (go:embed) and applied automatically on startup, so a deployed binary is self-contained — no migration files needed on the host.

To add a migration, create a new numbered pair, e.g.:

migrations/000002_virtual_keys.up.sql
migrations/000002_virtual_keys.down.sql

Providers

Name	Kind	Notes
llamacpp	local	OpenAI-compatible; routes through llama-swap
ollama	local	OpenAI-compatible
openrouter	api key	OpenAI-compatible; HTTP-Referer/X-Title attribution; reports pricing
openai	api key	reads cached_tokens from the usage response
anthropic	api key	Messages API ↔ OpenAI translation, including streamed tool calls; injects cache_control on the system prompt + penultimate message
github_copilot	subscription	exchanges a GitHub token for a ~30-min Copilot token, mutex-guarded refresh

Model addressing

Models are addressed as <provider>/<slug>, where <provider> is the configured provider instance name and <slug> is the upstream model id (which may itself contain slashes, e.g. openrouter/anthropic/claude-3.7-sonnet). The catalog is built at runtime by syncing each provider's model listing on a timer and cached in memory.

• any/<slug> — a reserved prefix that fans out across every provider serving that model and picks one by the configured routing strategy (round_robin | lowest_cost, set from the dashboard Routing page).
• Admin overrides — per-model enabled/blocked, parameter overrides, pricing, and rate limits live alongside the discovered data and survive syncs. Clearing an override in the dashboard falls back to the provider-reported price.
• Curated aliases — the schema has models / model_targets tables for friendly names with priority/weight failover. No code uses them yet.

Frontend

The management UI is a Vite + React + TypeScript project in backend/dashboard/. Its production build is written into backend/internal/web/dist/ (set via build.outDir in vite.config.ts) and embedded into the server binary, which serves it as a single-page app with history-API fallback (API routes take precedence).

cd backend/dashboard
npm install
npm run build      # type-checks (tsc), then outputs to ../internal/web/dist

During development, run the Vite dev server alongside the gateway:

npm run dev        # serves the UI on http://localhost:5173 with hot reload

The dev server proxies /v1, /api, and /auth to http://localhost:8080 (configured under server.proxy in vite.config.ts), so the running gateway backs the UI in development.

Project status

Implemented end to end and verified against Postgres: configuration, migrations, the OIDC login flow, virtual keys with model allowlists, the /v1 proxy (chat completions with streaming, embeddings, model listing), provider dispatch for all six provider types, the runtime catalog with any/<slug> routing, rate limits, request and content logging with retention, billing, and the management dashboard.

Not built yet: curated aliases (models / model_targets), group-scoped session keys, and per-key budgets.