localmodelroutex/README.md

LMR — Local Model Route

An OpenAI-compatible LLM gateway. It puts a single, authenticated, observable endpoint in front of any number of model backends — local servers and commercial APIs alike — and adds virtual-key auth, provider routing, request logging, usage-based billing, and a management UI.

Meant to run locally in a homelab environment as a way to securely-ish expose your local models and back them up with the big boys.

Clients talk to it exactly as they would to OpenAI (POST /v1/chat/completions, SSE streaming included); the gateway handles credentials, routing, and accounting behind the scenes.

Features

• OpenAI-compatible proxy — drop-in /v1/chat/completions with streaming.
• Virtual keys — issue, scope, revoke, and expire sk-… keys per user/group, each with its own model allowlist.
• Multi-provider routing — local (llama.cpp / Ollama) and API providers (OpenAI, Anthropic, GitHub Copilot) behind one interface.
• Provider abstraction — per-provider quirks handled internally (Anthropic cache_control injection, OpenAI cached_tokens, Copilot token exchange).
• Usage & billing — per-request cost from a pricing table, written asynchronously so it never blocks the request path.
• OIDC admin auth — Authentik / any OIDC provider, group-gated admin access.
• Encrypted secrets — provider credentials stored AES-256-GCM encrypted at rest.
• Single binary — the React management UI is embedded via go:embed.

Stack

• Go (chi, pgx/v5, viper, go-oidc, golang-migrate) — see backend/go.mod
• PostgreSQL 16 — keys, request logs, pricing
• React + TypeScript (Vite) — management UI, built into the server binary

Repo layout

.
├── backend/                 # Go module: git.govekar.net/LovroG05/localmodelroutex
│   ├── cmd/server/          # entrypoint: config → db → migrate → HTTP server
│   ├── internal/
│   │   ├── api/             # HTTP handlers + middleware (proxy, admin, auth)
│   │   ├── providers/       # modular provider layer (registry + capability ifaces)
│   │   │   ├── openaicompat/ #   shared OpenAI-compatible transport
│   │   │   ├── openai|openrouter|llamacpp|ollama|copilot/  # OpenAI-compatible
│   │   │   ├── anthropic/   #   Messages API + OpenAI<->Anthropic translation
│   │   │   └── all/         #   blank-import aggregator (self-registration)
│   │   ├── catalog/         # runtime model library: cache, sync, any/<slug> routing
│   │   ├── store/           # hand-written pgx data-access layer
│   │   ├── crypto/          # AES-256-GCM seal/open for provider secrets
│   │   ├── config/          # viper config, GATEWAY_ env prefix
│   │   ├── db/              # pgxpool setup + migration runner
│   │   ├── billing/         # cost calculation + async usage logging
│   │   └── web/             # go:embed of the built UI, served as SPA
│   ├── migrations/          # golang-migrate SQL, applied automatically at startup
│   └── dashboard/           # Vite + React TS project (builds into internal/web/dist)
└── README.md

Authentication

Two auth systems that never mix:

Surface	Header	Verification
Proxy routes (/v1/*)	Bearer sk-xxx	sha256 of the key → lookup in virtual_keys; check revoked / expired / model allowlist
Admin routes (/api/admin/*)	Bearer <OIDC JWT>	go-oidc Verify() → groups claim → must contain the admin group

Session keys: POST /auth/exchange trades an Authentik OIDC JWT for an sk-sess-xxx virtual key scoped to that group's model allowlist.

Provider credentials are encrypted with AES-256-GCM using the 32-byte key in GATEWAY_SECURITY_ENCRYPTION_KEY and decrypted only in memory at request time.

Configuration

All settings come from environment variables prefixed GATEWAY_. Dot-notation config keys map to underscores in env (e.g. oidc.client_id → GATEWAY_OIDC_CLIENT_ID). In development, a .env file in the working directory is loaded automatically; in production, set real environment variables.

Variable	Default	Required	Notes
GATEWAY_SERVER_PORT	8080	no	HTTP listen port
GATEWAY_SERVER_ENV	development	no	development | production (controls log format/level)
GATEWAY_DB_URL	—	yes	postgres://user:pass@host:5432/db
GATEWAY_DB_MAX_CONNS	20	no	pool max connections
GATEWAY_DB_MIN_CONNS	2	no	pool min connections
GATEWAY_DB_MAX_CONN_LIFETIME	30	no	minutes
GATEWAY_OIDC_ISSUER	—	yes	OIDC issuer URL
GATEWAY_OIDC_CLIENT_ID	—	yes	OIDC client ID
GATEWAY_OIDC_ADMIN_GROUP	—	yes	group required for admin access
GATEWAY_SECURITY_ENCRYPTION_KEY	—	yes	64 hex chars (32 bytes) for AES-256-GCM

Config is validated on startup; missing required values or a wrong-length encryption key cause the server to exit immediately.

Generate an encryption key:

openssl rand -hex 32

Running locally

Requires Go (see the go directive in backend/go.mod) and a PostgreSQL 16 instance.

# 1. Start Postgres
docker run -d --name lmr-postgres \
  -e POSTGRES_USER=gateway -e POSTGRES_PASSWORD=gateway -e POSTGRES_DB=gateway \
  -p 5432:5432 postgres:16

# 2. Configure
cd backend
cp .env.example .env   # fill in OIDC issuer/client + a 32-byte encryption key

# 3. Run — make loads .env, applies migrations, and starts the server
make run

The server listens on GATEWAY_SERVER_PORT (default :8080).

.env is consumed as environment variables: the Makefile loads it via include .env / export, and viper reads GATEWAY_* from the environment. For Docker/systemd, pass it with env_file / EnvironmentFile instead.

Make targets

Target	Action
make run	Run the gateway (migrations auto-applied on startup)
make build	Build the dashboard, then compile bin/gateway
make ui	Build the dashboard into the embedded asset dir
make migrate-up	Apply all pending migrations
make migrate-down	Roll back the most recent migration
make test / make lint / make tidy	Go tests / lint / module tidy

Health checks

Endpoint	Meaning
GET /healthz	Liveness — process is up
GET /readyz	Readiness — database reachable (pings the pool)

Database & migrations

SQL migrations live in backend/migrations/ using golang-migrate's {version}_{name}.{up,down}.sql naming. They are embedded into the binary (go:embed) and applied automatically on startup, so a deployed binary is self-contained — no migration files needed on the host.

To add a migration, create a new numbered pair, e.g.:

migrations/000002_virtual_keys.up.sql
migrations/000002_virtual_keys.down.sql

Providers

Name	Kind	Notes
llamacpp	local	OpenAI-compatible; routes through llama-swap
ollama	local	OpenAI-compatible
openrouter	api key	OpenAI-compatible; HTTP-Referer/X-Title attribution; reports pricing
openai	api key	reads cached_tokens from the usage response
anthropic	api key	Messages API ↔ OpenAI translation; injects cache_control on the system prompt + penultimate message
github-copilot	subscription	exchanges a GitHub PAT for a ~30-min Copilot token, mutex-guarded refresh

Model addressing

Models are addressed as <provider>/<slug>, where <provider> is the configured provider instance name and <slug> is the upstream model id (which may itself contain slashes, e.g. openrouter/anthropic/claude-3.7-sonnet). The catalog is built at runtime by syncing each provider's model listing on a timer and cached in memory.

• any/<slug> — a reserved prefix that fans out across every provider serving that model and picks one by the configured routing strategy (round_robin | lowest_cost | priority, in settings).
• Curated aliases — admins can define a friendly name mapping to one or more catalog entries with priority/weight for local-first/cloud-backup failover.
• Admin overrides — per-model enabled/blocked, parameter overrides, pricing, and rate limits live alongside the discovered data and survive syncs.

Frontend

The management UI is a Vite + React + TypeScript project in backend/dashboard/. Its production build is written into backend/internal/web/dist/ (set via build.outDir in vite.config.ts) and embedded into the server binary, which serves it as a single-page app with history-API fallback (API routes take precedence).

cd backend/dashboard
npm install
npm run build      # type-checks (tsc), then outputs to ../internal/web/dist

During development, run the Vite dev server alongside the gateway:

npm run dev        # serves the UI on http://localhost:5173 with hot reload

The dev server proxies /v1, /api, and /auth to http://localhost:8080 (configured under server.proxy in vite.config.ts), so the running gateway backs the UI in development.

Project status

Implemented: configuration, the database pool + migration runner, the server entrypoint (logging, health checks, graceful shutdown), the embedded dashboard handler, secret encryption (internal/crypto), the modular provider layer (internal/providers — OpenAI, OpenRouter, llama.cpp, Ollama, Anthropic, GitHub Copilot, with chat streaming + non-streaming and usage accounting), and the runtime model catalog (internal/catalog + internal/store: discovery sync, in-memory cache, <provider>/<slug> + any/<slug> routing) — all verified against Postgres.

Still to build: wiring the catalog syncer into the running server (build live providers from DB rows + decrypt credentials), and the HTTP API (internal/api — virtual-key auth, the /v1 proxy that ties keys → catalog resolution → provider dispatch, admin CRUD for providers/models/overrides, OIDC exchange), billing writes into request_logs, and the dashboard UI itself.