LovroG05/localmodelroutex
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

setup

Browse Source
This commit is contained in:
Lovro Govekar
2026-06-27 22:47:37 +02:00
parent 7d72606bb9
commit 2fa715d108
12 changed files with 442 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
README.md
View File

@@ -3,7 +3,7 @@
An OpenAI-compatible **LLM gateway**. It puts a single, authenticated, observable
endpoint in front of any number of model backends — local servers and commercial
APIs alike — and adds virtual-key auth, provider routing, request logging,
response caching, usage-based billing, and a management UI.
usage-based billing, and a management UI.
Meant to run locally in a homelab environment as a way to securely-ish expose your local models and back them up with the big boys.
@@ -20,7 +20,6 @@ behind the scenes.
(OpenAI, Anthropic, GitHub Copilot) behind one interface.
- **Provider abstraction** — per-provider quirks handled internally (Anthropic
`cache_control` injection, OpenAI `cached_tokens`, Copilot token exchange).
- **Response cache** — exact-match (sha256) and semantic (pgvector ANN) caching.
- **Usage & billing** — per-request cost from a pricing table, written
asynchronously so it never blocks the request path.
- **OIDC admin auth** — Authentik / any OIDC provider, group-gated admin access.
@@ -30,7 +29,7 @@ behind the scenes.
## Stack
- **Go** (chi, pgx/v5, viper, go-oidc, golang-migrate) — see [`backend/go.mod`](backend/go.mod)
- **PostgreSQL 16** + **pgvector** — keys, request logs, cache, pricing
- **PostgreSQL 16** — keys, request logs, pricing
- **React + TypeScript** (Vite) — management UI, built into the server binary
## Repo layout
@@ -45,7 +44,6 @@ behind the scenes.
│ │ ├── config/ # viper config, GATEWAY_ env prefix
│ │ ├── db/ # pgxpool setup + migration runner
│ │ ├── billing/ # cost calculation + async usage logging
│ │ ├── cache/ # exact (sha256) + semantic (pgvector) response cache
│ │ └── web/ # go:embed of the built UI, served as SPA
│ ├── migrations/ # golang-migrate SQL, applied automatically at startup
│ └── dashboard/ # Vite + React TS project (builds into internal/web/dist)
@@ -99,24 +97,39 @@ openssl rand -hex 32
## Running locally
Requires Go (see the `go` directive in [`backend/go.mod`](backend/go.mod)) and a
PostgreSQL 16 instance with the `pgvector` extension available.
PostgreSQL 16 instance.
```bash
# 1. Start Postgres (pgvector image includes the extension)
# 1. Start Postgres
docker run -d --name lmr-postgres \
-e POSTGRES_USER=gateway -e POSTGRES_PASSWORD=gateway -e POSTGRES_DB=gateway \
-p 5432:5432 pgvector/pgvector:pg16
-p 5432:5432 postgres:16
# 2. Configure
cd backend
cp .env .env.local # or edit .env directly; fill in OIDC + encryption key
cp .env.example .env # fill in OIDC issuer/client + a 32-byte encryption key
# 3. Run — migrations are applied automatically on startup
go run ./cmd/server
# 3. Run — make loads .env, applies migrations, and starts the server
make run
```
The server listens on `GATEWAY_SERVER_PORT` (default `:8080`).
`.env` is consumed as **environment variables**: the Makefile loads it via
`include .env` / `export`, and viper reads `GATEWAY_*` from the environment. For
Docker/systemd, pass it with `env_file` / `EnvironmentFile` instead.
### Make targets
| Target | Action |
|---|---|
| `make run` | Run the gateway (migrations auto-applied on startup) |
| `make build` | Build the dashboard, then compile `bin/gateway` |
| `make ui` | Build the dashboard into the embedded asset dir |
| `make migrate-up` | Apply all pending migrations |
| `make migrate-down` | Roll back the most recent migration |
| `make test` / `make lint` / `make tidy` | Go tests / lint / module tidy |
### Health checks
| Endpoint | Meaning |
@@ -177,5 +190,5 @@ backs the UI in development.
Early scaffolding. Implemented so far: configuration, the database connection
pool + migration runner, and the server entrypoint (logging, health checks,
graceful shutdown). The `api`, `providers`, `billing`, `cache`, and `web`
graceful shutdown). The `api`, `providers`, `billing`, and `web`
packages and the `dashboard/` UI are still to be built out.

26
backend/.env.example Normal file
View File

@@ -0,0 +1,26 @@
# Copy to .env and fill in. Loaded into the environment by the Makefile / Docker;
# real GATEWAY_* environment variables take precedence over this file.
#
# NOTE: keep comments on their own lines (not trailing a KEY=VALUE) so the
# Makefile's `include .env` parses values cleanly.
# --- Server ---
GATEWAY_SERVER_PORT=8080
# development | production (controls log format/level)
GATEWAY_SERVER_ENV=development
# --- Database ---
GATEWAY_DB_URL=postgres://gateway:gateway@localhost:5432/gateway
GATEWAY_DB_MAX_CONNS=20
GATEWAY_DB_MIN_CONNS=2
# Connection max lifetime, in minutes
GATEWAY_DB_MAX_CONN_LIFETIME=30
# --- OIDC (Authentik or any OIDC provider) ---
GATEWAY_OIDC_ISSUER=https://auth.example.com/application/o/llmgateway/
GATEWAY_OIDC_CLIENT_ID=your-client-id
GATEWAY_OIDC_ADMIN_GROUP=llm-admins
# --- Security ---
# 64 hex chars (32 bytes) for AES-256-GCM. Generate with: openssl rand -hex 32
GATEWAY_SECURITY_ENCRYPTION_KEY=

7
backend/.gitignore vendored
View File

@@ -1,5 +1,8 @@
# Compiled server binary
/bin/
# Embedded UI build output (generated by `cd dashboard && npm run build`)
/internal/web/dist/
# Embedded UI build output: ignore generated assets but keep the directory (and
# its placeholder) so `//go:embed dist` still compiles on a fresh checkout.
/internal/web/dist/*
!/internal/web/dist/.gitkeep
.claude

41
backend/Makefile Normal file
View File

@@ -0,0 +1,41 @@
# Load GATEWAY_* config from .env (if present) into the environment for recipes.
# Real environment variables already set take precedence over the file.
-include .env
export
BINARY := bin/gateway
.DEFAULT_GOAL := help
.PHONY: help run build ui migrate-up migrate-down test lint tidy clean
help: ## List available targets
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(firstword $(MAKEFILE_LIST)) \
| awk 'BEGIN{FS=":.*?## "}{printf " \033[36m%-14s\033[0m %s\n", $$1, $$2}'
run: ## Run the gateway (migrations applied automatically on startup)
go run ./cmd/server
build: ui ## Build the dashboard, then compile the server binary
go build -o $(BINARY) ./cmd/server
ui: ## Build the dashboard into the embedded asset dir
cd dashboard && npm run build
migrate-up: ## Apply all pending migrations
go run ./cmd/migrate up
migrate-down: ## Roll back the most recent migration
go run ./cmd/migrate down
test: ## Run Go tests
go test ./...
lint: ## Run golangci-lint
golangci-lint run
tidy: ## Tidy go.mod / go.sum
go mod tidy
clean: ## Remove build artifacts (keeps the embed placeholder)
rm -rf $(BINARY)
find internal/web/dist -mindepth 1 ! -name .gitkeep -delete 2>/dev/null || true

45
backend/cmd/migrate/main.go Normal file
View File

@@ -0,0 +1,45 @@
// Command migrate applies or rolls back the embedded database migrations.
//
// Usage:
//
// migrate up # apply all pending migrations (default)
// migrate down # roll back the most recent migration
//
// It reads the database URL from GATEWAY_DB_URL; the Makefile targets load the
// .env file into the environment first.
package main
import (
"fmt"
"os"
"git.govekar.net/LovroG05/localmodelroutex/internal/db"
)
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, "migrate:", err)
os.Exit(1)
}
}
func run() error {
dbURL := os.Getenv("GATEWAY_DB_URL")
if dbURL == "" {
return fmt.Errorf("GATEWAY_DB_URL is not set")
}
cmd := "up"
if len(os.Args) > 1 {
cmd = os.Args[1]
}
switch cmd {
case "up":
return db.Migrate(dbURL)
case "down":
return db.MigrateDown(dbURL, 1)
default:
return fmt.Errorf("unknown command %q (expected: up | down)", cmd)
}
}

4
backend/cmd/server/main.go
View File

@@ -17,6 +17,7 @@ import (
"git.govekar.net/LovroG05/localmodelroutex/internal/config"
"git.govekar.net/LovroG05/localmodelroutex/internal/db"
"git.govekar.net/LovroG05/localmodelroutex/internal/web"
)
func main() {
@@ -124,5 +125,8 @@ func newRouter(pool *pgxpool.Pool) http.Handler {
_, _ = w.Write([]byte("ok"))
})
// Dashboard SPA — catch-all, registered last so API routes take precedence.
r.Handle("/*", web.Handler())
return r
}

10
backend/internal/config/config.go
View File

@@ -39,13 +39,9 @@ type SecurityConfig struct {
func Load() (*Config, error) {
v := viper.New()
// env file
v.SetConfigName(".env")
v.SetConfigType("env")
v.AddConfigPath(".")
v.ReadInConfig() // silent fail — not present in prod
// env vars take precedence, map GATEWAY_SERVER_PORT → server.port
// Config comes from GATEWAY_* environment variables. A .env file, when used,
// is loaded into the environment by the launcher (see the Makefile / Docker
// env_file), not parsed here. Maps GATEWAY_SERVER_PORT → server.port, etc.
v.SetEnvPrefix("GATEWAY")
v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
v.AutomaticEnv()

44
backend/internal/db/migrate.go
View File

@@ -20,17 +20,10 @@ import (
// the application pool, so migrations run with a clean session and the pool's
// lifecycle stays independent.
func Migrate(databaseURL string) error {
src, err := iofs.New(migrations.FS, ".")
m, err := newMigrator(databaseURL)
if err != nil {
return fmt.Errorf("load embedded migrations: %w", err)
return err
}
m, err := migrate.NewWithSourceInstance("iofs", src, pgxURL(databaseURL))
if err != nil {
_ = src.Close()
return fmt.Errorf("init migrator: %w", err)
}
// m.Close closes both the source and the database connection it opened.
defer m.Close()
if err := m.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
@@ -39,6 +32,39 @@ func Migrate(databaseURL string) error {
return nil
}
// MigrateDown rolls back the n most recent migrations. n <= 0 rolls back one.
func MigrateDown(databaseURL string, n int) error {
if n <= 0 {
n = 1
}
m, err := newMigrator(databaseURL)
if err != nil {
return err
}
defer m.Close()
if err := m.Steps(-n); err != nil && !errors.Is(err, migrate.ErrNoChange) {
return fmt.Errorf("roll back migrations: %w", err)
}
return nil
}
// newMigrator builds a migrator over the embedded migrations. The caller must
// Close it (which closes both the source and the database connection).
func newMigrator(databaseURL string) (*migrate.Migrate, error) {
src, err := iofs.New(migrations.FS, ".")
if err != nil {
return nil, fmt.Errorf("load embedded migrations: %w", err)
}
m, err := migrate.NewWithSourceInstance("iofs", src, pgxURL(databaseURL))
if err != nil {
_ = src.Close()
return nil, fmt.Errorf("init migrator: %w", err)
}
return m, nil
}
// pgxURL rewrites a standard postgres connection URL to the "pgx5" scheme that
// golang-migrate's pgx/v5 driver registers itself under. The driver rewrites
// the scheme back to "postgres" before dialing.

0
backend/internal/web/dist/.gitkeep vendored Normal file
View File

51
backend/internal/web/web.go Normal file
View File

@@ -0,0 +1,51 @@
// Package web serves the embedded dashboard single-page application.
package web
import (
"embed"
"io/fs"
"net/http"
"path"
"strings"
)
// dist holds the production build of the dashboard (backend/dashboard) emitted
// by `npm run build`. The `all:` prefix also embeds dotfiles, so the directory
// still resolves on a fresh checkout where only .gitkeep is present.
//
//go:embed all:dist
var dist embed.FS
// Handler serves the embedded SPA: existing files (JS/CSS/assets) are returned
// directly, and any other path falls back to index.html so client-side routing
// works. If the dashboard has not been built, UI routes return 503 while the
// rest of the gateway keeps serving.
func Handler() http.Handler {
root, err := fs.Sub(dist, "dist")
if err != nil {
panic("web: " + err.Error()) // unreachable: dist is embedded at build time
}
index, _ := fs.ReadFile(root, "index.html") // nil when the UI isn't built
files := http.FileServer(http.FS(root))
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
name := strings.TrimPrefix(path.Clean("/"+r.URL.Path), "/")
// Serve a real asset when one exists (never a directory listing).
if name != "" {
if info, statErr := fs.Stat(root, name); statErr == nil && !info.IsDir() {
files.ServeHTTP(w, r)
return
}
}
// Otherwise it's a client-side route: hand back index.html.
if index == nil {
http.Error(w, "dashboard not built", http.StatusServiceUnavailable)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, _ = w.Write(index)
})
}

11
backend/migrations/000002_core.down.sql Normal file
View File

@@ -0,0 +1,11 @@
-- Drop in reverse dependency order.
DROP TABLE IF EXISTS request_logs;
DROP TABLE IF EXISTS group_models;
DROP TABLE IF EXISTS virtual_key_models;
DROP TABLE IF EXISTS virtual_keys;
DROP TABLE IF EXISTS model_targets;
DROP TABLE IF EXISTS models;
DROP TABLE IF EXISTS providers;
DROP TABLE IF EXISTS groups;
DROP TABLE IF EXISTS users;
DROP FUNCTION IF EXISTS set_updated_at();

197
backend/migrations/000002_core.up.sql Normal file
View File

@@ -0,0 +1,197 @@
-- Core gateway schema: identities, providers/models/routing, virtual keys and
-- their allowlists, and the request log that billing reads from.
-- Shared trigger to keep updated_at columns current on UPDATE.
CREATE OR REPLACE FUNCTION set_updated_at()
RETURNS trigger AS $$
BEGIN
NEW.updated_at = now();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- ===========================================================================
-- Identity
-- ===========================================================================
-- Provisioned lazily from OIDC (Authentik) on first login/exchange. Admin
-- status is NOT stored here: it is derived live from the JWT groups claim
-- against GATEWAY_OIDC_ADMIN_GROUP.
CREATE TABLE users (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
subject text NOT NULL UNIQUE, -- OIDC `sub`
email text,
name text,
created_at timestamptz NOT NULL DEFAULT now(),
last_login_at timestamptz
);
CREATE INDEX idx_users_email ON users (email);
-- OIDC groups the gateway knows about. Membership is dynamic (read from the
-- JWT); this table exists to attach model allowlists / budgets to a group so
-- session keys minted via /auth/exchange can be scoped.
CREATE TABLE groups (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL UNIQUE, -- matches the OIDC group name
description text,
allow_all_models boolean NOT NULL DEFAULT false,
monthly_budget numeric(14,6), -- NULL = unlimited
rpm_limit integer, -- requests/min, NULL = unlimited
tpm_limit integer, -- tokens/min, NULL = unlimited
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TRIGGER trg_groups_updated_at BEFORE UPDATE ON groups
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
-- ===========================================================================
-- Providers, models, and routing targets
-- ===========================================================================
-- An upstream backend. Credentials are AES-256-GCM encrypted by the app
-- (nonce || ciphertext over a JSON blob) using GATEWAY_SECURITY_ENCRYPTION_KEY;
-- the database never sees plaintext.
CREATE TABLE providers (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL UNIQUE, -- admin-facing identifier
type text NOT NULL CHECK (type IN ('llamacpp','ollama','openai','anthropic','github_copilot')),
base_url text,
credentials bytea, -- NULL for keyless local providers
enabled boolean NOT NULL DEFAULT true,
config jsonb NOT NULL DEFAULT '{}'::jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TRIGGER trg_providers_updated_at BEFORE UPDATE ON providers
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
-- A public, client-facing model alias (what callers put in the request "model"
-- field). Allowlists reference these, not the backends behind them.
CREATE TABLE models (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL UNIQUE, -- e.g. "qwen3-coder", "gpt-4o"
display_name text,
type text NOT NULL DEFAULT 'chat' CHECK (type IN ('chat','completion','embedding')),
enabled boolean NOT NULL DEFAULT true,
params jsonb NOT NULL DEFAULT '{}'::jsonb, -- default sampling params
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TRIGGER trg_models_updated_at BEFORE UPDATE ON models
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
-- A concrete backend for a model. Multiple targets per model enable failover
-- ("local first, cloud backup"): lower priority wins, weight balances ties.
-- Pricing lives here because cost depends on which backend served the request.
CREATE TABLE model_targets (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
model_id uuid NOT NULL REFERENCES models (id) ON DELETE CASCADE,
provider_id uuid NOT NULL REFERENCES providers (id) ON DELETE CASCADE,
upstream_model text NOT NULL, -- model id sent to the provider
priority integer NOT NULL DEFAULT 100, -- lower = preferred
weight integer NOT NULL DEFAULT 1, -- tie-break load balancing
enabled boolean NOT NULL DEFAULT true,
context_length integer,
input_price_per_mtok numeric(12,6) NOT NULL DEFAULT 0, -- per 1M prompt tokens
output_price_per_mtok numeric(12,6) NOT NULL DEFAULT 0, -- per 1M completion tokens
cached_price_per_mtok numeric(12,6), -- per 1M cached tokens, NULL = input price
currency char(3) NOT NULL DEFAULT 'USD',
config jsonb NOT NULL DEFAULT '{}'::jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (model_id, provider_id, upstream_model)
);
CREATE INDEX idx_model_targets_model ON model_targets (model_id, priority);
CREATE INDEX idx_model_targets_provider ON model_targets (provider_id);
CREATE TRIGGER trg_model_targets_updated_at BEFORE UPDATE ON model_targets
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
-- ===========================================================================
-- Virtual keys and allowlists
-- ===========================================================================
CREATE TABLE virtual_keys (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text,
key_hash bytea NOT NULL UNIQUE, -- sha256(raw key)
key_prefix text NOT NULL, -- e.g. "sk-a1b2c3" for display
type text NOT NULL DEFAULT 'standard' CHECK (type IN ('standard','session')),
user_id uuid REFERENCES users (id) ON DELETE CASCADE,
group_id uuid REFERENCES groups (id) ON DELETE SET NULL, -- source group for session keys
allow_all_models boolean NOT NULL DEFAULT false,
monthly_budget numeric(14,6),
rpm_limit integer,
tpm_limit integer,
created_by uuid REFERENCES users (id) ON DELETE SET NULL,
created_at timestamptz NOT NULL DEFAULT now(),
last_used_at timestamptz,
expires_at timestamptz,
revoked_at timestamptz
);
CREATE INDEX idx_virtual_keys_user ON virtual_keys (user_id);
CREATE INDEX idx_virtual_keys_group ON virtual_keys (group_id);
CREATE INDEX idx_virtual_keys_active ON virtual_keys (revoked_at, expires_at);
-- Per-key model allowlist (ignored when allow_all_models = true).
CREATE TABLE virtual_key_models (
virtual_key_id uuid NOT NULL REFERENCES virtual_keys (id) ON DELETE CASCADE,
model_id uuid NOT NULL REFERENCES models (id) ON DELETE CASCADE,
PRIMARY KEY (virtual_key_id, model_id)
);
CREATE INDEX idx_virtual_key_models_model ON virtual_key_models (model_id);
-- Per-group model allowlist; the scope snapshot source for session keys
-- (ignored when groups.allow_all_models = true).
CREATE TABLE group_models (
group_id uuid NOT NULL REFERENCES groups (id) ON DELETE CASCADE,
model_id uuid NOT NULL REFERENCES models (id) ON DELETE CASCADE,
PRIMARY KEY (group_id, model_id)
);
CREATE INDEX idx_group_models_model ON group_models (model_id);
-- ===========================================================================
-- Request log (billing source of truth; written async after the response)
-- ===========================================================================
CREATE TABLE request_logs (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
created_at timestamptz NOT NULL DEFAULT now(),
-- Soft references: keep logs even after the referent is deleted.
virtual_key_id uuid REFERENCES virtual_keys (id) ON DELETE SET NULL,
user_id uuid REFERENCES users (id) ON DELETE SET NULL,
model_id uuid REFERENCES models (id) ON DELETE SET NULL,
target_id uuid REFERENCES model_targets (id) ON DELETE SET NULL,
provider_id uuid REFERENCES providers (id) ON DELETE SET NULL,
-- Snapshots so a row stays meaningful after deletes.
key_prefix text,
model_name text,
provider_name text,
-- Outcome.
status_code integer,
streamed boolean NOT NULL DEFAULT false,
finish_reason text,
error text,
-- Usage and cost (snapshotted at request time).
prompt_tokens integer NOT NULL DEFAULT 0,
completion_tokens integer NOT NULL DEFAULT 0,
cached_tokens integer NOT NULL DEFAULT 0,
total_tokens integer NOT NULL DEFAULT 0,
cost numeric(14,6) NOT NULL DEFAULT 0,
currency char(3) NOT NULL DEFAULT 'USD',
-- Timing.
latency_ms integer,
ttft_ms integer, -- time to first token (streaming)
-- Request metadata.
request_id text,
client_ip inet
);
CREATE INDEX idx_request_logs_created ON request_logs (created_at DESC);
CREATE INDEX idx_request_logs_key ON request_logs (virtual_key_id, created_at DESC);
CREATE INDEX idx_request_logs_user ON request_logs (user_id, created_at DESC);
CREATE INDEX idx_request_logs_model ON request_logs (model_id, created_at DESC);