docs: add dashboard screenshots and architecture notes
Describe the request path, the provider capability interfaces, the catalog, auth, and logging for contributors. Reword the deploy section so the author's registry and cluster read as an example.
This commit is contained in:
45
README.md
45
README.md
@@ -11,6 +11,21 @@ Clients talk to it exactly as they would to OpenAI (`POST /v1/chat/completions`,
|
||||
SSE streaming included); the gateway handles credentials, routing, and accounting
|
||||
behind the scenes.
|
||||
|
||||

|
||||
|
||||
<details>
|
||||
<summary>More screenshots</summary>
|
||||
|
||||
Models page, with discovered models, pricing and per-model overrides:
|
||||
|
||||

|
||||
|
||||
Request log, with per-request tokens, cost and latency:
|
||||
|
||||

|
||||
|
||||
</details>
|
||||
|
||||
## Features
|
||||
|
||||
- **OpenAI-compatible proxy** — drop-in `/v1/chat/completions` with streaming,
|
||||
@@ -157,13 +172,19 @@ Docker/systemd, pass it with `env_file` / `EnvironmentFile` instead.
|
||||
|
||||
## Deploy
|
||||
|
||||
A push to `master` runs the Woodpecker pipeline in `.woodpecker.yml`. It builds
|
||||
the root `Dockerfile` and pushes `registry.govekar.net/lovrog05/localmodelroutex`
|
||||
with the tags `latest` and the commit SHA.
|
||||
Any OCI image host works. Build the root `Dockerfile` and run the image with the
|
||||
`GATEWAY_*` variables from [`.env.example`](.env.example). `compose.yml` runs it
|
||||
next to your own Postgres, and `compose.bundled.yml` runs Postgres inside the
|
||||
container for a single-host setup.
|
||||
|
||||
The gateway runs on k3s from [`deploy/k8s/lmr.yaml`](deploy/k8s/lmr.yaml). The
|
||||
manifest expects `configmap/lmr-config`, `secret/lmr-secret` and the pull secret
|
||||
`regcred2` to exist in the cluster. To roll out a new image:
|
||||
The files below are the author's own deployment. They are tracked as a worked
|
||||
example. Replace the registry and hostnames with yours.
|
||||
|
||||
- `.woodpecker.yml` runs the tests on every push and pull request, then builds
|
||||
the image and pushes it to a private registry on `master`.
|
||||
- [`deploy/k8s/lmr.yaml`](deploy/k8s/lmr.yaml) runs the gateway on k3s. The
|
||||
manifest expects `configmap/lmr-config`, `secret/lmr-secret` and a registry
|
||||
pull secret to exist in the cluster. To roll out a new image:
|
||||
|
||||
```bash
|
||||
kubectl apply -f deploy/k8s/lmr.yaml # only when the manifest changed
|
||||
@@ -252,3 +273,15 @@ management dashboard.
|
||||
|
||||
Not built yet: curated aliases (`models` / `model_targets`), group-scoped
|
||||
session keys, and per-key budgets.
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, the test and
|
||||
lint commands, and the pull request checklist. Report security problems as
|
||||
described in [SECURITY.md](SECURITY.md), not in public issues.
|
||||
|
||||
## License
|
||||
|
||||
LMR is released under the GNU Affero General Public License, version 3. See
|
||||
[LICENSE](LICENSE). If you run a modified LMR as a service for others, the AGPL
|
||||
requires you to offer them the source of your modified version.
|
||||
|
||||
148
docs/ARCHITECTURE.md
Normal file
148
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Architecture
|
||||
|
||||
This page explains how a request moves through LMR and where to add things.
|
||||
Read the [README](../README.md) first for the feature list and the repo layout.
|
||||
|
||||
## One request, end to end
|
||||
|
||||
A client sends `POST /v1/chat/completions` with `Authorization: Bearer sk-...`
|
||||
and `"model": "openrouter/anthropic/claude-sonnet-4.5"`.
|
||||
|
||||
1. **Key auth** (`internal/api/middleware.go`). The middleware hashes the
|
||||
bearer token with SHA-256 and loads the matching virtual key with its
|
||||
allowlist rules. Revoked and expired keys stop here.
|
||||
2. **Resolve** (`internal/catalog`). The model address is split into
|
||||
`<provider>/<slug>` and looked up in the in-memory catalog cache. The
|
||||
reserved `any/<slug>` prefix picks one provider by the routing strategy.
|
||||
3. **Allowlist and limits** (`internal/api`). The key's rules are matched
|
||||
against the resolved address. Then the per-model RPM and TPM limits from
|
||||
the catalog entry are checked in `internal/ratelimit`.
|
||||
4. **Dispatch** (`internal/providermgr`, `internal/providers`). The provider
|
||||
instance is fetched by name and type-asserted for the capability the route
|
||||
needs, for example `ChatCompleter`. The request payload is copied, the
|
||||
model is swapped for the upstream slug, and admin parameter overrides are
|
||||
merged in.
|
||||
5. **Relay**. The provider sends the upstream call and returns either a body
|
||||
or an SSE stream. The handler writes it to the client unchanged, apart from
|
||||
adding a `created` timestamp when the upstream omits it.
|
||||
6. **Account** (`internal/billing`, `internal/store`). Token usage is parsed
|
||||
from the response or the final stream chunk. Cost is the upstream-reported
|
||||
dollar amount when the provider gives one, else tokens times the catalog
|
||||
prices. The log row and the prompt and completion text are written in a
|
||||
goroutine so they never delay the response.
|
||||
|
||||
`/v1/embeddings` and `/v1/images/generations` follow the same six steps with
|
||||
their own capability interfaces.
|
||||
|
||||
## Components
|
||||
|
||||
```
|
||||
cmd/server/main.go config → db → migrate → providers → catalog → HTTP
|
||||
internal/config viper, GATEWAY_* environment variables
|
||||
internal/db pgxpool and the golang-migrate runner
|
||||
internal/store hand-written SQL, one file per table group
|
||||
internal/crypto AES-256-GCM seal and open for provider credentials
|
||||
internal/providermgr builds live provider clients from DB rows
|
||||
internal/providers provider abstraction and the per-type packages
|
||||
internal/catalog model library: sync, cache, address resolution
|
||||
internal/ratelimit per-model fixed-window RPM and TPM counters
|
||||
internal/billing cost calculation
|
||||
internal/api chi routes: /v1 proxy, /api/admin, /auth
|
||||
internal/oidcauth OIDC discovery, ID token verification, group check
|
||||
internal/web go:embed of the built dashboard, served as a SPA
|
||||
dashboard/ Vite + React + TypeScript management UI
|
||||
migrations/ numbered SQL pairs, applied at startup
|
||||
```
|
||||
|
||||
## The provider abstraction
|
||||
|
||||
`providers.Provider` is small on purpose. It has a name, a type, and a list of
|
||||
capabilities. Everything a route needs lives in a capability interface:
|
||||
|
||||
| Interface | Route | Method |
|
||||
|---|---|---|
|
||||
| `ChatCompleter` | `/v1/chat/completions` | `ChatCompletion`, `ChatCompletionStream` |
|
||||
| `Embedder` | `/v1/embeddings` | `Embeddings` |
|
||||
| `ImageGenerator` | `/v1/images/generations` | `ImageGeneration` |
|
||||
| `ModelLister` | catalog sync | `ListModels` |
|
||||
|
||||
A handler type-asserts for the interface it needs and answers 501 when the
|
||||
provider lacks it. New modalities add a new interface and a new route. Existing
|
||||
providers do not change.
|
||||
|
||||
Each provider type is a package under `internal/providers/` that registers a
|
||||
factory in `init()`. `internal/providers/all` blank-imports every package so
|
||||
the server binary links them all. The registry maps the `type` column of the
|
||||
providers table to the factory.
|
||||
|
||||
The OpenAI-compatible types (OpenAI, OpenRouter, llama.cpp, Ollama, Copilot)
|
||||
embed `openaicompat.Transport`, which holds the HTTP client, base URL, auth
|
||||
hook, static headers, and the flags for usage and cost reporting. A type only
|
||||
adds what differs. OpenRouter adds attribution headers and the image
|
||||
translation. Copilot adds a token exchange in its auth hook. Anthropic is the
|
||||
one type that does not embed the transport. It translates between the OpenAI
|
||||
and Messages schemas in both directions, including streamed tool calls.
|
||||
|
||||
### Why the image route translates
|
||||
|
||||
OpenRouter has no `/images/generations` endpoint. Its image models generate
|
||||
through chat completions with `modalities: ["image", "text"]` and return the
|
||||
image in `message.images`. The OpenRouter provider's `ImageGeneration` builds
|
||||
that chat request, maps OpenAI `size` and `quality` to OpenRouter's
|
||||
`image_config`, and renders the reply in the OpenAI images schema. Clients
|
||||
written for the OpenAI Images API work unchanged.
|
||||
|
||||
## The catalog
|
||||
|
||||
Admins configure providers, not models. On a timer, and after every admin
|
||||
change, the syncer asks each provider for its model list and upserts the rows
|
||||
into `provider_models`. The syncer extracts what the provider reports: display
|
||||
name, context length, pricing, and output modalities. Output modalities become
|
||||
the row's capabilities, which is how the image route knows which models accept
|
||||
an image request.
|
||||
|
||||
Admin overrides live in separate columns on the same row and survive syncs:
|
||||
enabled, blocked, parameter overrides, prices, and rate limits. A model that
|
||||
disappears from the upstream list is marked unavailable, not deleted.
|
||||
|
||||
The cache holds every servable row in memory, keyed by address. It is rebuilt
|
||||
after each sync, so the request path never queries the database for routing.
|
||||
|
||||
## Authentication
|
||||
|
||||
Two systems that do not mix:
|
||||
|
||||
- **Proxy routes** under `/v1` take a virtual key. Keys are stored as hashes.
|
||||
Each key is either allow-all or carries a list of address patterns, where
|
||||
`openrouter/*` matches a whole provider.
|
||||
- **Admin and user routes** under `/api` take an OIDC ID token. The login flow
|
||||
is the authorization code flow. The server does the code exchange and hands
|
||||
the ID token to the dashboard. Admin access needs the configured group in
|
||||
the `groups` claim. The dashboard's test chat uses a synthetic allow-all key
|
||||
bound to the signed-in user, so it needs no real API key.
|
||||
|
||||
Provider credentials are sealed with AES-256-GCM using the 32-byte key from
|
||||
`GATEWAY_SECURITY_ENCRYPTION_KEY`. They are opened in memory when the provider
|
||||
manager builds a client and are never returned by the admin API.
|
||||
|
||||
## Logging and retention
|
||||
|
||||
Every proxy request writes one row to `request_logs` with the key prefix,
|
||||
model, provider, status, token counts, cost, latency, and time to first token
|
||||
for streams. The prompt and completion text go to a second table. Inline
|
||||
image data in prompts is replaced by a size marker before the write.
|
||||
|
||||
A daily job deletes rows older than the `log.retention_days` setting. Zero
|
||||
keeps everything. Settings are rows in a key-value table and are edited from
|
||||
the dashboard.
|
||||
|
||||
## Dashboard
|
||||
|
||||
The dashboard is a Vite project in `backend/dashboard`. `npm run build` writes
|
||||
to `backend/internal/web/dist`, and the Go binary embeds that folder. In
|
||||
development, Vite proxies `/api`, `/v1`, and `/auth` to the Go server on port
|
||||
8080, so the UI hot-reloads against real data.
|
||||
|
||||
Pages map to admin routes one to one: Providers, Models, Routing, API Keys,
|
||||
Logs, Analytics, Settings, and a Chat page for testing a model through the
|
||||
same proxy path that clients use.
|
||||
BIN
docs/screenshots/logs.png
Normal file
BIN
docs/screenshots/logs.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 380 KiB |
BIN
docs/screenshots/models.png
Normal file
BIN
docs/screenshots/models.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 336 KiB |
BIN
docs/screenshots/overview.png
Normal file
BIN
docs/screenshots/overview.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 259 KiB |
Reference in New Issue
Block a user