docs: update README
This commit is contained in:
46
README.md
46
README.md
@@ -40,7 +40,14 @@ behind the scenes.
|
||||
│ ├── cmd/server/ # entrypoint: config → db → migrate → HTTP server
|
||||
│ ├── internal/
|
||||
│ │ ├── api/ # HTTP handlers + middleware (proxy, admin, auth)
|
||||
│ │ ├── providers/ # provider abstraction (llamacpp, ollama, openai, …)
|
||||
│ │ ├── providers/ # modular provider layer (registry + capability ifaces)
|
||||
│ │ │ ├── openaicompat/ # shared OpenAI-compatible transport
|
||||
│ │ │ ├── openai|openrouter|llamacpp|ollama|copilot/ # OpenAI-compatible
|
||||
│ │ │ ├── anthropic/ # Messages API + OpenAI<->Anthropic translation
|
||||
│ │ │ └── all/ # blank-import aggregator (self-registration)
|
||||
│ │ ├── catalog/ # runtime model library: cache, sync, any/<slug> routing
|
||||
│ │ ├── store/ # hand-written pgx data-access layer
|
||||
│ │ ├── crypto/ # AES-256-GCM seal/open for provider secrets
|
||||
│ │ ├── config/ # viper config, GATEWAY_ env prefix
|
||||
│ │ ├── db/ # pgxpool setup + migration runner
|
||||
│ │ ├── billing/ # cost calculation + async usage logging
|
||||
@@ -157,10 +164,27 @@ migrations/000002_virtual_keys.down.sql
|
||||
|---|---|---|
|
||||
| `llamacpp` | local | OpenAI-compatible; routes through llama-swap |
|
||||
| `ollama` | local | OpenAI-compatible |
|
||||
| `openrouter` | api key | OpenAI-compatible; HTTP-Referer/X-Title attribution; reports pricing |
|
||||
| `openai` | api key | reads `cached_tokens` from the usage response |
|
||||
| `anthropic` | api key | injects `cache_control` on the system prompt + penultimate message |
|
||||
| `anthropic` | api key | Messages API ↔ OpenAI translation; injects `cache_control` on the system prompt + penultimate message |
|
||||
| `github-copilot` | subscription | exchanges a GitHub PAT for a ~30-min Copilot token, mutex-guarded refresh |
|
||||
|
||||
## Model addressing
|
||||
|
||||
Models are addressed as **`<provider>/<slug>`**, where `<provider>` is the
|
||||
configured provider instance name and `<slug>` is the upstream model id (which
|
||||
may itself contain slashes, e.g. `openrouter/anthropic/claude-3.7-sonnet`). The
|
||||
**catalog** is built at runtime by syncing each provider's model listing on a
|
||||
timer and cached in memory.
|
||||
|
||||
- **`any/<slug>`** — a reserved prefix that fans out across every provider
|
||||
serving that model and picks one by the configured **routing strategy**
|
||||
(`round_robin` | `lowest_cost` | `priority`, in `settings`).
|
||||
- **Curated aliases** — admins can define a friendly name mapping to one or more
|
||||
catalog entries with priority/weight for local-first/cloud-backup failover.
|
||||
- **Admin overrides** — per-model `enabled`/`blocked`, parameter overrides,
|
||||
pricing, and rate limits live alongside the discovered data and survive syncs.
|
||||
|
||||
## Frontend
|
||||
|
||||
The management UI is a Vite + React + TypeScript project in
|
||||
@@ -188,7 +212,17 @@ backs the UI in development.
|
||||
|
||||
## Project status
|
||||
|
||||
Early scaffolding. Implemented so far: configuration, the database connection
|
||||
pool + migration runner, and the server entrypoint (logging, health checks,
|
||||
graceful shutdown). The `api`, `providers`, `billing`, and `web`
|
||||
packages and the `dashboard/` UI are still to be built out.
|
||||
Implemented: configuration, the database pool + migration runner, the server
|
||||
entrypoint (logging, health checks, graceful shutdown), the embedded dashboard
|
||||
handler, secret encryption (`internal/crypto`), the modular provider layer
|
||||
(`internal/providers` — OpenAI, OpenRouter, llama.cpp, Ollama, Anthropic, GitHub
|
||||
Copilot, with chat streaming + non-streaming and usage accounting), and the
|
||||
runtime model **catalog** (`internal/catalog` + `internal/store`: discovery
|
||||
sync, in-memory cache, `<provider>/<slug>` + `any/<slug>` routing) — all
|
||||
verified against Postgres.
|
||||
|
||||
Still to build: wiring the catalog syncer into the running server (build live
|
||||
providers from DB rows + decrypt credentials), and the HTTP API (`internal/api`
|
||||
— virtual-key auth, the `/v1` proxy that ties keys → catalog resolution →
|
||||
provider dispatch, admin CRUD for providers/models/overrides, OIDC exchange),
|
||||
billing writes into `request_logs`, and the dashboard UI itself.
|
||||
|
||||
Reference in New Issue
Block a user