docs: specify assistant model selector

2026-07-08 15:58:48 -05:00
parent 7c6b8d7c37
commit 1a58ef993e
1 changed files with 151 additions and 0 deletions
@@ -0,0 +1,151 @@
+# Assistant model selector design
+
+Date: 2026-07-08
+Status: approved for planning
+
+## Context
+
+The AI assistant chat currently uses one environment-configured model through `LLM_MODEL`.
+The chat UI posts messages to `/api/agent/chat`, and the server creates the OpenAI-compatible
+client without any request-time model choice.
+
+ginnoir wants a model selector in the assistant chat. The selector should discover available
+models from the configured OpenAI-compatible provider and save the selected model as the user's
+default.
+
+## Goals
+
+- Show a compact model selector in the assistant chat panel.
+- Discover models from the provider's `/models` endpoint server-side.
+- Persist the selected model per user so it works across browser sessions and devices.
+- Keep `LLM_MODEL` as the fallback when discovery fails, no model is saved, or the saved model
+  is no longer available.
+- Preserve mock-provider behavior in CI and local setups without `LLM_BASE_URL`.
+
+## Non-goals
+
+- Model hosting, training, or fine-tuning.
+- Multiple LLM providers in the same deployment.
+- Per-message experimental settings beyond selecting the model ID.
+- Exposing arbitrary browser-supplied model IDs to the provider.
+
+## User experience
+
+When the assistant bubble opens, the chat panel loads available model IDs from the server.
+The selector appears near the existing assistant status and clear-chat controls. It should be
+visible but compact enough not to reduce the message area materially.
+
+Changing the selector immediately saves the user's default model. The next message uses that
+model, and future assistant sessions start with the saved selection when it is still available.
+
+If model discovery fails, the panel remains usable with the `LLM_MODEL` fallback and shows a
+muted status that model discovery is unavailable. If the saved model has disappeared from the
+provider, the server and UI fall back to `LLM_MODEL`.
+
+## Architecture
+
+### Configuration
+
+`getLlmConfig()` remains the source for provider, base URL, API key, and fallback model.
+No additional allowlist environment variable is required because model IDs come from the
+provider's OpenAI-compatible `/models` endpoint.
+
+### Persistence
+
+Add nullable `assistant_model` storage to `users`.
+
+The existing assistant preference loader should return:
+
+- assistant enabled state
+- assistant display name
+- assistant system prompt
+- saved assistant model ID
+
+The value is nullable. `null` means "use the environment fallback model."
+
+### Model discovery API
+
+Add `GET /api/agent/models`.
+
+Behavior:
+
+- Require the same authenticated user/session or API auth shape as the chat endpoint.
+- Require assistant access to be enabled for the user.
+- If `LLM_BASE_URL` is missing or the provider is mock, return the fallback model as the only
+  available model.
+- Fetch `${LLM_BASE_URL}/models` with `Authorization: Bearer ${LLM_API_KEY}` when configured.
+- Accept OpenAI-style payloads with a top-level `data` array.
+- Normalize each model to `{ id: string, label: string }`, using the ID as the label.
+- Deduplicate, sort consistently, and include the fallback model if the provider omitted it.
+- If discovery fails, return the fallback model plus a degraded status instead of failing the
+  chat UI.
+
+The response should include enough metadata for the UI:
+
+```json
+{
+  "models": [{ "id": "llama3.2", "label": "llama3.2" }],
+  "selectedModel": "llama3.2",
+  "fallbackModel": "llama3.2",
+  "degraded": false
+}
+```
+
+### Saving the default model
+
+Add a server action for updating the user's assistant model, matching the existing assistant
+settings actions. The update path must:
+
+- Accept a model ID string or `null`.
+- Validate length and basic shape before touching the database.
+- Validate the requested model against the current discovered model list.
+- Save `null` when the selected model matches the fallback so `LLM_MODEL` changes take effect for
+  users who have not chosen a non-default model.
+- Revalidate assistant surfaces after saving.
+
+### Chat request flow
+
+Extend `clientChatInputSchema` with optional `model`.
+
+The chat route should:
+
+- Parse `model` from the request body.
+- Resolve the effective model from request model, saved user default, and fallback model.
+- Validate request model and saved user default against discovered models.
+- Reject an invalid request model with `400`.
+- Silently fall back when the saved user default is no longer available.
+- Pass the effective model into `runAgentChat`.
+
+`runAgentChat` should accept an optional model override. `createLlmClient` should support an
+override object or equivalent path that replaces only the model while preserving the configured
+provider, base URL, and API key.
+
+## Error handling
+
+- Missing auth: `401`.
+- Assistant disabled: `403`.
+- Invalid posted model: `400`.
+- Provider `/models` failure: return fallback model from the model-discovery API with
+  `degraded: true`; do not block chat startup.
+- LLM completion failure after a valid model is selected: keep the existing chat error behavior.
+
+## Testing
+
+Unit tests:
+
+- Model discovery normalizes OpenAI-compatible `/models` responses.
+- Discovery falls back to `LLM_MODEL` for mock or failed provider states.
+- Chat input schema accepts an optional valid model string and rejects invalid shapes.
+- Chat route rejects a model not returned by discovery.
+- `runAgentChat` passes the effective model override into the LLM client path.
+
+E2E smoke:
+
+- After assistant opt-in, opening the assistant shows the model selector.
+- Sending a message still renders the user message and assistant response with the mock provider.
+
+## Rollout notes
+
+This is additive. Existing deployments without a provider `/models` endpoint continue to use
+`LLM_MODEL`. The database migration is nullable, so existing users keep current behavior until
+they choose a model.