Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
46 lines
1.9 KiB
Markdown
46 lines
1.9 KiB
Markdown
# Audio / voice
|
|
|
|
Ollama can *understand* audio with speech-capable models (e.g. Gemma 4 `e4b`/`e2b`/
|
|
`12b`), but only via its **native `/api/chat`** with the audio base64 in the
|
|
`images` field — the OpenAI `input_audio` format is silently dropped, and there's
|
|
no native speech-to-text endpoint. This router bridges that gap.
|
|
|
|
Set the model with `ROUTER_AUDIO_MODEL` (default `gemma4:e4b`). It must be a model
|
|
whose `ollama show` capabilities include `audio`. **Note:** some fine-tunes/quants
|
|
have broken audio even if the flag is set — test before relying on one.
|
|
|
|
## Speech-to-text (Whisper-shaped)
|
|
|
|
```bash
|
|
curl http://<host>:8080/v1/audio/transcriptions \
|
|
-F file=@recording.wav \
|
|
-F model=whisper-1
|
|
# -> {"text": "the transcription"}
|
|
```
|
|
- `response_format=text` returns plain text instead of JSON.
|
|
- `/v1/audio/translations` does the same but translates to English.
|
|
- Works with the OpenAI SDK: set `base_url` to the router and call
|
|
`audio.transcriptions.create(...)`. The `model` field is ignored — the router
|
|
always uses `ROUTER_AUDIO_MODEL`.
|
|
|
|
## Audio inside a chat
|
|
|
|
Send OpenAI `input_audio` content to `/v1/chat/completions` and ask about it:
|
|
```json
|
|
{"model":"auto","messages":[{"role":"user","content":[
|
|
{"type":"text","text":"answer the question in this audio"},
|
|
{"type":"input_audio","input_audio":{"data":"<base64-wav>","format":"wav"}}
|
|
]}]}
|
|
```
|
|
The router detects the audio, translates it to Ollama's native call, runs it on
|
|
`ROUTER_AUDIO_MODEL`, and returns a normal OpenAI response (streaming supported).
|
|
|
|
## Limitations
|
|
|
|
- **WAV is verified.** Other formats (mp3/m4a/ogg) depend on Ollama's decoding —
|
|
transcode to WAV if they fail.
|
|
- **Not streaming STT** — it transcribes a complete clip per request (like
|
|
Whisper's file API), not a live mic stream. Perfect for record-then-send.
|
|
- Audio only works through the router's native bridge, so audio requests always
|
|
target `ROUTER_AUDIO_MODEL` (they bypass the normal model-selection map).
|