Initial commit: llm-router — smart OpenAI/Anthropic/Ollama router

A stdlib pre-router in front of Ollama with LiteLLM backend:
- auto model selection by content/tools/modality, with fallbacks
- OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints
- Whisper-shaped /v1/audio/transcriptions + in-chat audio
- key-based fleet policies (e.g. force a client onto uncensored models)
- optional Bearer auth; launchd/systemd service install
- benchmark harnesses (speed, quality, agentic tool use) with sample results

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Joseph Costa
2026-07-05 02:05:16 -05:00
co-authored by Claude Opus 4.8
commit 9938d46a67
32 changed files with 2419 additions and 0 deletions
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Create a copy of an Ollama model with a fixed context window (num_ctx).
# Useful because Ollama's default context is small and large prompts get truncated.
#
# Usage: ./scripts/make-context-variant.sh <base-model> <num_ctx> [new-tag]
# Example: ./scripts/make-context-variant.sh glm-4.7-flash 131072 glm-4.7-flash:128k
#
# Memory note: KV cache grows with num_ctx. On a 36 GB unified-memory Mac, keep
# the model + KV under ~27 GB (the default GPU wired ceiling) to avoid swapping.
set -euo pipefail
BASE="${1:?usage: make-context-variant.sh <base-model> <num_ctx> [new-tag]}"
NCTX="${2:?need a num_ctx value, e.g. 65536}"
TAG="${3:-${BASE%%:*}:${NCTX}ctx}"
TMP="$(mktemp -t Modelfile.XXXXXX)"
printf 'FROM %s\nPARAMETER num_ctx %s\n' "$BASE" "$NCTX" > "$TMP"
ollama create "$TAG" -f "$TMP"
rm -f "$TMP"
echo ">> created $TAG (num_ctx=$NCTX from $BASE)"
ollama show "$TAG" | grep -i num_ctx || true