Initial commit: llm-router — smart OpenAI/Anthropic/Ollama router
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Create a copy of an Ollama model with a fixed context window (num_ctx).
|
||||
# Useful because Ollama's default context is small and large prompts get truncated.
|
||||
#
|
||||
# Usage: ./scripts/make-context-variant.sh <base-model> <num_ctx> [new-tag]
|
||||
# Example: ./scripts/make-context-variant.sh glm-4.7-flash 131072 glm-4.7-flash:128k
|
||||
#
|
||||
# Memory note: KV cache grows with num_ctx. On a 36 GB unified-memory Mac, keep
|
||||
# the model + KV under ~27 GB (the default GPU wired ceiling) to avoid swapping.
|
||||
set -euo pipefail
|
||||
BASE="${1:?usage: make-context-variant.sh <base-model> <num_ctx> [new-tag]}"
|
||||
NCTX="${2:?need a num_ctx value, e.g. 65536}"
|
||||
TAG="${3:-${BASE%%:*}:${NCTX}ctx}"
|
||||
|
||||
TMP="$(mktemp -t Modelfile.XXXXXX)"
|
||||
printf 'FROM %s\nPARAMETER num_ctx %s\n' "$BASE" "$NCTX" > "$TMP"
|
||||
ollama create "$TAG" -f "$TMP"
|
||||
rm -f "$TMP"
|
||||
echo ">> created $TAG (num_ctx=$NCTX from $BASE)"
|
||||
ollama show "$TAG" | grep -i num_ctx || true
|
||||
Reference in New Issue
Block a user