Joseph CostaandClaude Opus 4.8
9938d46a67
A stdlib pre-router in front of Ollama with LiteLLM backend: - auto model selection by content/tools/modality, with fallbacks - OpenAI /v1, Anthropic /v1/messages, and Ollama-native /api/* endpoints - Whisper-shaped /v1/audio/transcriptions + in-chat audio - key-based fleet policies (e.g. force a client onto uncensored models) - optional Bearer auth; launchd/systemd service install - benchmark harnesses (speed, quality, agentic tool use) with sample results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
21 lines
923 B
Bash
Executable File
21 lines
923 B
Bash
Executable File
#!/usr/bin/env bash
|
|
# Create a copy of an Ollama model with a fixed context window (num_ctx).
|
|
# Useful because Ollama's default context is small and large prompts get truncated.
|
|
#
|
|
# Usage: ./scripts/make-context-variant.sh <base-model> <num_ctx> [new-tag]
|
|
# Example: ./scripts/make-context-variant.sh glm-4.7-flash 131072 glm-4.7-flash:128k
|
|
#
|
|
# Memory note: KV cache grows with num_ctx. On a 36 GB unified-memory Mac, keep
|
|
# the model + KV under ~27 GB (the default GPU wired ceiling) to avoid swapping.
|
|
set -euo pipefail
|
|
BASE="${1:?usage: make-context-variant.sh <base-model> <num_ctx> [new-tag]}"
|
|
NCTX="${2:?need a num_ctx value, e.g. 65536}"
|
|
TAG="${3:-${BASE%%:*}:${NCTX}ctx}"
|
|
|
|
TMP="$(mktemp -t Modelfile.XXXXXX)"
|
|
printf 'FROM %s\nPARAMETER num_ctx %s\n' "$BASE" "$NCTX" > "$TMP"
|
|
ollama create "$TAG" -f "$TMP"
|
|
rm -f "$TMP"
|
|
echo ">> created $TAG (num_ctx=$NCTX from $BASE)"
|
|
ollama show "$TAG" | grep -i num_ctx || true
|