feat(llm): serve 64k context (YaRN) to meet Hermes' 64K minimum

Hermes Agent rejects models with <64K context. Qwen2.5-14B is 32k native, so
enable YaRN rope-scaling (2x → 65536) and drop the V-cache to q4_0 for VRAM
headroom on the 16GB P100.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ginnoir
2026-06-26 16:39:56 -05:00
co-authored by Claude Opus 4.8
parent 605c6d3709
commit 265b407e8d
+7 -1
View File
@@ -35,13 +35,19 @@ services:
- "-ngl" - "-ngl"
- "99" - "99"
- "--ctx-size" - "--ctx-size"
- "65536"
- "--rope-scaling"
- "yarn"
- "--rope-scale"
- "2"
- "--yarn-orig-ctx"
- "32768" - "32768"
- "--flash-attn" - "--flash-attn"
- "on" - "on"
- "--cache-type-k" - "--cache-type-k"
- "q8_0" - "q8_0"
- "--cache-type-v" - "--cache-type-v"
- "q8_0" - "q4_0"
- "--host" - "--host"
- "0.0.0.0" - "0.0.0.0"
- "--port" - "--port"