feat(llm): serve 64k context (YaRN) to meet Hermes' 64K minimum
Hermes Agent rejects models with <64K context. Qwen2.5-14B is 32k native, so enable YaRN rope-scaling (2x → 65536) and drop the V-cache to q4_0 for VRAM headroom on the 16GB P100. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
605c6d3709
commit
265b407e8d
@@ -35,13 +35,19 @@ services:
|
||||
- "-ngl"
|
||||
- "99"
|
||||
- "--ctx-size"
|
||||
- "65536"
|
||||
- "--rope-scaling"
|
||||
- "yarn"
|
||||
- "--rope-scale"
|
||||
- "2"
|
||||
- "--yarn-orig-ctx"
|
||||
- "32768"
|
||||
- "--flash-attn"
|
||||
- "on"
|
||||
- "--cache-type-k"
|
||||
- "q8_0"
|
||||
- "--cache-type-v"
|
||||
- "q8_0"
|
||||
- "q4_0"
|
||||
- "--host"
|
||||
- "0.0.0.0"
|
||||
- "--port"
|
||||
|
||||
Reference in New Issue
Block a user