ginnoir/homelabstack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(llm): --parallel 1 so a single request gets the full 64k context

Browse Source 
With the default 4 slots, llama-server splits ctx into 32k per sequence, which
fails Hermes' 64K minimum. One slot serves the full 65536 per request (serial
agent use; concurrent calls queue).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ginnoir
2026-06-26 16:42:41 -05:00
co-authored by Claude Opus 4.8
parent 265b407e8d
commit 74791105b3
1 changed files with 2 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/llm/docker-compose.yml
+2
View File
@@ -32,6 +32,8 @@ services:
- "/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf" - "/models/Qwen2.5-14B-Instruct-Q4_K_M.gguf"
- "--alias" - "--alias"
- "qwen2.5-14b-instruct" - "qwen2.5-14b-instruct"
- "--parallel"
- "1"
- "-ngl" - "-ngl"
- "99" - "99"
- "--ctx-size" - "--ctx-size"