ginnoir/homelabstack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(llm): q8_0 V-cache (q4_0 tanked generation to 1.3 tok/s on Pascal)

Browse Source 
The q4_0 V-cache + flash-attention path is pathological on the GP100: 1.28
tok/s generation at 5-8% GPU util. q8_0 V-cache gives 9.2 tok/s and still fits
64k context in 16GB (15.3GB used, ~950MB free).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ginnoir
2026-06-26 17:04:17 -05:00
co-authored by Claude Opus 4.8
parent 13424fcf75
commit 4aa2cd9468
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/llm/docker-compose.yml
+1 -1
View File
@@ -51,7 +51,7 @@ services:
- "--cache-type-k"
- "q8_0"
- "--cache-type-v"
- "q4_0"
- "q8_0"
- "--host"
- "0.0.0.0"
- "--port"