Files
homelabstack/stacks
ginnoirandClaude Opus 4.8 4aa2cd9468 fix(llm): q8_0 V-cache (q4_0 tanked generation to 1.3 tok/s on Pascal)
The q4_0 V-cache + flash-attention path is pathological on the GP100: 1.28
tok/s generation at 5-8% GPU util. q8_0 V-cache gives 9.2 tok/s and still fits
64k context in 16GB (15.3GB used, ~950MB free).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:04:17 -05:00
..
2026-06-15 22:15:28 -05:00