| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| llama 13B Q8_0                 |  23.33 GiB |    23.57 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |           pp512 |     5480.47 ± 455.77 |
| llama 13B Q8_0                 |  23.33 GiB |    23.57 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |          pp4096 |       5427.57 ± 7.74 |
| llama 13B Q8_0                 |  23.33 GiB |    23.57 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |           tg128 |         61.36 ± 0.03 |
| llama 13B Q8_0                 |  23.33 GiB |    23.57 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |  pp512 @ d16384 |      4690.50 ± 63.96 |
| llama 13B Q8_0                 |  23.33 GiB |    23.57 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 | pp4096 @ d16384 |       4258.19 ± 6.39 |
| llama 13B Q8_0                 |  23.33 GiB |    23.57 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |  tg128 @ d16384 |         56.17 ± 0.03 |

build: d2462f8f7 (9590)