| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| seed_oss 36B Q8_0              |  35.78 GiB |    36.15 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |           pp512 |      3469.74 ± 12.21 |
| seed_oss 36B Q8_0              |  35.78 GiB |    36.15 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |          pp4096 |       3116.09 ± 6.05 |
| seed_oss 36B Q8_0              |  35.78 GiB |    36.15 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |           tg128 |         39.52 ± 0.02 |
| seed_oss 36B Q8_0              |  35.78 GiB |    36.15 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |  pp512 @ d16384 |      1809.95 ± 15.50 |
| seed_oss 36B Q8_0              |  35.78 GiB |    36.15 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 | pp4096 @ d16384 |       1635.41 ± 5.24 |
| seed_oss 36B Q8_0              |  35.78 GiB |    36.15 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |  tg128 @ d16384 |         32.06 ± 0.01 |

build: d2462f8f7 (9590)