| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| glm4moe 355B.A32B Q2_K - Medium | 125.43 GiB |   356.79 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |           pp512 |      936.68 ± 105.72 |
| glm4moe 355B.A32B Q2_K - Medium | 125.43 GiB |   356.79 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |          pp4096 |       1147.47 ± 3.17 |
| glm4moe 355B.A32B Q2_K - Medium | 125.43 GiB |   356.79 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |           tg128 |         61.14 ± 0.09 |
| glm4moe 355B.A32B Q2_K - Medium | 125.43 GiB |   356.79 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |  pp512 @ d16384 |        712.15 ± 3.03 |
| glm4moe 355B.A32B Q2_K - Medium | 125.43 GiB |   356.79 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 | pp4096 @ d16384 |        812.77 ± 2.41 |
| glm4moe 355B.A32B Q2_K - Medium | 125.43 GiB |   356.79 B | CUDA       | 999 |   q8_0 |   q8_0 |   1 |  tg128 @ d16384 |         37.84 ± 0.06 |

build: d2462f8f7 (9590)