llama.cpp Benchmark Report

Generated 2026-06-13 10:35:20 · 12 model(s) with data

System

GPUs
2 x NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU driver
610.43.02
GPU power limit
600 W (set via nvidia-smi -pl; persistence mode on)
CPU
AMD Ryzen 9 9950X 16-Core Processor
System RAM
128 GiB (OS-visible: 123 GiB)
llama.cpp build
version: 9590 (d2462f8f7) built with GNU 14.2.0 for Linux x86_64

Benchmark configuration

GPU power limit (both GPUs)
600 W
Flash attention
on (`-fa 1`)
KV cache type (K and V)
q8_0 (`-ctk q8_0 -ctv q8_0`)
CPU threads
16 (`-t 16`)
GPU layers
all (`-ngl 999`)
Batch / micro-batch
llama-bench defaults (`-b 2048 -ub 512`)
Multi-GPU split (dual)
layer (`-sm layer`)
Repetitions
5 (`-r 5`)
Cool-down between models
300 s
nvidia-smi sampling
every 500 ms, scoped to the run's GPUs

Models under test

AliasHF repo : quantPlacementNote
devstral-small-2-24bunsloth/Devstral-Small-2507-GGUF:Q8_0gpu024B dense, Q8 ~26 GB (verify repo)
glm-4.6-355bunsloth/GLM-4.6-GGUF:UD-Q2_K_XLdual355B-A32B, Q2_K_XL ~135 GB, dual
gpt-oss-120bggml-org/gpt-oss-120b-GGUFgpu0MXFP4 native, ~60 GB, single card
hermes-4-70bunsloth/Hermes-4-70B-GGUF:Q8_0gpu070B dense (Llama 3.1), Q8 ~75 GB, single card
hermes-4.3-36bNousResearch/Hermes-4.3-36B-GGUF:Q8_0gpu036B dense (Seed-OSS-36B), Q8 ~38 GB, single card
minimax-m2.7unsloth/MiniMax-M2.7-GGUF:UD-Q4_K_XLdual~230B-A10B, Q4, dual (verify tag)
qwen2.5-coder-32bunsloth/Qwen2.5-Coder-32B-Instruct-GGUF:Q8_0gpu032B dense, Q8 ~35 GB
qwen3-235b-a22bunsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:Q4_K_Mdual235B-A22B, Q4 ~133 GB, dual
qwen3-coder-480bunsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF:UD-Q2_K_XLdual480B-A35B, Q2 ~180 GB, dual (tight)
qwen3-coder-next-80bunsloth/Qwen3-Coder-Next-GGUF:Q8_0gpu080B-A3B, Q8 ~85 GB, single card
qwen3.6-27bunsloth/Qwen3.6-27B-GGUF:Q8_0gpu027B dense, Q8 ~30 GB, single card
qwen3.6-35b-a3bunsloth/Qwen3.6-35B-A3B-GGUF:Q8_0gpu035B-A3B MoE, Q8 ~38 GB, single card

Prefill throughput (prompt processing)

Prefill 512 @ d0Prefill 4096 @ d00.00200040006000800010000tokens / secondqwen3.6-35b-a3bqwen3.6-35b-a3b — Prefill 512 @ d0: 99199919qwen3.6-35b-a3b — Prefill 4096 @ d0: 98109810gpt-oss-120bgpt-oss-120b — Prefill 512 @ d0: 95059505gpt-oss-120b — Prefill 4096 @ d0: 95419541devstral-small-2-24bdevstral-small-2-24b — Prefill 512 @ d0: 58445844devstral-small-2-24b — Prefill 4096 @ d0: 57185718qwen3-coder-next-80bqwen3-coder-next-80b — Prefill 512 @ d0: 48694869qwen3-coder-next-80b — Prefill 4096 @ d0: 48874887qwen3.6-27bqwen3.6-27b — Prefill 512 @ d0: 42764276qwen3.6-27b — Prefill 4096 @ d0: 42994299qwen2.5-coder-32bqwen2.5-coder-32b — Prefill 512 @ d0: 39773977qwen2.5-coder-32b — Prefill 4096 @ d0: 38503850minimax-m2.7minimax-m2.7 — Prefill 512 @ d0: 18371837minimax-m2.7 — Prefill 4096 @ d0: 37273727hermes-4.3-36bhermes-4.3-36b — Prefill 512 @ d0: 36393639hermes-4.3-36b — Prefill 4096 @ d0: 32823282qwen3-235b-a22bqwen3-235b-a22b — Prefill 512 @ d0: 19771977qwen3-235b-a22b — Prefill 4096 @ d0: 26132613hermes-4-70bhermes-4-70b — Prefill 512 @ d0: 20062006hermes-4-70b — Prefill 4096 @ d0: 19151915qwen3-coder-480bqwen3-coder-480b — Prefill 512 @ d0: 928928qwen3-coder-480b — Prefill 4096 @ d0: 14441444glm-4.6-355bglm-4.6-355b — Prefill 512 @ d0: 960960glm-4.6-355b — Prefill 4096 @ d0: 11781178

Decode throughput (token generation)

Decode 128 @ d0Decode 128 @ d163840.0050.0100150200250300tokens / secondgpt-oss-120bgpt-oss-120b — Decode 128 @ d0: 275275gpt-oss-120b — Decode 128 @ d16384: 235235qwen3.6-35b-a3bqwen3.6-35b-a3b — Decode 128 @ d0: 239239qwen3.6-35b-a3b — Decode 128 @ d16384: 225225qwen3-coder-next-80bqwen3-coder-next-80b — Decode 128 @ d0: 195195qwen3-coder-next-80b — Decode 128 @ d16384: 183183minimax-m2.7minimax-m2.7 — Decode 128 @ d0: 129129minimax-m2.7 — Decode 128 @ d16384: 89.689.6qwen3-235b-a22bqwen3-235b-a22b — Decode 128 @ d0: 83.983.9qwen3-235b-a22b — Decode 128 @ d16384: 53.653.6qwen3-coder-480bqwen3-coder-480b — Decode 128 @ d0: 73.473.4qwen3-coder-480b — Decode 128 @ d16384: 48.948.9devstral-small-2-24bdevstral-small-2-24b — Decode 128 @ d0: 61.461.4devstral-small-2-24b — Decode 128 @ d16384: 56.256.2glm-4.6-355bglm-4.6-355b — Decode 128 @ d0: 61.161.1glm-4.6-355b — Decode 128 @ d16384: 37.837.8qwen3.6-27bqwen3.6-27b — Decode 128 @ d0: 50.550.5qwen3.6-27b — Decode 128 @ d16384: 48.948.9qwen2.5-coder-32bqwen2.5-coder-32b — Decode 128 @ d0: 43.443.4qwen2.5-coder-32b — Decode 128 @ d16384: 38.638.6hermes-4.3-36bhermes-4.3-36b — Decode 128 @ d0: 39.539.5hermes-4.3-36b — Decode 128 @ d16384: 32.132.1hermes-4-70bhermes-4-70b — Decode 128 @ d0: 20.820.8hermes-4-70b — Decode 128 @ d16384: 18.418.4

Peak temperatures and power

Maxima are taken over the active benchmark window of each run; only GPUs that actively participated are shown, so single-card runs omit the idle second card. Peak power samples a few percent above the configured limit are normal: nvidia-smi reports instantaneous draw while NVIDIA's power controller regulates a time-averaged budget, so brief transients above the cap are expected and not a fault. Compare the average power chart against the limit instead.

Max core temperature

GPU 0GPU 10.0020.040.060.080.0100degrees Cdevstral-small-2-24bdevstral-small-2-24b — GPU 0: 65.065.0glm-4.6-355bglm-4.6-355b — GPU 0: 78.078.0glm-4.6-355b — GPU 1: 71.071.0gpt-oss-120bgpt-oss-120b — GPU 0: 56.056.0hermes-4-70bhermes-4-70b — GPU 0: 79.079.0hermes-4.3-36bhermes-4.3-36b — GPU 0: 74.074.0minimax-m2.7minimax-m2.7 — GPU 0: 68.068.0minimax-m2.7 — GPU 1: 65.065.0qwen2.5-coder-32bqwen2.5-coder-32b — GPU 0: 71.071.0qwen3-235b-a22bqwen3-235b-a22b — GPU 0: 68.068.0qwen3-235b-a22b — GPU 1: 63.063.0qwen3-coder-480bqwen3-coder-480b — GPU 0: 81.081.0qwen3-coder-480b — GPU 1: 69.069.0qwen3-coder-next-80bqwen3-coder-next-80b — GPU 0: 56.056.0qwen3.6-27bqwen3.6-27b — GPU 0: 70.070.0qwen3.6-35b-a3bqwen3.6-35b-a3b — GPU 0: 56.056.0

Max power draw (instantaneous samples)

GPU 0GPU 10.00100200300400500600700wattsdevstral-small-2-24bdevstral-small-2-24b — GPU 0: 603603glm-4.6-355bglm-4.6-355b — GPU 0: 482482glm-4.6-355b — GPU 1: 496496gpt-oss-120bgpt-oss-120b — GPU 0: 582582hermes-4-70bhermes-4-70b — GPU 0: 607607hermes-4.3-36bhermes-4.3-36b — GPU 0: 605605minimax-m2.7minimax-m2.7 — GPU 0: 569569minimax-m2.7 — GPU 1: 541541qwen2.5-coder-32bqwen2.5-coder-32b — GPU 0: 607607qwen3-235b-a22bqwen3-235b-a22b — GPU 0: 478478qwen3-235b-a22b — GPU 1: 438438qwen3-coder-480bqwen3-coder-480b — GPU 0: 558558qwen3-coder-480b — GPU 1: 469469qwen3-coder-next-80bqwen3-coder-next-80b — GPU 0: 467467qwen3.6-27bqwen3.6-27b — GPU 0: 603603qwen3.6-35b-a3bqwen3.6-35b-a3b — GPU 0: 540540

Average power draw (over the active window)

GPU 0GPU 10.00100200300400500600wattsdevstral-small-2-24bdevstral-small-2-24b — GPU 0: 460460glm-4.6-355bglm-4.6-355b — GPU 0: 239239glm-4.6-355b — GPU 1: 259259gpt-oss-120bgpt-oss-120b — GPU 0: 327327hermes-4-70bhermes-4-70b — GPU 0: 501501hermes-4.3-36bhermes-4.3-36b — GPU 0: 492492minimax-m2.7minimax-m2.7 — GPU 0: 387387minimax-m2.7 — GPU 1: 377377qwen2.5-coder-32bqwen2.5-coder-32b — GPU 0: 474474qwen3-235b-a22bqwen3-235b-a22b — GPU 0: 364364qwen3-235b-a22b — GPU 1: 347347qwen3-coder-480bqwen3-coder-480b — GPU 0: 433433qwen3-coder-480b — GPU 1: 365365qwen3-coder-next-80bqwen3-coder-next-80b — GPU 0: 307307qwen3.6-27bqwen3.6-27b — GPU 0: 466466qwen3.6-35b-a3bqwen3.6-35b-a3b — GPU 0: 311311

Per-model thermals over time

Charts are trimmed to the active benchmark window: the model-loading phase and idle tails are cut (the amount removed is noted per model), so the measured runs fill the chart instead of being squeezed by minutes of GGUF loading. SM clock dropping during a run is the clearest sign of throttling. Power-cap limited is expected whenever the power limit is set below the card's maximum — the card is staying within budget, not faulting. Thermal / HW slowdown means the card hit a temperature or hardware limit and reduced clocks; investigate cooling and airflow.

devstral-small-2-24bGPU 0power-cap limited (expected at your set power limit)

-hf unsloth/Devstral-Small-2507-GGUF:Q8_0 | placement: gpu0

showing the 42s active window; 72s of model-load/idle trimmed from a 114s recording

Temperature (°C)
0.0010.020.030.040.050.060.070.080.00.0010.020.030.040.050.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.0010.020.030.040.050.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.0010.020.030.040.050.0MHzseconds (active window)GPU 0
glm-4.6-355bGPU 0 + 1power-cap limited (expected at your set power limit)

-hf unsloth/GLM-4.6-GGUF:UD-Q2_K_XL | placement: dual

showing the 173s active window; 415s of model-load/idle trimmed from a 588s recording

Temperature (°C)
0.0020.040.060.080.01000.0050.0100150200°Cseconds (active window)GPU 0 coreGPU 1 core
Power draw (W)
0.001002003004005006007000.0050.0100150200Wseconds (active window)600 W limitGPU 0GPU 1
SM clock (MHz)
0.005001000150020002500300035000.0050.0100150200MHzseconds (active window)GPU 0GPU 1
gpt-oss-120bGPU 0no throttling

-hf ggml-org/gpt-oss-120b-GGUF | placement: gpu0

showing the 23s active window; 186s of model-load/idle trimmed from a 209s recording

Temperature (°C)
0.0010.020.030.040.050.060.070.00.005.0010.015.020.025.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.005.0010.015.020.025.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.005.0010.015.020.025.0MHzseconds (active window)GPU 0
hermes-4-70bGPU 0power-cap limited (expected at your set power limit)

-hf unsloth/Hermes-4-70B-GGUF:Q8_0 | placement: gpu0

showing the 121s active window; 228s of model-load/idle trimmed from a 349s recording

Temperature (°C)
0.0020.040.060.080.01000.0020.040.060.080.0100120140°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.0020.040.060.080.0100120140Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.0020.040.060.080.0100120140MHzseconds (active window)GPU 0
hermes-4.3-36bGPU 0power-cap limited (expected at your set power limit)

-hf NousResearch/Hermes-4.3-36B-GGUF:Q8_0 | placement: gpu0

showing the 75s active window; 115s of model-load/idle trimmed from a 190s recording

Temperature (°C)
0.0020.040.060.080.00.0020.040.060.080.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.0020.040.060.080.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.0020.040.060.080.0MHzseconds (active window)GPU 0
minimax-m2.7GPU 0 + 1power-cap limited (expected at your set power limit)

-hf unsloth/MiniMax-M2.7-GGUF:UD-Q4_K_XL | placement: dual

showing the 40s active window; 856s of model-load/idle trimmed from a 896s recording

Temperature (°C)
0.0010.020.030.040.050.060.070.080.00.0010.020.030.040.050.0°Cseconds (active window)GPU 0 coreGPU 1 core
Power draw (W)
0.001002003004005006007000.0010.020.030.040.050.0Wseconds (active window)600 W limitGPU 0GPU 1
SM clock (MHz)
0.005001000150020002500300035000.0010.020.030.040.050.0MHzseconds (active window)GPU 0GPU 1
qwen2.5-coder-32bGPU 0power-cap limited (expected at your set power limit)

-hf unsloth/Qwen2.5-Coder-32B-Instruct-GGUF:Q8_0 | placement: gpu0

showing the 63s active window; 101s of model-load/idle trimmed from a 164s recording

Temperature (°C)
0.0020.040.060.080.00.0010.020.030.040.050.060.070.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.0010.020.030.040.050.060.070.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.0010.020.030.040.050.060.070.0MHzseconds (active window)GPU 0
qwen3-235b-a22bGPU 0 + 1power-cap limited (expected at your set power limit)

-hf unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:Q4_K_M | placement: dual

showing the 58s active window; 814s of model-load/idle trimmed from a 871s recording

Temperature (°C)
0.0010.020.030.040.050.060.070.080.00.0010.020.030.040.050.060.0°Cseconds (active window)GPU 0 coreGPU 1 core
Power draw (W)
0.001002003004005006007000.0010.020.030.040.050.060.0Wseconds (active window)600 W limitGPU 0GPU 1
SM clock (MHz)
0.005001000150020002500300035000.0010.020.030.040.050.060.0MHzseconds (active window)GPU 0GPU 1
qwen3-coder-480bGPU 0 + 1power-cap limited (expected at your set power limit)

-hf unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF:UD-Q2_K_XL | placement: dual

showing the 85s active window; 1113s of model-load/idle trimmed from a 1198s recording

Temperature (°C)
0.0020.040.060.080.01000.0020.040.060.080.0100°Cseconds (active window)GPU 0 coreGPU 1 core
Power draw (W)
0.001002003004005006007000.0020.040.060.080.0100Wseconds (active window)600 W limitGPU 0GPU 1
SM clock (MHz)
0.005001000150020002500300035000.0020.040.060.080.0100MHzseconds (active window)GPU 0GPU 1
qwen3-coder-next-80bGPU 0no throttling

-hf unsloth/Qwen3-Coder-Next-GGUF:Q8_0 | placement: gpu0

showing the 32s active window; 250s of model-load/idle trimmed from a 282s recording

Temperature (°C)
0.0010.020.030.040.050.060.070.00.005.0010.015.020.025.030.035.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.005.0010.015.020.025.030.035.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.005.0010.015.020.025.030.035.0MHzseconds (active window)GPU 0
qwen3.6-27bGPU 0power-cap limited (expected at your set power limit)

-hf unsloth/Qwen3.6-27B-GGUF:Q8_0 | placement: gpu0

showing the 50s active window; 85s of model-load/idle trimmed from a 135s recording

Temperature (°C)
0.0020.040.060.080.00.0010.020.030.040.050.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.0010.020.030.040.050.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.0010.020.030.040.050.0MHzseconds (active window)GPU 0
qwen3.6-35b-a3bGPU 0no throttling

-hf unsloth/Qwen3.6-35B-A3B-GGUF:Q8_0 | placement: gpu0

showing the 20s active window; 110s of model-load/idle trimmed from a 130s recording

Temperature (°C)
0.0010.020.030.040.050.060.070.00.005.0010.015.020.025.0°Cseconds (active window)GPU 0 core
Power draw (W)
0.001002003004005006007000.005.0010.015.020.025.0Wseconds (active window)600 W limitGPU 0
SM clock (MHz)
0.005001000150020002500300035000.005.0010.015.020.025.0MHzseconds (active window)GPU 0

Summary table

ModelQuantGPUsPrefill 512Prefill 4096Decode @d0Decode @d16384t/s per W (avg)Avg power WPeak power WMax VRAM GiBMax core °CThrottled
devstral-small-2-24bQ8_005844571861.456.20.1346060325.265.0power cap
glm-4.6-355bUD-Q2_K_XL0+1960117861.137.80.1249849613078.0power cap
gpt-oss-120b–0950595412752350.8432758259.956.0none
hermes-4-70bQ8_002006191520.818.40.0450160773.079.0power cap
hermes-4.3-36bQ8_003639328239.532.10.0849260538.674.0power cap
minimax-m2.7UD-Q4_K_XL0+11837372712989.60.1776456913568.0power cap
qwen2.5-coder-32bQ8_003977385043.438.60.0947460735.371.0power cap
qwen3-235b-a22bQ4_K_M0+11977261383.953.60.1271147813668.0power cap
qwen3-coder-480bUD-Q2_K_XL0+1928144473.448.90.0979855817281.0power cap
qwen3-coder-next-80bQ8_00486948871951830.6430746780.056.0none
qwen3.6-27bQ8_004276429950.548.90.1146660327.370.0power cap
qwen3.6-35b-a3bQ8_00991998102392250.7731154035.356.0none