Performance of llama.cpp on Nvidia CUDA #15013
Replies: 122 comments 101 replies
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
|
Here's the results for my devices. Not sure how to get a "cuda info string" though. CUDA Scoreboard for Llama 2 7B, Q4_0 (no FA)
CUDA Scoreboard for Llama 2 7B, Q4_0 (with FA)
|
|
While technically not directly related, there may also be value in comparing AMD ROCM build here too, as ROCM acts a replacement (sometimes a directly compatible layer) for most CUDA calls. I admit risk of confusion for Nvidia users in the thread if this path is taken. |
|
As I know you cannot run ROCm on Nvidia GPU. If you would like to see compared results check Vulkan thread. You can find there results for Vulkan/CUDA and Vulkan/ROCm. UPD: Created ROCm discussion. |
|
Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes
build: 9c35706 (6060) Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes
build: 9c35706 (6060) |
|
Device 0: NVIDIA GeForce RTX 4070 Ti SUPER, compute capability 8.9, VMM: yes
build: 9c35706 (647) |
|
@olegshulyakov One more benchmark for RTX 4080: Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9, VMM: yes
build: 20638e4 (2) |
|
@Ristovski why so slow? Have you undervolted it? It pretty as RTX 3080, I expected somewhere between RTX 3090 and 3080 Ti =( |
Hmm indeed, I didn't give much thought to the score at first. It should be stock but not completely sure as that is one of our work machines. I didn't have much time to investigate today, will check again tomorrow! |
|
Device 0: 3090. Power limit to 250w
build: 9c35706 (6060) Device 2: 5090. Power limit to 400w
build: 9c35706 (6060) |
|
Can you please launch them without a limit on full power? |
|
Sure, results with defaults power limits: 3090 at 390W
build: 9c35706 (6060) 5090 at 600W
build: 9c35706 (6060) |
|
crazy, the additional 200W on the 5090 were likely consumed but the performance change was irrelevantly small |
|
Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes
Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1, VMM: yes
build: 9c35706 (6060) |
|
@olegshulyakov To help users quickly understand the approximate largest models that can run on each GPU, I suggest adding a VRAM column next to the GPU name on the main scoreboard. Example:
|
|
Made it a little bit better 🙂 |
|
Device 0: NVIDIA GeForce RTX 2060 SUPER, compute capability 7.5, VMM: yes
build: 5c0eb5e (6075) |
|
@ggerganov Can you please add "performance" label? |
|
@olegshulyakov I see you grabbed some of my numbers from the Vulkan thread. However, I flooded that post with a bunch of data that probably came across as noise. While you quoted my correct numbers for Non-FA, the FA results you grabbed were actually when run on two GPUs instead of one. To make things easier, here are the numbers from a single card: RTX 5060 Ti 16 GB
And here's another GPU for the collection: RTX 4060 Ti 8 GB
|
|
Nice 64GB VRAM setup you got there!
We all be here showing off our GPU collections 😅 |
|
Thanks. It isn't the fastest setup around, especially when working with 70B+ models, but it is completely usable for inference. There are also some benefits I like about these particular cards (Gigabyte Windforce):
|
|
Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
build: 9c35706 (6060) |
|
Yeah also saw numbers for my 4090 taken from the Vulkan thread. Re-ran CUDA results so you can get the latest FA and non-FA results from same build: FA:
Non-FA:
nvidia-dkms 575.64.03-1 ❯ nvcc --version |
|
NVIDIA P106-100 I ran two times, took the best on 2 different build
build: 5fd160b (6106)
build: 860a9e4 (5688) Sadly, nvidia was not supporting this device for the vulkan driver |
|
I just bricked my gtx 1070 Ti :( so i would not be able to reproduce the result with newer build |
|
@pebaryan I've taken the last build one. |
|
Would like to participate with a slightly exotic one from my cute server cube.. :-) (RTX 2000 Ada, 16GB, 75W) I did two runs:
gml_cuda_init: GGML_CUDA_FORCE_MMQ: no
build: 756cfea (6105)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
build: 1d72c84 (6109) Seems to make no big difference... ^^ |
|
I finally got my hands on similar card as before (NP106) but with display output NVIDIA GTX 1060
build: 5fd160b (6106) |
|
just realized i didn't use the latest build, not that difference though
build: 79c1160 (6123) |
|
NVIDIA CMP 170HX (GA100, sm_80, 8 GB HBM2e) Mining card (same die gen as A100)
Despite supposedly coming from the same die, the card is compute-clamped in some way, I'm interested in digging into it further. I measured FP16 GEMM ≈ 6.5 TFLOP/s, which is supposedly ~2.1% of the A100's peak. HBM2e bandwidth is supposedly intact. |
|
CMP 170HX compute rate is basically nuked, Nvidia intentionally restricted the card to 1/32 its available compute... Most of the time that translates to 6x slower performance... |
|
if there's a will there's a way...
just under 2x speedup |
|
the roofline of this card really is interesting, the TG:PP TPS ratio makes it really strange. |
|
Hi! Please check my patch that partly restores tg on cmp90hx/cmp170hx? Can your kernel hacking be combined with that? |
|
well... ggml_cuda_init: found 1 CUDA devices (Total VRAM: 40622 MiB):
build: 571d0d5 (10068) |
|
Hello, four more results on laptops : ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15984 MiB):
build: 1acee6b (9293) And, ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16110 MiB):
build: b22ff4b (9299) And ggml_cuda_init: found 1 CUDA devices (Total VRAM: 4031 MiB):
build: dbe9c0c (9341) And the last one : ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16266 MiB):
build: 0d18aaa (9351) This test is not working with T2000, llama-bench is showing 3715 MiB, but nvidia-smi is showing 4096MiB and no program loaded (How did the results of T1000 were collected, does it have more vRam ?) ggml_cuda_init: found 1 CUDA devices (Total VRAM: 3715 MiB):
An other error on GTX 980M with 4096MiB of vRam
|
|
pp256 - 4GB VRAM isn't enough for this model... |
CMP 100-210 (Mining GV100 - sm_70/Volta - 16GB HBM2)I thought I'd chip in with a card I haven't seen good stats on: the CMP 100-210, a mining version of the V100. Like other CMP cards, the card is gimped to PCIe 1.1 x1 and has its Tensor Cores disabled, but seems to otherwise retain decent FP32 + FP16 performance and fully working 16GB of HBM2 memory. (FP64 however is mega gimped to ~0.377 TFLOPS unlike the real V100's 7.8 TFLOPs).
build: bb28c1f (9281) However, more is possible: setting a small uBatch size on this card absurdly increased PP speed for some models. In my testing,
Full uBatch StatsFull uBatch stats for lulz. PP only, as TG remained a constant ~109 TPS and would just clutter up the table.
|
|
Hi! Can you please check my patch that partly restores tg on cmp90hx/cmp170hx? Hope it will work for cmp210 too. I will be very grateful if you measure numbers on your cmp100-210 and post them here. |
|
Commented on suggestion since this concerns future changes rather than current state of LlamaCPP. TLDR, on the CMP100 I saw either no performance change or even a loss of performance (based on uBatch size). The benchmarks provided with #24616 show stronger performance improvements on other CMP models which are more heavily nerfed than mine; it's probably just not needed on the CMP100. |
|
The CMP100 (Volta) probably needs a different patch than the CMP90HX/CMP170HX (Ampere). |
|
Thanks for this insight. I posted a code to set the micro batch size to 56 during compilation for CMP100 cards. #25139 When using MMQ large tiles, the CMP100 is power usage is lower even though it says 100% utilization but when ubatch is =<56, the power draw increases. |
|
@jmuffins great graphs. on these cards where |
|
And a small laptop Blackwell on Windows : llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 192d8ae (9334) |
|
Device 0: NVIDIA GeForce RTX 5090 Laptop GPU, compute capability 12.0, VMM: yes, VRAM: 24462 MiB Win11, 95W Power Mode
Win11, 175W Power Mode
build: 29f1482 (9253) |
|
Hello, What is the model of the laptop please ?, looking to buy one, Legion or Alienware... !? |
|
https://www.hp.com/de-de/shop/products/laptops/omen-max-gaming-laptop-16-ah0790ng-be6g0ea-abd (20% discount with WBW20 code) |
|
must be plugged in AC to have full power of course |
|
|
|
Server version Nvidia RTX PRO 6000 (300W)
build: d2462f8 (1) |
|
GPU: Asus Dual GeForce RTX 5060 OC 8GB GDDR7 DLSS4 @ PCIE 5.0x8 (CUDA 13.3)
build: 3a3edc9 (9715) |
|
GPU: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1, VMM: yes, VRAM: 4031 MiB
build: 894bb27 (9783) Also for fun I ran my 970, but it couldn't handle the q4_0 quant (thanks nvidia with your 3.5 + .5 vram) so I had to drop to q3_K:
|
|
exec /app/llama-bench -m /models/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -p 512,1024,2048,4096,8192 -n 128,256,512,1024,2048
build: 050ee92 (9821) |
|
RTX 3050 6 GB 96 bit ggml_cuda_init: found 1 CUDA devices (Total VRAM: 5804 MiB):
build: 12127de (10015) |
|
where is 2080ti 22g |
|
Device 0: NVIDIA GeForce GTX 1050, compute capability 6.1, VMM: yes, VRAM: 4095 MiB
build: a4ce259 (10013) Lenovo Yoga 730-15IKB laptop |
A10G (Custom A10 variant - sm_86/Ampere - 24GB GDDR6)This is a customer-specific SKU (AWS, others) based on the A10 but with a 2-slot width (compared to A10's 1-slot) and higher TDP of 300W. Memory bandwidth maxes out at 498 GB/s like the regular A10. Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes, VRAM: 22502 MiB
build: 571d0d5 (10068) |
|
the result seems very low... have you tried to run mmapeak on it? |
|
I think if I'm reading this right I see 50% INT4/INT8 performance compared to the 3090's 568 INT4/284 INT8 (based on the 3090 datasheet, Appendix A), although the datasheet does not list the instruction(s) used to arrive at that result so I could be making an apples to oranges comparison. Update: The A10G datasheet lists lower performance stats of 140 INT8/280 INT4, which is around what I'm getting. The card is just nerfed. |
|
It is nerfed since on my 3080 I can get 280 TOPS |
|
NVIDIA CMP 170HX (GA100, sm_80, 64 GB HBM2e) unlocked Mining card (same die gen as A100) ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -p 512,1024,2048,4096,8192,16384,32768 -n 128,256,512,1024,2048
build: 7ef790f (10149) |
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This is similar to the Performance of llama.cpp on Apple Silicon M-series, Performance of llama.cpp on AMD ROCm(HIP) and Performance of llama.cpp with Vulkan, but for CUDA! I think it's good to consolidate and discuss our results here.
We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our CUDA releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and CUDA info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same device I'll prioritize newer commits with substantial CUDA updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!
CUDA Scoreboard for Llama 2 7B, Q4_0 (no FA)
CUDA Scoreboard for Llama 2 7B, Q4_0 (with FA)
More detailed test
The main idea of this test is to show a decrease in performance with increasing size.
All reactions