Performance of llama.cpp on Apple Silicon M-series

[ggerganov]

Summary

LLaMA 7B

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]
✅ M1 1	68	7			108.21	7.92	107.81	14.19
✅ M1 1	68	8			117.25	7.91	117.96	14.15
✅ M1 Pro 1	200	14	262.65	12.75	235.16	21.95	232.55	35.52
✅ M1 Pro 1	200	16	302.14	12.75	270.37	22.34	266.25	36.41
✅ M1 Max 1	400	24	453.03	22.55	405.87	37.81	400.26	54.61
✅ M1 Max 1	400	32	599.53	23.03	537.37	40.2	530.06	61.19
✅ M1 Ultra 1	800	48	875.81	33.92	783.45	55.69	772.24	74.93
✅ M1 Ultra 1	800	64	1168.89	37.01	1042.95	59.87	1030.04	83.73
✅ M2 2	100	8			147.27	12.18	145.91	21.7
✅ M2 2	100	10	201.34	6.72	181.4	12.21	179.57	21.91
✅ M2 Pro 2	200	16	312.65	12.47	288.46	22.7	294.24	37.87
✅ M2 Pro 2	200	19	384.38	13.06	344.5	23.01	341.19	38.86
✅ M2 Max 2	400	30	600.46	24.16	540.15	39.97	537.6	60.99
✅ M2 Max 2	400	38	755.67	24.65	677.91	41.83	671.31	65.95
✅ M2 Ultra 2	800	60	1128.59	39.86	1003.16	62.14	1013.81	88.64
✅ M2 Ultra 2	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27
🟥 M3 3	100	8
🟨 M3 3	100	10			187.52	12.27	186.75	21.34
🟨 M3 Pro 3	150	14			272.11	17.44	269.49	30.65
✅ M3 Pro 3	150	18	357.45	9.89	344.66	17.53	341.67	30.74
✅ M3 Max 3	300	30	589.41	19.54	566.4	34.3	567.59	56.58
✅ M3 Max 3	400	40	779.17	25.09	757.64	42.75	759.7	66.31
✅ M3 Ultra 3	800	60	1121.80	42.24	1085.76	63.55	1073.09	88.40
✅ M3 Ultra 3	800	80	1538.34	39.78	1487.51	63.93	1471.24	92.14
🟥 M4 4	120	8
✅ M4 4	120	10	230.18	7.43	223.64	13.54	221.29	24.11
✅ M4 Pro 4	273	16	381.14	17.19	367.13	30.54	364.06	49.64
✅ M4 Pro 4	273	20	464.48	17.18	449.62	30.69	439.78	50.74
✅ M4 Max 4	410	32	736.25	24.29	718.56	43.87	713.93	69.95
✅ M4 Max 4	546	40	922.83	31.64	891.94	54.05	885.68	83.06
🟥 M4 Ultra	820	64
🟥 M4 Ultra	1092	80
🟥 M5 5	154	8
🟥 M5 5	154	10
🟥 M5 Pro 5	307	16
🟥 M5 Pro 5	307	20
🟥 M5 Max 5	460	32
🟥 M5 Max 5	614	40
🟥 M5 Ultra	?	?
🟥 M5 Ultra	?	?

plot.py

# GPT-4 Generated Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Creating DataFrame from the provided data
data = {
    "Chip": ["M1", "M1", "M1 Pro", "M1 Pro", "M1 Max", "M1 Max", "M1 Ultra", "M2", "M2 Pro", "M2 Pro", "M2 Max", "M2 Max", "M2 Ultra", "M2 Ultra", "M3", "M3 Pro", "M3 Pro", "M3 Max"],
    "BW (GB/s)":     [68, 68, 200, 200, 400, 400, 800, 100, 200, 200, 400, 400, 800, 800, 100, 150, 150, 400],
    "GPU Cores":     [7, 8, 14, 16, 24, 32, 48, 10, 16, 19, 30, 38, 60, 76, 10, 14, 18, 40],
    "F16 PP (t/s)":  [None, None, None, 302.14, 453.03, 599.53, 875.81, 201.34, 312.65, 384.38, 600.46, 755.67, 1128.59, 1401.85, None, None, 357.45, 779.17],
    "F16 TG (t/s)":  [None, None, None, 12.75, 22.55, 23.03, 33.92, 6.72, 12.47, 13.06, 24.16, 24.65, 39.86, 41.02, None, None, 9.89, 25.09],
    "Q8_0 PP (t/s)": [108.21, 117.25, 235.16, 270.37, 405.87, 537.37, 783.45, 181.4, 288.46, 344.5, 540.15, 677.91, 1003.16, 1248.59, 187.52, 272.11, 344.66, 757.64],
    "Q8_0 TG (t/s)": [7.92, 7.91, 21.95, 22.34, 37.81, 40.2, 55.69, 12.21, 22.7, 23.01, 39.97, 41.83, 62.14, 66.64, 12.27, 17.44, 17.53, 42.75],
    "Q4_0 PP (t/s)": [107.81, 117.96, 232.55, 266.25, 400.26, 530.06, 772.24, 179.57, 294.24, 341.19, 537.6, 671.31, 1013.81, 1238.48, 186.75, 269.49, 341.67, 759.7],
    "Q4_0 TG (t/s)": [14.19, 14.15, 35.52, 36.41, 54.61, 61.19, 74.93, 21.91, 37.87, 38.86, 60.99, 65.95, 88.64, 94.27, 21.34, 30.65, 30.74, 66.31]
}
df = pd.DataFrame(data)

# Helper function to plot and annotate multiple data series in the same plot
def plot_multi_series(ax, x, y_series, labels, xlabel, ylabel, title, poly_power=1):
    colors = ['r', 'g', 'b']  # Colors for different series
    for i, y in enumerate(y_series):
        # Sorting data for regression
        sorted_indices = np.argsort(x)
        x_sorted = x[sorted_indices]
        y_sorted = y[sorted_indices]

        # Masking NaN values
        mask = ~np.isnan(y_sorted)
        x_sorted = x_sorted[mask]
        y_sorted = y_sorted[mask]

        # Fitting a polynomial regression model
        coefficients = np.polyfit(x_sorted, y_sorted, poly_power)
        polynomial = np.poly1d(coefficients)

        # Creating a range of x-values for a smoother trendline
        x_range = np.linspace(x_sorted.min(), x_sorted.max(), 500)
        trendline = polynomial(x_range)

        # Plotting
        ax.scatter(x, y, color=colors[i], label=labels[i], s=20)
        ax.plot(x_range, trendline, f"{colors[i]}-", linewidth=1)  # Trendline in the same color

    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    ax.set_title(title)
    ax.legend()

    # Annotating points with the number of GPU cores and Bandwidth
    for i, txt in enumerate(df["Chip"]):
        ax.annotate(f"{df['GPU Cores'][i]} Cores, {df['BW (GB/s)'][i]} GB/s", (x[i], y_series[0][i]))


# Creating plots for PP vs Cores and TG vs Bandwidth
fig, axs = plt.subplots(1, 2, figsize=(15, 6))
fig.suptitle('PP vs GPU Cores and TG vs Bandwidth for F16, Q8_0, and Q4_0')

# PP vs GPU Cores
y_series_cores_pp = [df["F16 PP (t/s)"], df["Q8_0 PP (t/s)"], df["Q4_0 PP (t/s)"]]
plot_multi_series(axs[0], df["GPU Cores"], y_series_cores_pp,
                  ['F16 PP', 'Q8_0 PP', 'Q4_0 PP'], 'GPU Cores', 'Performance (t/s)',
                  'PP Performance vs GPU Cores', 1)

# TG vs Bandwidth
y_series_bw_tg = [df["F16 TG (t/s)"], df["Q8_0 TG (t/s)"], df["Q4_0 TG (t/s)"]]
plot_multi_series(axs[1], df["BW (GB/s)"], y_series_bw_tg,
                  ['F16 TG', 'Q8_0 TG', 'Q4_0 TG'], 'Bandwidth (GB/s)', 'Performance (t/s)',
                  'TG Performance vs Bandwidth', 2)

plt.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.show()

---

Description

This is a collection of short llama.cpp benchmarks on various Apple Silicon hardware. It can be useful to compare the performance that llama.cpp achieves across the M-series chips and hopefully answer questions of people wondering if they should upgrade or not. Collecting info here just for Apple Silicon for simplicity. Similar collection for A-series chips is available here: #4508

If you are a collaborator to the project and have an Apple Silicon device, please add your device, results and optionally username for the following command directly into this post (requires LLaMA 7B v2):

git checkout 8e672efe
make clean && make -j llama-bench && ./llama-bench \
  -m ./models/llama-7b-v2/ggml-model-f16.gguf  \
  -m ./models/llama-7b-v2/ggml-model-q8_0.gguf \
  -m ./models/llama-7b-v2/ggml-model-q4_0.gguf \
  -p 512 -n 128 -ngl 99 2> /dev/null

• Make sure to run the benchmark on commit 8e672ef
• Please also include the F16 model as shown, not just the quantum models
• Contributors can post the same results in the comments below
• If a device is already benchmarked and your results are comparable, there is no need to add it again
• PP means "prompt processing" (bs = 512), TG means "text-generation" (bs = 1), t/s means "tokens per second"
• ✅ means the data has been added to the summary

Note that in this benchmark we are evaluating the performance against the same build 8e672ef (2023 Nov 21) in order to keep all performance factors even. Since then, there have been multiple improvements resulting in better absolute performance. As an example, here is how the same test compares over time on M2 Ultra:

	BW [GB/s]	GPU Cores	F16 PP [t/s]	F16 TG [t/s]	Q8_0 PP [t/s]	Q8_0 TG [t/s]	Q4_0 PP [t/s]	Q4_0 TG [t/s]
2023 Nov 21
M2 Ultra 8e672ef	800	76	1401.85	41.02	1248.59	66.64	1238.48	94.27
2024 Nov 12
M2 Ultra 86ed72d + FA	800	76	1525.95	43.15	1368.18	73.11	1391.78	108.80
2025 Aug 02
M2 Ultra 5c0eb5e + FA	800	76	1561.35	43.24	1386.97	73.35	1412.42	109.41

---

M1 Pro, 8+2 CPU, 16 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	302.14 ± 0.07
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.75 ± 0.00
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	270.37 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.34 ± 0.00
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	266.25 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	36.41 ± 0.01

build: 8e672ef (1550)

M2 Ultra, 16+8 CPU, 76 GPU (@ggerganov) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1401.85 ± 1.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	41.02 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1248.59 ± 0.73
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	66.64 ± 0.02
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1238.48 ± 0.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	94.27 ± 0.05

build: 8e672ef (1550)

M3 Max (MBP 14), 12+4 CPU, 40 GPU (@slaren) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	794.26 ± 3.16
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.27 ± 0.07
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	749.37 ± 8.35
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	43.00 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	690.99 ± 33.76
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.85 ± 0.22

build: d103d93 (1553)

Footnotes

1. https://en.wikipedia.org/wiki/Apple_M1#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. https://en.wikipedia.org/wiki/Apple_M2#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

3. https://en.wikipedia.org/wiki/Apple_M3#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

4. https://en.wikipedia.org/wiki/Apple_M4#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

5. https://en.wikipedia.org/wiki/Apple_M5#Variants ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

[QueryType]

M2 Mac Mini, 4+4 CPU, 10 GPU, 24 GB Memory (@QueryType) ✅

---

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	201.34 ± 0.21
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	6.72 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	181.40 ± 0.05
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	12.21 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	179.57 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	21.91 ± 0.02

build: 8e672ef (1550)

[brozkrut]

M2 Max Studio, 8+4 CPU, 38 GPU ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	755.67 ± 0.11
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.65 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	677.91 ± 0.26
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.83 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	671.31 ± 0.20
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	65.95 ± 0.08

build: 8e672ef (1550)

[maver1ck]

Wow. I wasn't aware that 4090 is so fast.

[vitali-fridman]

This is from one/two generations old hardware but it's for 70B model which might be of interest.

CPU: AMD 3995WX, GPU: 2x Nvidia 3090, Ubuntu 23.10, Kernel 6.5.0-14, NV Driver: 545.23.08, CUDA: 12.3.1

model	size	params	backend	ngl	test	t/s
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	pp 512	179.29 ± 2.83
llama 70B Q4_0	36.20 GiB	68.98 B	CUDA	99	tg 128	21.17 ± 0.04

For comparison, 7B model on the same hardware

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp 512	1178.60 ± 88.08
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg 128	87.34 ± 0.89

[zotona]

could you try at 7b model for correct comparation? Thanks!

[pukhrajvansh]

what the hell this is lower than m4 max, i mean 2x 3090 whatt..??

[atlas5301]

what the hell this is lower than m4 max, i mean 2x 3090 whatt..??

Probably because llama.cpp is not well optimized on gpus. You can expect significantly better throughput with sglang and vllm.

[crasm]

M2 Ultra, 16+8 CPU, 60 GPU (@crasm) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1128.59 ± 0.82
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	39.86 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1003.16 ± 0.39
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	62.14 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	1013.81 ± 0.92
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	88.64 ± 0.06

build: 8e672ef (1550)

[ymcui]

M3 Max (MBP 16), 12+4 CPU, 40 GPU (@ymcui) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	779.17 ± 0.49
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.09 ± 0.01
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	757.64 ± 1.03
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.75 ± 0.06
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	759.70 ± 2.26
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.31 ± 0.12

build: 55978ce (1555)

---

Short Note: mostly similar to the one reported by @slaren . But for Q4_0 pp 512, my result is 759.70 ± 2.26, while the one in the main post is 690.99 ± 33.76. Not sure about the source of the difference.

[slaren]

I am not sure why, but the results that I get are not very consistent. I suspect that it may due to the cooling limitations of the smaller laptop. I repeated the test now and the results are very similar to yours.

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	787.24 ± 0.84
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	25.15 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	755.88 ± 1.56
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	42.64 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	760.65 ± 0.77
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	66.35 ± 0.24

[Azirine]

In the graph, why is PP t/s plotted against bandwidth and TG t/s plotted against GPU cores? Seems like GPU cores have more effect on PP t/s.

[Azirine]

How about also sharing the largest model sizes and context lengths people can run with their amount of RAM? It's important to get the amount of RAM right when buying Apple computers because you can't upgrade later.

[ggerganov]

You can compute these. By default, you can use ~75% of the total RAM with the GPU. You can use more if you do some tricks

[minosvasilias]

M2 Pro, 6+4 CPU, 16 GPU (@minosvasilias) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	312.65 ± 15.75
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	12.47 ± 0.71
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	288.46 ± 0.06
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	22.70 ± 0.12
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	294.24 ± 0.10
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	37.87 ± 0.10

build: e9c13ff (1560)

[to3d]

Would love to see how M1 Max and M1 Ultra fare given their high memory bandwidth.

[MrSparc]

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 96 GB RAM (@MrSparc) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.50 ± 0.58
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	41.79 ± 0.04
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	669.51 ± 1.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	64.55 ± 1.36

build: e9c13ff (1560)

[rlippmann]

I'm also using a MBP16 M2Max with the same CPU/GPU specs, but only 32 gb ram and my results are roughly the same:

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 32 GB RAM ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	747.99 ± 0.28
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.54 ± 0.22
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	674.37 ± 0.63
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.67 ± 0.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	668.28 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	62.98 ± 0.06

build: 22da055 (1566)

[MrSparc]

Yes, it is expected that the same cpu/gpu spec will have similar performance values for same models to be compared regardless of RAM, as long as the size of the model to be used can be loaded into memory.
The amount of RAM is a limiting factor in the size of the model that can be loaded, as only 75% (by default) of the unified memory can be used as VRAM on the GPU
https://github.com/ggerganov/llama.cpp#memorydisk-requirements

[CedricYauLBD]

M1 Max (MBP 16) 8+2 CPU, 32 GPU, 64GB RAM (@CedricYauLBD) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	599.53 ± 0.86
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	23.03 ± 0.09
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	537.37 ± 0.19
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.20 ± 0.03
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	530.06 ± 0.17
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	61.19 ± 0.15

build: e9c13ff (1560)

Note: M1 Max RAM Bandwidth is 400GB/s

[philipturner]

Look at what I started

[yxzwayne]

off topic, but your benchmark output is my desktop rn :D

[paramaggarwal]

M3 Pro (MBP 14), 5+6 CPU, 14 GPU (@paramaggarwal) ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	272.11 ± 1.40
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	17.44 ± 0.42
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	269.49 ± 1.14
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	30.65 ± 0.20

build: e9c13ff (1560)

[ggerganov]

This one has 150 GB/s memory bandwidth, correct?

[paramaggarwal]

Yes, that's correct. (source)

[Kaszebe]

Could it run a Q5 quant of llama3 70b Instruct at ~2 tokens per second?

[mladencucakSYN]

I'm also interested to see if it can run a bit bigger model with some kind of reasonable outcome. Just don't want to spend MCB Max money

[bagobones]

The old models give excellent comparative numbers but I wonder if the benchmark needs to be re-based around the current most popular models at some point.

Not just bigger ones for finding the biggest but popular sets / distillations that go from small to very large.

It looks like 96-128 ish gigs of shared memory will be practical on Apple / AMD / nvidia digits going forward.

[brozkrut]

Chip (vs. Predecessor)	F16 PP	F16 TG	Q8_0 PP	Q8_0 TG	Q4_0 PP	Q4_0 TG
M2 Pro (16) vs. M1 Pro (16)	312.65 302.14	12.47 12.75	288.46 270.37	22.7 22.34	294.24 266.25	37.87 36.41
	+3.48%	-2.20%	+6.69%	+1.61%	+10.51%	+4.01%
M2 Max (38) vs. M1 Max (32)	755.67 599.53	24.65 23.03	677.91 537.37	41.83 40.2	671.31 530.06	65.95 61.19
	+26.04%	+7.03%	+26.15%	+4.05%	+26.65%	+7.78%
M2 Ultra (60) vs. M2 Max (38)	1128.59 755.67	39.86 24.65	1003.16 677.91	62.14 41.83	1013.81 671.31	88.64 65.95
	+49.34%	+61.90%	+48.04%	+48.48%	+51.03%	+34.41%
M2 Ultra (76) vs. M2 Max (38)	1401.85 755.67	41.02 24.65	1248.59 677.91	66.64 41.83	1238.48 671.31	94.27 65.95
	+85.67%	+66.45%	+84.24%	+59.47%	+84.53%	+43.06%
M2 Ultra (76) vs. M2 Ultra (60)	1401.85 1128.59	41.02 39.86	1248.59 1003.16	66.64 62.14	1238.48 1013.81	94.27 88.64
	+24.25%	+2.91%	+24.43%	+7.23%	+22.19%	+6.33%
M3 Pro (14) vs. M2 Pro (16)			272.11 288.46	17.44 22.7	269.49 294.24	30.65 37.87
			-5.67%	-23.17%	-8.41%	-19.07%
M3 Max (40) vs. M2 Max (38)	779.17 755.67	25.09 24.65	757.64 677.91	42.75 41.83	759.7 671.31	66.31 65.95
	+3.11%	+1.78%	+11.76%	+2.20%	+13.17%	+0.55%

[pudepiedj]

### M2 MAX (MBP 16) 38 Core 32GB ✅

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	754.39 ± 0.36
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	24.31 ± 0.38
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	671.33 ± 2.65
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	40.85 ± 0.32
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	664.07 ± 9.11
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	63.29 ± 0.15

build: 795cd5a (1493)

[MrSparc]

I'm looking at the summary plot about "PP performance vs GPU cores" and evidence that original unquantised fp16 model always delivers more performance than quantized models.
Sorry if my question is silly, I'm new to this area, but can someone explain to me why original model delivers more performance than quantized models? Thanks

[ggerganov]

The question is not silly - the observation is expected. At large batch size (PP means batch size of 512) the computation is compute bound. I.e. the speed depends on how many FLOPS you can utilize. For quantum models, the existing kernels require extra compute to dequantize the data compared to F16 models where the data is already in F16 format.

[Basten7]

Old Mac Intel with AMD GPU is not dead, yet !

That old Mac with Intel and an AMD GPU might be showing its age, but it’s far from useless and can still pack a punch today with the right tweaks.

export GGML_METAL_DEVICE_INDEX=1
./build/bin/llama-bench -ngl 99 -m ~/Models/llama-2-7b-q4_0.gguf
ggml_metal_init: found device: AMD Radeon RX 6900 XT

./build/bin/llama-bench -ngl 99 -m ~/Models/llama-2-7b-q4_0.gguf

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	12	pp512	331.84 ± 0.65
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	12	tg128	98.54 ± 0.83

build: 79c1160 (6123 with metal3 patch)

export GGML_METAL_DEVICE_INDEX=2
./build/bin/llama-bench -ngl 99 -m ~/Models/llama-2-7b-q4_0.gguf
ggml_metal_init: found device: AMD Radeon PRO W6800

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	12	pp512	246.19 ± 0.79
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	12	tg128	85.82 ± 0.23

build: 79c1160 (6123 with metal3 patch)

export GGML_METAL_DEVICE_INDEX=3
./build/bin/llama-bench -ngl 99 -m ~/Models/llama-2-7b-q4_0.gguf
ggml_metal_init: found device: AMD Radeon PRO W6800X Duo

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	12	pp512	251.48 ± 0.19
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	12	tg128	96.24 ± 0.19

build: 79c1160 (6123 with metal3 patch)

The tweaks : https://gist.github.com/Basten7/f316fef96aac9a6614032a65c9825eaf

[netrunnereve]

There's some speculation online about how the new A19 chip has the NPU cores inside the GPU instead of being a seperate module, which might indicate that it can be used like tensor cores to help with prompt processing.

[bluemoehre]

So, let's reignite the discussion:
M5 is already there and M5 Pro / M5 Ultra is just around the corner.
Personally I don't believe rumors about M5 Extreme, but lets see what Apple puts into the next gen Mac Pro having space for like 4x NVIDIA GPUs? 😁

It looks like we have a good chance of ~30% LLM performance increase:
https://browser.geekbench.com/v6/compute/4916492

[ggerganov]

Definitely interesting. I suppose the newly introduced tensor types in MSL4 will be the way to utilize the new NPU cores (https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf). Hopefully we can easily adapt the implementation to support this.

[bluemoehre]

Would be awesome if both xPUs could work together. Maybe things would already speed up if one handles PP and the other TG? I mean, we’ve already got Performance and Efficiency Cores in CPUs, so maybe in the future you could even pick which processing / layers / experts run where.

I hope the M5 Pro supports 128GB+ at about 325GB/s (and that Apple doesn't get too greedy with $). That way you could run multiple models in the background and still use the machine normally. That said, I've still got pre-orders for the AMD Strix Halo 395+ 🥲

[bluemoehre]

@ggerganov time to update the table 😉 - https://www.apple.com/macbook-pro/

• M5: 153 GB/s

[bluemoehre]

M5 Max has 1,5x PP performance vs M3 Ultra / 2,3x performance vs M4 Max ... now I need to order one.

[Anemll]

M5 (base) benchmark , 2.6 speedup, FA is slower for TG, but faster for prefill?

lama-bench
-m llama-7b-v2/ggml-model-f16.gguf
-m llama-7b-v2/ggml-model-Q8_0.gguf
-m llama-7b-v2/ggml-model-Q4_0.gguf
-p 512 -n 128 -ngl 99 -o md 2>/dev/null

M5 Without Neural Accelerator

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	257.79 ± 0.94
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	9.43 ± 0.12
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	264.15 ± 2.26
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	16.62 ± 0.15
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	247.68 ± 2.63
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	29.62 ± 0.33
build: 8e672ef (1550)

M5 with Neural Accelerator

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	Metal,BLAS	4	pp512	374.35 ± 1.95
llama 7B F16	12.55 GiB	6.74 B	Metal,BLAS	4	tg128	9.67 ± 0.06
llama 7B Q8_0	6.67 GiB	6.74 B	Metal,BLAS	4	pp512	489.78 ± 3.90
llama 7B Q8_0	6.67 GiB	6.74 B	Metal,BLAS	4	tg128	17.50 ± 0.38
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	4	pp512	636.36 ± 1.92
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	4	tg128	31.02 ± 0.37
build: 9fce244 (6817)

Also Metal4 plus -fa 1

model	size	params	backend	threads	fa	test	t/s
llama 7B F16	12.55 GiB	6.74 B	Metal,BLAS	4	1	pp512	378.36 ± 4.69
llama 7B F16	12.55 GiB	6.74 B	Metal,BLAS	4	1	tg128	9.56 ± 0.11
llama 7B Q8_0	6.67 GiB	6.74 B	Metal,BLAS	4	1	pp512	512.15 ± 2.73
llama 7B Q8_0	6.67 GiB	6.74 B	Metal,BLAS	4	1	tg128	17.55 ± 0.24
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	4	1	pp512	650.65 ± 5.06
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	4	1	tg128	30.67 ± 0.25
build: 9fce244 (6817))

[ggerganov]

FA is slower for TG, but faster for prefill?

Looks like noise to me.

[Hassan-A-K]

1 | model                          |       size |     params | backend    | ngl | test       |              t/s |
2 | ------------------------------ | ---------: | ---------: | ---------- | --: | ---------- | ---------------: |
3 | llama 7B mostly F16            |  12.55 GiB |     6.74 B | Metal      |  99 | pp 512     |    736.25 ± 1.88 |
4 | llama 7B mostly F16            |  12.55 GiB |     6.74 B | Metal      |  99 | tg 128     |     24.29 ± 0.18 |
5 | llama 7B mostly Q8_0           |   6.67 GiB |     6.74 B | Metal      |  99 | pp 512     |    718.56 ± 0.78 |
6 | llama 7B mostly Q8_0           |   6.67 GiB |     6.74 B | Metal      |  99 | tg 128     |     43.87 ± 0.02 |
7 | llama 7B mostly Q4_0           |   3.56 GiB |     6.74 B | Metal      |  99 | pp 512     |    713.93 ± 0.25 |
8 | llama 7B mostly Q4_0           |   3.56 GiB |     6.74 B | Metal      |  99 | tg 128     |     69.95 ± 0.15 |

build: 8e672ef (1550)
M4 Max 14C 32C 36GB 1TB ✅

[arashakb]

We need a comparison between llama.cpp inference speed and PyTorch inference speed. A fair comparison.
No one cares about the llama.cpp speed alone. if Pytroch inference speed is faster or on par, honestly there is no clear motivation on why people should use llama.cpp

All of the quantization algorithms can be written in PyTorch too. it's just engineering work. But if someone shows that having the inference in pure C++ has clear benefits and we can actually run the inference faster (on CPU or GPU or both) that is something people will invest on.

We do not have any clear proof showing llama.cpp is actually faster than the PyTorch implementation on either CPU or GPU. if we have that please educate me.

[tarruda]

here is no clear motivation on why people should use llama.cpp

How about :

• Not having to deal with python dependencies
• builtin-web UI. Nothing else needs to be installed to start using LLMs interactively
• Multiple inference backends, mix CPU/GPU inference.
• Easy distributed inference with RPC. You can even mix different platforms.

[arashakb]

@j05hau
Jury's out on "convenience" part lol
I do agree with portability but still I don't think people would sacrifice inference speed over portability. or at least people want to know if their model will become slower or not.

I actually like llama.cpp but the fact that there is no benchmark for speed compared to PyTorch is just so weird and I think many people agree with me on this.

[arashakb]

here is no clear motivation on why people should use llama.cpp

How about :

• Not having to deal with python dependencies
• builtin-web UI. Nothing else needs to be installed to start using LLMs interactively
• Multiple inference backends, mix CPU/GPU inference.
• Easy distributed inference with RPC. You can even mix different platforms.

these are valid. Thank you for mentioning them.
still I think since llama.cpp is all about running inference, the most important thing is the inference speed. that't the main reason why different inference engines got created (vLLM, TensorRT-LLM, etc) and yet there is no clear answer or comparison on this for llama.cpp compared to even a basic transformers forward pass.
I hope this will become more clear in future.

[tarruda]

Most of the times the inference speed of llama.cpp is enough for me.

The best thing about llama.cpp are the amazing dynamic quants. A recent example was Qwen 3.5 397B which maintains decent amount of intelligence even with a 2-bit quant as I've shown here: https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF/discussions/8. I can run a 400B parameter model on my 128G mac because of those quants. There's simply no alternative.

[arthurhjorth]

What a weird comment to leave here.

First, this is not the right place for it. This thread is for performance measures across M-series chipsets, not for meta discussions.

Second, people do care. This helps people shop for M-chipsets that best suit their purpose/budget.

Third, and most important, if this is so important to you, why don't you write up the code to run comparable benchmarks? This is an open source project. Showing up and demanding that other people do the work to show you data just because you want them is.. lol.

If this is important to you, create an issue. The heading could be something like "Comparable benchmarks across inference frameworks?". Content: Hey guys, I think it would be really useful to have comparable inference data across llama.cpp, mlx, pytorch, etc. Here's a repo where I run these comparisons. Comparing across frameworks is tricky because they don't all support the same dtypes and quants, so here are my thoughts on fair comparisons. ... If I maintain this repo and keep it up to date for new models, quants etc., would there be interest in somehow making my data easily accessible here, so people can get the most accurate information possible?"

You're welcome.

[elfarolab]

and you are PhD too ..

[hnfong]

I came here trying to get a sense of the M5 speedup given the new Metal API in M5 chips.

ggerganov hasn't updated the post yet, but I found useful info in #16634

[d-shehu]

I'm holding out for a benchmark of M5 Max and assuming there is a Mac Studio launching soon. I'd like to see some numbers comparing it to a Spark and Strix Halo respectively.

Anyone have $5K laying around for a new Macbook Pro ... ;)

https://www.apple.com/shop/buy-mac/macbook-pro/14-inch

[olegshulyakov]

There is Strix Halo result.

[Hassan-A-K]

Done sorry for the delay also doing M5 10C soon

[JJJ-M]

This is a great project you created, I am contributing.
It would be great to add the quantization and MLX or GGUF, so we can make exact comparisons.

[uncSoft]

Added! there was a bug in sparkle, please manually update to the latest release: https://github.com/uncSoft/anubis-oss/releases/latest

[Hassan-A-K]

M5 Max (MBP 16"), 6+12 CPU, 40 GPU, 64 GB

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1018.30 ± 0.89
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	37.58 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1051.59 ± 0.67
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	64.61 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	987.10 ± 1.05
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	102.93 ± 0.17

build: 8e672ef (1550)
M5 Max 18C 40C 64GB 1TB ✅

[ggerganov]

Hm, interesting observation. I suppose the tensor API does a better job at interleaving memory reads with the computation.

[arthurhjorth]

Ah, thanks! If this holds for all models (and I don't see why it wouldn't) then that would really help alleviate the M-chipsets' Achilles heel.

[CSendner]

just to extend/confirm the initial data:

M5 Max (MBP 16"), 6+12 CPU, 40 GPU, 128 GB

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	1032.18 ± 1.34
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	38.06 ± 0.04
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	1054.83 ± 0.70
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	65.11 ± 0.07
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	990.53 ± 0.44
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	104.03 ± 0.09

build: 8e672ef (1550)

[Razvan-H-Andrei]

What about M5 Pro? does anyone have any data?

[bnggge]

What about M5 Pro? does anyone have any data?

M5 Pro (MBP 14“), 10+5 CPU, 16 GPU, 24 GB

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	pp512	1310.28 ± 8.21
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	tg128	21.11 ± 0.18
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	pp512	1304.66 ± 1.03
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	tg128	37.78 ± 0.01
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	pp512	1350.43 ± 1.97
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	tg128	63.86 ± 0.06

build: c3a1128 (8509)

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	pp512	695.51 ± 0.74
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	5	tg128	21.37 ± 0.01
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	pp512	964.18 ± 0.87
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	5	tg128	37.70 ± 0.14
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	pp512	1135.71 ± 9.26
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	5	tg128	63.74 ± 0.06

build: 073bb2c (8762)

model	size	params	backend	ngl	test	t/s
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	pp 512	423.30 ± 0.14
llama 7B mostly F16	12.55 GiB	6.74 B	Metal	99	tg 128	20.22 ± 0.02
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	pp 512	431.14 ± 0.10
llama 7B mostly Q8_0	6.67 GiB	6.74 B	Metal	99	tg 128	35.86 ± 0.24
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	pp 512	403.19 ± 0.22
llama 7B mostly Q4_0	3.56 GiB	6.74 B	Metal	99	tg 128	60.04 ± 0.21

build: 8e672ef (1550)

[SpyderMYK]

Apple M-series benchmark — mac-mini-m4

Hardware: Apple M4, 10 cores, 24 GB RAM, macOS 26.4
llama-bench:
Settings: -p 512 -n 128 -ngl 99 -r 3 (Metal backend, full GPU offload)

llama3.2:3b

GGUF: sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff

model	size	params	backend	threads	test	t/s
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	BLAS,MTL	4	pp512	530.01 ± 0.86
llama 3B Q4_K - Medium	1.87 GiB	3.21 B	BLAS,MTL	4	tg128	45.95 ± 0.02

build: 2e97c5f (9100)

qwen2.5-coder:7b

GGUF: sha256-60e05f2100071479f596b964f89f510f057ce397ea22f2833a0cfe029bfc2463

model	size	params	backend	threads	test	t/s
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	BLAS,MTL	4	pp512	239.73 ± 2.07
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	BLAS,MTL	4	tg128	22.60 ± 0.01

build: 2e97c5f (9100)

[michaelasper]

Apple M5 Max - 128gb - 16"
Total Number of Cores: 18 (6 Super and 12 Performance)
Total Number of GPU Cores: 40

model	size	params	backend	threads	test	t/s
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	pp512	3189.90 ± 8.62
llama 7B F16	12.55 GiB	6.74 B	MTL,BLAS	6	tg128	35.39 ± 0.10
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	pp512	3219.94 ± 2.06
llama 7B Q8_0	6.67 GiB	6.74 B	MTL,BLAS	6	tg128	72.52 ± 0.11
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	6	pp512	3347.44 ± 1.53
llama 7B Q4_0	3.56 GiB	6.74 B	MTL,BLAS	6	tg128	119.08 ± 0.69

build: 52b3df0 (9754)

./bin/llama-bench \
  -m ../models/llama-7b-v2/ggml-model-f16.gguf  \
  -m ../models/llama-7b-v2/ggml-model-q8_0.gguf \
  -m ../models/llama-7b-v2/ggml-model-q4_0.gguf \
  -p 512 -n 128 -ngl 99

[bluemoehre]

Holy cow ... we're gonna get milked 😅

Apple plans a new Mac Studio by the end of the year, codenamed J775, which will feature an M5 Ultra for the first time (codename “Sotra D” or H17D). It is planned to have up to 36 CPU cores and 80 GPU cores. Apple is said to have planned to deliver the computers with up to 768 GByte of RAM [...]

Source: Heise