T
Models

Win Rate
humaneval-python
java
javascript
Throughput (tokens/s)
🔴
DeepSeek-Coder-33b-instruct

39.58
80.02
52.03
65.13
25.2
🔴
DeepSeek-Coder-7b-instruct

38.75
80.22
53.34
65.8
51
🔶
Phind-CodeLlama-34B-v2

37.04
71.95
54.06
65.34
15.1
🔶
Phind-CodeLlama-34B-v1

36.12
65.85
49.47
64.45
15.1
🔶
Phind-CodeLlama-34B-Python-v1

35.27
70.22
48.72
66.24
15.1
🔴
DeepSeek-Coder-33b-base

35
52.45
43.77
51.28
25.2
🔶
WizardCoder-Python-34B-V1.0

33.96
70.73
44.94
55.28
15.1
🔴
DeepSeek-Coder-7b-base

31.75
45.83
37.72
45.9
51
🔶
CodeLlama-34b-Instruct

30.96
50.79
41.53
45.85
15.1
🔶
WizardCoder-Python-13B-V1.0

30.58
62.19
41.77
48.45
25.3
🟢
CodeLlama-34b

30.35
45.11
40.19
41.66
15.1
🟢
CodeLlama-34b-Python

29.65
53.29
39.46
44.72
15.1
🔶
WizardCoder-15B-V1.0

28.92
58.12
35.77
41.91
43.7
🔶
CodeLlama-13b-Instruct

27.88
50.6
33.99
40.92
25.3
🟢
CodeLlama-13b

26.19
35.07
32.23
38.26
25.3
🟢
CodeLlama-13b-Python

24.73
42.89
33.56
40.66
25.3
🔶
CodeLlama-7b-Instruct

23.69
45.65
28.77
33.11
33.1
🟢
CodeLlama-7b

22.31
29.98
29.2
31.8
33.1
🔴
CodeShell-7B

22.31
34.32
30.43
33.17
33.9
🔶
OctoCoder-15B

21.15
45.3
26.03
32.8
44.4
🟢
Falcon-180B

20.9
35.37
28.48
31.68
-1
🟢
CodeLlama-7b-Python

20.62
40.48
29.15
36.34
33.1
🟢
StarCoder-15B

20.58
33.57
30.22
30.79
43.9
🟢
StarCoderBase-15B

20.15
30.35
28.53
31.7
43.8
🟢
CodeGeex2-6B

17.42
33.49
23.46
29.9
32.7
🟢
StarCoderBase-7B

16.85
28.37
24.44
27.35
46.9
🔶
OctoGeeX-7B

16.65
42.28
19.33
28.5
32.7
🔶
WizardCoder-3B-V1.0

15.73
32.92
24.34
26.16
50
🟢
CodeGen25-7B-multi

15.35
28.7
26.01
26.27
32.6
🔶
Refact-1.6B

14.85
31.1
22.78
22.36
50
🔴
DeepSeek-Coder-1b-base

14.42
32.13
27.16
28.46
-1
🟢
StarCoderBase-3B

11.65
21.5
19.25
21.32
50
🔶
WizardCoder-1B-V1.0

10.35
23.17
19.68
19.13
71.4
🟢
Replit-2.7B

8.54
20.12
21.39
20.18
42.2
🟢
CodeGen25-7B-mono

8.15
33.08
19.75
23.22
34.1
🟢
StarCoderBase-1.1B

8.12
15.17
14.2
13.38
71.4
🟢
CodeGen-16B-Multi

7.08
19.26
22.2
19.15
17.2
🟢
Phi-1

6.25
51.22
10.76
19.25
-1
🟢
StableCode-3B

6.04
20.2
19.54
18.98
30.2
🟢
DeciCoder-1B

5.81
19.32
15.3
17.85
54.6
🟢
SantaCoder-1.1B

4.58
18.12
15
15.47
50.8