Arabic + English speech recognition on phone calls: eval-v1 benchmarks

Word error rates of our fine-tunes (SFT, then RL) and their base models on the eval-v1 test sets: real phone calls in Arabic, English and code-switched speech, accented call-centre English, noisy Arabic, and the Arabic and English Arenas. A community project: not made, affiliated with or endorsed by Cohere. Built 2026-10-29.

Models

Column	Checkpoint	What it is
Cohere soup (main)	Cohere-AR/cohere-transcribe-ar-soup-v4c70 @ `main`	Ours, the best on the Arena: a weight average of 0.7 x Cohere RL v4c and 0.3 x Cohere RL (round 4, step 800), no further training. Its own page is Cohere-AR/cohere-transcribe-ar-soup-v4c70; the same weights are the main of Cohere-AR/cohere-transcribe-ar-rl since 2026-10-29.
Cohere RL v4c	Cohere-AR/cohere-transcribe-ar-rl-v4c @ `main`	Ours: SFT v4 (every human-annotated SIP train folder but the selection dev set's, the order calls twice), then two MWER stages of 800 steps (v4b, v4c) on pools with the SIP calls cut with the benchmark's edge padding, their digit grouping scored, and Arabic-English code-switching clips. Its own page: Cohere-AR/cohere-transcribe-ar-rl-v4c.
Cohere RL	Cohere-AR/cohere-transcribe-ar-rl @ `r4-step-800`	Ours, the best on the Arabic and English Arena: Cohere SFT step 10,000, then three MWER rounds (rounds 1, 2 and 4: 800 steps each on pools of ~45-48k human-labelled clips, with a penalty for English words not written in Latin script); the decoder learns, the encoder is frozen. Branch r4-step-800 (main until 2026-10-29).
Cohere SFT	Cohere-AR/cohere-transcribe-ar-sft @ `step-10000`	Ours: a full supervised fine-tune of Cohere Transcribe Arabic, 10,000 steps over 69 sources (8,535 h of audio seen: Arabic dialects, Arabic and English phone calls, accented and code-switched speech). Branch step-10000 is main.
Cohere base	CohereLabs/cohere-transcribe-arabic-07-2026 @ `c3e911b`	The starting point of the SFT.
Qwen3-ASR base	Qwen/Qwen3-ASR-1.7B-hf @ `bcd2b5b`	Qwen3-ASR-1.7B in the transformers layout.
Qwen3-ASR run20	Arabic-STT/qwen3-asr-1.7b-ar-run20-mwer-s800 @ `1e523e8`	Ours, the previous best: Qwen3-ASR-1.7B after several SFT rounds and MWER post-training (step 800).

Overview

WER %, lower is better; the best number of each row in bold. Hover a number for its 95 % interval, the raw number and the substitution / deletion / insertion split.

Cohere soup (main) is best of the 7 in 13 of 57 rows.
Cohere soup (main) vs Cohere RL v4c: lower in 35 rows, higher in 19, equal in 3.
Cohere soup (main) vs Cohere RL: lower in 28 rows, higher in 25, equal in 4.
Cohere soup (main) vs Cohere SFT: lower in 47 rows, higher in 8, equal in 2.
Cohere soup (main) vs Cohere base: lower in 51 rows, higher in 6.
Cohere soup (main) vs Qwen3-ASR base: lower in 48 rows, higher in 9.
Cohere soup (main) vs Qwen3-ASR run20: lower in 37 rows, higher in 19, equal in 1.

Benchmark	Headline slice	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
Arabic Arena (official board)	v2 Arena, 87 ranked models	13.03	13.13	13.13	13.28	17.54	28.97	14.16
English Arena (official board)	v2 Arena, 53 ranked models	5.80	5.83	5.81	5.89	6.69	6.82	7.09
Arabic + English Arena (average)	mean of the two v2 headlines, 51 ranked models on both boards	9.41	9.48	9.47	9.59	12.12	17.90	10.62
Audioluna	20 calls	12.20	12.20	11.98	12.65	16.60	20.93	11.32
SIP test	all scorable segments	14.66	14.52	15.83	15.93	30.28	45.76	14.15
test-arabic	all segments	9.50	9.54	9.28	9.78	13.87	18.24	10.04
CallHome EN	140 excerpts	12.54	12.66	12.53	12.95	16.49	12.47	13.75
AppTek EN	mean of 14 accents	9.28	9.18	9.45	12.58	13.72	8.39	9.28
SVQ AR	all environments	22.07	22.21	22.12	21.75	20.85	30.84	21.31
Arabic Arena v2	arena-clean headline (19 sets)	12.27	12.39	12.35	12.73	16.83	28.32	13.45
English Arena v2	arena-clean headline (24 sets)	5.43	5.49	5.43	5.55	6.26	6.36	6.73

Arabic Arena (official board)

The board of the Arena Space itself (kadirnar/arabic-arena), read when this page was built: 88 models, each scored by the Arena's own code on every one of its 37 weighted set conditions (the eval-v1 rows below hold only some of them, cleaned); the v2 headline is a weighted geometric mean over the tracks (clean 50 %, code-switch 15 %, noisy 15 %, telephony 10 %, call-centre 10 %). Closed APIs are unranked references.

Measure	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
Arena v2 (the board's headline)	13.03	13.13	13.13	13.28	17.54	28.97	14.16
Rank among 87 ranked models	1	3	2	5	32	49	10
Real-audio sets only	13.10	13.17	13.19	13.35	17.81	28.38	14.41
Zero-shot sets only	16.57	16.67	16.63	16.89	22.25	32.45	17.80
CER Arena	6.78	6.83	6.79	6.89	10.13	13.80	7.09
Track: clean (50 %)	12.57	12.65	12.69	12.68	15.87	25.85	13.96
Track: code-switch (15 %)	14.58	14.69	14.69	15.53	28.22	41.74	15.91
Track: noisy (15 %)	11.60	11.65	11.68	11.73	13.85	23.65	12.26
Track: telephony (10 %)	11.37	11.54	11.34	11.43	14.64	28.35	12.15
Track: call-centre (10 %)	17.98	18.23	18.10	18.46	24.23	41.10	18.51
Noise-only clips with 3+ invented words, %	51.00	50.33	50.33	50.33	64.33	37.33	41.67

Arabic Arena (official board) per set (37 weighted set conditions, WER v2)

Set / condition	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
fleurs_ar/clean	3.87	3.88	3.97	4.11	4.04	8.41	4.43
cv18_ar/clean	4.00	4.08	4.16	4.02	4.32	14.04	6.17
masc_clean/clean	3.77	3.84	3.81	3.68	4.01	10.40	4.88
mgb2/clean	7.11	7.04	7.38	7.44	9.75	14.50	8.75
sada/clean	22.02	22.00	22.00	22.57	29.51	38.97	21.37
casablanca/clean	32.64	32.81	32.69	32.66	36.78	55.52	34.70
mgb3_egy/clean	7.24	7.35	7.29	7.54	12.92	29.02	8.22
mgb5_mor/clean	24.99	25.43	25.27	25.06	24.24	57.47	25.47
svq_ar/clean	14.68	14.78	14.46	14.01	14.01	22.02	15.64
shams_lev/clean	18.37	18.36	18.19	18.30	32.03	31.78	18.88
hassaniya/clean	52.29	52.21	52.34	52.17	67.15	66.57	54.55
wasil/clean	11.24	11.35	11.17	11.27	11.70	24.03	12.01
echonova/clean	15.51	15.51	15.63	15.55	33.63	26.39	16.97
massive_ar/clean	10.93	11.08	11.28	11.26	12.66	21.15	11.22
gsb_ar/clean	21.23	21.23	21.31	21.30	29.24	38.24	22.26
mixat_cs/clean	10.75	10.74	10.68	11.02	14.23	17.86	12.03
algerian_cs/clean	25.19	25.54	25.71	25.88	38.76	48.73	27.94
perle_cs/clean	7.62	7.74	7.76	9.18	15.67	44.45	9.50
tedxtn_cs/clean	21.90	21.93	21.86	22.22	73.31	78.43	20.06
masc_noisy/clean	12.87	12.85	12.83	12.99	13.52	22.20	13.75
svq_ar_noisy/clean	15.97	15.82	15.60	15.36	14.92	23.28	15.82
fleurs_ar/noisy	5.13	5.20	5.22	5.27	5.31	11.77	5.91
cv18_ar/noisy	8.35	8.68	8.95	9.02	11.76	25.78	9.82
mgb2/noisy	9.79	9.73	9.65	9.57	13.89	21.47	10.34
sada/noisy	28.22	27.99	28.16	28.75	40.39	51.97	26.01
fleurs_ar/telephony_v2	4.58	4.64	4.60	4.72	5.01	11.99	5.51
cv18_ar/telephony_v2	6.88	7.15	7.01	6.95	7.68	21.58	7.62
mgb2/telephony_v2	7.86	7.95	7.84	7.89	10.48	17.82	8.74
sada/telephony_v2	25.92	25.84	25.84	26.55	35.62	47.03	24.02
casablanca/telephony_v2	34.09	34.38	34.00	33.58	39.80	64.53	35.05
mgb3_egy/telephony_v2	9.86	10.09	9.59	9.67	17.21	37.11	10.41
fleurs_ar/callcenter_v2	7.46	7.74	7.37	7.48	8.26	20.17	8.90
cv18_ar/callcenter_v2	14.09	14.50	14.63	14.90	18.44	35.94	15.83
mgb2/callcenter_v2	12.65	12.68	12.68	12.92	17.71	30.57	12.76
sada/callcenter_v2	33.86	33.71	33.96	35.26	47.72	59.53	30.53
casablanca/callcenter_v2	43.84	44.36	44.32	43.49	52.05	72.58	42.78
mgb3_egy/callcenter_v2	17.13	17.27	17.11	17.94	30.20	50.31	17.11

Top 10 of the board

Rank	Model	Weights	Arena v2
1	cohere-transcribe-ar-soup-v4c70	gated	13.03
2	cohere-transcribe-ar-rl-r4-step-800	gated	13.13
3	cohere-transcribe-ar-rl-v4c	gated	13.13
4	cohere-transcribe-ar-rl-step-800	gated	13.18
5	cohere-transcribe-ar-sft-step-10000	gated	13.28
6	cohere-transcribe-ar-sft-step-6000	gated	13.54
7	arabic-stt-qwen3-asr-1.7b-best-grpo2	gated	13.65
8	arabic-stt-qwen3-asr-1.7b-best-task06	gated	13.75
9	arabic-stt-qwen3-asr-1.7b-run20-mwer-s1000	gated	14.16
10	arabic-stt-qwen3-asr-1.7b-run20-mwer-s800	gated	14.16
–	soniox-stt-async-v5	api: unranked reference	17.37

English Arena (official board)

The board of the Arena Space itself (kadirnar/arabic-arena), read when this page was built: 54 models, each scored by the Arena's own code on every one of its 29 weighted set conditions (the eval-v1 rows below hold only some of them, cleaned); the v2 headline is a weighted geometric mean over the tracks (clean 35 %, accented 5 %, noisy 15 %, telephony 22 %, call-centre 22 %). Closed APIs are unranked references.

Measure	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
Arena v2 (the board's headline)	5.80	5.83	5.81	5.89	6.69	6.82	7.09
Rank among 53 ranked models	1	3	2	5	7	9	13
Real-audio sets only	4.41	4.44	4.43	4.53	4.90	4.92	5.59
Zero-shot sets only	5.41	5.47	5.40	5.67	6.76	5.52	6.29
CER Arena	3.49	3.49	3.50	3.57	4.20	4.25	4.32
Track: clean (35 %)	4.03	4.05	4.07	4.07	4.37	4.65	5.30
Track: accented (5 %)	12.03	12.02	12.10	12.30	13.84	12.45	13.20
Track: noisy (15 %)	6.05	6.12	5.98	6.08	6.69	7.04	7.06
Track: telephony (22 %)	4.42	4.44	4.40	4.59	4.92	4.71	5.35
Track: call-centre (22 %)	11.10	11.12	11.12	11.21	15.09	15.33	12.89
Noise-only clips with 3+ invented words, %	45.67	46.67	45.00	97.67	45.00	44.33	35.33

English Arena (official board) per set (29 weighted set conditions, WER v2)

Set / condition	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
librispeech_clean/clean	1.02	1.03	1.03	1.02	1.04	1.17	1.79
librispeech_other/clean	2.04	2.04	2.10	2.09	2.00	2.92	4.24
fleurs_en/clean	5.16	5.29	5.25	5.25	6.09	4.32	5.85
common_voice_en/clean	6.60	6.70	6.69	6.62	6.53	7.03	8.84
voxpopuli_en/clean	5.48	5.56	5.49	5.47	5.06	5.73	6.92
earnings22/clean	7.31	7.37	7.30	7.39	9.60	9.77	7.88
gigaspeech/clean	7.57	7.58	7.58	7.58	7.82	7.58	8.24
spgispeech/clean	1.95	1.94	1.97	1.96	2.95	3.07	2.66
ami_sdm/clean	6.65	6.61	6.76	6.86	6.24	7.48	7.07
edacc/clean	12.03	12.02	12.10	12.30	13.84	12.45	13.20
urgent2024/clean	8.91	8.96	8.72	9.10	9.71	9.75	9.93
librispeech_clean/noisy	2.09	2.07	2.07	2.09	1.67	1.79	2.49
fleurs_en/noisy	6.96	7.05	6.85	7.04	9.21	8.48	8.16
voxpopuli_en/noisy	6.60	6.86	6.50	6.48	6.32	8.25	8.65
earnings22/noisy	9.48	9.60	9.48	9.56	14.15	14.11	10.05
minds14_tel/clean	1.20	1.22	1.16	1.34	1.49	0.95	1.50
vystadial_voip/clean	4.02	4.10	3.99	4.68	5.22	4.83	4.70
librispeech_clean/telephony	1.39	1.36	1.46	1.48	1.46	1.65	2.19
fleurs_en/telephony	5.56	5.54	5.65	5.73	6.67	5.12	6.53
voxpopuli_en/telephony	6.53	6.56	6.50	6.51	5.48	6.73	7.73
earnings22/telephony	7.27	7.48	7.05	7.09	9.42	9.62	7.88
common_voice_en/telephony	11.25	11.18	11.04	10.93	11.25	12.80	12.91
gigaspeech/telephony	7.29	7.35	7.27	7.34	7.72	7.61	8.40
librispeech_clean/callcenter	4.72	4.77	4.65	4.84	7.71	8.23	6.01
fleurs_en/callcenter	11.74	11.85	12.09	11.96	16.72	14.07	13.52
voxpopuli_en/callcenter	8.99	8.87	8.96	8.94	10.07	11.94	11.19
earnings22/callcenter	13.26	13.26	13.26	13.32	20.06	19.49	14.20
common_voice_en/callcenter	25.02	24.78	24.74	25.05	31.07	33.08	26.68
gigaspeech/callcenter	11.33	11.46	11.44	11.51	14.61	14.55	13.29

Top 10 of the board

Rank	Model	Weights	Arena v2
1	cohere-transcribe-ar-soup-v4c70	gated	5.80
2	cohere-transcribe-ar-rl-r4-step-800	gated	5.81
3	cohere-transcribe-ar-rl-v4c	gated	5.83
4	cohere-transcribe-ar-rl-step-800	gated	5.84
5	cohere-transcribe-ar-sft-step-10000	gated	5.89
6	cohere-transcribe-ar-sft-step-6000	gated	5.94
7	cohere-transcribe-arabic-07-2026	open	6.69
8	cohere-transcribe-03-2026	open	6.77
9	qwen3-asr-1.7b	open	6.82
10	arabic-stt-qwen3-asr-1.7b-best-task06	gated	6.91
–	soniox-stt-async-v5	api: unranked reference	9.18

Arabic + English Arena (average)

The plain mean of the two official v2 headlines above (Arabic and English weigh the same), for every model on both boards (51 ranked). Closed APIs are unranked references.

Measure	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
Average of the Arabic and English v2 headlines	9.41	9.48	9.47	9.59	12.12	17.90	10.62
Rank among 51 ranked models on both boards	1	3	2	5	31	34	10
Arabic Arena v2	13.03	13.13	13.13	13.28	17.54	28.97	14.16
English Arena v2	5.80	5.83	5.81	5.89	6.69	6.82	7.09

Top 10 by the average

Rank	Model	Weights	Arabic v2	English v2	Average
1	cohere-transcribe-ar-soup-v4c70	gated	13.03	5.80	9.41
2	cohere-transcribe-ar-rl-r4-step-800	gated	13.13	5.81	9.47
3	cohere-transcribe-ar-rl-v4c	gated	13.13	5.83	9.48
4	cohere-transcribe-ar-rl-step-800	gated	13.18	5.84	9.51
5	cohere-transcribe-ar-sft-step-10000	gated	13.28	5.89	9.59
6	cohere-transcribe-ar-sft-step-6000	gated	13.54	5.94	9.74
7	arabic-stt-qwen3-asr-1.7b-best-grpo2	gated	13.65	6.94	10.29
8	arabic-stt-qwen3-asr-1.7b-best-task06	gated	13.75	6.91	10.33
9	arabic-stt-qwen3-asr-1.7b-run20-mwer-s1000	gated	14.16	7.08	10.62
10	arabic-stt-qwen3-asr-1.7b-run20-mwer-s800	gated	14.16	7.09	10.62
–	soniox-stt-async-v5	api: unranked reference	17.37	9.18	13.27

Audioluna

20 phone calls with human gold-standard transcripts (kadirnar/audioluna-asr-call-samples): 10 English, 6 code-switched Arabic-English and 4 Arabic. A two-channel call is mixed to mono, cut at pauses into chunks of at most 25 s and decoded whole. Primary metric: the v5 scorer (word edits per call, pooled over the calls); the last rows score the same output against the dataset's own references or with other normalisers.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
20 calls	v5 scorer, gold standard	12.20	12.20	11.98	12.65	16.60	20.93	11.32
EN calls (10)	v5 scorer, gold standard	7.20	7.17	7.09	7.46	7.96	7.89	7.11
MIX calls (6)	v5 scorer, gold standard	23.54	23.95	23.32	24.87	36.32	45.80	20.65
AR calls (4)	v5 scorer, gold standard	11.41	10.80	10.74	11.07	15.03	27.21	11.02
verified calls (8)	v5 scorer, gold standard	5.67	5.69	5.61	5.67	6.58	7.15	5.42
calls with unresolved spans (4)	v5 scorer, gold standard	9.96	9.73	9.83	10.52	11.35	12.67	10.36
draft calls (8)	v5 scorer, gold standard	22.71	22.87	22.21	23.69	34.20	45.96	19.97
20 calls	v5 scorer, the dataset's references	12.25	12.25	12.04	12.71	16.64	20.97	11.38
20 calls	v4 generic normaliser	13.32	13.39	13.20	13.86	18.19	23.02	12.50
20 calls	leaderboard normaliser	24.48	24.54	24.33	24.81	21.55	26.26	23.90

SIP test

The test split of kadirnar/sip_annotated: 1,002 human-annotated Arabic phone calls (mostly Levantine), 11,466 scorable segments, each cut from its call with 0.15 s on either side. extnorm is the extended Arabic normaliser (numbers as digits, letter variants folded); the leaderboard row uses the Open Universal Arabic ASR Leaderboard's normaliser.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
all scorable segments	extnorm	14.66	14.52	15.83	15.93	30.28	45.76	14.15
without the 35 calls whose texts are in training sources	extnorm	14.22	14.09	15.43	15.43	29.53	45.29	14.19
all scorable segments	extnorm + digit runs joined	13.62	13.59	14.21	14.67	30.15	46.61	12.43
all scorable segments	leaderboard	19.58	19.38	20.70	20.78	39.16	54.90	18.93
AB_VA_SIP	extnorm	12.20	12.08	13.64	13.26	25.77	40.37	13.16
Orders_Batch	extnorm	19.23	18.84	20.66	21.57	38.52	54.96	16.33
AB_DataCollection	extnorm	10.97	11.21	11.03	11.18	27.60	44.05	9.88
AB_Dummy_Calls	extnorm	21.20	21.03	21.85	22.42	37.02	53.31	19.74

test-arabic

A diagnostic, in no composite: the Audioluna recordings cut into segments, with machine references (a five-system consensus, 13.5 % WER against the human transcripts).

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
all segments	extnorm	9.50	9.54	9.28	9.78	13.87	18.24	10.04
Arabic recordings (10)	extnorm	16.85	16.98	16.18	17.19	23.69	34.21	17.23
English recordings (10)	extnorm	4.54	4.52	4.62	4.78	7.23	7.46	5.19
report half (10 recordings)	extnorm	9.93	9.85	9.51	10.40	15.08	19.12	10.07

CallHome EN

BrunoHays/callhome-telephony, English: 140 telephone excerpts (8 kHz, 20.3 h), decoded whole. The references lose their CHAT mark-up, hesitations and backchannels are removed on both sides, then the Whisper English normaliser.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
140 excerpts	CHAT clean-up + fillers + Whisper EN	12.54	12.66	12.53	12.95	16.49	12.47	13.75
without the 2 low words/s excerpts	CHAT clean-up + fillers + Whisper EN	12.39	12.51	12.37	12.80	16.35	12.33	13.59
140 excerpts	Whisper EN only	13.43	13.55	13.41	13.82	17.95	13.90	14.62

AppTek EN

apptek-com/apptek_callcenter_dialogues: accented English call-centre dialogues, 4 calls for each of 14 accents; every channel (agent or customer) is decoded on its own and scored with AppTek's scorer. Headline: the mean over the 14 accents.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
mean of 14 accents	AppTek scorer	9.28	9.18	9.45	12.58	13.72	8.39	9.28
all channels pooled	AppTek scorer	9.12	9.02	9.26	12.15	13.46	8.21	9.16
agent channels	AppTek scorer	7.76	7.76	7.80	9.07	11.27	7.59	8.25
customer channels	AppTek scorer	11.32	11.06	11.61	17.15	17.02	9.21	10.64

SVQ AR

google/svq 2.0.0, Arabic: 8,973 utterances of held-out speakers in four environments; extnorm.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
clean	extnorm	21.59	21.96	21.33	21.05	21.03	30.26	22.11
background speech	extnorm	22.77	22.57	23.30	22.90	19.61	28.31	18.81
media noise	extnorm	23.93	24.23	23.72	23.00	24.21	39.67	24.54
traffic noise	extnorm	20.30	20.43	20.39	20.18	19.69	28.56	21.10
all environments	extnorm	22.07	22.21	22.12	21.75	20.85	30.84	21.31

Arabic Arena v2

The v2 scoring of the arabic-arena Space (kadirnar/arabic-arena) on eval-v1's copies of its test sets: per set a capped corpus WER (L2sp); the headlines are weighted geometric means over the tracks. arena-clean drops duplicate rows, the broken sudanese_sdn, the SVQ copies (SVQ AR is its own benchmark) and mtedx_ar (reported apart). Since 2026-10-28 the Space does not weight sudanese_sdn and mtedx_ar either; eval-v1 holds 21 of its 37 weighted Arabic set conditions, so no headline here is the Space's leaderboard number (the row of the earlier weighting averages 23 of the 39 it had). † marks the sets whose training split is in our fine-tunes' data.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
arena-clean headline (19 sets)	L2sp, geometric	12.27	12.39	12.35	12.73	16.83	28.32	13.45
earlier weighting: v2 formula on the 23 of the 39 sets weighted until 2026-10-28	L2sp, geometric	14.51	14.67	14.60	14.99	19.15	31.08	15.68
arena-clean, report halves where defined	L2sp, geometric	12.32	12.44	12.40	12.74	16.82	28.34	13.40
arena-clean without † sets	L2sp, geometric	15.84	16.00	15.91	16.97	23.66	36.14	17.23
track clean (10 sets clean / 13 raw)	L2sp, geometric	12.05	12.17	12.13	12.29	15.40	24.39	13.58
track codeswitch (2 sets clean / 2 raw)	L2sp, geometric	13.05	13.34	13.21	15.10	33.71	59.02	13.76
track noisy (1 sets clean / 2 raw)	L2sp, geometric	12.86	12.82	12.79	12.86	13.62	22.18	13.70
track telephony (3 sets clean / 3 raw)	L2sp, geometric	9.27	9.33	9.26	9.41	11.07	23.03	9.94
track callcenter (3 sets clean / 3 raw)	L2sp, geometric	15.08	15.37	15.49	15.65	19.29	35.25	16.25
mtedx_ar/clean (reported apart)	L2sp	63.82	63.78	63.73	63.74	61.76	67.42	64.28

Arabic Arena v2 per set (26 set conditions, cleaned WER v2)

Set / condition	Track	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20	Note
casablanca/clean	clean	32.61	32.68	32.62	32.56	36.82	55.64	34.76
cv18_ar/clean †	clean	3.94	4.12	4.08	4.04	4.26	13.96	6.05
echonova/clean	clean	15.44	15.44	15.53	15.50	33.64	26.36	16.93
fleurs_ar/clean †	clean	3.64	3.71	3.76	4.04	4.01	8.55	4.42	27 of 428 clips are also in one of our training sets
hassaniya/clean	clean	52.35	52.38	52.48	52.18	67.05	66.51	54.53
masc_clean/clean †	clean	3.69	3.73	3.69	3.64	4.01	10.45	4.83
massive_ar/clean	clean	11.12	11.03	11.08	11.41	12.58	21.25	11.24
mtedx_ar/clean †	clean	63.82	63.78	63.73	63.74	61.76	67.42	64.28	suspicious (every model >= 61.8 % WER, cuts inside speech): reported apart, in no composite
sada/clean †	clean	21.84	21.98	21.86	22.61	29.52	38.83	21.36	noisy references; half of its episodes served our earlier model selection
shams_lev/clean	clean	18.28	18.31	18.02	18.36	32.16	31.85	18.91
sudanese_sdn/clean	clean	–	–	–	–	–	–	–	broken: 959 of 1,000 references belong to a neighbouring clip; dropped from arena-clean
svq_ar/clean	clean	14.65	14.78	14.43	13.97	14.06	22.16	15.63	copies of svq-ar utterances: outside the clean composite (SVQ counted once, as svq-ar)
wasil/clean	clean	10.42	10.60	10.45	10.58	11.06	23.33	11.37	arena-clean drops the 24 rows of 12 byte-identical pairs
perle_cs/clean	codeswitch	7.85	8.13	8.04	10.34	15.59	44.37	9.43	half of it served our earlier model selection
tedxtn_cs/clean †	codeswitch	21.72	21.89	21.72	22.06	72.89	78.49	20.09
masc_noisy/clean †	noisy	12.86	12.82	12.79	12.86	13.62	22.18	13.70
svq_ar_noisy/clean	noisy	15.60	15.80	15.68	15.46	14.87	23.25	15.89	copies of svq-ar utterances: outside the clean composite (SVQ counted once, as svq-ar)
cv18_ar/telephony_v2 †	telephony	6.95	7.08	6.88	6.95	7.75	21.64	7.48
fleurs_ar/telephony_v2 †	telephony	4.44	4.46	4.48	4.55	5.01	11.95	5.51	27 of 428 clips are also in one of our training sets
sada/telephony_v2 †	telephony	25.80	25.73	25.77	26.42	34.95	47.24	23.87	noisy references; half of its episodes served our earlier model selection
cv18_ar/callcenter_v2 †	callcenter	14.03	14.43	14.83	14.63	18.50	36.34	15.97
fleurs_ar/callcenter_v2 †	callcenter	7.25	7.46	7.32	7.51	8.20	20.25	8.86	27 of 428 clips are also in one of our training sets
sada/callcenter_v2 †	callcenter	33.75	33.75	34.23	34.86	47.26	59.55	30.34	noisy references; half of its episodes served our earlier model selection
cv18_ar/telephony †	(telephony_v1, not weighted)	7.01	6.81	7.15	6.88	8.48	21.98	8.28
fleurs_ar/telephony †	(telephony_v1, not weighted)	4.74	4.69	4.83	4.76	5.22	12.79	5.70	27 of 428 clips are also in one of our training sets
sada/telephony †	(telephony_v1, not weighted)	26.51	26.97	26.49	26.74	35.83	49.29	24.60	noisy references; half of its episodes served our earlier model selection

English Arena v2

The same scoring for the English Arena (eval-v1 holds 24 of its 29 weighted set conditions). fleurs_en is scored with its bracketed (spoken) words kept.

Slice	Metric	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20
arena-clean headline (24 sets)	L2sp, geometric	5.43	5.49	5.43	5.55	6.26	6.36	6.73
as published: v2 formula on the 24 of 29 weighted sets held	L2sp, geometric	5.66	5.72	5.66	5.79	6.47	6.64	6.98
arena-clean, report halves where defined	L2sp, geometric	5.44	5.49	5.51	5.59	6.23	6.24	6.57
arena-clean without † sets	L2sp, geometric	4.67	4.73	4.67	4.89	6.23	4.55	5.45
track clean (7 sets clean / 7 raw)	L2sp, geometric	3.87	3.90	3.91	3.91	4.06	4.31	5.21
track accented (1 sets clean / 1 raw)	L2sp, geometric	12.09	12.07	12.16	12.44	13.72	12.43	13.23
track noisy (4 sets clean / 4 raw)	L2sp, geometric	5.22	5.32	5.19	5.19	5.94	6.21	6.21
track telephony (7 sets clean / 7 raw)	L2sp, geometric	3.98	4.03	3.95	4.27	4.44	4.25	4.84
track callcenter (5 sets clean / 5 raw)	L2sp, geometric	10.76	10.88	10.73	10.93	15.08	15.24	12.64
fleurs_en/callcenter (brackets_kept)	L2sp	10.28	10.36	10.52	10.69	15.80	13.10	12.22
fleurs_en/clean (brackets_kept)	L2sp	3.65	3.69	3.58	3.60	4.44	2.73	4.07
fleurs_en/noisy (brackets_kept)	L2sp	5.65	5.76	5.65	5.70	8.45	7.10	6.79
fleurs_en/telephony (brackets_kept)	L2sp	4.23	4.30	4.32	4.32	5.02	3.84	5.11

English Arena v2 per set (24 set conditions, cleaned WER v2)

Set / condition	Track	Cohere soup (main)	Cohere RL v4c	Cohere RL	Cohere SFT	Cohere base	Qwen3-ASR base	Qwen3-ASR run20	Note
ami_sdm/clean †	clean	6.59	6.67	6.76	6.75	6.28	7.50	7.08
common_voice_en/clean †	clean	6.58	6.71	6.66	6.70	6.55	7.03	8.89
earnings22/clean †	clean	7.33	7.43	7.28	7.37	9.62	9.76	7.89
fleurs_en/clean	clean	3.65	3.69	3.58	3.60	4.44	2.73	4.07	arena-clean scores it with the bracketed (spoken) words kept
librispeech_clean/clean †	clean	1.01	1.02	1.03	1.02	1.02	1.18	1.76
librispeech_other/clean †	clean	2.01	2.00	2.07	2.06	2.00	2.94	4.23
voxpopuli_en/clean †	clean	5.50	5.52	5.53	5.50	5.09	5.67	6.93	arena-clean drops one clip of each of 12 near-duplicate pairs
edacc/clean †	accented	12.09	12.07	12.16	12.44	13.72	12.43	13.23
earnings22/noisy †	noisy	9.56	9.56	9.56	9.50	14.01	14.09	9.97
fleurs_en/noisy	noisy	5.65	5.76	5.65	5.70	8.45	7.10	6.79	arena-clean scores it with the bracketed (spoken) words kept
librispeech_clean/noisy †	noisy	2.04	2.12	2.04	2.04	1.64	1.83	2.56
voxpopuli_en/noisy †	noisy	6.76	6.83	6.58	6.58	6.42	8.13	8.60	arena-clean drops one clip of each of 12 near-duplicate pairs
common_voice_en/telephony †	telephony	11.28	11.18	10.83	11.00	11.18	12.98	12.91
earnings22/telephony †	telephony	7.37	7.45	7.03	7.01	9.42	9.70	7.86
fleurs_en/telephony	telephony	4.23	4.30	4.32	4.32	5.02	3.84	5.11	arena-clean scores it with the bracketed (spoken) words kept
librispeech_clean/telephony †	telephony	1.41	1.45	1.45	1.46	1.46	1.72	2.18
minds14_tel/clean	telephony	1.21	1.22	1.19	1.66	1.49	0.94	1.52
voxpopuli_en/telephony †	telephony	6.52	6.58	6.51	6.58	5.60	6.70	7.80	arena-clean drops one clip of each of 12 near-duplicate pairs
vystadial_voip/clean	telephony	4.06	4.16	4.10	4.83	5.28	4.83	4.68
common_voice_en/callcenter †	callcenter	24.74	25.16	24.67	25.02	31.35	33.11	26.44
earnings22/callcenter †	callcenter	13.36	13.26	13.21	13.36	20.43	19.22	14.13
fleurs_en/callcenter	callcenter	10.28	10.36	10.52	10.69	15.80	13.10	12.22	arena-clean scores it with the bracketed (spoken) words kept
librispeech_clean/callcenter †	callcenter	4.75	4.87	4.63	4.82	7.71	8.32	6.25
voxpopuli_en/callcenter †	callcenter	8.92	9.03	8.96	9.06	9.99	11.86	11.31	arena-clean drops one clip of each of 12 near-duplicate pairs

Training curve: Cohere SFT

The eval-v1 dev set during the SFT: 7 groups (Arabic and English calls, dialects, broadcast and accented speech), 150 clips per group for the WER with the trainer's greedy decode; the loss is the text-token loss on every dev clip. The learning rate decays over the last 30 % of the steps.

Step	Dev WER %	95 % CI	Dev loss
0	27.49	25.62–29.63	0.692
2,000	18.00	16.47–19.54	0.295
4,000	18.05	16.51–19.61	0.285
6,000	17.99	16.38–19.66	0.280
8,000	17.20	15.61–18.80	0.273
10,000	17.07	15.53–18.61	0.270

Training curve: Cohere RL (round 1)

The same dev set during the first MWER round. MWER lowers the expected word error of sampled outputs, not the cross-entropy, so the dev loss rises slightly while the error falls. Later rounds moved the dev WER little while the test sets improved: the dev groups are small.

Step	Dev WER %	95 % CI	Dev loss
0	17.16	15.60–18.74	0.270
400	17.14	15.59–18.72	0.271
800	16.94	15.46–18.48	0.273

How these numbers were made

The official Arena rows and sections are read from the Arena Space (kadirnar/arabic-arena) when this page is built. Cohere RL (round 4, step 800) was added to that board on 2026-10-28, Cohere soup and Cohere RL v4c on 2026-10-29, with the Arena's own code on 4 x H100 (the other entries ran on an A10, so only its speed differs); its confidence interval and rank set appear after the next full rebuild of the board.
Cohere soup and Cohere RL v4c were run on the eval-v1 suite on 2026-10-29 (4 x H100), with the same code as the other columns.
The eval-v1 rows are our own runs of the eval-v1 suite (2026-10-27, 4 x H100), not a submission to any public leaderboard.
Decoding: greedy, a single pass, automatic language (the model writes its own language tag); no beam search, sampling, rescoring or test-time augmentation. Qwen3-ASR decodes with vLLM and Cohere Transcribe with transformers (bf16); the Arena rows use the Arena's own backends with the language forced.
The numbers are cleaned: the eval-v1 audit removes broken references and counts repeated audio once (each tooltip also gives the raw number). WER is a corpus rate: errors summed over the items, divided by the reference words.
95 % intervals: a cluster bootstrap of 2,000 resamples; the clusters are calls, recordings, speakers or videos.