Word error on formal and casual Kinyarwanda

Read-speech numbers are a poor guide to Kinyarwanda ASR. I scored the same systems on two registers with human references: formal parliamentary debate, and casual conversation.

Setup

Formal set. Ten parliamentary sittings from 2022, with audio aligned to the official minutes. Splits are whole sittings. The test set is two sittings: 456 segments, 0.91 hours, 6,036 reference words. Fifty of those segments have numbers written as digits in the minutes, which inflates word error, so I also report a no-digit subset of 406 segments (4,874 words). Development is one sitting (354 segments) and was used only to select decoding settings and vote weights. The minutes are clean verbatim: fillers and repetitions have been edited out, so some of the remaining error is editorial rather than acoustic.

Casual set. A held-out set of informal conversations: 478 scored clips, 1.12 hours, 5,995 reference words, from 148 recordings. The references are human and near-verbatim, but they use informal orthography, in which locatives such as mu and ku are often joined to the next word and apostrophes are mostly dropped. Ordinary word error therefore confounds recognition mistakes with spelling conventions.

Metrics. On the formal set I report corpus word error rate (WER), and I emphasize the no-digit subset. On the casual set, a claim has to move both an orthography-tolerant word error rate, wer_canon (apostrophes deleted and mu/ku joined to the following word on both the hypothesis and the reference), and a space-insensitive character error rate, cer_nospace. I also quote raw WER, but I do not decide claims on it. Differences are paired-bootstrap 95% confidence intervals (2,000 resamples, seed 0; formal by segment, casual by recording). All rates are in percent; a negative delta means the first system is better.

Protocol. Decoding hyperparameters were chosen on development data; the test set was scored once per system for a given experiment. The formal test set has since been reused across later experiments, so I treat it as a pilot test, not a sealed one. Headline open-model rates come from a 16 October 2026 re-score of the original hypotheses under these rules. The vote, the closed-model bake-off, and a later 23.4-hour student were scored with the same scorer after that date.

Systems. I compare Meta's Omnilingual ASR models (LLM-7B, LLM-3B, CTC-7B), public Digital Umuganda CTC models, a four-system vote, a CTC student trained on 23.4 hours of agreement-filtered labels, and closed transcription APIs. The vote combines Omnilingual LLM-7B, Omnilingual CTC-7B, and two fine-tuned CTC students; its weights were fit on the development sitting.

Results

Table 1. Word error on formal parliamentary speech and casual conversation.

System	Formal WER ↓	Casual wer_canon ↓	Casual cer_nospace ↓
Four-system vote	13.89	30.26†	8.96†
Omnilingual LLM-7B	15.57	33.52	10.12
Gemini 4 Argon, 60 s windows	17.76*	32.03*	11.79*
CTC student, 23.4 h labels + LM	18.51	32.85	9.95
Omnilingual LLM-3B	20.87	37.09	11.51
Digital Umuganda noise-robust + LM	21.32	32.19	10.06
Omnilingual CTC-7B	21.99	36.81	10.46
Digital Umuganda 660h + LM	22.36	33.43	11.21

Notes. Formal WER is the no-digit subset of the parliamentary test set (406 segments, 4,874 reference words), except where marked. Casual rates are on the full held-out set (478 clips, 148 recordings), except ^† the vote, which was scored only on a held-out evaluation half (229 clips, 74 recordings). ^* Gemini 4 Argon is scored on 60-second windows of consecutive clips, not clip by clip. Lower is better. I do not mark a casual winner: among the leading full-set systems the differences are small, and several of the pairwise intervals include zero.

The register gap is the main result. Omnilingual LLM-7B, the strongest single open system on formal speech, has a no-digit WER of 15.57% on the parliamentary test set and a wer_canon of 33.52% on casual conversation (raw WER 43.60%; cer_nospace 10.12%). The four-system vote lowers formal no-digit WER to 13.89%, a reduction of 1.68 points against LLM-7B (95% CI [−2.50, −0.91]). On casual speech, scored only on the evaluation half, the vote reaches 30.26 wer_canon against 33.11 for LLM-7B on the same clips; the interval is [−6.92, 0.00], and a post-hoc check found that most of the point-estimate gain comes from three recordings in which LLM-7B entered a repetition loop.

Table 2. Closed transcription APIs against the open stack on formal speech.

System	Screen WER ↓	Test WER, no-digit ↓
Four-system vote	—	13.89
Omnilingual LLM-7B	16.2	15.57
Gemini 4 Argon, 60 s windows	19.69a	17.76
Gemini 4 Argon, clip-level	19.69	22.51
Gemini 3.5 Transcribe	38.83	screened out
Gemini 3.8 Flash	49.72	screened out
gpt-transcribe	79.89	screened out
gpt-4o-transcribe	83.52	screened out
gpt-audio-1.5	89.39	screened out
Grok STT	101.82	screened out
MAI Transcribe 2	106.15	screened out

Notes. The screen is 40 parliamentary development clips (405 seconds of audio). A system was dropped if its screen WER was above 30 and more than 1.5 times LLM-7B on the same clips (LLM-7B screen WER 16.2). Test numbers are no-digit WER on the parliamentary test set. ^a Argon's screen number is the clip-level run; the 60-second windowing was selected later, on the full development sitting, and then scored once on the test set.

Closed APIs did not overtake the open stack. On the 40-clip formal screen, three OpenAI transcribers landed between 79.89 and 89.39% WER and were not run on the test set. Grok STT and MAI Transcribe 2 were above 100%. Only Gemini 4 Argon survived (19.69%). Scored clip by clip on the parliamentary test set, Argon has a no-digit WER of 22.51%, 8.62 points behind the vote ([7.04, 10.20]). Much of that gap is a clip-end failure. Mid-clip, on the same test comparison, Argon ties LLM-7B (−0.18 [−1.67, +1.31]). On the development sitting, Argon got the last word wrong in 41% of no-digit clips, against 6% for LLM-7B. After selecting 60-second windows of consecutive clips on development data, Argon reaches 17.76% no-digit WER on the test set. That is 3.50 points behind the vote ([1.28, 5.97]) and statistically tied with LLM-7B (+1.28 [−1.46, +4.36]). On casual conversation the windowed system does not beat the pre-registered best open student on both casual metrics (wer_canon 32.03 vs 33.27, interval [−2.99, +1.35]; cer_nospace 11.79 vs 10.31, interval [−0.13, +3.86] in the other direction).

Table 3. Fine-tuning the Digital Umuganda 660-hour model on human minutes versus agreement-filtered machine labels, LM decode.

Training labels	Hours	Formal WER ↓	Δ vs untrained	Casual wer_canon ↓	Δ vs untrained
None (untrained)	—	22.36	—	33.43	—
Human official minutes	6.0	19.37	−3.00 [−3.89, −2.09]	36.32	+2.89 [+2.04, +3.75]
All LLM-7B labels	6.0	19.14	−3.22 [−4.10, −2.31]	35.06	+1.63 [+0.79, +2.52]
Agreement-filtered LLM-7B	2.4	19.22	−3.14 [−3.94, −2.31]	33.30	−0.13 [−0.85, +0.56]

Notes. Formal WER is no-digit, parliamentary test, LM decode. Casual wer_canon is the full casual set. Deltas are paired-bootstrap 95% intervals (2,000 resamples, seed 0). The filtered 2.4-hour arm is not different from the 6.0-hour human arm on formal no-digit WER under greedy decode (−0.18 [−0.96, +0.60]); the LM-decode point estimates above are likewise within about a quarter of a point of one another.

Agreement-filtered machine labels match human labels on formal speech and do not close the casual gap. Fine-tuning on 6.0 hours of official minutes cuts formal no-digit WER by 3.00 points with a language model, but it raises casual wer_canon (+2.89 [+2.04, +3.75]). The same student trained on 2.4 hours of LLM-7B labels, kept only where a second system agrees, matches the human-label arm on the formal test set and does not reliably improve casual wer_canon against the untrained model. Looser keep thresholds, and machine labels drawn from casual audio, similarly fail to move both casual metrics together.

A later student trained on 23.4 hours of filtered labels continues the formal trend (no-digit WER 18.51 with a language model, 3.86 points better than the untrained baseline, interval [−4.79, −3.00]) and still leaves casual wer_canon at 32.85, within noise of the untrained model (−0.59 [−1.44, +0.30]).

Analysis

Formal Kinyarwanda, at least in this parliamentary domain, is already in a usable range for an open 7B speech LLM, and a vote takes another one to two points off. Casual conversation is a different problem: even an orthography-tolerant metric stays in the low thirties, and neither closed APIs, nor fine-tuning on formal minutes, nor scaling agreement-filtered labels to a few tens of hours closes it. The vote's apparent casual gain is too concentrated to treat as a general improvement.

A screen on short segments misrepresented Gemini 4 Argon as a long-form system. Fixing the clip-end artifact was not enough to beat the open vote on formal speech.

Limitations

The official minutes are edited, so formal WER includes an unknown editorial floor. The casual references are informally spelled; wer_canon and cer_nospace are designed for that, but they remain automatic metrics against one annotation style. The parliamentary test set has been scored in many decisions and is no longer clean. The 60-second Argon condition is not a clip-level comparison. The vote's casual result uses only half the casual set. I do not yet have a sealed, human-transcribed casual or broadcast test set of comparable size.

Next measurements

The next formal claims should be scored once on a sealed set that has not been used for selection. The casual gap needs human-corrected conversational transcripts, not more machine labels of the current quality. I will also keep the vote as a teacher for a larger CTC student, and only then re-measure both registers.