KinyaBench: open language models on Kinyarwanda
Preliminary chat, 0-shot, single-prompt scores for AfriqueQwen3.5 4B and 9B on the Kinyarwanda slice of public benchmarks.
I needed a reproducible number for how current open models handle Kinyarwanda, not an average over African languages. KinyaBench is a public-track harness that runs the Kinyarwanda slices of standard benchmarks under a fixed protocol and records the model revision.
Setup
Tasks. The core suite is the AfroBench-Lite Kinyarwanda slice used in the AfriqueLLM paper: AfriMGSM (250 items, exact match, chain-of-thought), AfriMMLU (500, 4-way accuracy), AfriXNLI (600, 3-way accuracy), Belebele (900, 4-way accuracy), SIB-200 (204, 7-way accuracy), Injongo intent (640, 40-way accuracy), and FLORES-200 English↔Kinyarwanda (1,012 sentences per direction, chrF++). FLORES is loaded from Meta's official FLORES-200 release, not the gated Hugging Face copy.
Protocol. Instruct checkpoints are run in the model's own chat template, 0-shot, greedy. Thinking is forced off (enable_thinking=False as a top-level model argument; nesting it inside chat-template arguments is silently overwritten by lm-eval's vLLM wrapper). Every generated answer is scanned for thinking markers; a leak would invalidate the run. This first pass uses prompt variant 1 only, so these scores are for a single prompt, not a best-of-prompts number. Multiple-choice tasks are scored by log-likelihood after the assistant prefix. I used EleutherAI lm-eval 0.4.13 and vLLM 0.31.0, on one A40, in bfloat16, with a 16,384-token context.
Models. McGill-NLP/AfriqueQwen3.5-4B-50Langs-Instruct-v1 (revision 0ff0ff64) and McGill-NLP/AfriqueQwen3.5-9B-50Langs-Instruct-v1 (revision 693f12e4). Both are CC-BY-4.0. For context I also copy the paper's 5-shot scores for the 4B base 50-language checkpoint. That row is a different model and a different protocol; it is not a direct baseline.
Results
Table 1. KinyaBench public-track scores, 10 October 2026. Chat protocol, 0-shot, prompt 1, thinking off.
| Task (n) | Metric | 4B instruct ↑ | 9B instruct ↑ | Paper 4B base, 5-shot |
|---|---|---|---|---|
| AfriMGSM (250) | exact match | 46.0 ± 3.2 | 39.2 ± 3.1 | 36.08 |
| AfriMMLU (500) | acc | 34.0 ± 2.1 | 39.2 ± 2.2 | 39.44 |
| AfriXNLI (600) | acc | 43.7 ± 2.0 | 56.2 ± 2.0 | 34.57 |
| Belebele (900) | acc | 35.6 ± 1.6 | 62.0 ± 1.6 | 64.58 |
| SIB-200 (204) | acc | 66.2 ± 3.3 | 80.4 ± 2.8 | 81.00 |
| Injongo (640) | acc | 58.4 ± 1.9 | 66.9 ± 1.9 | 65.81 |
| FLORES en→kin (1,012) | chrF++ | 46.12 ± 0.41 | 48.17 ± 0.40 | 46.39 |
| FLORES kin→en (1,012) | chrF++ | 51.81 ± 0.49 | 54.13 ± 0.45 | 53.04 |
Notes. Intervals are the lm-eval standard error, not a 95% confidence interval. Bold marks the better of the two instruct runs, which are the only scores from this protocol. The paper column is AfriqueQwen3.5-4B-50Langs, base, 5-shot (AfriMGSM 8-shot), from appendix tables of arXiv 2601.06395 v3. It is shown for orientation, not as a matched baseline. AfriMGSM is flexible-extract exact match; strict-match is 0.0 for both instruct models.
Figure 1. Public-track scores for AfriqueQwen3.5 instruct, chat, 0-shot, prompt 1, thinking off. Error bars are one lm-eval standard error. Left: accuracy or exact match. Right: FLORES-200 chrF++. The 9B model leads on seven of eight tasks; the 4B model leads only on AfriMGSM.
The 9B model is ahead on seven of the eight tasks. The gap is largest on Belebele: 62.0% against 35.6% (standard errors 1.6 points). Because the same harness, on the same 4B checkpoint, lands close to the published FLORES numbers (46.12 chrF++ for English→Kinyarwanda against 46.39 in the paper, 51.81 against 53.04 in the other direction), the 4B's weak multiple-choice scores are a property of the instruct model under this protocol, not a scoring bug.
AfriMGSM is the exception: 46.0% exact match for the 4B, 39.2% for the 9B. These are thinking-off, greedy, 0-shot figures and are not comparable to the McGill model-card numbers, which use thinking and a long generation budget. AfriXNLI, which sits near chance (33.3%) for every model in the paper, is 43.7% for the 4B and 56.2% for the 9B under chat 0-shot.
Thinking markers leaked in none of the audited generations (FLORES both ways, and AfriMGSM). The 9B AfriMGSM run had one empty completion out of 250.
Analysis
A 9B African-continued Qwen3.5 is, under this protocol, a clearly stronger Kinyarwanda reader than its 4B counterpart, especially on Belebele and SIB-200. Translation is already close to the published base-model figures at 4B, and a little higher at 9B. I have not yet run the paper's own protocol (raw 5-shot, on the base checkpoint), so I cannot claim a reproduction of the comprehension rows. I have also not yet run a Qwen3.5-9B instruct model without the African continued pretraining, so this note does not isolate that training stage.
Limitations
One prompt variant, two models, no base-protocol validation run. AfriXNLI remains a weak signal in the literature even when the 9B is well above chance here. The native, locally written track has no automatic scorer yet, so I am not reporting assistant-quality numbers. SSA-COMET, the paper's headline FLORES metric, is not implemented; I compare chrF++ with the paper's appendix.
Next measurements
The same protocol with all prompt variants; a base-protocol run of AfriqueQwen3.5-4B-50Langs; Qwen3.5-9B instruct without African continued pretraining; and Gemma 4 26B-A4B and 31B, which have no published Kinyarwanda scores I can find. After that, continued-pretraining models from my own pack can be placed on the same scale.