Building a Kinyarwanda corpus and training stack

From 15 February to 27 April 2026 I built a Kinyarwanda text factory, packed a pretraining corpus, trained models from that pack, distilled a 32,433-row supervision set, and started a custom tokenizer. The account below is limited to what the files and training logs still show.

The aim was a native Kinyarwanda language model: continued pretraining of an open 8–9 billion-parameter base on a few hundred million clean tokens, then supervised fine-tuning. I finished a full-parameter run of Qwen3-8B, a LoRA adaptation of Qwen3.5-9B, and an instruction-tuning run that I served to testers. Held-out pack perplexity on the 8B run fell from 10.44 to 5.50; the LoRA run moved only 8.07 to 7.84. The served system failed a manual quality review. When I returned to the work in October I treated those results as a reason to measure existing models before training new base weights.

The pipeline

The factory mined, language-identified, filtered, and deduplicated documents, then packed them as input_ids sequences with no residual text column. A new base therefore needs retokenization from the document shards. Training and evaluation lived in separate codebases: a continued-pretraining loop with configs, checkpoints, and a promotion gate that I only ever ran dry; a supervision pipeline that generated, cleaned, and split instruction data; and a small evaluation and serving stack for a tester chat surface.

The pack

On 27 February 2026 I exported a packed training split of 183,083,577 tokens under the Qwen3 tokenizer (411,133 documents, 44,699 sequences, maximum length 4,096), with 2,026,316 validation tokens and 2,047,384 test tokens. The source-token estimate before packing is 178,795,577 tokens across 420,301 documents.

Table 1. Estimated source-token composition of the pack, by type.

Source type	Documents	Tokens (est.)	Share
Public collection (mbazaNLP v01.1)	390,562	125,091,104	70.0%
Books	684	18,356,385	10.3%
Targeted web text	13,806	17,010,273	9.5%
News (KINNEWS)	10,314	8,849,327	4.9%
Official documents	458	5,016,073	2.8%
Wikipedia	3,556	3,211,803	1.8%
Web crawl	921	1,260,612	0.7%

Notes. Token counts are the factory's pre-pack estimates, summing to 178,795,577. The official packed training split is 183,083,577 tokens. Official documents combine institutional publications; web crawl combines index-targeted and language-filtered Common Crawl slices.

A random sample of Common Crawl WET files produced no Kinyarwanda: 0 documents in 114,254, even at a language-identification threshold of 0.50. The web-crawl tokens that did enter the pack came from index-targeted and language-filtered extraction, and they remain 0.7% of the estimate.

Filters included language identification, length, boilerplate and template phrases, and near-duplicate removal. A mid-pipeline deduplication pass on 29 of 32 shards flagged 11,052 exact and 1,212 fuzzy duplicates of 409,332 documents (3.0%). I do not treat that as a statistic for the final pack, because three shards were not in that report.

On 28 February I ran a recall-first clean of the mbazaNLP slice: 210,832 of 390,562 documents were kept (54.0%), holding 103,609,913 of 125,091,104 estimated tokens (82.8%). Most discards were short fragments; only 35 documents were hard-rejected, all on a combined template-phrase and low language-ID signal. That clean was packed as a 161,244,925-token training split. It is not the 183-million-token export above, which still includes the uncleaned mbazaNLP slice.

A later, more aggressive retokenization under Qwen3.5, in four cleaning versions, ends at 144,146,803 tokens (304,411 of 420,301 documents kept). That pack is not the continued-pretraining run below.

Tokenizer cost

On FLORES-200 devtest (1,012 parallel sentences; 20,077 Kinyarwanda words and 21,901 English words, whitespace-split) the candidate base tokenizers all cost about twice as many tokens for Kinyarwanda as for the same English content. Gemma 4 26B-A4B is the most compact of the three, and the spread among them is small: 2.66 to 2.90 Kinyarwanda tokens per word, an 8% difference in sequence length between Gemma 4 and Qwen3. Qwen2.5-7B and Qwen3-8B produce identical token counts on this set; so do Qwen3.5-9B and AfriqueQwen3.5-9B-Instruct.

Table 2. Tokenizer fertility on FLORES-200 devtest.

Tokenizer	Vocab	Kin. tokens/word ↓	Kin. chars/token ↑	Kin. ÷ Eng. tokens
Qwen2.5-7B / Qwen3-8B	151,665 / 151,669	2.90	2.50	2.11
Qwen3.5-9B / AfriqueQwen3.5	248,070 / 248,077	2.77	2.62	2.02
Gemma 4 26B-A4B	262,144	2.66	2.74	1.97

Notes. Encoded with the tokenizers 0.23.3 library from the local Hugging Face tokenizer files, without adding special tokens (adding them does not change the counts). Words are whitespace-split. Qwen2.5-7B and Qwen3-8B differ in vocabulary size but not in token counts on this set; the same is true of Qwen3.5-9B and AfriqueQwen3.5-9B-Instruct. English tokens per word are 1.26, 1.26, and 1.24 respectively.

I also trained 16,000-piece Unigram models on a stratified shard of the corpus. On the held-out slice in Figure 2, a custom trainer reached 5.08 characters per token against 4.39 for SentencePiece trained on the same shard. The custom trainer was about two orders of magnitude slower (166.6 s versus 1.79 s on a 1,024-document shard, and 359.8 s versus 3.31 s on a 3,072-document shard) and hit RecursionError on larger representative shards. A custom vocabulary also cannot be dropped into a pretrained model without relearning embeddings. I parked the tokenizer.

Training

I configured two continued-pretraining recipes: full-parameter Qwen3-8B on a 183-million-token target (learning rate 1×10⁻⁵, sequence length 2,048, Adafactor), and LoRA on Qwen3.5-9B (rank 32, α 16, dropout 0.05, on q_proj and v_proj only; learning rate 2×10⁻⁵) with a 150-million-token target. A later, higher-capacity LoRA configuration (rank 256, α 512) was started and crashed early.

The Qwen3-8B run finished on 2 March 2026: 11,170 steps, 183,007,801 tokens seen (one pass over the February pack, short by 75,776 tokens) in 22.46 hours (2,267 tokens per second). Validation was logged every 500 steps. I have no measurement of the unadapted base on the same split, so the first point is step 500, not step 0. Held-out perplexity fell from 10.44 at step 500 to 5.50 at step 11,000. The best logged value is 5.45 at step 6,500, after 106.5 million tokens; after that the curve is nearly flat, ending at 5.50. Mean training loss was 3.72 over the first 50 steps and 1.89 over the last 50. A 20-million-token sanity on the same recipe, earlier that day, had already moved validation perplexity from 14.42 at step 200 to 9.25 at step 1,200. An earlier full-run start stopped at 161 steps with no validation logged. No test-split metric was logged for any of these runs.

The Qwen3.5-9B LoRA adaptation ran overnight on 13 March 2026: 2,289 steps, 150,010,227 tokens seen, 11.02 hours, 3,780 tokens per second. The training file was a retokenized pack under the Qwen3.5-9B tokenizer, not the February Qwen3 export directly. Validation was logged every 500 steps. Held-out perplexity fell from 8.07 at step 500 to 7.84 at step 2,000, a drop of 0.23 over 1,500 steps. Mean training loss was 2.252 over the first 50 steps and 2.099 over the last 50, and the curve had already flattened in the second half of the run. No downstream task evaluation of this adapter was logged.

The higher-capacity LoRA attempt crashed after a last logged optimizer step of 240, with training loss 2.27 → 2.09 over the twelve logged points. No validation was recorded: evaluation was scheduled every 500 steps.

Table 3. Training runs from the February–April programme.

Model	Method	Tokens / rows	Steps	Final train loss	Val. PPL or loss ↓
Qwen3-8B	Full CPT	183,007,801 seen	11,170	2.074	10.44 → 5.50
Qwen3.5-9B	LoRA r=32, q/v	150,010,227 seen	2,289	2.168	8.07 → 7.84
Qwen3.5-9B	LoRA r=256, α=512	—	240 (crashed)	2.088	No val logged
Qwen3-8B	Full SFT, 3 epochs (failed)	12,144,794 seen	6,000	0.696	1.402 → 1.182

Notes. Validation figures are the first and last logged held-out values, except the SFT row, which ends at the best validation loss (step 4,000; perplexity 3.26). The 8B continued-pretraining best is 5.45 at step 6,500; the last logged value, at step 11,000, is 5.50. LoRA perplexities are at steps 500 and 2,000; the run continued to step 2,289 with no further validation. The higher-capacity LoRA log has no tokens-seen field. The SFT run failed during the second of three planned epochs; at step 6,000, validation loss was 1.199. No step-0 base-model validation was recorded. GPU type was not recorded. An instruction-tuned checkpoint from the SFT recipe was served and reviewed, as described below.

The two completed continued-pretraining runs are not on a comparable vertical scale. The 8B full-parameter run moves held-out perplexity by 4.94 points; the LoRA run moves it by 0.23. Neither includes a step-0 base-model point, and neither was followed by a task evaluation of the continued-pretraining checkpoint itself.

Supervision

Between 7 and 9 March I built a 32,433-row instruction set, split 29,146 / 1,601 / 1,686 for train, validation, and test. Every response was written by GPT-5.4. Automatic cleaning kept 26,515 of 26,525 single-turn answers (nine rejected as too short, one as a duplicate prompt–response pair) and all 5,918 multi-turn chat pairs.

A pre-registered mix check against the lab's own plan failed eleven slices. Language composition is the clearest of those failures.

Table 4. Language modes in the 32,433-row supervision set, against the planned floors.

Language mode	Rows	Share	Planned floor
Kinyarwanda	30,141	93.0%	45%
Mixed Kinyarwanda	2,234	6.9%	12%
Control	39	0.12%	2%
English	16	0.05%	3%
French	3	0.01%	2%
Swahili	0	0.00%	1%

Notes. Floors are from the lab mix plan. The other failed slices are content types below their floors: multi-turn chat 18.2% against 20%; safety refusals 0.84% against 6%; noisy-input robustness 0.61% against 6%; code-switching 3.8% against 7%; multilingual retention 0.88% against 3%; language control 0.84% against 2%. Kinyarwanda instruction rows, at 31.8%, cleared their 20% floor.

The set is almost entirely Kinyarwanda teacher text, with 16 English rows, three French, and no Swahili. I configured a three-epoch full-parameter SFT run of the Qwen3-8B continued-pretraining checkpoint on this set (learning rate 8×10⁻⁶, sequence length 2,048). A 200-step sanity finished the same day with validation loss 1.318. The main run failed at step 6,000, during the second of the three planned epochs, after 12,144,794 supervised tokens and 3.49 hours. Validation loss fell from 1.402 at step 500 to a best of 1.182 at step 4,000 (perplexity 3.26), then rose slightly to 1.199 at the last logged step. Training and validation loss both jumped near the epoch boundary at step 3,645; the best validation point is early in the second epoch. Training loss kept falling, to 0.696 at step 6,000. An instruction-tuned checkpoint from that recipe was served.

Evaluation

On 11 March I reviewed the seven conversations captured from the tester chat surface (34 generations; a broader export of ten conversations contains 37 generations). One of the seven was clearly acceptable. The others failed on identity, on basic facts about Rwanda, on instruction following, or on repetition and unnatural Kinyarwanda. The review called this a failed quality pass, and not a problem that a prompt change would fix.

A concurrent 200-item internal evaluation set was run on the base model, on the instruction-tuned system, and later on untuned Qwen3.5-9B. Those runs were never quality-scored: the pipeline recorded HTTP success, not graded answers. I do not report those automatic pass rates here.

What this implied

A perplexity drop on a held-out pack is not a quality gate. Full-parameter continued pretraining moved held-out perplexity from 10.44 to 5.50, and after supervised fine-tuning on the GPT-5.4 responses, the served system failed a human review. 150–183 million Kinyarwanda tokens, used as full-parameter or low-rank continued pretraining, did not produce a usable model at this scale of measurement. A 32,433-row teacher-written supervision set, almost without English or French replay and already off its own mix plan, did not produce acceptable assistant behaviour in that review.

When I returned to the work in October I treated those findings as settled. I stopped training new base weights on a few hundred million tokens, parked the custom tokenizer, and turned to measuring public-track scores for open models, and to a speech-recognition lab, before any further continued pretraining. The public-track results are in a separate note.