<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
    <channel>
        <title>Cedric Mutesa</title>
        <link>https://www.mcedric.dev</link>
        <description>Writing and research notes by Cedric Mutesa.</description>
        <lastBuildDate>Tue, 10 Nov 2026 10:28:19 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <image>
            <title>Cedric Mutesa</title>
            <url>https://www.mcedric.dev/favicon.ico</url>
            <link>https://www.mcedric.dev</link>
        </image>
        <copyright>All rights reserved 2026</copyright>
        <item>
            <title><![CDATA[Word error on formal and casual Kinyarwanda]]></title>
            <link>https://www.mcedric.dev/research/formal-casual-kinyarwanda-asr</link>
            <guid>https://www.mcedric.dev/research/formal-casual-kinyarwanda-asr</guid>
            <pubDate>Mon, 19 Oct 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[Open and closed recognizers on formal parliamentary speech and casual conversation, including a four-system vote.]]></description>
            <content:encoded><![CDATA[<p>Read-speech numbers are a poor guide to Kinyarwanda ASR. I scored the same systems on two registers with human references: formal parliamentary debate, and casual conversation.</p>
<h2>Setup</h2>
<p><strong>Formal set.</strong> Ten parliamentary sittings from 2022, with audio aligned to the official minutes. Splits are whole sittings. The test set is two sittings: 456 segments, 0.91 hours, 6,036 reference words. Fifty of those segments have numbers written as digits in the minutes, which inflates word error, so I also report a no-digit subset of 406 segments (4,874 words). Development is one sitting (354 segments) and was used only to select decoding settings and vote weights. The minutes are clean verbatim: fillers and repetitions have been edited out, so some of the remaining error is editorial rather than acoustic.</p>
<p><strong>Casual set.</strong> A held-out set of informal conversations: 478 scored clips, 1.12 hours, 5,995 reference words, from 148 recordings. The references are human and near-verbatim, but they use informal orthography, in which locatives such as <em>mu</em> and <em>ku</em> are often joined to the next word and apostrophes are mostly dropped. Ordinary word error therefore confounds recognition mistakes with spelling conventions.</p>
<p><strong>Metrics.</strong> On the formal set I report corpus word error rate (WER), and I emphasize the no-digit subset. On the casual set, a claim has to move both an orthography-tolerant word error rate, wer_canon (apostrophes deleted and <em>mu</em>/<em>ku</em> joined to the following word on both the hypothesis and the reference), and a space-insensitive character error rate, cer_nospace. I also quote raw WER, but I do not decide claims on it. Differences are paired-bootstrap 95% confidence intervals (2,000 resamples, seed 0; formal by segment, casual by recording). All rates are in percent; a negative delta means the first system is better.</p>
<p><strong>Protocol.</strong> Decoding hyperparameters were chosen on development data; the test set was scored once per system for a given experiment. The formal test set has since been reused across later experiments, so I treat it as a pilot test, not a sealed one. Headline open-model rates come from a 16 October 2026 re-score of the original hypotheses under these rules. The vote, the closed-model bake-off, and a later 23.4-hour student were scored with the same scorer after that date.</p>
<p><strong>Systems.</strong> I compare Meta&#x27;s Omnilingual ASR models (LLM-7B, LLM-3B, CTC-7B), public Digital Umuganda CTC models, a four-system vote, a CTC student trained on 23.4 hours of agreement-filtered labels, and closed transcription APIs. The vote combines Omnilingual LLM-7B, Omnilingual CTC-7B, and two fine-tuned CTC students; its weights were fit on the development sitting.</p>
<h2>Results</h2>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 1.</strong> Word error on formal parliamentary speech and casual conversation.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">System</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Formal WER ↓</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Casual wer_canon ↓</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Casual cer_nospace ↓</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Four-system vote</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>13.89</strong></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">30.26<sup>†</sup></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">8.96<sup>†</sup></td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Omnilingual LLM-7B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">15.57</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">33.52</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">10.12</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Gemini 4 Argon, 60 s windows</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">17.76<sup>*</sup></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">32.03<sup>*</sup></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">11.79<sup>*</sup></td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">CTC student, 23.4 h labels + LM</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">18.51</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">32.85</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">9.95</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Omnilingual LLM-3B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">20.87</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">37.09</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">11.51</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Digital Umuganda noise-robust + LM</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">21.32</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">32.19</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">10.06</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Omnilingual CTC-7B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">21.99</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">36.81</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">10.46</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">Digital Umuganda 660h + LM</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">22.36</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">33.43</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">11.21</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Formal WER is the no-digit subset of the parliamentary test set (406 segments, 4,874 reference words), except where marked. Casual rates are on the full held-out set (478 clips, 148 recordings), except <sup>†</sup> the vote, which was scored only on a held-out evaluation half (229 clips, 74 recordings). <sup>*</sup> Gemini 4 Argon is scored on 60-second windows of consecutive clips, not clip by clip. Lower is better. I do not mark a casual winner: among the leading full-set systems the differences are small, and several of the pairwise intervals include zero.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-asr-formal-casual-light.svg" type="image/svg+xml"/><img src="/images/research/fig-asr-formal-casual-light.png" alt="Horizontal bar charts of formal no-digit word error and casual orthography-tolerant word error for six Kinyarwanda ASR systems. The four-system vote is lowest on formal speech at 13.89 percent." width="1489" height="679" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 1.</strong> Formal no-digit WER (left) and casual wer_canon (right) for the main systems in Table 1. The vote&#x27;s casual bar (<sup>†</sup>, hatched) is the evaluation half of the casual set. Gemini 4 Argon (<sup>*</sup>) is scored on 60-second windows. n = 406 formal no-digit segments; n = 478 casual clips, except the hatched bar (n = 229).</p></figcaption></figure>
<p>The register gap is the main result. Omnilingual LLM-7B, the strongest single open system on formal speech, has a no-digit WER of 15.57% on the parliamentary test set and a wer_canon of 33.52% on casual conversation (raw WER 43.60%; cer_nospace 10.12%). The four-system vote lowers formal no-digit WER to 13.89%, a reduction of 1.68 points against LLM-7B (95% CI [−2.50, −0.91]). On casual speech, scored only on the evaluation half, the vote reaches 30.26 wer_canon against 33.11 for LLM-7B on the same clips; the interval is [−6.92, 0.00], and a post-hoc check found that most of the point-estimate gain comes from three recordings in which LLM-7B entered a repetition loop.</p>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 2.</strong> Closed transcription APIs against the open stack on formal speech.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">System</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Screen WER ↓</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Test WER, no-digit ↓</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Four-system vote</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">—</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>13.89</strong></td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Omnilingual LLM-7B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>16.2</strong></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">15.57</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Gemini 4 Argon, 60 s windows</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">19.69<sup>a</sup></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">17.76</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Gemini 4 Argon, clip-level</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">19.69</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">22.51</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Gemini 3.5 Transcribe</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">38.83</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">screened out</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Gemini 3.8 Flash</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">49.72</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">screened out</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">gpt-transcribe</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">79.89</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">screened out</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">gpt-4o-transcribe</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">83.52</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">screened out</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">gpt-audio-1.5</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">89.39</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">screened out</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Grok STT</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">101.82</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">screened out</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">MAI Transcribe 2</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">106.15</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">screened out</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> The screen is 40 parliamentary development clips (405 seconds of audio). A system was dropped if its screen WER was above 30 and more than 1.5 times LLM-7B on the same clips (LLM-7B screen WER 16.2). Test numbers are no-digit WER on the parliamentary test set. <sup>a</sup> Argon&#x27;s screen number is the clip-level run; the 60-second windowing was selected later, on the full development sitting, and then scored once on the test set.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-asr-closed-models-light.svg" type="image/svg+xml"/><img src="/images/research/fig-asr-closed-models-light.png" alt="Bar charts of closed and open ASR word error on a 40-clip formal development screen and on the parliamentary test set. OpenAI, Grok, and MAI systems are far above the open stack on the screen; only Gemini 4 Argon reaches the test set, where it remains behind the four-system vote." width="1540" height="747" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 2.</strong> Formal WER for closed APIs and the open stack. Left: development screen, 40 clips (405 s). Right: parliamentary test, no-digit WER. The hatched bar is clip-level Gemini 4 Argon; the solid slate bar is the same model with 60-second windows. Systems to the right of the screen threshold were not run on the test set.</p></figcaption></figure>
<p>Closed APIs did not overtake the open stack. On the 40-clip formal screen, three OpenAI transcribers landed between 79.89 and 89.39% WER and were not run on the test set. Grok STT and MAI Transcribe 2 were above 100%. Only Gemini 4 Argon survived (19.69%). Scored clip by clip on the parliamentary test set, Argon has a no-digit WER of 22.51%, 8.62 points behind the vote ([7.04, 10.20]). Much of that gap is a clip-end failure. Mid-clip, on the same test comparison, Argon ties LLM-7B (−0.18 [−1.67, +1.31]). On the development sitting, Argon got the last word wrong in 41% of no-digit clips, against 6% for LLM-7B. After selecting 60-second windows of consecutive clips on development data, Argon reaches 17.76% no-digit WER on the test set. That is 3.50 points behind the vote ([1.28, 5.97]) and statistically tied with LLM-7B (+1.28 [−1.46, +4.36]). On casual conversation the windowed system does not beat the pre-registered best open student on both casual metrics (wer_canon 32.03 vs 33.27, interval [−2.99, +1.35]; cer_nospace 11.79 vs 10.31, interval [−0.13, +3.86] in the other direction).</p>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 3.</strong> Fine-tuning the Digital Umuganda 660-hour model on human minutes versus agreement-filtered machine labels, LM decode.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">Training labels</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Hours</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Formal WER ↓</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Δ vs untrained</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Casual wer_canon ↓</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Δ vs untrained</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">None (untrained)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">—</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">22.36</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">—</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">33.43</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">—</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Human official minutes</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">6.0</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">19.37</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">−3.00 [−3.89, −2.09]</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">36.32</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">+2.89 [+2.04, +3.75]</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">All LLM-7B labels</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">6.0</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>19.14</strong></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">−3.22 [−4.10, −2.31]</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">35.06</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">+1.63 [+0.79, +2.52]</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">Agreement-filtered LLM-7B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">2.4</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">19.22</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">−3.14 [−3.94, −2.31]</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor"><strong>33.30</strong></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">−0.13 [−0.85, +0.56]</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Formal WER is no-digit, parliamentary test, LM decode. Casual wer_canon is the full casual set. Deltas are paired-bootstrap 95% intervals (2,000 resamples, seed 0). The filtered 2.4-hour arm is not different from the 6.0-hour human arm on formal no-digit WER under greedy decode (−0.18 [−0.96, +0.60]); the LM-decode point estimates above are likewise within about a quarter of a point of one another.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-asr-pseudo-labels-light.svg" type="image/svg+xml"/><img src="/images/research/fig-asr-pseudo-labels-light.png" alt="Bar charts showing that fine-tuning on 2.4 hours of agreement-filtered machine labels matches 6 hours of human official minutes on formal parliamentary speech, while none of the label sets lower casual word error reliably." width="1467" height="547" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 3.</strong> Digital Umuganda 660-hour students after fine-tuning, LM decode. Left: formal no-digit WER on the parliamentary test set. Right: casual wer_canon on the full informal set. Agreement-filtered machine labels (2.4 h) match 6.0 h of human minutes on formal speech and do not reliably change casual error.</p></figcaption></figure>
<p>Agreement-filtered machine labels match human labels on formal speech and do not close the casual gap. Fine-tuning on 6.0 hours of official minutes cuts formal no-digit WER by 3.00 points with a language model, but it raises casual wer_canon (+2.89 [+2.04, +3.75]). The same student trained on 2.4 hours of LLM-7B labels, kept only where a second system agrees, matches the human-label arm on the formal test set and does not reliably improve casual wer_canon against the untrained model. Looser keep thresholds, and machine labels drawn from casual audio, similarly fail to move both casual metrics together.</p>
<p>A later student trained on 23.4 hours of filtered labels continues the formal trend (no-digit WER 18.51 with a language model, 3.86 points better than the untrained baseline, interval [−4.79, −3.00]) and still leaves casual wer_canon at 32.85, within noise of the untrained model (−0.59 [−1.44, +0.30]).</p>
<h2>Analysis</h2>
<p>Formal Kinyarwanda, at least in this parliamentary domain, is already in a usable range for an open 7B speech LLM, and a vote takes another one to two points off. Casual conversation is a different problem: even an orthography-tolerant metric stays in the low thirties, and neither closed APIs, nor fine-tuning on formal minutes, nor scaling agreement-filtered labels to a few tens of hours closes it. The vote&#x27;s apparent casual gain is too concentrated to treat as a general improvement.</p>
<p>A screen on short segments misrepresented Gemini 4 Argon as a long-form system. Fixing the clip-end artifact was not enough to beat the open vote on formal speech.</p>
<h2>Limitations</h2>
<p>The official minutes are edited, so formal WER includes an unknown editorial floor. The casual references are informally spelled; wer_canon and cer_nospace are designed for that, but they remain automatic metrics against one annotation style. The parliamentary test set has been scored in many decisions and is no longer clean. The 60-second Argon condition is not a clip-level comparison. The vote&#x27;s casual result uses only half the casual set. I do not yet have a sealed, human-transcribed casual or broadcast test set of comparable size.</p>
<h2>Next measurements</h2>
<p>The next formal claims should be scored once on a sealed set that has not been used for selection. The casual gap needs human-corrected conversational transcripts, not more machine labels of the current quality. I will also keep the vote as a teacher for a larger CTC student, and only then re-measure both registers.</p>]]></content:encoded>
            <author>cmutesa@connect.ust.hk (Cedric Mutesa)</author>
        </item>
        <item>
            <title><![CDATA[KinyaBench: open language models on Kinyarwanda]]></title>
            <link>https://www.mcedric.dev/research/kinyabench-open-models</link>
            <guid>https://www.mcedric.dev/research/kinyabench-open-models</guid>
            <pubDate>Sat, 10 Oct 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[Preliminary chat, 0-shot, single-prompt scores for AfriqueQwen3.5 4B and 9B on the Kinyarwanda slice of public benchmarks.]]></description>
            <content:encoded><![CDATA[<p>I needed a reproducible number for how current open models handle Kinyarwanda, not an average over African languages. KinyaBench is a public-track harness that runs the Kinyarwanda slices of standard benchmarks under a fixed protocol and records the model revision.</p>
<h2>Setup</h2>
<p><strong>Tasks.</strong> The core suite is the AfroBench-Lite Kinyarwanda slice used in the AfriqueLLM paper: AfriMGSM (250 items, exact match, chain-of-thought), AfriMMLU (500, 4-way accuracy), AfriXNLI (600, 3-way accuracy), Belebele (900, 4-way accuracy), SIB-200 (204, 7-way accuracy), Injongo intent (640, 40-way accuracy), and FLORES-200 English↔Kinyarwanda (1,012 sentences per direction, chrF++). FLORES is loaded from Meta&#x27;s official FLORES-200 release, not the gated Hugging Face copy.</p>
<p><strong>Protocol.</strong> Instruct checkpoints are run in the model&#x27;s own chat template, 0-shot, greedy. Thinking is forced off (<code>enable_thinking=False</code> as a top-level model argument; nesting it inside chat-template arguments is silently overwritten by lm-eval&#x27;s vLLM wrapper). Every generated answer is scanned for thinking markers; a leak would invalidate the run. This first pass uses prompt variant 1 only, so these scores are for a single prompt, not a best-of-prompts number. Multiple-choice tasks are scored by log-likelihood after the assistant prefix. I used EleutherAI lm-eval 0.4.13 and vLLM 0.31.0, on one A40, in bfloat16, with a 16,384-token context.</p>
<p><strong>Models.</strong> <code>McGill-NLP/AfriqueQwen3.5-4B-50Langs-Instruct-v1</code> (revision <code>0ff0ff64</code>) and <code>McGill-NLP/AfriqueQwen3.5-9B-50Langs-Instruct-v1</code> (revision <code>693f12e4</code>). Both are CC-BY-4.0. For context I also copy the paper&#x27;s 5-shot scores for the 4B <em>base</em> 50-language checkpoint. That row is a different model and a different protocol; it is not a direct baseline.</p>
<h2>Results</h2>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 1.</strong> KinyaBench public-track scores, 10 October 2026. Chat protocol, 0-shot, prompt 1, thinking off.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">Task (n)</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Metric</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">4B instruct ↑</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">9B instruct ↑</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Paper 4B base, 5-shot</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">AfriMGSM (250)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">exact match</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>46.0</strong> ± 3.2</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">39.2 ± 3.1</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">36.08</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">AfriMMLU (500)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">acc</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">34.0 ± 2.1</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>39.2</strong> ± 2.2</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">39.44</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">AfriXNLI (600)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">acc</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">43.7 ± 2.0</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>56.2</strong> ± 2.0</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">34.57</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Belebele (900)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">acc</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">35.6 ± 1.6</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>62.0</strong> ± 1.6</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">64.58</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">SIB-200 (204)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">acc</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">66.2 ± 3.3</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>80.4</strong> ± 2.8</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">81.00</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Injongo (640)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">acc</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">58.4 ± 1.9</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>66.9</strong> ± 1.9</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">65.81</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">FLORES en→kin (1,012)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">chrF++</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">46.12 ± 0.41</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>48.17</strong> ± 0.40</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">46.39</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">FLORES kin→en (1,012)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">chrF++</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">51.81 ± 0.49</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor"><strong>54.13</strong> ± 0.45</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">53.04</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Intervals are the lm-eval standard error, not a 95% confidence interval. Bold marks the better of the two instruct runs, which are the only scores from this protocol. The paper column is AfriqueQwen3.5-4B-50Langs, base, 5-shot (AfriMGSM 8-shot), from appendix tables of arXiv 2601.06395 v3. It is shown for orientation, not as a matched baseline. AfriMGSM is flexible-extract exact match; strict-match is 0.0 for both instruct models.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-kinyabench-tasks-light.svg" type="image/svg+xml"/><img src="/images/research/fig-kinyabench-tasks-light.png" alt="Grouped bar charts of AfriqueQwen3.5 4B and 9B instruct scores on six Kinyarwanda understanding tasks and on FLORES translation. The 9B model leads on all tasks except AfriMGSM, with the largest gap on Belebele." width="1593" height="728" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 1.</strong> Public-track scores for AfriqueQwen3.5 instruct, chat, 0-shot, prompt 1, thinking off. Error bars are one lm-eval standard error. Left: accuracy or exact match. Right: FLORES-200 chrF++. The 9B model leads on seven of eight tasks; the 4B model leads only on AfriMGSM.</p></figcaption></figure>
<p>The 9B model is ahead on seven of the eight tasks. The gap is largest on Belebele: 62.0% against 35.6% (standard errors 1.6 points). Because the same harness, on the same 4B checkpoint, lands close to the published FLORES numbers (46.12 chrF++ for English→Kinyarwanda against 46.39 in the paper, 51.81 against 53.04 in the other direction), the 4B&#x27;s weak multiple-choice scores are a property of the instruct model under this protocol, not a scoring bug.</p>
<p>AfriMGSM is the exception: 46.0% exact match for the 4B, 39.2% for the 9B. These are thinking-off, greedy, 0-shot figures and are not comparable to the McGill model-card numbers, which use thinking and a long generation budget. AfriXNLI, which sits near chance (33.3%) for every model in the paper, is 43.7% for the 4B and 56.2% for the 9B under chat 0-shot.</p>
<p>Thinking markers leaked in none of the audited generations (FLORES both ways, and AfriMGSM). The 9B AfriMGSM run had one empty completion out of 250.</p>
<h2>Analysis</h2>
<p>A 9B African-continued Qwen3.5 is, under this protocol, a clearly stronger Kinyarwanda reader than its 4B counterpart, especially on Belebele and SIB-200. Translation is already close to the published base-model figures at 4B, and a little higher at 9B. I have not yet run the paper&#x27;s own protocol (raw 5-shot, on the base checkpoint), so I cannot claim a reproduction of the comprehension rows. I have also not yet run a Qwen3.5-9B instruct model without the African continued pretraining, so this note does not isolate that training stage.</p>
<h2>Limitations</h2>
<p>One prompt variant, two models, no base-protocol validation run. AfriXNLI remains a weak signal in the literature even when the 9B is well above chance here. The native, locally written track has no automatic scorer yet, so I am not reporting assistant-quality numbers. SSA-COMET, the paper&#x27;s headline FLORES metric, is not implemented; I compare chrF++ with the paper&#x27;s appendix.</p>
<h2>Next measurements</h2>
<p>The same protocol with all prompt variants; a base-protocol run of AfriqueQwen3.5-4B-50Langs; Qwen3.5-9B instruct without African continued pretraining; and Gemma 4 26B-A4B and 31B, which have no published Kinyarwanda scores I can find. After that, continued-pretraining models from my own pack can be placed on the same scale.</p>]]></content:encoded>
            <author>cmutesa@connect.ust.hk (Cedric Mutesa)</author>
        </item>
        <item>
            <title><![CDATA[Building a Kinyarwanda corpus and training stack]]></title>
            <link>https://www.mcedric.dev/research/kinyarwanda-pretraining-stack</link>
            <guid>https://www.mcedric.dev/research/kinyarwanda-pretraining-stack</guid>
            <pubDate>Mon, 27 Apr 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[The February–April 2026 programme that built a Kinyarwanda text pack, measured tokenizer cost, trained models from the pack, and distilled a 32,433-row supervision set.]]></description>
            <content:encoded><![CDATA[<p>From 15 February to 27 April 2026 I built a Kinyarwanda text factory, packed a pretraining corpus, trained models from that pack, distilled a 32,433-row supervision set, and started a custom tokenizer. The account below is limited to what the files and training logs still show.</p>
<p>The aim was a native Kinyarwanda language model: continued pretraining of an open 8–9 billion-parameter base on a few hundred million clean tokens, then supervised fine-tuning. I finished a full-parameter run of Qwen3-8B, a LoRA adaptation of Qwen3.5-9B, and an instruction-tuning run that I served to testers. Held-out pack perplexity on the 8B run fell from 10.44 to 5.50; the LoRA run moved only 8.07 to 7.84. The served system failed a manual quality review. When I returned to the work in October I treated those results as a reason to measure existing models before training new base weights.</p>
<h2>The pipeline</h2>
<p>The factory mined, language-identified, filtered, and deduplicated documents, then packed them as <code>input_ids</code> sequences with no residual text column. A new base therefore needs retokenization from the document shards. Training and evaluation lived in separate codebases: a continued-pretraining loop with configs, checkpoints, and a promotion gate that I only ever ran dry; a supervision pipeline that generated, cleaned, and split instruction data; and a small evaluation and serving stack for a tester chat surface.</p>
<h2>The pack</h2>
<p>On 27 February 2026 I exported a packed training split of 183,083,577 tokens under the Qwen3 tokenizer (411,133 documents, 44,699 sequences, maximum length 4,096), with 2,026,316 validation tokens and 2,047,384 test tokens. The source-token estimate before packing is 178,795,577 tokens across 420,301 documents.</p>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 1.</strong> Estimated source-token composition of the pack, by type.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">Source type</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Documents</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Tokens (est.)</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Share</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Public collection (mbazaNLP v01.1)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">390,562</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">125,091,104</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>70.0%</strong></td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Books</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">684</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">18,356,385</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">10.3%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Targeted web text</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">13,806</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">17,010,273</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">9.5%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">News (KINNEWS)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">10,314</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">8,849,327</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">4.9%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Official documents</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">458</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">5,016,073</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.8%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Wikipedia</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">3,556</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">3,211,803</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">1.8%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">Web crawl</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">921</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">1,260,612</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">0.7%</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Token counts are the factory&#x27;s pre-pack estimates, summing to 178,795,577. The official packed training split is 183,083,577 tokens. Official documents combine institutional publications; web crawl combines index-targeted and language-filtered Common Crawl slices.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-corpus-composition-light.svg" type="image/svg+xml"/><img src="/images/research/fig-corpus-composition-light.png" alt="Horizontal bar chart of estimated source tokens in the Kinyarwanda pack. A public monolingual collection, mbazaNLP v01.1, accounts for 125.1 million tokens, or 70 percent of the total." width="1417" height="679" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 1.</strong> Estimated source tokens by type, from the same totals as Table 1. The highlighted bar is the public monolingual collection that accounts for 70% of the pack.</p></figcaption></figure>
<p>A random sample of Common Crawl WET files produced no Kinyarwanda: 0 documents in 114,254, even at a language-identification threshold of 0.50. The web-crawl tokens that did enter the pack came from index-targeted and language-filtered extraction, and they remain 0.7% of the estimate.</p>
<p>Filters included language identification, length, boilerplate and template phrases, and near-duplicate removal. A mid-pipeline deduplication pass on 29 of 32 shards flagged 11,052 exact and 1,212 fuzzy duplicates of 409,332 documents (3.0%). I do not treat that as a statistic for the final pack, because three shards were not in that report.</p>
<p>On 28 February I ran a recall-first clean of the mbazaNLP slice: 210,832 of 390,562 documents were kept (54.0%), holding 103,609,913 of 125,091,104 estimated tokens (82.8%). Most discards were short fragments; only 35 documents were hard-rejected, all on a combined template-phrase and low language-ID signal. That clean was packed as a 161,244,925-token training split. It is not the 183-million-token export above, which still includes the uncleaned mbazaNLP slice.</p>
<p>A later, more aggressive retokenization under Qwen3.5, in four cleaning versions, ends at 144,146,803 tokens (304,411 of 420,301 documents kept). That pack is not the continued-pretraining run below.</p>
<h2>Tokenizer cost</h2>
<p>On FLORES-200 devtest (1,012 parallel sentences; 20,077 Kinyarwanda words and 21,901 English words, whitespace-split) the candidate base tokenizers all cost about twice as many tokens for Kinyarwanda as for the same English content. Gemma 4 26B-A4B is the most compact of the three, and the spread among them is small: 2.66 to 2.90 Kinyarwanda tokens per word, an 8% difference in sequence length between Gemma 4 and Qwen3. Qwen2.5-7B and Qwen3-8B produce identical token counts on this set; so do Qwen3.5-9B and AfriqueQwen3.5-9B-Instruct.</p>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 2.</strong> Tokenizer fertility on FLORES-200 devtest.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">Tokenizer</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Vocab</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Kin. tokens/word ↓</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Kin. chars/token ↑</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Kin. ÷ Eng. tokens</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Qwen2.5-7B / Qwen3-8B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">151,665 / 151,669</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.90</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.50</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.11</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Qwen3.5-9B / AfriqueQwen3.5</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">248,070 / 248,077</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.77</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.62</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.02</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">Gemma 4 26B-A4B</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">262,144</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor"><strong>2.66</strong></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor"><strong>2.74</strong></td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">1.97</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Encoded with the <code>tokenizers</code> 0.23.3 library from the local Hugging Face tokenizer files, without adding special tokens (adding them does not change the counts). Words are whitespace-split. Qwen2.5-7B and Qwen3-8B differ in vocabulary size but not in token counts on this set; the same is true of Qwen3.5-9B and AfriqueQwen3.5-9B-Instruct. English tokens per word are 1.26, 1.26, and 1.24 respectively.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-tokenizer-fertility-light.svg" type="image/svg+xml"/><img src="/images/research/fig-tokenizer-fertility-light.png" alt="Two-panel figure. Left: tokens per word on FLORES-200 devtest for Kinyarwanda and English, for Qwen3-8B, Qwen3.5-9B, and Gemma 4 26B-A4B. Right: characters per token for a custom 16k Unigram and SentencePiece baselines on the same held-out shard." width="1644" height="636" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 2.</strong> Left: tokens per word on FLORES-200 devtest (n = 1,012 sentences). Right: characters per token for 16,000-piece Unigram models on a held-out shard of the corpus (1,025 documents, 1,676,192 characters). Higher characters per token is better compression. The custom trainer is more compact than SentencePiece on this shard and cannot be swapped into a pretrained model.</p></figcaption></figure>
<p>I also trained 16,000-piece Unigram models on a stratified shard of the corpus. On the held-out slice in Figure 2, a custom trainer reached 5.08 characters per token against 4.39 for SentencePiece trained on the same shard. The custom trainer was about two orders of magnitude slower (166.6 s versus 1.79 s on a 1,024-document shard, and 359.8 s versus 3.31 s on a 3,072-document shard) and hit <code>RecursionError</code> on larger representative shards. A custom vocabulary also cannot be dropped into a pretrained model without relearning embeddings. I parked the tokenizer.</p>
<h2>Training</h2>
<p>I configured two continued-pretraining recipes: full-parameter Qwen3-8B on a 183-million-token target (learning rate 1×10⁻⁵, sequence length 2,048, Adafactor), and LoRA on Qwen3.5-9B (rank 32, α 16, dropout 0.05, on <code>q_proj</code> and <code>v_proj</code> only; learning rate 2×10⁻⁵) with a 150-million-token target. A later, higher-capacity LoRA configuration (rank 256, α 512) was started and crashed early.</p>
<p>The Qwen3-8B run finished on 2 March 2026: 11,170 steps, 183,007,801 tokens seen (one pass over the February pack, short by 75,776 tokens) in 22.46 hours (2,267 tokens per second). Validation was logged every 500 steps. I have no measurement of the unadapted base on the same split, so the first point is step 500, not step 0. Held-out perplexity fell from 10.44 at step 500 to 5.50 at step 11,000. The best logged value is 5.45 at step 6,500, after 106.5 million tokens; after that the curve is nearly flat, ending at 5.50. Mean training loss was 3.72 over the first 50 steps and 1.89 over the last 50. A 20-million-token sanity on the same recipe, earlier that day, had already moved validation perplexity from 14.42 at step 200 to 9.25 at step 1,200. An earlier full-run start stopped at 161 steps with no validation logged. No test-split metric was logged for any of these runs.</p>
<p>The Qwen3.5-9B LoRA adaptation ran overnight on 13 March 2026: 2,289 steps, 150,010,227 tokens seen, 11.02 hours, 3,780 tokens per second. The training file was a retokenized pack under the Qwen3.5-9B tokenizer, not the February Qwen3 export directly. Validation was logged every 500 steps. Held-out perplexity fell from 8.07 at step 500 to 7.84 at step 2,000, a drop of 0.23 over 1,500 steps. Mean training loss was 2.252 over the first 50 steps and 2.099 over the last 50, and the curve had already flattened in the second half of the run. No downstream task evaluation of this adapter was logged.</p>
<p>The higher-capacity LoRA attempt crashed after a last logged optimizer step of 240, with training loss 2.27 → 2.09 over the twelve logged points. No validation was recorded: evaluation was scheduled every 500 steps.</p>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 3.</strong> Training runs from the February–April programme.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">Model</th><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0.6em 0.45em 0.6em;font-variant-numeric:tabular-nums">Method</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Tokens / rows</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Steps</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Final train loss</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Val. PPL or loss ↓</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Qwen3-8B</td><td style="text-align:left;padding:0.32em 0.6em">Full CPT</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">183,007,801 seen</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">11,170</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.074</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums"><strong>10.44 → 5.50</strong></td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Qwen3.5-9B</td><td style="text-align:left;padding:0.32em 0.6em">LoRA r=32, q/v</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">150,010,227 seen</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2,289</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.168</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">8.07 → 7.84</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Qwen3.5-9B</td><td style="text-align:left;padding:0.32em 0.6em">LoRA r=256, α=512</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">—</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">240 (crashed)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2.088</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">No val logged</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">Qwen3-8B</td><td style="text-align:left;padding:0.32em 0.6em;border-bottom:1px solid currentColor">Full SFT, 3 epochs (failed)</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">12,144,794 seen</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">6,000</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">0.696</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">1.402 → <strong>1.182</strong></td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Validation figures are the first and last logged held-out values, except the SFT row, which ends at the best validation loss (step 4,000; perplexity 3.26). The 8B continued-pretraining best is 5.45 at step 6,500; the last logged value, at step 11,000, is 5.50. LoRA perplexities are at steps 500 and 2,000; the run continued to step 2,289 with no further validation. The higher-capacity LoRA log has no tokens-seen field. The SFT run failed during the second of three planned epochs; at step 6,000, validation loss was 1.199. No step-0 base-model validation was recorded. GPU type was not recorded. An instruction-tuned checkpoint from the SFT recipe was served and reviewed, as described below.</p></p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-cpt-curves-light.svg" type="image/svg+xml"/><img src="/images/research/fig-cpt-curves-light.png" alt="Two-column training curves. Left: Qwen3-8B full continued pretraining, train loss falling and held-out perplexity dropping from 10.44 to 5.50, with a best of 5.45. Right: Qwen3.5-9B LoRA, a nearly flat train-loss mean and held-out perplexity falling only from 8.07 to 7.84. The vertical scales differ." width="1678" height="1009" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 3.</strong> The two completed continued-pretraining runs. Top: training loss at every step, with a 50-step trailing mean. Bottom: held-out perplexity at the logged validation steps. The vertical scales are independent: the 8B full-parameter run moves 10.44 to 5.50 (best 5.45 at step 6,500); the LoRA run moves 8.07 to 7.84. Neither panel has a step-0 base-model point.</p></figcaption></figure>
<p>The two completed continued-pretraining runs are not on a comparable vertical scale. The 8B full-parameter run moves held-out perplexity by 4.94 points; the LoRA run moves it by 0.23. Neither includes a step-0 base-model point, and neither was followed by a task evaluation of the continued-pretraining checkpoint itself.</p>
<h2>Supervision</h2>
<p>Between 7 and 9 March I built a 32,433-row instruction set, split 29,146 / 1,601 / 1,686 for train, validation, and test. Every response was written by GPT-5.4. Automatic cleaning kept 26,515 of 26,525 single-turn answers (nine rejected as too short, one as a duplicate prompt–response pair) and all 5,918 multi-turn chat pairs.</p>
<p>A pre-registered mix check against the lab&#x27;s own plan failed eleven slices. Language composition is the clearest of those failures.</p>
<p style="margin:1.7em 0 0.45em 0"><p><strong>Table 4.</strong> Language modes in the 32,433-row supervision set, against the planned floors.</p></p>
<table style="width:100%;border-collapse:collapse;font-variant-numeric:tabular-nums"><thead><tr><th style="text-align:left;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 1.05em 0.45em 0;font-variant-numeric:tabular-nums">Language mode</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Rows</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Share</th><th style="text-align:right;font-weight:600;border-top:1px solid currentColor;border-bottom:1px solid currentColor;padding:0.45em 0 0.45em 1.05em;font-variant-numeric:tabular-nums">Planned floor</th></tr></thead><tbody><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Kinyarwanda</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">30,141</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">93.0%</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">45%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Mixed Kinyarwanda</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2,234</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">6.9%</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">12%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">Control</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">39</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">0.12%</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">English</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">16</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">0.05%</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">3%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0">French</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">3</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">0.01%</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums">2%</td></tr><tr><td style="text-align:left;padding:0.32em 1.05em 0.32em 0;border-bottom:1px solid currentColor">Swahili</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">0</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">0.00%</td><td style="text-align:right;padding:0.32em 0 0.32em 1.05em;font-variant-numeric:tabular-nums;border-bottom:1px solid currentColor">1%</td></tr></tbody></table>
<p style="margin:0.4em 0 1.7em 0;font-size:0.88em"><p><em>Notes.</em> Floors are from the lab mix plan. The other failed slices are content types below their floors: multi-turn chat 18.2% against 20%; safety refusals 0.84% against 6%; noisy-input robustness 0.61% against 6%; code-switching 3.8% against 7%; multilingual retention 0.88% against 3%; language control 0.84% against 2%. Kinyarwanda instruction rows, at 31.8%, cleared their 20% floor.</p></p>
<p>The set is almost entirely Kinyarwanda teacher text, with 16 English rows, three French, and no Swahili. I configured a three-epoch full-parameter SFT run of the Qwen3-8B continued-pretraining checkpoint on this set (learning rate 8×10⁻⁶, sequence length 2,048). A 200-step sanity finished the same day with validation loss 1.318. The main run failed at step 6,000, during the second of the three planned epochs, after 12,144,794 supervised tokens and 3.49 hours. Validation loss fell from 1.402 at step 500 to a best of 1.182 at step 4,000 (perplexity 3.26), then rose slightly to 1.199 at the last logged step. Training and validation loss both jumped near the epoch boundary at step 3,645; the best validation point is early in the second epoch. Training loss kept falling, to 0.696 at step 6,000. An instruction-tuned checkpoint from that recipe was served.</p>
<figure class="research-figure"><picture><source srcSet="/images/research/fig-sft-curves-light.svg" type="image/svg+xml"/><img src="/images/research/fig-sft-curves-light.png" alt="Two-panel figure of the failed Qwen3-8B supervised fine-tuning run on the 32,433-row set written by GPT-5.4. Train loss keeps falling to 0.696. Held-out loss falls from 1.402 at step 500 to a best of 1.182 at step 4,000, then rises slightly to 1.199 when the run fails at step 6,000." width="1368" height="986" loading="lazy" decoding="async"/></picture><figcaption><p><strong>Figure 4.</strong> Supervised fine-tuning of the Qwen3-8B continued-pretraining checkpoint on the 32,433-row set. Every response in that set was written by GPT-5.4. Top: training loss at every step, with a 50-step trailing mean. Bottom: held-out loss every 500 steps. The run failed at step 6,000, in the second of three planned epochs. A jump in both curves near step 3,645 marks the start of that epoch. The best validation loss is 1.182 at step 4,000; training loss continued to fall after that point.</p></figcaption></figure>
<h2>Evaluation</h2>
<p>On 11 March I reviewed the seven conversations captured from the tester chat surface (34 generations; a broader export of ten conversations contains 37 generations). One of the seven was clearly acceptable. The others failed on identity, on basic facts about Rwanda, on instruction following, or on repetition and unnatural Kinyarwanda. The review called this a failed quality pass, and not a problem that a prompt change would fix.</p>
<p>A concurrent 200-item internal evaluation set was run on the base model, on the instruction-tuned system, and later on untuned Qwen3.5-9B. Those runs were never quality-scored: the pipeline recorded HTTP success, not graded answers. I do not report those automatic pass rates here.</p>
<h2>What this implied</h2>
<p>A perplexity drop on a held-out pack is not a quality gate. Full-parameter continued pretraining moved held-out perplexity from 10.44 to 5.50, and after supervised fine-tuning on the GPT-5.4 responses, the served system failed a human review. 150–183 million Kinyarwanda tokens, used as full-parameter or low-rank continued pretraining, did not produce a usable model at this scale of measurement. A 32,433-row teacher-written supervision set, almost without English or French replay and already off its own mix plan, did not produce acceptable assistant behaviour in that review.</p>
<p>When I returned to the work in October I treated those findings as settled. I stopped training new base weights on a few hundred million tokens, parked the custom tokenizer, and turned to measuring public-track scores for open models, and to a speech-recognition lab, before any further continued pretraining. The public-track results are in a <a href="/research/kinyabench-open-models">separate note</a>.</p>]]></content:encoded>
            <author>cmutesa@connect.ust.hk (Cedric Mutesa)</author>
        </item>
        <item>
            <title><![CDATA[Forking Recoil]]></title>
            <link>https://www.mcedric.dev/writing/recoil-next</link>
            <guid>https://www.mcedric.dev/writing/recoil-next</guid>
            <pubDate>Mon, 28 Jul 2025 00:00:00 GMT</pubDate>
            <description><![CDATA[A TypeScript continuation of Recoil, with React 18 and 19 support, a Rollup ESM and CJS build, and the same public API.]]></description>
            <content:encoded><![CDATA[<p>The product I was working on, <a href="https://chatmaid.net/">Chatmaid</a>, schedules and sends WhatsApp messages from your own number, through a mobile app, a web dashboard, and an API. The whole stack used <a href="https://recoiljs.org/">Recoil</a> for state. A React 18 upgrade broke it. Upstream Recoil had already stopped moving, so I forked the library.</p>
<p>React 18 changed how a library must subscribe to store updates. The supported path is <code>useSyncExternalStore</code>, which Recoil adopted in 0.6 on 28 January 2022 (<a href="https://github.com/facebookexperimental/Recoil/pull/1488">#1488</a>). The hook exists so an external store cannot tear under concurrent rendering: React reads a consistent snapshot for that render. It also requires a <code>getServerSnapshot</code> for server render. Without that function, Next.js apps failed during SSR (<a href="https://github.com/facebookexperimental/Recoil/issues/1526">#1526</a>). Some host environments, including nested renderers, exposed a React 18 dispatcher but not the hook itself, which produced <code>useSyncExternalStore is not a function</code>. Recoil added a fallback in 0.7.6 (<a href="https://github.com/facebookexperimental/Recoil/pull/2001">#2001</a>, <a href="https://github.com/facebookexperimental/Recoil/pull/2010">#2010</a>), and further Next.js 13 SSR fixes in 0.7.7 (<a href="https://github.com/facebookexperimental/Recoil/issues/2082">#2082</a>, <a href="https://github.com/facebookexperimental/Recoil/issues/2086">#2086</a>). Application and test code had their own breaks: <code>ReactDOM.render</code> gave way to <code>createRoot</code>, and <code>act</code> moved from <code>ReactTestUtils</code> into the <code>react</code> package. Chatmaid&#x27;s upgrade hit this class of failure.</p>
<p>The last Recoil release, 0.7.7, shipped on 1 March 2023. The last upstream commit was on 7 September 2023. A planned handover to external maintainers stalled after the remaining public maintainers were laid off (<a href="https://github.com/facebookexperimental/Recoil/discussions/2171">discussion #2171</a>). Meta archived <a href="https://github.com/facebookexperimental/Recoil"><code>facebookexperimental/Recoil</code></a> on 1 January 2025.</p>
<h2>The fork</h2>
<p>I created <a href="https://github.com/Mutesa-Cedric/Recoil-next"><code>Mutesa-Cedric/Recoil-next</code></a> on 27 November 2024, a month before the archive. The next day, in <a href="https://github.com/Mutesa-Cedric/Recoil-next/commit/fb238a59833b96c3eb9877e08a70d5896a8e0258">fb238a5</a>, I raised the peer dependency from <code>react &gt;= 16.13.1</code> to <code>react &gt;= 18.3.1</code>, switched the example app to <code>createRoot</code>, and replaced <code>ReactTestUtils.act</code> with <code>act</code> from <code>react</code>. That commit updates the package metadata together with the test and example trees, 44 files in all. The current peer range is <code>react &gt;= 18</code>. The development tree uses React 19.1.1, and the test matrix covers React 18 and React 19 on Node 20 and 22.</p>
<h2>TypeScript</h2>
<p>Recoil was written in Flow. On 6 July 2025 I started a port to TypeScript (<a href="https://github.com/Mutesa-Cedric/Recoil-next/commit/c09d731ae">c09d731</a>). This was a rewrite of the sources, not a types layer over Flow. Shared utilities, the ADT and cache layers, atoms, selectors, hooks, and <code>RecoilRoot</code> all moved before the first publish. The core package was complete on 16 July (<a href="https://github.com/Mutesa-Cedric/Recoil-next/commit/ecb69f038">ecb69f0</a>). I then ported <code>recoil-relay</code>, <code>recoil-sync</code>, and <code>refine</code>. <a href="https://github.com/Mutesa-Cedric/Recoil-next/pull/1">Pull request #1</a> landed on 8 August and made the TypeScript tree the repository root.</p>
<p>The build moved to Rollup, emitting ESM and CommonJS with generated typings. Tests moved from Jest to Vitest. The workspace uses pnpm. ESLint is on v9. The ported core test tree has 43 files and 284 <code>test</code>/<code>it</code> cases. I published the rewritten package as <a href="https://www.npmjs.com/package/recoil-next"><code>recoil-next</code></a> <code>0.1.0</code> on 28 July 2025. It has been downloaded more than 350,000 times since then.</p>
<p>Migration is an import change, and that is the only required code change. The Recoil API is unchanged, so the original Recoil documentation still applies:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// Before</span>
<span class="token keyword module">import</span> <span class="token imports"><span class="token punctuation">{</span>atom<span class="token punctuation">,</span> selector<span class="token punctuation">,</span> useRecoilState<span class="token punctuation">}</span></span> <span class="token keyword module">from</span> <span class="token string">&#x27;recoil&#x27;</span><span class="token punctuation">;</span>

<span class="token comment">// After</span>
<span class="token keyword module">import</span> <span class="token imports"><span class="token punctuation">{</span>atom<span class="token punctuation">,</span> selector<span class="token punctuation">,</span> useRecoilState<span class="token punctuation">}</span></span> <span class="token keyword module">from</span> <span class="token string">&#x27;recoil-next&#x27;</span><span class="token punctuation">;</span>
</code></pre>
<p>Version 0.2.0, on 10 August, stopped <code>react/jsx-runtime</code> from being bundled into the output (<a href="https://github.com/Mutesa-Cedric/Recoil-next/commit/f327be4ad">f327be4</a>) and updated the todo example to use <code>recoil-next</code>. On 15 August the ported <code>recoil-core</code>, <code>recoil-sync</code>, and <code>recoil-relay</code> tests were passing. I added <code>__DEV__</code> support and published 0.3.0.</p>
<p><code>recoil-next@0.4.4</code> ships an unminified ESM build of 204 KB, 41.7 KB gzipped. Recoil 0.7.7&#x27;s unminified <code>es/index.js</code> is 275 KB, 63.0 KB gzipped.</p>
<h2>The NaN loop</h2>
<p>An atom whose default was <code>NaN</code> could throw <code>Maximum update depth exceeded</code> as soon as a component read it:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> <span class="token maybe-class-name">Atom</span> <span class="token operator">=</span> <span class="token function">atom</span><span class="token punctuation">(</span><span class="token punctuation">{</span>
  <span class="token literal-property property">key</span><span class="token operator">:</span> <span class="token string">&#x27;myKey&#x27;</span><span class="token punctuation">,</span>
  <span class="token keyword module">default</span><span class="token operator">:</span> <span class="token number">NaN</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

<span class="token keyword module">export</span> <span class="token keyword module">default</span> <span class="token keyword">function</span> <span class="token function"><span class="token maybe-class-name">Test</span></span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
  <span class="token keyword">const</span> val <span class="token operator">=</span> <span class="token function">useRecoilValue</span><span class="token punctuation">(</span><span class="token maybe-class-name">Atom</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
  <span class="token keyword control-flow">return</span> <span class="token operator">&lt;</span>div<span class="token operator">&gt;</span><span class="token punctuation">{</span>val<span class="token punctuation">}</span><span class="token operator">&lt;</span><span class="token operator">/</span>div<span class="token operator">&gt;</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
</code></pre>
<p><code>Loadable.is()</code> compared contents with <code>===</code>. In JavaScript, <code>NaN === NaN</code> is false, so two loadables holding <code>NaN</code> looked different on every check. Recoil uses that equality to decide whether a value has changed and subscribers must re-render, so the store kept updating. The fix is <code>Object.is</code>, which treats <code>NaN</code> as equal to itself:</p>
<pre class="language-ts"><code class="language-ts"><span class="token keyword">is</span><span class="token punctuation">(</span>other<span class="token operator">:</span> <span class="token maybe-class-name">Loadable</span><span class="token operator">&lt;</span><span class="token builtin">any</span><span class="token operator">&gt;</span><span class="token punctuation">)</span><span class="token operator">:</span> <span class="token builtin">boolean</span> <span class="token punctuation">{</span>
  <span class="token keyword control-flow">return</span> <span class="token punctuation">(</span>
    <span class="token punctuation">(</span>other <span class="token keyword module">as</span> <span class="token builtin">any</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token property-access">state</span> <span class="token operator">===</span> <span class="token punctuation">(</span><span class="token keyword">this</span> <span class="token keyword module">as</span> <span class="token builtin">any</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token property-access">state</span> <span class="token operator">&amp;&amp;</span>
    <span class="token known-class-name class-name">Object</span><span class="token punctuation">.</span><span class="token keyword">is</span><span class="token punctuation">(</span><span class="token punctuation">(</span>other <span class="token keyword module">as</span> <span class="token builtin">any</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token property-access">contents</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token keyword">this</span> <span class="token keyword module">as</span> <span class="token builtin">any</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token property-access">contents</span><span class="token punctuation">)</span>
  <span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
</code></pre>
<p>That change is <a href="https://github.com/Mutesa-Cedric/Recoil-next/commit/2342fff6e9c43b29608d1391c2f78a440e7fd85d">2342fff</a>, documented in <a href="https://github.com/Mutesa-Cedric/Recoil-next/issues/3">#3</a>. It shipped in 0.4.0 on 7 November 2025.</p>
<h2>One package</h2>
<p>After the TypeScript port, <code>recoil-relay</code>, <code>recoil-sync</code>, and <code>refine</code> still lived in the same workspace as core. Those packages are optional integrations, not the Recoil API. I published the companions as <a href="https://www.npmjs.com/package/recoil-relay-next"><code>recoil-relay-next</code></a> and <a href="https://www.npmjs.com/package/refine-next"><code>refine-next</code></a> on 18 August 2025. On 28 December, <a href="https://github.com/Mutesa-Cedric/Recoil-next/pull/4">pull request #4</a> removed them from this repository. A core install is the Recoil API and nothing else. <code>recoil-next@0.4.4</code> was published on 26 January 2026.</p>
<p>The public API is still the Recoil one, including atoms, selectors, hooks, and <code>RecoilRoot</code>. The sources are TypeScript, and the build emits ESM and CommonJS for React 18 and 19.</p>]]></content:encoded>
            <author>cmutesa@connect.ust.hk (Cedric Mutesa)</author>
        </item>
        <item>
            <title><![CDATA[.mjs, .cjs, and how Node decides]]></title>
            <link>https://www.mcedric.dev/writing/mjs-cjs</link>
            <guid>https://www.mcedric.dev/writing/mjs-cjs</guid>
            <pubDate>Mon, 16 Oct 2023 00:00:00 GMT</pubDate>
            <description><![CDATA[How Node chooses ESM or CommonJS from a file extension and package.json, and what require(esm) changed for packages that ship both.]]></description>
            <content:encoded><![CDATA[<p>Node has two module systems: CommonJS (<code>require</code>, <code>module.exports</code>) and ECMAScript modules (<code>import</code> / <code>export</code>). I first wrote this in 2023 because <code>.mjs</code> is the most direct way to opt a file into ESM. That is still true. What people actually trip over is the <code>&quot;type&quot;</code> field, <code>&quot;exports&quot;</code> conditions, and <code>require()</code> of ESM, and parts of that have changed.</p>
<h2>How Node chooses</h2>
<p>The <a href="https://nodejs.org/api/packages.html#determining-module-system">packages documentation</a> is the source of truth. For a file Node is asked to run or import:</p>
<ul>
<li><code>.mjs</code> is always an ES module, regardless of the nearest <code>package.json</code>.</li>
<li><code>.cjs</code> is always CommonJS, for the same reason.</li>
<li><code>.js</code> follows the nearest parent <code>package.json</code>. <code>&quot;type&quot;: &quot;module&quot;</code> makes it ESM; <code>&quot;type&quot;: &quot;commonjs&quot;</code>, or no <code>&quot;type&quot;</code> field at all, makes it CommonJS.</li>
</ul>
<p><code>.mjs</code> therefore needs no package configuration. This is the example I used in 2023, and it still runs with <code>node main.mjs</code>:</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// mathUtils.mjs</span>
<span class="token keyword module">export</span> <span class="token keyword">function</span> <span class="token function">add</span><span class="token punctuation">(</span><span class="token parameter">a<span class="token punctuation">,</span> b</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
  <span class="token keyword control-flow">return</span> a <span class="token operator">+</span> b
<span class="token punctuation">}</span>

<span class="token keyword module">export</span> <span class="token keyword">function</span> <span class="token function">multiply</span><span class="token punctuation">(</span><span class="token parameter">a<span class="token punctuation">,</span> b</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
  <span class="token keyword control-flow">return</span> a <span class="token operator">*</span> b
<span class="token punctuation">}</span>
</code></pre>
<pre class="language-js"><code class="language-js"><span class="token comment">// main.mjs</span>
<span class="token keyword module">import</span> <span class="token imports"><span class="token punctuation">{</span> add<span class="token punctuation">,</span> multiply <span class="token punctuation">}</span></span> <span class="token keyword module">from</span> <span class="token string">&#x27;./mathUtils.mjs&#x27;</span>

<span class="token console class-name">console</span><span class="token punctuation">.</span><span class="token method function property-access">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Sum: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token console class-name">console</span><span class="token punctuation">.</span><span class="token method function property-access">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Product: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span><span class="token function">multiply</span><span class="token punctuation">(</span><span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
</code></pre>
<p>The extension is the signal. The same <code>import</code> in a <code>.js</code> file is a syntax error unless that file is already in an ESM scope. <a href="https://nodejs.org/api/packages.html#packagejson-and-file-extensions">File extensions override <code>&quot;type&quot;</code></a>, which is how you mix formats in one package: a <code>&quot;type&quot;: &quot;module&quot;</code> package can still keep a CommonJS file by naming it <code>.cjs</code>, and a CommonJS package can keep an ESM file by naming it <code>.mjs</code>.</p>
<p>Package authors should set <code>&quot;type&quot;</code> explicitly, even when every file is CommonJS. Node may change the default later, and tools should not have to guess. Since Node 22.7 and 20.19, an ambiguous <code>.js</code> file (no <code>&quot;type&quot;</code> in scope) is also subject to <a href="https://nodejs.org/api/packages.html#syntax-detection">syntax detection</a>: if the source can only parse as ESM, Node retries it as ESM. That has a cost, and it is a fallback, not a reason to omit <code>&quot;type&quot;</code>.</p>
<p><code>import</code> and <code>require</code> also resolve specifiers differently. <a href="https://nodejs.org/api/esm.html#mandatory-file-extensions">A relative or absolute <code>import</code> must include the file extension</a>, and directory indexes have to be written out (<code>./startup/index.js</code>, not <code>./startup</code>). <code>require(&#x27;./startup&#x27;)</code> will still search <code>.js</code>, <code>.json</code>, and <code>.node</code>, and will still treat a folder as a module.</p>
<h2>What <code>.mjs</code> is not</h2>
<p>ESM can be statically analysed. Bundlers use that to drop unused exports, which is tree shaking. If <code>main.mjs</code> only imports <code>add</code>, a bundler can omit <code>multiply</code> from the output. That is a property of <code>import</code> / <code>export</code>, not of the <code>.mjs</code> extension, and it is not something Node does when you run <code>node main.mjs</code>. I overstated this in 2023.</p>
<p>Dynamic <code>import()</code> is not an <code>.mjs</code> feature either. It is <a href="https://nodejs.org/api/esm.html#import-expressions">available in both CommonJS and ESM</a>, and it is the way a CommonJS file loads an ES module when it needs to do so asynchronously:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> add <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token keyword control-flow">await</span> <span class="token keyword module">import</span><span class="token punctuation">(</span><span class="token string">&#x27;./mathUtils.mjs&#x27;</span><span class="token punctuation">)</span>
</code></pre>
<p>In a browser, the decision is the script&#x27;s <code>type=&quot;module&quot;</code> (and the JavaScript MIME type the server sends), not the file extension. <code>.mjs</code> is not special there. Bare specifiers such as <code>import &#x27;lodash&#x27;</code> also do not resolve in a browser without an import map, and Node built-ins do not exist there. ESM is a shared syntax, not a promise that the same file runs unchanged on both sides without a bundler.</p>
<p>An ES module has to name its dependencies in <code>import</code> statements, and top-level bindings stay in the module. They are not written onto a shared <code>exports</code> object, and they do not leak into other files. That is the practical difference from CommonJS, and it is why I preferred ESM in 2023.</p>
<h2>Packages: <code>&quot;exports&quot;</code> and the dual-package hazard</h2>
<p>For a package, <code>&quot;main&quot;</code> is only the default entry. <a href="https://nodejs.org/api/packages.html#exports"><code>&quot;exports&quot;</code></a> replaces it when present: it can list subpaths, and it hides anything not listed. Consumers of <code>pkg/hidden.js</code> get <code>ERR_PACKAGE_PATH_NOT_EXPORTED</code>.</p>
<p><a href="https://nodejs.org/api/packages.html#conditional-exports">Conditional exports</a> then pick a file based on how the package is loaded. <code>&quot;import&quot;</code> and <code>&quot;require&quot;</code> are mutually exclusive. Key order matters; more specific conditions go first, and <code>&quot;default&quot;</code> last:</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
  <span class="token property">&quot;type&quot;</span><span class="token operator">:</span> <span class="token string">&quot;module&quot;</span><span class="token punctuation">,</span>
  <span class="token property">&quot;exports&quot;</span><span class="token operator">:</span> <span class="token punctuation">{</span>
    <span class="token property">&quot;import&quot;</span><span class="token operator">:</span> <span class="token string">&quot;./index.mjs&quot;</span><span class="token punctuation">,</span>
    <span class="token property">&quot;require&quot;</span><span class="token operator">:</span> <span class="token string">&quot;./index.cjs&quot;</span>
  <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
</code></pre>
<p>That pattern is common, and it has a cost. If one part of an application <code>import</code>s the package and a dependency <code>require</code>s it, Node can evaluate two different files and keep two copies in memory. Constructors then fail <code>instanceof</code> across the boundary, and mutations of a singleton on one copy are invisible to the other. Node&#x27;s name for this is the <a href="https://nodejs.org/api/packages.html#dual-commonjses-module-packages">dual package hazard</a>; the current docs point at the <a href="https://nodejs.github.io/package-examples/">package examples</a> repository for the worked cases. The usual mitigations are to wrap a single CommonJS implementation for <code>import</code>, or to isolate any state in one shared file that both entry points load.</p>
<h2><code>require(esm)</code></h2>
<p>Until recently, <code>require()</code> of a native ES module threw <code>ERR_REQUIRE_ESM</code>. That is why so many packages shipped a separate <code>.cjs</code> build. <a href="https://nodejs.org/api/modules.html#loading-ecmascript-modules-using-require">Loading ESM with <code>require()</code></a> was unflagged in <a href="https://nodejs.org/en/blog/release/v22.12.0">Node 22.12</a> and <a href="https://nodejs.org/en/blog/release/v20.19.0">20.19</a>, and in current Node it is stable (as of 25.4). The module must be synchronous. If the graph contains top-level <code>await</code>, <code>require()</code> throws <code>ERR_REQUIRE_ASYNC_MODULE</code> and you should use <code>import()</code> instead.</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> add <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">&#x27;./mathUtils.mjs&#x27;</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> <span class="token maybe-class-name">Point</span> <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">&#x27;./point.mjs&#x27;</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token keyword module">default</span>
</code></pre>
<p>The return value is the module namespace, so a default export is on <code>.default</code>. A package that only needs to support these Node versions can ship ESM and let both <code>import</code> and <code>require</code> load the same file, which removes the split that causes the hazard:</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
  <span class="token property">&quot;type&quot;</span><span class="token operator">:</span> <span class="token string">&quot;module&quot;</span><span class="token punctuation">,</span>
  <span class="token property">&quot;exports&quot;</span><span class="token operator">:</span> <span class="token string">&quot;./index.js&quot;</span><span class="token punctuation">,</span>
  <span class="token property">&quot;engines&quot;</span><span class="token operator">:</span> <span class="token punctuation">{</span>
    <span class="token property">&quot;node&quot;</span><span class="token operator">:</span> <span class="token string">&quot;^20.19.0 || &gt;=22.12.0&quot;</span>
  <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
</code></pre>
<p>The <code>&quot;module-sync&quot;</code> <a href="https://nodejs.org/api/packages.html#conditional-exports">exports condition</a> matches that same synchronous-ESM case for both loaders.</p>
<p><code>.mjs</code> remains the local way to force a format. <code>&quot;type&quot;</code> does the same at package scope. A split <code>&quot;import&quot;</code> / <code>&quot;require&quot;</code> pair is a compatibility choice with a real failure mode, and it should not be the default shape of a new package.</p>]]></content:encoded>
            <author>cmutesa@connect.ust.hk (Cedric Mutesa)</author>
        </item>
    </channel>
</rss>