Bantu ITN: shared kernel with a test-only language profile by cal3bdev · Pull Request #518 · NVIDIA/NeMo-text-processing

cal3bdev · 2026-10-25T15:47:51Z

What does this PR do ?

Adds a shared inverse text normalization (ITN) kernel for Bantu languages in
nemo_text_processing/inverse_text_normalization/bantu/, tested with a small made-up
test language. Luganda and Swahili will be built on it by supplying a language profile
(word lists and settings), instead of each writing its own grammars.

This is stage 1 of the staged plan agreed with @tbartley94: the kernel and a small
test-only language profile, with tests that check the kernel on its own. Luganda and
Swahili follow in their own PRs.

The kernel is laid out like a regular language but is not added to the language
codes, so it cannot be selected with --lang. The tests build the classifier and
verbalizer directly and run them through NeMo's inverse_normalize runtime.

Collection: ITN

Why a shared kernel

Bantu languages share the hard parts of number reading:

Number words agree with the noun class of what is counted, so the same digit has
different spoken forms (Luganda: abantu babiri "2 people", amagi abiri "2 eggs").
The same word can mean different numbers in different contexts (Luganda abiri is 2
after amagi "eggs", but 20 after a currency word).
The join word between number parts is also an ordinary word (Luganda mu joins
"twenty mu five" but also means "in").
Large amounts are counted with scale nouns (Luganda mitwalo ebiri,
"two ten-thousands" = 20,000).

The kernel handles these once; a language supplies only its words and settings.

How it works

The classifier reads the whole sentence. Runs of number words are marked first,
then each run is read in the context of the word before it: a counted noun
(CardinalFst), a currency phrase or money cue (MoneyFst), or nothing, in which
case the words stay unchanged (WordFst). Output is the standard token format:
tokens { name: "bakopa" } tokens { cardinal { integer: "21" } }.
Values are resolved in the tagger; the verbalizer only formats them
(digit grouping, <amount> <ISO code>).
No weights decide between readings. Weights only prefer a normalization over
leaving words unchanged. Readings are made unique by structure, and every test also
checks that its input has exactly one best reading. (NeMo's runtime picks one reading
silently when there is a tie, so this check matters.)
Input: the kernel reads lower-cased words. Any other word (capitals, digits,
punctuation, any script) passes through unchanged, and repeated spaces are collapsed.
Punctuation attached to a number word (bapi.) leaves that number unchanged; full
cased and punctuation handling comes with each language.
Supported classes in this PR: cardinal, money and plain words.

Files

nemo_text_processing/inverse_text_normalization/bantu/
  profile.py                  the settings a language supplies
  taggers/
    tokenize_and_classify.py  ClassifyFst(profile): reads the whole sentence
    cardinal.py               CardinalFst: a noun and the number that agrees with it
    money.py                  MoneyFst: amounts after a currency phrase, implied currency
    word.py                   WordFst: words left unchanged
    number.py                 place-value numbers built from the profile
    scale.py                  scale nouns with a count (ten-thousands, millions)
    sentence.py               number-word runs, token formats, money-signal check
  verbalizers/
    cardinal.py, money.py, word.py, verbalize.py, verbalize_final.py
tests/nemo_text_processing/bantu/
  toy_profile.py              the made-up test language
  utils.py                    NeMo's runtime with the kernel's classifier and verbalizer
  data_inverse_text_normalization/test_cases_{cardinal,money,word}.txt
  test_cardinal.py, test_money.py, test_word.py, test_verbalizer.py, test_profile.py
Jenkinsfile                   new stage: L1: Run Bantu kernel ITN tests

The test-only language

toy_profile.py is a small made-up language. Every word is invented, but it is shaped
like a Bantu language so that each kernel feature is used: noun classes with agreement,
a join word that is also an ordinary word, a currency phrase that licenses a special
reading, ten-thousands and millions counted with scale nouns, implied currency with
money cues, contracted joins, and a word that continues a number but is not handled
yet. Its currency code is XTS, the ISO 4217 code reserved for testing.

Because the words are invented, a failing test points at the kernel, not at real
language data. The test profile is written in Python because it exists only for the
tests; real languages will keep their word lists in data/*.tsv files.

Tests

pytest tests/nemo_text_processing/bantu --cpu

102 tests, all passing in about 7 seconds, with every line of the kernel covered.
Spoken/written pairs are in test_cases_*.txt, as for other languages: counted
numbers, money, words left unchanged, numbers inside sentences, malformed number
runs, and input with capitals, digits, punctuation and other scripts.
The verbalizer is tested on its own, including rejecting invalid values.
The classifier's FAR cache is saved and restored with the same results.
Invalid profiles are rejected with a clear error.
A new Jenkinsfile stage runs these tests in CI.

Outside this PR, we also ran our 205 Luganda development cases (cardinals and money)
through this kernel in NeMo's runtime: all pass, each with exactly one best reading.
That data will come with the Luganda PR after native-speaker review.

Questions for the reviewer

Base branch: this PR targets main. Other languages use a staging branch (e.g.
staging/telugu_itn_v1). Should we retarget to one, e.g. staging/bantu_itn_v1?
Sparrowhawk: the kernel is not a registered language, so pynini_export.py cannot
export it as-is. How would you like the test language exported for the Sparrowhawk tests?

Not in this PR

Sparrowhawk tests: next step, as agreed, so the code layout can be reviewed first
(see question 2).
Languages: Luganda (stage 2) and Swahili (stage 3) come in their own PRs, each with
its data and tests. Swahili test cases are being reviewed by Ernest Omondi.
Cased input and attached punctuation: with each language.
Other classes (e.g. decimals): added later per language, updating the kernel as needed.

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

🤖 Generated with Claude Code

Adds nemo_text_processing/inverse_text_normalization/bantu/, a shared inverse text normalization kernel that Bantu languages configure with a language profile. It supports cardinal and money, resolved in a sentence-level classifier; the verbalizers only format. The kernel is not listed in the language codes. Tests build the classifier and verbalizer directly with a made-up test language (currency code XTS), run through NeMo's inverse_normalize runtime, and check that every test input has exactly one best reading. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Cal3bdev <[email protected]>

- Pass any other word (capitals, digits, punctuation, any script) through unchanged and collapse repeated spaces, instead of failing on such input. - Split the taggers into CardinalFst, MoneyFst and WordFst classes, with the place-value builders in number.py. - Move test cases into test_cases_*.txt files read by parse_test_case_file. - Add a Jenkinsfile stage for the kernel tests. - Use typing.Optional/Tuple, NeMo's logger and the {lang}_itn_lower_cased.far cache name; move the vowel list into the profile. - Fix the money-signal docstring and remove project-internal notes. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Cal3bdev <[email protected]>

The test language now has a ten-thousands number place and counts a listed noun with its ten-thousands scale, as for millions, so a count of 10 or more after a listed noun is read like after any other word. Corrects the comment on how listed nouns are counted and adds test cases. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Cal3bdev <[email protected]>

tbartley94

Minor changes needed:

Expansion of cardinal cases
attribution for crane labs (optional)
Left questions about reasoning for semiotic class distinctions. Not necessarily a problem but curious about justification.

I'm also uncertain if the profile approach should be used. It's rather clean but departs from our normal convention so adds maintenance overhead. Any further explanation on the approach?

tbartley94 · 2026-10-28T21:10:16Z

@@ -0,0 +1,13 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#


You'll want to add Crane Labs contribution here just so it doesn't all go to Nvidia.

Done in d2752d1.

tbartley94 · 2026-10-28T21:10:27Z

@@ -0,0 +1,69 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#


and elsewhere.

Done in d2752d1.

tbartley94 · 2026-10-28T21:13:22Z

+bakopa emina deka k'epi~bakopa 120,000
+bakopa emina pideka vo ema~bakopa 210,000
+bakopa emina pideka vo milo fu~bakopa 205,000
+bakopa emina pideka vo baima~bakopa emina pideka vo baima


please add a random sample for a million or so

Done in d2752d1.

tbartley94 · 2026-10-28T21:13:57Z

+zikopa pideka vo kasa~zikopa 20 vo kasa
+rana bakopa bako vo kasa lena~rana bakopa 3 vo kasa lena
+rana bakopa bapi lena zikopa ziko~rana bakopa 2 lena zikopa 3
+bakopa bapi zikopa ziko~bakopa 2 zikopa 3


2 is a bit too frequent here, please just enumerate cases up to 20 for a basic sanity checks

Done in d2752d1.

tbartley94 · 2026-10-28T21:14:47Z

@@ -0,0 +1,39 @@
+koin zipi~2 XTS


let's add base cases for dollars, pound, euros, yen

Done in d2752d1.

tbartley94 · 2026-10-28T21:35:32Z

@@ -0,0 +1,88 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.


why is this distinct from a cardinal?

number.py holds the place-value builders that cardinal, money and scale all use. CardinalFst can't be the shared piece as it stands, because it also decides the context (the noun before the number). Other languages pass a CardinalFst into MoneyFst and reuse its graph, and we can follow that: move these builders into CardinalFst and have MoneyFst take the cardinal. I'd do that as part of the restructure in my main comment.

tbartley94 · 2026-10-28T21:36:21Z

+
+from nemo_text_processing.inverse_text_normalization.bantu.profile import CardinalProfile, Forms, NounContext
+
+DIGIT = pynini.union(*"0123456789").optimize()


please use strings.digit

Done in d2752d1.

tbartley94 · 2026-10-28T21:36:45Z

+from nemo_text_processing.inverse_text_normalization.bantu.profile import CardinalProfile, Forms, NounContext
+
+DIGIT = pynini.union(*"0123456789").optimize()
+NONZERO = pynini.union(*"123456789").optimize()


strings.digit[1:]

Done in d2752d1.

tbartley94 · 2026-10-28T21:37:47Z

+    for place in profile.places:
+        if place.power != expected_power:
+            raise ValueError("places must be consecutive ascending decimal powers")
+        if any(value not in "123456789" or len(value) != 1 for _, value in place.forms):


string already defined as global

Done in d2752d1.

tbartley94 · 2026-10-28T21:39:26Z

+    return amount.optimize()
+
+
+def money_token(amount: pynini.Fst, code: str, marker: str = "") -> pynini.Fst:


believe you can reuse the token context helper from other cases no?

Done in d2752d1.

- add Crane Labs to the licence headers - move taggers/sentence.py to bantu/graph_utils.py - use NEMO_SPACE, string.digits and GraphFst.add_tokens - move the thousands separator into the profile - cardinal cases: enumerate 1-20, add million-range samples - money cases: base cases for USD, GBP, EUR and JPY Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Cal3bdev <[email protected]>

cal3bdev · 2026-10-29T10:59:44Z

Thanks for the comments, @tbartley94. I've pushed d2752d1 with the smaller fixes and replied to the questions inline.

On the profile: it was simply the quickest way for each language to hand its words and settings to the shared kernel. You're right that it departs from the usual interface: the profile argument on the constructors, a language written in Python instead of data files, and the test helper that builds the normalizer by hand.

Proposal: keep the kernel's builders in bantu/, and make each language a normal NeMo language around them:

Each language (lg/, sw/) has the standard ClassifyFst(cache_dir, overwrite_cache, input_case) and VerbalizeFinalFst(), keeps its word lists in data/*.tsv, and is registered like any other language.
It reads its TSVs and calls the shared builders directly, with no inheritance. The profile becomes an internal detail.
The test language gets the same layout and is hidden from the CLI, as you suggested. That removes the build_normalizer helper and the profile arguments.
MoneyFst takes the CardinalFst, as in other languages.

Adding a Bantu language would then mean adding its TSVs and tests. Does this work for you? If so, I'll restructure the PR in one pass.

cal3bdev and others added 3 commits October 25, 2026 10:56

cal3bdev marked this pull request as ready for review October 27, 2026 06:09

tbartley94 self-requested a review October 28, 2026 17:29

tbartley94 requested changes Oct 28, 2026

View reviewed changes

		@@ -0,0 +1,13 @@
		# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
		#

		@@ -0,0 +1,69 @@
		# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
		#

		@@ -0,0 +1,88 @@
		# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.


		from nemo_text_processing.inverse_text_normalization.bantu.profile import CardinalProfile, Forms, NounContext

		DIGIT = pynini.union(*"0123456789").optimize()

		return amount.optimize()


		def money_token(amount: pynini.Fst, code: str, marker: str = "") -> pynini.Fst:

Conversation

cal3bdev commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Why a shared kernel

How it works

Suggested reading order

Files

The test-only language

Tests

Questions for the reviewer

Not in this PR

Before your PR is "Ready for review"

Uh oh!

tbartley94 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cal3bdev commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cal3bdev commented Oct 25, 2026 •

edited

Loading