Skip to content

feat: add Hebrew TN/ITN with raw-field tagging - #398

Merged
pengzhendong merged 6 commits into
masterfrom
feat/hebrew-normalization
Oct 31, 2026
Merged

pengzhendong merged 6 commits into
masterfrom
feat/hebrew-normalization

Conversation

@pengzhendong

Copy link
Copy Markdown
Member

Summary

  • Add native Hebrew (he) ITN and a new TN, completing the direction missing from NeMo. Both use raw-field taggers and semantic verbalizers, with independent language graphs and no NeMo dependency.
  • Register both directions in the CLI, packaging, cache-manifest tests and README.
  • TN counts with feminine forms and joins ו by a consistent rule; masculine readings are used for thousands/millions counts and for the noun איש (15 איש → חמישה עשר איש). Prefix letters written with a hyphen (ב-2.5 מיליון) are glued to the reading.

Coverage and limitations

  • All 337 original ITN fixture lines (7 files) are retained verbatim.
  • New TN is checked against 296 reversed ITN cases. 25 are excluded, each with a reason: they contradict other fixtures (inconsistent ו, gender, decimal or date readings), glue words without a space, or end with a space.
  • Hand-written raw-field, prefix, measure, time, date and n-best/mapping tests cover the new direction.

Test plan

  • Original, reversed, raw-field and mapping tests
  • CLI and cache registration regressions with NeMo imports blocked
  • Full project suite on the merged master baseline (20576 passed, NeMo imports blocked)
  • flake8 tn itn and git diff --check

@pengzhendong
pengzhendong merged commit 9d2bae6 into master Oct 31, 2026
1 check passed
@pengzhendong
pengzhendong deleted the feat/hebrew-normalization branch October 31, 2026 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant