Skip to content

feat: add Hindi-English code-mixed ITN with raw-field tagging - #396

Merged
pengzhendong merged 5 commits into
masterfrom
feat/hindi-english-normalization
Oct 31, 2026
Merged

pengzhendong merged 5 commits into
masterfrom
feat/hindi-english-normalization

Conversation

@pengzhendong

Copy link
Copy Markdown
Member

Summary

  • Add native Hindi–English code-mixed ITN (hi_en), reusing the existing standalone Hindi and English ITN rules. No TN pipeline is introduced.
  • Keep source text in raw fields; local English extensions support Indian numbering, date punctuation and plural temperature units without modifying the standalone English grammar.
  • Register the ITN-only CLI entry, package data, configuration-aware cache tests and README. --cased defaults to false, and TN rejects hi_en.

Test plan

  • All 486 original fixture lines (14 files) retained verbatim, including 70 cased date lines
  • Hand-written mixed-language, raw-field, escaping and n-best/mapping cases
  • CLI and cache registration regressions with NeMo imports blocked
  • Full project suite on the merged master baseline (20644 passed, NeMo imports blocked)
  • flake8 tn itn and git diff --check

@pengzhendong pengzhendong changed the title feat: add Hindi-English code-mixed ITN feat: add Hindi-English code-mixed ITN with raw-field tagging Oct 31, 2026
@pengzhendong
pengzhendong merged commit fcb3a50 into master Oct 31, 2026
1 check passed
@pengzhendong
pengzhendong deleted the feat/hindi-english-normalization branch October 31, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant