Skip to content

feat: add Marathi TN/ITN with raw-field tagging - #395

Merged
pengzhendong merged 3 commits into
masterfrom
feat/marathi-normalization
Oct 31, 2026
Merged

pengzhendong merged 3 commits into
masterfrom
feat/marathi-normalization

Conversation

@pengzhendong

Copy link
Copy Markdown
Member

Summary

  • Add native Marathi (mr) ITN and a new TN, completing the direction missing from NeMo. Both use raw-field taggers and semantic verbalizers, with independent language graphs and no NeMo dependency.
  • Register both directions in the CLI, packaging, cache-manifest tests and README.
  • Support Devanagari, ASCII and mixed digits, lakh/crore quantities, dates, decimals and Marathi time idioms. Preserve explicit punctuation spacing through the shared joiner.

Coverage and limitations

  • All 131 original ITN fixture lines are retained verbatim.
  • New TN is checked against reversed ITN fixtures and hand-written cases. Seven reverse cases remain excluded with explicit reasons: conflicting hundreds-count/colloquial time readings, singular minute wording and a trailing-space variant.
  • Leading-zero or oversized digit strings are read digit by digit. No constructor options are added.

Test plan

  • Original, reversed, raw-field and mapping tests
  • CLI and cache registration regressions with NeMo imports blocked
  • Full project suite on the merged master baseline (19824 passed, NeMo imports blocked)
  • flake8 tn itn and git diff --check

@pengzhendong pengzhendong changed the title feat: add Marathi TN and register bidirectional pipelines feat: add Marathi TN/ITN with raw-field tagging Oct 31, 2026
@pengzhendong
pengzhendong merged commit 28e20ec into master Oct 31, 2026
1 check passed
@pengzhendong
pengzhendong deleted the feat/marathi-normalization branch October 31, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant