Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds support for minor denomination currencies #250

Closed
wants to merge 288 commits into from
Closed

Adds support for minor denomination currencies #250

wants to merge 288 commits into from

Conversation

zoobereq
Copy link
Collaborator

@zoobereq zoobereq commented Dec 5, 2024 •
edited
Loading

What does this PR do ?

This PR adds support for minor denomination currencies (e.g. cents for USD). The major - minor currency denomination pairs are stored in a dictionary under data, and may be expanded according to the need and use case. The list of currency symbol - currency name mappings stored in the same directory can likewise be expanded.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

github-advanced-security[bot]
github-advanced-security bot found potential problems Dec 5, 2024
View reviewed changes
Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

ngachchi and others added 23 commits December 5, 2024 15:48
@ngachchi
added unit test cases and minor fixes
6ab1eca 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
added missing units to improve accuracy for measure class
39dde2e 
Signed-off-by: Namrata Gachchi <[email protected]>
@zoobereq
Updates the cache
730cd04 
Signed-off-by: Simon Zuberek <[email protected]>
@ngachchi
Merge branch 'hi_tn' of https://github.com/ngachchi/NeMo-text-processing
b0d3e63 
 into hi_tn
@ngachchi
fixed the sparrowhawk to trim extra space
a8f1a57 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
removed unused english whitelist files
2b9f657 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
45e5d3b 
for more information, see https://pre-commit.ci
@ngachchi
reverted to previous logic
b5d887d
@BuyuanCui @pre-commit-ci @ngachchi
Jp tn 20241017 (#240)
a8b8b48 
* ja tn

Signed-off-by: Alex Cui <[email protected]>

* adding ja

Signed-off-by: Alex Cui <[email protected]>

* removing

Signed-off-by: Alex Cui <[email protected]>

* updated tests

Signed-off-by: Alex Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* addressing comment

Signed-off-by: Alex Cui <[email protected]>

* addressing ci

Signed-off-by: Alex Cui <[email protected]>

* addressing ci

Signed-off-by: Alex Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* addresing comment

Signed-off-by: Alex Cui <[email protected]>

* removing

Signed-off-by: Alex Cui <[email protected]>

* adresing comment

Signed-off-by: Alex Cui <[email protected]>

* removing unused import

Signed-off-by: Alex Cui <[email protected]>

* addressing comment

Signed-off-by: Alex Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* addressing comment;

Signed-off-by: Alex Cui <[email protected]>

* addressing comment

Signed-off-by: Alex Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* date for ja

Signed-off-by: Alex Cui <[email protected]>

* addresing comment

Signed-off-by: Alex Cui <[email protected]>

* addressing comment

Signed-off-by: Alex Cui <[email protected]>

* jenkins

Signed-off-by: Alex Cui <[email protected]>

* addresing comment

Signed-off-by: Alex Cui <[email protected]>

* addressing comment

Signed-off-by: Alex Cui <[email protected]>

* typo

Signed-off-by: Alex Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* adressing comment

Signed-off-by: Alex Cui <[email protected]>

* addressing comment

Signed-off-by: Alex Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ci

Signed-off-by: Alex Cui <[email protected]>

---------

Signed-off-by: Alex Cui <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <[email protected]>
@zoobereq @pre-commit-ci @ngachchi
Fixes issue 228 (#234)
3f05501 
* Fixes issue 228

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixes issue 228

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Expands Whitelist and removes redundant [0-9]+-[A-Za-z] (and in reverse) pattern matching from ELECTRONIC

Signed-off-by: Simon Zuberek <[email protected]>

* Updates the cache

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removes unused imports and variables

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removes redundant abbreviation mappings

Signed-off-by: Simon Zuberek <[email protected]>

* Updates the cache

Signed-off-by: Simon Zuberek <[email protected]>

---------

Signed-off-by: Simon Zuberek <[email protected]>
Co-authored-by: Simon Zuberek <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Hindi TN changes
8dc2d32 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci @ngachchi
[pre-commit.ci] auto fixes from pre-commit.com hooks
72611ad 
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Updated date for Hindi TN cache
a48787c 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
additional whitelist class .tsv files and unused imports removed
a2a5acf 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci @ngachchi
[pre-commit.ci] auto fixes from pre-commit.com hooks
b2d4dea 
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
incorporated suggestions for unused statements and another for closin…
add48be 
…g the file opened

Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
98d657f 
for more information, see https://pre-commit.ci
@ngachchi @pre-commit-ci
Hindi ITN Support for Cardinal, Decimal, Ordinal, Fraction, Date, Tim…
2249724 
…e, Money and Measure (#223)

* Hindi ITN Support for Cardinal, Decimal, Ordinal, Fraction, Date, Time

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Committing all changes made

Signed-off-by: Tarushi V <[email protected]>

* Updated date.py and added more test cases to cardinal for improved accuracy

Signed-off-by: Tarushi V <[email protected]>

* Updated date.py

Signed-off-by: Tarushi V <[email protected]>

* Added hi to Jenkins and cleanup

Signed-off-by: Tarushi V <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Changes and cleanup based on feedback

Signed-off-by: Tarushi V <[email protected]>

* Changes and cleanup based on feedback

Signed-off-by: Tarushi V <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Resolved conflicts

Signed-off-by: Tarushi V <[email protected]>

* Committing code for measure.py

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* changes to run_evaluate.py

Signed-off-by: Tarushi V <[email protected]>

* Hindi ITN for money.py

Signed-off-by: Tarushi V <[email protected]>

* Changes and cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup date verbalizer

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* pushing .gitignore file from main branch

Signed-off-by: Tarushi V <[email protected]>

---------

Signed-off-by: Tarushi V <[email protected]>
Signed-off-by: tarushi2k2 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Combined Hindi TN and ITN seperate blocks into single
62a6e19 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
[pre-commit.ci] auto fixes from pre-commit.com hooks
5829bbf 
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Added init.py files and removed unused commented lines
8fc3a0c 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
commented irrevelant references and unused snippets from whitelist an…
fa41100 
…d word file

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Whitelist and Word class changes
f41357e 
Signed-off-by: Namrata Gachchi <[email protected]>
zoobereq and others added 11 commits December 17, 2024 12:11
@zoobereq @ngachchi
Updates the cache
7f10a30 
Signed-off-by: Simon Zuberek <[email protected]>
Signed-off-by: Namrata Gachchi <[email protected]>
@zoobereq @ngachchi
Updates the cache again
6e8e045 
Signed-off-by: Simon Zuberek <[email protected]>
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
dedh and dhai implementation approach
166773a 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci @ngachchi
[pre-commit.ci] auto fixes from pre-commit.com hooks
1cddd06 
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>
@zoobereq @anand-nv @ngachchi
Fix space issue with ZH ITN (#244)
ab94f8f 
* Fix space issue with ZH ITN

Signed-off-by: Anand Joseph <[email protected]>

* Update Jenkinsfile

Update FST paths

Signed-off-by: anand-nv <[email protected]>

---------

Signed-off-by: Anand Joseph <[email protected]>
Signed-off-by: anand-nv <[email protected]>
Signed-off-by: Simon Zuberek <[email protected]>
Co-authored-by: Anand Joseph <[email protected]>
Co-authored-by: anand-nv <[email protected]>
Signed-off-by: Namrata Gachchi <[email protected]>
@tbartley94 @ngachchi
contributing update (#251)
ff881b6 
* contributing update

Signed-off-by: tbartley94 <[email protected]>

* adding edits

Signed-off-by: tbartley94 <[email protected]>

* spelling

Signed-off-by: tbartley94 <[email protected]>

---------

Signed-off-by: tbartley94 <[email protected]>
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
Merge branch 'hi_tn' of https://github.com/ngachchi/NeMo-text-processing
738214e 
 into hi_tn

Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
reverted code and added zero to the hour tsv file
1465ec1 
Signed-off-by: Namrata Gachchi <[email protected]>
@ngachchi
reverted to previous logic
5851784 
Signed-off-by: Namrata Gachchi <[email protected]>
@tarushi2k2 @pre-commit-ci @ngachchi
Hindi ITN - Addition of Whitelist and Word (#248)
4025556 
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Updation

Signed-off-by: Tarushi V <[email protected]>

* Updation

Signed-off-by: Tarushi V <[email protected]>

---------

Signed-off-by: Tarushi V <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <[email protected]>
@tarushi2k2 @pre-commit-ci @ngachchi
Hindi ITN - Addition of Whitelist and Word (#248)
483f1f4 
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Updation

Signed-off-by: Tarushi V <[email protected]>

* Updation

Signed-off-by: Tarushi V <[email protected]>

---------

Signed-off-by: Tarushi V <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <[email protected]>
tarushi2k2 and others added 8 commits December 19, 2024 09:39
@tarushi2k2 @pre-commit-ci @ngachchi
Hindi ITN - Addition of Whitelist and Word (#248)
ef54e78 
* Addition of whitelist and word classes

Signed-off-by: Tarushi V <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updation of Jenkins date

Signed-off-by: Tarushi V <[email protected]>

* Cleanup

Signed-off-by: Tarushi V <[email protected]>

* Updation

Signed-off-by: Tarushi V <[email protected]>

* Updation

Signed-off-by: Tarushi V <[email protected]>

---------

Signed-off-by: Tarushi V <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci @ngachchi
[pre-commit.ci] auto fixes from pre-commit.com hooks
c3dfcff 
for more information, see https://pre-commit.ci

Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
c54a7ad 
for more information, see https://pre-commit.ci
@ngachchi
Merge branch 'hi_tn' of https://github.com/ngachchi/NeMo-text-processing
0ca2774 
 into hi_tn
@ngachchi
Date further implementation (BC, B.C.) added
999ef8c 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
844ac7e 
for more information, see https://pre-commit.ci
@ngachchi
added date range implementation
d4fd55b 
Signed-off-by: Namrata Gachchi <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
d92f502 
for more information, see https://pre-commit.ci
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 7, 2025

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jan 7, 2025
@github-actions github-actions bot removed the Stale label Jan 10, 2025
@zoobereq zoobereq closed this Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.