Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

extracted iev-data functionality and add it to iev #31

Merged
merged 2 commits into from
Mar 8, 2024
Merged

extracted iev-data functionality and add it to iev #31

merged 2 commits into from
Mar 8, 2024

Conversation

HassanAkbar
Copy link
Member

Move all three commands (xlsx2yaml, xlsx2db, db2yaml) from iev-data to iev and use glossarist V2 for reading and writing concept to YAML files.

closes #30

github-advanced-security[bot]
github-advanced-security bot found potential problems Mar 7, 2024
View reviewed changes
lib/iev/utilities.rb
Comment on lines +17 to +47
text.gsub(
/<a href="?(IEV)\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, \1:\2}}',
).gsub(
/<a href="?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, IEV:\2}}',
).gsub(
# To handle <a> tags without ending tag like
# `Voir <a href=IEV103-05-21>IEV 103-05-21`
# for concept '702-03-11' in `fr`
/<a href="?(IEV)?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)$/,
'{{\3, IEV:\2}}',
).gsub(
/<a href="?(.*?)"?>(.*?)<\/a>/,
'\1[\2]',
).gsub(
Regexp.new([SIMG_PATH_REGEX, "\\s*", FIGURE_TWO_REGEX].join),
"#{IMAGE_PATH_PREFIX}/#{term_domain}/\\1[Figure \\2 - \\3; \\6]",
).gsub(
Regexp.new([SIMG_PATH_REGEX, "\\s*", FIGURE_ONE_REGEX].join),
"#{IMAGE_PATH_PREFIX}/#{term_domain}/\\1[Figure \\2 - \\3]",
).gsub(
/<img\s+(.+?)\s*>/,
"#{IMAGE_PATH_PREFIX}/#{term_domain}/\\1[]",
).gsub(
/<br>/,
"\n",
).gsub(
/<b>(.*?)<\/b>/,
"*\\1*",
)

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '' and with many repetitions of 'a'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '' and with many repetitions of 'a'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '' and with many repetitions of 'a'.
lib/iev/utilities.rb
Comment on lines +17 to +41
text.gsub(
/<a href="?(IEV)\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, \1:\2}}',
).gsub(
/<a href="?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, IEV:\2}}',
).gsub(
# To handle <a> tags without ending tag like
# `Voir <a href=IEV103-05-21>IEV 103-05-21`
# for concept '702-03-11' in `fr`
/<a href="?(IEV)?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)$/,
'{{\3, IEV:\2}}',
).gsub(
/<a href="?(.*?)"?>(.*?)<\/a>/,
'\1[\2]',
).gsub(
Regexp.new([SIMG_PATH_REGEX, "\\s*", FIGURE_TWO_REGEX].join),
"#{IMAGE_PATH_PREFIX}/#{term_domain}/\\1[Figure \\2 - \\3; \\6]",
).gsub(
Regexp.new([SIMG_PATH_REGEX, "\\s*", FIGURE_ONE_REGEX].join),
"#{IMAGE_PATH_PREFIX}/#{term_domain}/\\1[Figure \\2 - \\3]",
).gsub(
/<img\s+(.+?)\s*>/,
"#{IMAGE_PATH_PREFIX}/#{term_domain}/\\1[]",
).gsub(

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<img\\t' and with many repetitions of '\\t\\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<img\\t;' and with many repetitions of '\\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<img\\t' and with many repetitions of '\\t\\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<img\\t;' and with many repetitions of '\\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<img\\t' and with many repetitions of '\\t\\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<img\\t;' and with many repetitions of '\\t'.
lib/iev/utilities.rb
Comment on lines +17 to +32
text.gsub(
/<a href="?(IEV)\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, \1:\2}}',
).gsub(
/<a href="?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, IEV:\2}}',
).gsub(
# To handle <a> tags without ending tag like
# `Voir <a href=IEV103-05-21>IEV 103-05-21`
# for concept '702-03-11' in `fr`
/<a href="?(IEV)?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)$/,
'{{\3, IEV:\2}}',
).gsub(
/<a href="?(.*?)"?>(.*?)<\/a>/,
'\1[\2]',
).gsub(

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '<a href=>' and with many repetitions of '<a href=>a'.
lib/iev/utilities.rb
Comment on lines +17 to +29
text.gsub(
/<a href="?(IEV)\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, \1:\2}}',
).gsub(
/<a href="?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, IEV:\2}}',
).gsub(
# To handle <a> tags without ending tag like
# `Voir <a href=IEV103-05-21>IEV 103-05-21`
# for concept '702-03-11' in `fr`
/<a href="?(IEV)?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)$/,
'{{\3, IEV:\2}}',
).gsub(

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '' and with many repetitions of 'a'.
lib/iev/utilities.rb
Comment on lines +17 to +23
text.gsub(
/<a href="?(IEV)\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, \1:\2}}',
).gsub(
/<a href="?\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, IEV:\2}}',
).gsub(

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '' and with many repetitions of 'a'.
lib/iev/utilities.rb
Comment on lines +17 to +20
text.gsub(
/<a href="?(IEV)\s*(\d\d\d-\d\d-\d\d\d?)"?>(.*?)<\/?a>/,
'{{\3, \1:\2}}',
).gsub(

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings starting with '' and with many repetitions of 'a'.
lib/iev/utilities.rb
Comment on lines +52 to +54
input.gsub('\n', "\n\n")
.gsub(/<[pbr]+>/, "\n\n")
.gsub(/\s*\n[\n\s]+/, "\n\n")

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings with many repetitions of '\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings with many repetitions of '\t'.
This
regular expression
Loading
that depends on a
library input
Loading
may run slow on strings with many repetitions of '\t'.
@HassanAkbar
Copy link
Member Author

The comments are related to regex that I have copied over from the iev-data gem, and it is not taking that much time.

I have tested it on the actual data file from iev-data and it took around 7 minutes with relaton fetching records and around 4 minutes with relaton local cache, which I think is acceptable time considering the file has more then 2.6 Lacs records.

The test cases that are failing are on windows and are failing because of sqlite3 unable to install on windows.

@HassanAkbar HassanAkbar requested a review from ronaldtse March 8, 2024 10:13
@ronaldtse ronaldtse merged commit 90260ba into main Mar 8, 2024
11 of 16 checks passed
@ronaldtse ronaldtse deleted the extract_iev_functionality_in_iev branch March 8, 2024 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ronaldtse ronaldtse ronaldtse approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Combine the iev gem with the iev-data gem, use Glossarist Ruby for providing IEV content
2 participants
@HassanAkbar @ronaldtse