-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make simple-parsing
faster
#278
Comments
Thanks a lot @anivegesana , You're more than welcome to submit a PR for the caching, I think the current tests are sufficient in convering the docstring creation, I dont think additional tests are needed. |
Fixes #278 Signed-off-by: Fabrice Normandin <[email protected]>
Fixes #278 Signed-off-by: Fabrice Normandin <[email protected]>
Fixes #278 Signed-off-by: Fabrice Normandin <[email protected]>
* Add pytest-benchmark test dependency Signed-off-by: Fabrice Normandin <[email protected]> * Add performance files for before the changes Signed-off-by: Fabrice Normandin <[email protected]> * Use cached versions of inspect.getdoc/getsource Fixes #278 Signed-off-by: Fabrice Normandin <[email protected]> * Make numpy import lazy Signed-off-by: Fabrice Normandin <[email protected]> * Simplify the benchmark code a bit Signed-off-by: Fabrice Normandin <[email protected]> * Remove .benchmarks file from git history Signed-off-by: Fabrice Normandin <[email protected]> * Playing around with GitHub actions for benchmark Signed-off-by: Fabrice Normandin <[email protected]> * Remove upload workflow, add benchmark Signed-off-by: Fabrice Normandin <[email protected]> * Tweak benchmark.yml and build.yml Signed-off-by: Fabrice Normandin <[email protected]> * Only run workflow on push to master Signed-off-by: Fabrice Normandin <[email protected]> --------- Signed-off-by: Fabrice Normandin <[email protected]>
Hey @anivegesana , just FYI, I did make the numpy import lazy, and added the lru_cache to all the inspect functions in #279 |
Thank you! I noticed that my Thank you for your awesome work! |
Oops. Thanks for pointing it out |
Is your feature request related to a problem? Please describe.
Thank you for your work on #276! In the parser that our team wrote, we noticed that most of the time needed to generate the
--help
menu was spent inside of theinspect.getsource
function. I think the issue is that the issue is that caching is done on_get_attribute_docstring
, but the majority of the time is spent ininspect.getsource
, which has to be done multiple times for the same dataclass since the key of the cache is(dataclass, field_name)
.Describe the solution you'd like
Constructing caches for the
inspect.getsource
,inspect.getdoc
, anddp.parse
functions dramatically speeds up the construction of the parser. I ran a simple experiment where I didn't change any code of thesimple-parsing
library and simply prepended the following lines of code to my script:Also, you import numpy even if you don't use it. It is possible to import numpy lazily so that
--help
menus will be more snappy. (I haven't checked to see if this is a major source of unnecessary waiting, but, on my machine, it seems to take up ~0.105s/0.211s to construct the parser and parse the arguments.This may not may not also apply to yaml.)To lazily load numpy, you can create a module called
lazy_numpy.py
and use the following code:Importing
lazy_numpy
does not importnumpy
but accessing any attribute withinlazy_numpy
does.Describe alternatives you've considered
I haven't tried other locations that caching could be improved. I just think that, if three lines of code can speed things up this dramatically, might as well add them.
Additional context
I can open a PR, but I do not have a good idea on how to implement test cases for this.
The text was updated successfully, but these errors were encountered: