Map disk cache #132
Map disk cache #132
Conversation
…peed things up between calls of the same code. There needs to be some more work on this to make sure the the cache is invalidated when it needs to be.
…dated the date in the boilerplate.
…er-kernel storage format. The per-kernel storage foramt consists of a '.inf' which hold information about the file and a '.ptx' file that stores the actual kernel.
…sed but I feel like it should be in there.
… the Python to make it easy to clear the map cache.
…k cache before running.
Codecov Report
@@ Coverage Diff @@
## master #132 +/- ##
==========================================
- Coverage 58.78% 58.77% -0.01%
==========================================
Files 63 63
Lines 5095 5097 +2
==========================================
+ Hits 2995 2996 +1
- Misses 2100 2101 +1
Continue to review full report at Codecov.
|
|
Oh, the "coverage" thing again. |
…ileutils.hpp a la PR ledatelescope#132.
|
+1 for this functionality -- as this is C++ level I would defer to others for review though |
Codecov Report
@@ Coverage Diff @@
## master #132 +/- ##
=======================================
Coverage 61.62% 61.62%
=======================================
Files 64 64
Lines 5300 5300
=======================================
Hits 3266 3266
Misses 2034 2034 Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #132 +/- ##
==========================================
- Coverage 61.62% 58.75% -2.88%
==========================================
Files 64 64
Lines 5300 5302 +2
==========================================
- Hits 3266 3115 -151
- Misses 2034 2187 +153
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #132 +/- ##
==========================================
- Coverage 58.50% 58.36% -0.14%
==========================================
Files 67 67
Lines 5731 5753 +22
==========================================
+ Hits 3353 3358 +5
- Misses 2378 2395 +17
Continue to review full report at Codecov.
|
|
I should be able to try this PR again today, on cuda 11. I'll also start to look at one of the others, whatever is most relevant or ready: is that mmap ring space #135 ? |
|
Tests worked great for me on cuda11, tried with and without map cache enabled. It makes a huge difference in speed. Just out of curiosity, I messed with overwriting one of the files with random bytes and sure, it core dumped. Not surprising, and not worth detecting… just trying to push the limits, find the "robustness boundary." Then I tried editing one of the numbers in If I delete the whole directory and start over, it works again of course. Then if I change I started to experiment with adding a |
|
I remember seeing an issue about machines with >1 GPU, and I'm wondering if the map cache makes that any worse… is it embedding GPU-0 architecture info. Or what if $HOME is shared across machines with different GPUs. Maybe not worth worrying about, since if something goes wrong it's a simple matter to wipe the directory and start fresh. Just thinking out loud about potential complications. |
The hang is surprising. The
I had thought about this too. The options are:
|
My g++ 10.3 was throwing this warning, and I'm reading it could mean
that subclass exceptions would not match. There are other places in
the code that use `const&` in a `catch` so I assume fix is compatible
with other compilers we encounter.
```
map.cpp:425:21: warning: catching polymorphic
type ‘class std::exception’ by value [
-Wcatch-value=]
425 | } catch( std::exception ) {}
| ^~~~~~~~~
```
It shouldn't make things worse on multi-GPU systems but it might be problematic if those GPUs are heterogeneous. The cache is stored as PTX which should be forward compatible. The hitch is that when The alternative would be to do something like always generate the PTX for the lowest compute capability that Bifrost was compiled for if the disk cache is enabled. That would solve my above problems, maybe with a little slowdown, but be largely unchanged in the homogeneous GPU case. |
I don't think any of this necessarily has to hold up this branch. Could leave a TODO here or there as a reminder for things to pursue in the future or revisit if there's a relevant bug report. Or write it as a low-priority github issue with appropriate label (wish list, improvement, for future consideration, etc.)? |
…y was configured for.
… into map-disk-cache
|
Your point on splitting up the cache by arch. does make sense but I'm not sure how necessary that is if we stick with PTX for the cache. As far as I can tell PTX gets interpreted by the CUDA driver and JIT compiled to a binary when it is run. So as long as we aren't using features in the PTX that the card cannot support it should be ok to have a flat cache. That should be true if we always target the lowest arch. requested during the install. Getting fancier is an option but I think how we do that is going to need some more thought about the details of compatibility and what CUDA really does with PTX files/binaries. Maybe a "TODO" is the way to go. |
I couldn't get access to it by importing `bifrost.map`, because of the collision between the module `map` and the function `map` within `bifrost.map`.
|
Diagnosed the hang that I experienced before. After it core-dumped because I corrupted a ptx file, it left behind the |
|
I ran through my tests again this morning and I think I'm happy with this. @league if you are happy with it too then I will merge it. |
|
LGTM, ship it. 🚢 |
Adds a disk-based caching system for
bifrost.map()calls to help reduce start up time in pipelines that are run in the same configuration multiple times. This feature can also be turned off by setting the "NOMAPCACHE" variable to one in theuser.mkfile.