-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor properties.py - 4d peak labelling #265
Comments
To make some progress try to break this up into a bunch of smaller tasks:
Properties.py is more challenging:
|
Note : multiprocessing + shared memory seems to be buggy. The remove from tracker monkeypatch does not work. Abandon it.
|
There is still the evil monkeypatch in properties.py |
Within #389 the monkeypatch is suppressed by: This is not ideal, but it appears to run with or without the patch in cases where the code works. This could be about cleanup during OOM problems. Cleaning all this up would be a natural case for also developing #252 |
In sinograms/properties.py and sinograms/point_by_point.py the code works via shared memory.
This does not scale beyond one node and it fails on python2.7.
For global read-only memory we could use mmap with numpy on a non-compressed hdf5 file (https://gist.github.com/maartenbreddels/09e1da79577151e5f7fec660c209f06e):
This may be useful for reducing some out-of-memory problems.
Another upgrade path could be looking into dask.dataframe for distributed processing.
The text was updated successfully, but these errors were encountered: