Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not run tasks with jobs_limit==1 in the multiprocessing.Pool when multiprocessing is enabled #81

Open
Jerry-Ma opened this issue Aug 31, 2017 · 0 comments

Comments

@Jerry-Ma
Copy link

Jerry-Ma commented Aug 31, 2017

This will give the users the choice of implementing their own multiprocessing procedure within some certain task functions (decorated with @jobs_limit(1)), while allowing the pipeline to be executed in the multiprocessing mode, i.e. allowing other tasks to be arranged to run in parallel by ruffus.

Currently, if one implements his/her own mp code with in a task function, and the pipeline is run with
multiprocessing enabled, an exception about not being able to create child process within daemon process is thrown. Also, a number of frameworks/libraries, notably, matplotlib, are not able to run
in a child process through multiprocessing.Pool. Another case of this kind, according to my test, is lmfit.

The work around is to allow the jobs to be scheduled either on to the Pool, or run directly, depending on the jobs_limit param of the task. I did a quick hack of the code to enable this. If anyone is interested, I can submit a pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant