You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This will give the users the choice of implementing their own multiprocessing procedure within some certain task functions (decorated with @jobs_limit(1)), while allowing the pipeline to be executed in the multiprocessing mode, i.e. allowing other tasks to be arranged to run in parallel by ruffus.
Currently, if one implements his/her own mp code with in a task function, and the pipeline is run with
multiprocessing enabled, an exception about not being able to create child process within daemon process is thrown. Also, a number of frameworks/libraries, notably, matplotlib, are not able to run
in a child process through multiprocessing.Pool. Another case of this kind, according to my test, is lmfit.
The work around is to allow the jobs to be scheduled either on to the Pool, or run directly, depending on the jobs_limit param of the task. I did a quick hack of the code to enable this. If anyone is interested, I can submit a pull request.
The text was updated successfully, but these errors were encountered:
This will give the users the choice of implementing their own multiprocessing procedure within some certain task functions (decorated with
@jobs_limit(1)
), while allowing the pipeline to be executed in the multiprocessing mode, i.e. allowing other tasks to be arranged to run in parallel by ruffus.Currently, if one implements his/her own mp code with in a task function, and the pipeline is run with
multiprocessing enabled, an exception about not being able to create child process within daemon process is thrown. Also, a number of frameworks/libraries, notably,
matplotlib
, are not able to runin a child process through
multiprocessing.Pool
. Another case of this kind, according to my test, islmfit
.The work around is to allow the jobs to be scheduled either on to the Pool, or run directly, depending on the
jobs_limit
param of the task. I did a quick hack of the code to enable this. If anyone is interested, I can submit a pull request.The text was updated successfully, but these errors were encountered: