bsub.poll prints to screen

[duttk]

Hello,

Firstly thanks for this module. I can avoid rolling my own LSF wrapper.
I tested bsub 0.3.5 and when I use the bsub.poll, I see the line below when the polling is finished:
No unfinished job found

Could we perhaps get rid of this ?
Thanks!

[brentp]

yes. care to make a PR?

[duttk]

Sure thing.
Seems simple enough, I just need to redirect stderr to stdout in the 'bsub -w' call.
I'll make one over the weekend.

[nextgenusfs]

I'm seeing the 'No unfinished job found' and then it raises an exception, am I doing something wrong? Here is the wrapper function I'm using.

def submitLSF(cmd, cpus=1, verbose=False):
	#wrapper for bsub to run like subprocess module, cmd is a list
	jobName = bsub('job-'+str(os.getpid()), n=str(cpus), verbose=verbose)
	job = jobName(' '.join(cmd))
	bsub.poll(job.job_id)

Here is what error looks like:

No unfinished job found
Traceback (most recent call last):
  File "/home/xyxxx/.local/lib/python2.7/site-packages/bsub/bsub.py", line 119, in poll
    while job_ids.intersection(self.running_jobs(names=names)):
  File "/home/xyxxx/.local/lib/python2.7/site-packages/bsub/bsub.py", line 105, in running_jobs
    for x in sp.check_output(["bjobs", "-w"])\
  File "/bioinfo/apps/all_apps/anaconda2/envs/gtools/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['bjobs', '-w']' returned non-zero exit status 255

[brentp]

I no longer even have access to an LSF system so I can't be much help. perhaps your job is finishing before the polling starts? The poll function might need to check for an empty queue.

[nextgenusfs]

Okay thanks, I'll try a few things. The job I tried it on runs for awhile actually, say 20 minutes and it completes successfully, so just seems like an error with the polling.

[brentp]

what does bjobs -w return for you when the job is running?

[nextgenusfs]

So when no job is running:

$ bjobs -w
No unfinished job found

When something is running, then gives status like:

$ bjobs -w
JOBID    USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
2633934  xxyxx  RUN   premium     xxx103      32*xxxx07   pb-asm-6123 Dec 13 22:55

I'm not trying to do anything fancy -- just want to launch the job and wait for it to finish before running the next step in pipeline. The wrapper above was just to keep my same subprocess syntax that I was using...

[nextgenusfs]

Small update, so I manually did the "polling" with bjobs -w and seems like I have to ignore the exit code 255 -- not sure why 255 is being returned after the job is finished, but if I ignore that exit code then seems to run.

This code works to poll on my system:

def bsub_poll(jobid):
    DEVNULL = open(os.devnull, 'w')
    jobfinished = False
    while not jobfinished:
        p1 = subprocess.Popen(['bjobs', '-w'], stdout=subprocess.PIPE, stderr=DEVNULL).communicate()[0].split()
        if not jobid in p1: 
            jobfinished = True
        time.sleep(1)