Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TBVaccine produces insane reporting when Dask or Dask Distributed task crashes #11

Open
frol opened this issue Jun 1, 2017 · 0 comments

Comments

@frol
Copy link

frol commented Jun 1, 2017

Here is the reproduction:

import tbvaccine
tbvaccine.add_hook(isolate=False)

import dask

def x(arg1=123):
    assert arg1 == 122, "qwe"

if __name__ == '__main__':
    dask.delayed(x)().compute()

The error:

  File "/tmp/env/lib/python3.5/site-packages/dask/async.py", line 272, in _execute_task
    return func(*args2)
  File "qq5.py", line 7, in x
    assert arg1 == 122, "qwe"
|     worker_id                = None
AssertionError: qwe

I don't understand at all where worker_id came from.

Running Dask Distributed, I see the output, which reports things which don't exist in that scope at all...

import tbvaccine
tbvaccine.add_hook(isolate=False)

import dask
import dask.distributed


def x(arg1=123):
    assert arg1 == 122, "qwe"

if __name__ == '__main__':
    client = dask.distributed.Client()
    dask.delayed(x)().compute(get=client.get)
  File "/tmp/env/lib/python3.5/site-packages/dask/base.py", line 203, in compute
    results = get(dsk, keys, **kwargs)
|     x = b'\x80\x04\x95\xa1\x00\x00\x00\x00\x00\x00\x00\x8c\x16tblib.pickling_support\x94\x8c\x12unpickle_traceback\x94\x93\x94\x8c\x05tblib\x94\x8c\x05Frame\x94\x93\x94)\x81\x94}\x94(\x8c\x06f_code\x94h\x03\x8c\x04Code\x94\x93\x94)\x81\x94}\x94(\x8c\x07co_name\x94\x8c\x01x\x94\x8c\x0bco_filename\x94\x8c\x06qq4.py\x94ub\x8c\tf_globals\x94}\x94ubK\tN\x87\x94R\x94.'
  File "/tmp/env/lib/python3.5/site-packages/distributed/client.py", line 1590, in get
    resources=resources)
|     ret       = <tblib.Traceback object at 0x7f95aeadfcf8>
|     tb_frame  = <tblib.Frame object at 0x7f95aeadf908>
|     tb_lineno = 9
|     tb_next   = None
  File "/tmp/env/lib/python3.5/site-packages/distributed/utils.py", line 223, in sync
    six.reraise(*error[0])
|     code   = <code object x at 0x7f95aeb33270, file "qq4.py", line 9>
|     f_code = <tblib.Code object at 0x7f95aeadf9b0>
|     self   = <tblib.Traceback object at 0x7f95aeadfcf8>
|     tb     = <traceback object at 0x7f95aeaeb148>
  File "/tmp/env/lib/python3.5/site-packages/six.py", line 686, in reraise
    raise value
AssertionError: qwe

The direct call to x() produces a sane output, so it should be some magic involved with Dask. Better-Exceptions module just hangs in this situation, so TBVaccine is better in this respect, but it is still completely misleading and unhelpful.

@frol frol mentioned this issue Jun 1, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant