python3 how to find out which line in apply_async target function caused error when error_callback is applied

Question

Using python 3.4

I have a target function that fails and apply_async calls my error callback function instead of my callback function.

The problem is that the exception captured and sent to the error_callback is not helpful and I can't figure out where the error in the target function occurs.

My error callback

  def error_callback(self,e):
    print('error_callback()',e)
    # tested one of these at a time
    traceback.print_exc()
    traceback.print_last()
    traceback.print_stack()

And the traceback outputs

print_exc

  File "/usr/lib/python3.4/traceback.py", line 125, in _iter_chain
    context = exc.__context__
  AttributeError: 'NoneType' object has no attribute '__context__'

print_last

  File "/usr/lib/python3.4/traceback.py", line 262, in print_last
    raise ValueError("no last exception")
  ValueError: no last exception

print_stack

  File "./relative/path/to/my/script.py", line 71, in traceback
    traceback.print_stack()

I'd have to confess that I ain't really comfortable with the traceback library and how to use it but it seems to me that it's not possible to use traceback in this case?

What should I do?

Answer 1

The traceback functions you're calling are all expecting to get the exception info from sys.exec_info, which is where exception information is usually stored while an exception is being caught. However, since the actual exception in your case came from a child process, they're not finding any exception information when you call them in the parent process.

You need to use extract the relevant information from the Exception object e that you received from the child process. Try using traceback.print_exception with appropriate arguments:

traceback.print_exception(type(e), e, e.__traceback__)

Answer 2

I can't figure out where the error in the target function occurs.

Here is an example of what you can do:

import multiprocessing as mp

def do_stuff(num):
    if False:
        x = 10
    else: 
        x = 20

    result = 5/(2 - num)  #Line 9
    return result 

def success_handler(result):
    print('success')

def error_handler(e):
    print('error')
    print(dir(e), "\n")
    print("-->{}<--".format(e.__cause__))

with mp.Pool(2) as my_pool:
    results =  [
        my_pool.apply_async(
            do_stuff, 
            args=(i,),
            callback=success_handler, 
            error_callback=error_handler
       ) 
       for i in range(3)
    ]


    try:
        for result_obj in results:
            print("result is: {}".format(result_obj.get()))
    except ZeroDivisionError as e:
        print("Oh, boy.  This is the second time I've seen this error.")


--output:--
success
success
result is: 2.5
result is: 5.0
error
['__cause__', '__class__', '__context__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__suppress_context__', '__traceback__', 'args', 'with_traceback'] 

-->
"""
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "1.py", line 10, in do_stuff
    result = 5/(2 - num)  #Line 9
ZeroDivisionError: division by zero
"""<--
Oh, boy.  This is the second time I've seen this error.

Note the line numbers of the target function in the error message.

I don't really understand the purpose of the error_callback because after calling the error_callback, multiprocessing reraises the error, so you have to catch it anyway. I guess the error_calback must be used for doing process specific cleanup.