Using QueueLogger with Python JSON Logger
When logging from multiple processes (via multiprocessing module), using QueueHandler is one approach with Python 2.
QueueHandler however sets exc_info attribute of a LogRecord to None since it is not "pickleable" (more on this later). This becomes a problem when you use python-json-logger to format your logs as JSON since it relies on exc_info being set. The result is you don't get exc_info in your logs. This is no however no longer true since this PR was merged. Sample Code.
Solution #1: Modify Python JSON logger
The first solution is to look for exc_text which is set by the standard Formatter class as per my PR and set that as the value of exc_info which means at least we get the string representation of the traceback. This PR is now merged.
Solution #2: Subclass QueueHandler
The second solution is it to subclass logutils.queue.QueueHandler as follows and add pickling support for exc_info via tblib:
# Bring in support for serializing/deserializing tracebacks# needed by QueueHandlerfromtblibimportpickling_supportpickling_support.install()classQueueHandlerWithTraceback(logutils.queue.QueueHandler):""" QueueHandler with support for pickling/unpickling Tracebacks via tblib. We only override the prepare() method to *not* set `exc_info=None` """def__init__(self,*args,**kwargs):logutils.queue.QueueHandler.__init__(self,*args,**kwargs)defprepare(self,record):self.format(record)record.msg=record.messagerecord.args=Nonereturnrecord
Instead of logutils.queue.QueueHandler, we will then use QueueHandlerWithTraceback instead above (Sample Code).