Known Issues and Workarounds¶
Don’t Call Twisted APIs from non-Twisted threads¶
As is the case in any Twisted program, you should never call Twisted APIs (e.g.
reactor.callLater) from non-Twisted threads.
Only call Twisted APIs from functions decorated by
@wait_for and friends.
Preventing deadlocks on shutdown¶
To ensure a timely process exit, during reactor shutdown Crochet will try to
interrupt calls to
EventualResult.wait() or functions decorated with
@wait_for with a
crochet.ReactorStopped exception. This is still not a
complete solution, unfortunately. If you are shutting down a thread pool as
part of Twisted’s reactor shutdown, this will wait until all threads are
done. If you’re blocking indefinitely, this may rely on Crochet interrupting
those blocking calls… but Crochet’s shutdown may be delayed until the thread
pool finishes shutting down, depending on the ordering of shutdown events.
The solution is to interrupt all blocking calls yourself. You can do this by
firing or canceling any
Deferred instances you are waiting on as part of
your application shutdown, and do so before you stop any thread pools.
Reducing Twisted log messages¶
Twisted can be rather verbose with its log messages. If you wish to reduce the message flow you can limit them to error messages only:
import logging logging.getLogger('twisted').setLevel(logging.ERROR)
In order to prevent massive memory leaks, Twisted currently wipes out the traceback from exceptions it captures (see https://tm.tl/7873 for ideas on improving this).
This means that often exceptions re-raised by Crochet will be missing their tracebacks.
You can however get access to a string version of the traceback, suitable for logging, from
EventualResult objects returned by
from crochet import run_in_reactor, TimeoutError @run_in_reactor def download_page(url): from twisted.web.client import getPage return getPage(url) result = download_page("https://github.com") try: page = result.wait(timeout=1000) except TimeoutError: # Handle timeout ... except: # Something else happened: print(result.original_failure().getTraceback())
uWSGI, multiprocessing, Celery¶
uWSGI, the standard library
multiprocessing.py library and Celery by default use
exec() to create child processes on Unix systems.
This means they effectively clone a running parent Python process, preserving all existing imported modules.
This is a fundamentally broken thing to do, e.g. it breaks the standard library’s
It also breaks Crochet.
You have two options for dealing with this problem. The ideal solution is to avoid this “feature”:
- Use the
- Use the
forkserver) start methods when using Python 3. See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods for more details.
Alternatively, you can ensure you only start Crochet inside the child process:
- Only run
crochet.setup()inside the WSGI application function.
- Only run
crochet.setup()in the child process.
- Only run