Django + Celery + Eventlet - Reduce memory usage
Running celery with django seems to eat a lot of memory. So, I looked for a way to manage (reduce) the concurrent celery workers. Then, I found Eventlet:
"Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it."
"Celery supports Eventlet as an alternative execution pool implementation. It is in some cases superior to multiprocessing, but you need to ensure your tasks do not perform blocking calls, as this will halt all other operations in the worker until the blocking call returns.
The multiprocessing pool can take use of multiple processes, but how many is often limited to a few processes per CPU. With Eventlet you can efficiently spawn hundreds, or thousands of green threads. In an informal test with a feed hub system the Eventlet pool could fetch and process hundreds of feeds every second, while the multiprocessing pool spent 14 seconds processing 100 feeds. Note that is one of the applications evented I/O is especially good at (asynchronous HTTP requests). You may want a mix of both Eventlet and multiprocessing workers, and route tasks according to compatibility or what works best."
So, here are what I need to do to make use of Eventlet:
1. Install Eventlet inside the python environment (virtualenv):
$ pip install eventlet
2. Run django celery beat with eventlet parameter:
python manage.py celeryd --pool=eventlet -v 2 -B -s celery -E -l info
Turn on htop and you will see the magic!
(1): running django celery normal way (5-7 processes)
(2): running django celery with eventlet as execution pool
References:
* Eventlet: http://eventlet.net/
* Celery concurrency with Eventlet: http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
"Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it."
"Celery supports Eventlet as an alternative execution pool implementation. It is in some cases superior to multiprocessing, but you need to ensure your tasks do not perform blocking calls, as this will halt all other operations in the worker until the blocking call returns.
The multiprocessing pool can take use of multiple processes, but how many is often limited to a few processes per CPU. With Eventlet you can efficiently spawn hundreds, or thousands of green threads. In an informal test with a feed hub system the Eventlet pool could fetch and process hundreds of feeds every second, while the multiprocessing pool spent 14 seconds processing 100 feeds. Note that is one of the applications evented I/O is especially good at (asynchronous HTTP requests). You may want a mix of both Eventlet and multiprocessing workers, and route tasks according to compatibility or what works best."
So, here are what I need to do to make use of Eventlet:
1. Install Eventlet inside the python environment (virtualenv):
$ pip install eventlet
2. Run django celery beat with eventlet parameter:
python manage.py celeryd --pool=eventlet -v 2 -B -s celery -E -l info
Turn on htop and you will see the magic!
(1): running django celery normal way (5-7 processes)
(2): running django celery with eventlet as execution pool
References:
* Eventlet: http://eventlet.net/
* Celery concurrency with Eventlet: http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
Comments
Post a Comment