I am busy changing the world

For some reason when I access the celery's tasks page in the Django's admin (.../djcelery/taskstate/), I got this error: Database returned an invalid value in QuerySet.datetimes(). Are time zone definitions for your database and pytz installed? I have to do the following to get rid of the error: 1. Uncomment these settings: # CELERY_TIMEZONE = 'UTC' # CELERY_ENABLE_UTC = True 2. Add this under mysqld section in /etc/mysql/my.cnf : default-time-zone = "+07:00" 3. Run the following command (without sudo): $ mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p  mysql 4. Restart mysql: $ sudo service mysql restart

In my experience, It's a good idea that you can store the celery tasks information into the database so you can have a better idea about what going in the background without touching the enormous logs pool. Here is how easy it is to do it: 1. In your Django project's settings.py: CELERY_RESULT_BACKEND = 'db+mysql://mydbuser:mydbpasswd@localhost/mydb' CELERY_TRACK_STARTED = True CELERY_SEND_TASK_SENT_EVENT = True 2. Along side with celeryd, you need to run the celerycam app: (venv)$ ./manage.py celerycam 3. And then in your Django admin's dashboard, go to tasks under Djcelery section and enjoy the greatness:

To delete all the celery tasks in Django, run the following command: (venv): ./manage.py celery purge

Yesterday, after deploying my Django apps stack to a new Ubuntu server (14.04 from 12.04), I found that the Celery task was not executed. What happened? Finally, I managed to make it work again following these changes: 1. Upgrade these python modules to the latest version (come with Ubuntu 14.04): Django 1.6.6 , django-celery 3.1.10 , celery 3.1.13 ,  librabbitmq 1.5.2 (to avoid warning about rabbitmq compatibility) 2. Add these CELERY settings (in red color) in my Django app's settings.py: ... CELERY_IMPORTS = ("myapp.tasks",) CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend' CELERY_ACCEPT_CONTENT = ['pickle', 'json'] # new in celery 3.1 CELERY_TRACK_STARTED = True ... (remember to restart the celery beat after making those changes)

In the latest stable version of celery (3.1.13), you cannot run a celery beat as a daemon (I used supervisord) using root user without a special environment variable. That special variable is C_FORCE_ROOT. So, in your supervisord configuration file, add this: ... environment=C_FORCE_ROOT="yes" ... This is not encouraged but in some cases like mine, I just need the system works, and avoiding any extra configuration.

When I executed a celery task in my django project (South installed), I got this error: ... Programming Error ... DatabaseError: no such table: myapp.celery_taskmeta... ... The problem here is actually that South manages the djcelery tables. So, I need to migrate djcelery to it's new schema python manage.py migrate djcelery 0001 --fake python manage.py migrate djcelery python manage.py migrate djcelery 0002 --fake python manage.py migrate djcelery ... (try to do the fake migration until the error disappears. I had to do it 4 times (0004) in my case) Reference: http://stackoverflow.com/questions/6959702/why-are-celery-taskmeta-and-other-tables-not-being-created-when-running-a-syncdb

Sometimes we cannot measure the progress or percentage completed of a job when using celery task (http://iambusychangingtheworld.blogspot.com/2013/07/django-celery-display-progress-bar-of.html). You can only say if it done or not by checking the status of the task. Here is a case, where the business code returns a failed result, but the task is still executed successfully. How can you tell? Luckily, you can use the result attribute of the task to get the return value of the business code. For example: Run the celery worker: celery -A tasks worker --loglevel = info Try to call the task which will return a fail result: >> from tasks import add >> job = add.delay(5,4) >> job.state SUCCESS >> job.result FAIL With this you can figure exactly what the result of the task is.

This morning, I tried to send around 1/2 thousand of emails using an Google App Email account from my Django app along with celery and django-mailer . But after 100 messages were sent, the celery task cannot send email, and I got this error: Google SMTP Error: 454 4.7.0 Too many login attempts, please try again later There are something I need to do to solve this: 1. Add a DKIM record in Google Apps: following this stackoverflow question: http://serverfault.com/questions/543007/google-smtp-error-454-4-7-0-too-many-login-attempts-please-try-again-later In order to add a DKIM record in Google Apps, you need to do the following: Go to the Admin Console Click on "Google Apps" Click on "Gmail" Scroll down until you see "Authenticate Email" and click that Select the domain you wish to add DKIM to When it asks what prefix you want to use, simply use the default of 'google' You will then see a TXT record in two parts, one piece has...

Just noted when I stopped the django-celery app which monitored by supervisord with: # supervisorctl stop myappcelery The status show myappcelery had stopped: # supervisorctl status myappcelery                         STOPPED    Nov 20 10:48 AM But, the python processed of django-celery were still running: # ps aux | grep celery root     25729  9.0  1.8 225676 38188 ?        S    11:07   0:00 python /mydjangoapp/manage.py celeryd --pool=eventlet -v 2 -B -s celery -E -l info root     25748  1.0  1.7 164640 36596 ?        S    11:07   0:00 python /mydjangoapp/ manage.py celeryd --pool=eventlet -v 2 -B -s celery -E Then, I take a look at the supervisord configuratio...

Running celery with django seems to eat a lot of memory. So, I looked for a way to manage (reduce) the concurrent celery workers. Then, I found Eventlet: "Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it." "Celery supports Eventlet as an alternative execution pool implementation. It is in some cases superior to multiprocessing, but you need to ensure your tasks do not perform blocking calls, as this will halt all other operations in the worker until the blocking call returns. The multiprocessing pool can take use of multiple processes, but how many is often limited to a few processes per CPU. With Eventlet you can efficiently spawn hundreds, or thousands of green threads. In an informal test with a feed hub system the Eventlet pool could fetch and process hundreds of feeds every second, while the multiprocessing pool spent 14 seconds processing 100 feeds. Note that is one of the applicat...

I'm using Celery ( http://www.celeryproject.org/ ) to do the users's long-running task in Django. It eases the users by not let them wait the process done. When a user call the action (the view to execute a certain task), Django immediately response back to the user and it's done. The  job which has to be done was sent to the message broker where it will be actually executed. We all know all of the things mentioned above. Question is "How does the user know when the task will finish?" Here is the solution: - In your task code, continuously updating state of the task (over a for loop) , define a custom state names PROGRESS - Writing an infinite loop of Ajax call in your template to check the state of the current process and update the progress bar. I also make a complete and simple example to implement the above idea, call testcele project (and create  celery app inside testcele project). The main (and only) functionality of testcele is that it...

It's good to blog in a beautiful Sunday. This is the second one. A popular situation a webapp can face is to send thoundsands of emails to the users. This action can block the system a while, and this is annoying! To solve this problem, there are many solutions. One of them is sending email to a queue and send them later in the background (automatically), so the system don't have to be freezed. Pretty cool huh? I will setup a message queue in Django with: