Excellent recommendation. In my experience Celery is an overkill most of the times and will force you to spend more time doing ops guaranteed.

Thanks for this, I had considered using celery for a recent project but ultimately backed away because I got the feeling it was more trouble than it was worth. As a point of reference would you say the learning curve for a celery setup is similar to that of django? Not that theres anything terribly hard about django, but Id agree that its probably overkill if youre relatively new to python and are just looking for a quick way to produce some html with no intent on developing it further.

I wouldn't say Celery's learning curve is steeper than Django's, but it definitely seems like overkill for your case. If you need to do some time-consuming action periodically (and making an HTTP request by hand each time is not an option), then you could just use cron for the start if your project is relatively simple. And if you literally need to just produce some HTML when asked for, then why are you considering using an async task processor such as Celery?

Oh no I would never use celery for that. It was just comparing the learning curve of celery to django.

I just started using Celery this week for the first time, to handle parallel processing of thousands of tasks in a data pipeline.

Three days in, I can tell you that it does work, but it does take a lot of searching through the docs to optimize. It's very hard to run with class objects too, so we just created long-scripted functions for the worker.

Even now I'm trying to figure out why the worker is unable to refresh access tokens after 60 minutes, and tempted to just have it run as root.

Django is kind of slow learning curve, but understandable. Celery for me was quicker, but more along the lines of: follow the instructions, and google / stack overflow until it works. A lot less understanding involved.

It's somewhat akin to a Django vs. Flask discussion really. But yes for more light weight stuff I too would recommend rq.

I had the same feeling, though I am using celery for different projects, still it needs time for me to figure out "what is going on there?". Specially I have used for simple task queue system which was overkill. And python-rq definitely a good choice. It does one thing, API is quiet simple and short and it does the task well.

Thanks! I didn't know about RQ, from a cursory glance it does look a lot simpler than Celery. Sidenote: I decided to follow RQ's author on Github, and discoverd Gitflow as well. So, double thanks!

This! Celery is well supported and powerful but often it is just too much to manage. Everytime we have an error crop up in our deployment it takes too much time to figure out whats going on.