Sometimes you may have a periodic task being managed by Celery. A typical example is you are polling some API every minute to get data you would like to use on your website. Setting up a periodic task with Celery and Django in this manner is reasonably well documented, so I won't go into that here.
However, what if you need to adjust this periodic frequency dynamically, i.e. at runtime? Let's say you hit the rate-limit of the API you are polling and want to decrease your polling frequency from 1 minute to 10 minutes to give the API a break. This can be achieved quite easily if you are using djcelery with the database backend as the scheduler, i.e. in your Django
settings.py you have:
Now you should happily see your
Periodic Tasks etc in the Django admin. In your
tasks.py file you can now update your periodic tasks much like you would any other Django model:
#myapp/tasks.py @periodic_task(run_every=(timedelta(minutes=1))) def scrape_task(): logger.info("Start polling...") pt = PeriodicTask.objects.filter(name='myapp.tasks.scrape_task') result = pollAPI(...) # do actual polling work if 'Rate-limit failure' in result: #Decrease frequency. intv, created = IntervalSchedule.objects.get_or_create( every=1, period='hours') pt.interval_id = intv.id pt.save() elif pt.schedule.run_every.seconds != 60: # If no problems we go back to usual freq intv, created = IntervalSchedule.objects.get_or_create( every=60, period='seconds') pt.interval_id = intv.id pt.save() logger.info("Finished polling...")
What's going on here: in your function that does the actual polling
pollAPI(...), you need to determine if you hit the rate-limit (usually by the response status code, headers or an error msg in the body), if so make the result contain the message 'Rate-limit failure'. Now we need to get or create the relevant intervals; here we have a period of 1hr for the decreased freq and 1 minute for the usual polling freq. Finally, don't forget to save!