Sometimes you may have a periodic task being managed by Celery. A typical example is you are polling some API every minute to get data you would like to use on your website. Setting up a periodic task with Celery and Django in this manner is reasonably well documented, so I won't go into that here.

However, what if you need to adjust this periodic frequency dynamically, i.e. at runtime? Let's say you hit the rate-limit of the API you are polling and want to decrease your polling frequency from 1 minute to 10 minutes to give the API a break. This can be achieved quite easily if you are using djcelery with the database backend as the scheduler, i.e. in your Django settings.py you have:

CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend'

Now you should happily see your Intervals, Periodic Tasks etc in the Django admin. In your tasks.py file you can now update your periodic tasks much like you would any other Django model:

#myapp/tasks.py
@periodic_task(run_every=(timedelta(minutes=1)))
def scrape_task(): 
    logger.info("Start polling...")
    pt = PeriodicTask.objects.filter(name='myapp.tasks.scrape_task')[0]
    result = pollAPI(...)  # do actual polling work
    if 'Rate-limit failure' in result:
        #Decrease frequency.
        intv, created = IntervalSchedule.objects.get_or_create(
                                                     every=1, period='hours')
        pt.interval_id = intv.id
        pt.save()
    elif pt.schedule.run_every.seconds != 60:
        # If no problems we go back to usual freq
        intv, created = IntervalSchedule.objects.get_or_create(
                                                    every=60, period='seconds')
        pt.interval_id = intv.id
        pt.save()
    logger.info("Finished polling...")

What's going on here: in your function that does the actual polling pollAPI(...), you need to determine if you hit the rate-limit (usually by the response status code, headers or an error  msg in the body), if so make the result contain the message 'Rate-limit failure'. Now we need to get or create the relevant intervals; here we have a period of 1hr for the decreased freq and 1 minute for the usual polling freq. Finally, don't forget to save!

Current rating: 3.5

About Lee

I am a Theoretical Physics PhD graduate now working in the technology sector. I have strong mathematical skills and originally started in heavy-duty scientific computing, but now I work mostly with Python and the Django framework. I am available for hire now, so check out my resume and get in touch.

Comments