Using Python magic to improve the deferred API
Posted by Nick Johnson | Filed under python, deferred, app-engine, coding, celery
Recently, my attention was drawn, via a blog post to a Python task queue implementation called Celery. The object of my interest was not so much Celery itself - though it does look both interesting and well written - but the syntax it uses for tasks.
While App Engine's deferred library takes the 'higher level function' approach - that is, you pass your function and its arguments to the 'defer' function - I've never been entirely happy with that approach. Celery, in contrast, uses Python's support for decorators (one of my favorite language features) to create what, in my view, is a much neater and more flexible interface. While defining and calling a deferred function looks like this:
def my_task_func(some_arg): # do something defer(my_task_func, 123)
Doing the same in Celery looks like this:
@task def my_task_func(some_arg): # do something my_task_func.delay(123)
Using a decorator, Celery is able to modify the function it's decorating such that you can now call it on the task queue using a much more intuitive syntax, with the function's original calling convention preserved. Let's take a look at how this works, first, and then explore how we might make use of it ...