Blogging on App Engine, part 8: PubSubHubbub

Posted by Nick Johnson | Filed under tech, app-engine, coding, bloggart

This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.

Notice anything different? That's right, this blog has now been migrated to Bloggart - after all, if I won't run it, why should I expect anyone else to? ;)

Migrating comments to Disqus

In the previous post, I promised I'd cover migrating comments in today's post. Unfortunately, doing so proved to be both more complicated and less interesting than anticipated, so I'm going to instead provide an overview of the required steps. If you're really determined to see all the nitty-gritty, you can examine the change yourself.

Import of comments to disqus is through the disqus API. The API uses a straightforward RESTful model, with requests URL-encoded as either GET or POST requests, and responses returned as JSON strings. To make our lives easier, we'll define a straightforward wrapper function to make Disqus API calls:

def disqus_request(method, request_type=urlfetch.GET, **kwargs):
  kwargs['api_version'] = '1.1'
  if request_type == urlfetch.GET:
    url = "http://disqus.com/api/%s?%s" % (method, urllib.urlencode(kwargs))
    payload = None
  else:
    url = "http://disqus.com/api/%s/" % (method,)
    payload = urllib.urlencode(kwargs)
  response = urlfetch.fetch(url, payload, method=request_type)
  if response.status_code != 200:
    raise Exception("Invalid status code", response.status_code, response.content)
  result = simplejson.loads(response.content)
  if not result['succeeded']:
    raise Exception("RPC did not succeed", result)
  return result

This function takes the method name, optionally the type (GET or POST) and any number of keyword arguments. It converts the arguments to a URL-encoded string, uses the urlfetch API to send the request, then decodes and returns the result. Here's an example of using this function to call 'get_forum_list':

forums = disqus_request('get_forum_list', user_api_key=disqus_user_key)

The API documentation isn't the most complete, so I'll go over the required steps to import comments here:

Obtain your user API key
Call get_forum_list to obtain a list of forums you own.
Locate the forum you want to import entries for by finding the entry with the correct 'shortname' field.
Call get_forum_api_key with the user key and forum ID to get the forum's API key.
For each post you want to import comments for:
1. Call thread_by_identifier to get a unique identifier for a thread.
2. Call update_thread to set the URL for the thread just created.
3. For each comment, call create_post, supplying the forum API key, thread id, post data, and optional parent post ID.

PubSubHubbub support

As you've probably heard by now, the cool new kid on the block is PubSubHubbub, a protocol for instant 'push' updating of Atom feeds. Since we naturally don't want to be left behind, we're going to be implementing PubSubHubbub ('hubbub' for short) support on Bloggart. Fortunately, the hubbub protocol puts an emphasis on making things easy to implement, especially for the publisher.

The first thing we need to do is add a configuration option for enabling hubbub support, and setting the hub URL to use. Add the following to the end of config.py:

# If you want to use PubSubHubbub, supply the hub URL to use here.
hubbub_hub_url = 'http://pubsubhubbub.appspot.com/'

The next thing we need to do is announce that we support hubbub, by adding a link tag to our atom feed. Add this to themes/default/atom.xml after the existing link tags:

<link rel="hub" href="{{config.hubbub_hub_url}}" />

Finally, we need to notify the hub whenever we publish or update an item. Open up generators.py and make the following changes (highlighted in yellow) to AtomContentGenerator:

  @classmethod
  def generate_resource(cls, post, resource):
    import models
    q = models.BlogPost.all().order('-updated')
    posts = q.fetch(10)
    template_vals = {
        'posts': posts,
    }
    rendered = utils.render_template("atom.xml", template_vals)
    static.set('/feeds/atom.xml', rendered,
               'application/atom+xml; charset=utf-8')
    if config.hubbub_hub_url:
      cls.send_hubbub_ping(config.hubbub_hub_url)
 
  @classmethod
  def send_hubbub_ping(cls, hub_url):
    data = urllib.urlencode({
        'hub.url': 'http://%s/feeds/atom.xml' % (config.host,),
        'hub.mode': 'publish',
    })
    response = urlfetch.fetch(hub_url, data, urlfetch.POST)
    if response.status_code / 100 != 2:
      raise Exception("Hub ping failed", response.status_code, response.content)

That's all that's required. Whenever we update our Atom feed, we ping the hub, which will fetch the feed and send the modified parts to any subscribers. Easy!

You can view the blog-so-far at http://bloggart-demo.appspot.com/, and see the code for this stage here.

In the next post, we'll handle sitemap support, and discuss the blog-so-far, and where to go next.

21 October, 2009

Previous Post Next Post

Nick's Blog