Trip reports: Stack Overflow Amsterdam, and Oredev
Posted by Nick Johnson | Filed under tech, conferences
I spent this week attending and speaking at two different conferences - the Stack Overflow Dev Day in Amsterdam, and Øredev, in Malmö, Sweden.
Stack Overflow was a lot of fun. Talks covered topics such as FogBugz, jQuery, Fog Creek Software, creators of FogBugz, QT (pronounced 'cute'), FogBugz, Python, App Engine (my own talk), Yahoo! developer tools, and, of course, FogBugz.
Particular highlights for me were Simon Willison's talk about Python, and Christian Heilmann's talk on Yahoo! developer tools. Simon works at The Guardian, a UK newspaper that is particularly keen on exposing and consuming data and APIs, and used his hour to introduce Python by demonstrating how he uses it, particularly the interactive interpreter, on a day to day basis to extract and munge statistical data and produce content for news stories, such as infographics. Although I know Python very well, his presentation style was extremely engaging, and I had to restrain myself from going up to him and asking if he was looking for apprentices in the fine art of presenting. He's also a co-author of Django, so he showed off some of Django's features in his talk as well.
Christian's talk was completely ...
Server-side JavaScript with Rhino
Posted by Nick Johnson | Filed under coding, app-engine, cookbook, tech, java
I've only made limited use of the Java runtime for App Engine so far: The two runtimes are largely equivalent in terms of features, and my personal preference is for Python. There are, however, a few really cool things that you can only do on the Java runtime. One of them is embedding an interpreter for a scripting language.
Rhino is a JavaScript interpreter implemented in Java. It supports sandboxing, meaning you can create an environment in which it's safe to execute user-provided code, and it allows you to expose arbitrary Java objects to the JavaScript runtime, meaning you can easily provide interfaces with which the JavaScript code can carry out whatever actions it's designed to carry out.
Rhino also supports several rather cool additional features. You can set callback functions that count the number of operations executed by the JavaScript code, and terminate it if it takes too long, you can serialize a context or individual objects and deserialize them later, and you can even use continuations to pause execution when waiting for data, and pick it up again later - possibly even in another request! Hopefully we'll show off some of these features in future ...
Blogging on App Engine, part 10: Recap
Posted by Nick Johnson | Filed under tech, app-engine, coding, bloggart
This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.
Over the last 9 posts, we've covered building a fully functional blogging system for App Engine from scratch, and I've even migrated this blog to it. In the process, we've covered many important components of App Engine, including the new deferred library (and through it, Task Queues), important design principles, such as pre-generation of content, and interesting technologies such as PubSubHubbub, CSEs, Disqus, and sitemaps.
There's also been significant community involvement, for which I am very grateful. Amongst the contributors were:
- Moraes, who ported bloggart to werkzeug, calling it bloggartzeug
- Sylvain, who ported bloggart to tornado, calling it bloggartornado
- andialbrecht, who contributed - and continues to contribute - enhancements and bugfixes for bloggart
- Everyone who contributed a name suggestion on the first post, and everyone who has provided feedback during the series
- Everyone who filed bug reports and feature requests in the issue tracker
To give some sort of objective assessment of how well Bloggart measures up, we need to compare it to our original goals, outlined in the very first post:
- Simple ...
Blogging on App engine, part 9: Sitemaps and verification
Posted by Nick Johnson | Filed under coding, app-engine, tech, bloggart
This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.
Today we're going to cover basic sitemap support and verifying your site with Google.
Sitemaps
Sitemaps are a recent innovation that aim to make it easier for search engines to find and index your site. The format is a very straightforward XML file. Several optional attributes can be present, such as the last-modified date and update frequency; for this first attempt we're not going to use any of them, and just provide a basic listing of URLs. Future enhancements could provide more sitemap information, and break the sitemap into multiple files for extensibility.
For the purposes of generating a complete sitemap, we have a significant advantage: Our static serving infrastructure provides us with a convenient means of getting a list of all valid URLs. Not all URLs should be indexed, however, so we should make it possible to specify what content should be indexed. Add a new property to the StaticContent model in static.py:
indexed = db.BooleanProperty(required=True, default=True)
We'll need to enhance our set() method to take this ...
Blogging on App Engine, part 8: PubSubHubbub
Posted by Nick Johnson | Filed under coding, app-engine, tech, bloggart
This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.
Notice anything different? That's right, this blog has now been migrated to Bloggart - after all, if I won't run it, why should I expect anyone else to? ;)
Migrating comments to Disqus
In the previous post, I promised I'd cover migrating comments in today's post. Unfortunately, doing so proved to be both more complicated and less interesting than anticipated, so I'm going to instead provide an overview of the required steps. If you're really determined to see all the nitty-gritty, you can examine the change yourself.
Import of comments to disqus is through the disqus API. The API uses a straightforward RESTful model, with requests URL-encoded as either GET or POST requests, and responses returned as JSON strings. To make our lives easier, we'll define a straightforward wrapper function to make Disqus API calls:
def disqus_request(method, request_type=urlfetch.GET, **kwargs):
kwargs['api_version'] = '1.1'
if request_type == urlfetch.GET:
url = "http://disqus.com/api/%s?%s" % (method, urllib.urlencode(kwargs))
payload = None
else:
url = "http://disqus.com/api/%s ...
Blogging on App Engine, part 7: Migration
Posted by Nick Johnson | Filed under coding, app-engine, tech, bloggart
This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.
We're finally going to tackle (at least part of) that big bugbear of blogging systems: Migrating from the old system to the new one. In this post, we'll cover the necessary pre-requisites, briefly cover the theory of importing from a blogging system hosted outside App Engine, then go over a practical example of migrating from Bloog (since that's what this blog is hosted on).
Regenerating posts
Before we can write migration or import scripts, we need to improve (again) our dependency regeneration code. One thing that's probably occurred to you if you've been following this series is that there's currently no easy way to regenerate all the resources when something global changes such as the theme or the configuration. One could simply call .publish() on each blog post, but that would result in regenerating the common resources, such as the index and tags pages, over and over again - potentially hundreds of times. The same applies to migration: We could publish each new post as we process it, but this ...
Blogging on App Engine, part 6: Comments and Search
Posted by Nick Johnson | Filed under tech, app-engine, coding, bloggart
This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.
Today we're going to tackle two separate issues: Support for commenting on posts, and support for search. Commenting is fairly straightforward, so we'll deal with that first.
Commenting
Rather than implement our own comments system, we're going to take advantage of an existing 'SaaS' commenting offering, Disqus. Disqus provides simple drop in Javascript powered comment support, and has, by now, a rather impressive feature set, incorporating support for various login schemes - their own, OpenID, facebook connect, twitter, and others - as well as advanced functionality like finding and displaying 'reactions' from social sites around the web along with comments from users.
Integrating disqus support is straightforward. Since some people might not want to use it, or might want to use an alternate system, however, we're going to use a new config setting to ensure we only enable it if it's wanted. Add the following to the bottom of config.py:
# To use disqus for comments, set this to the 'short name' of the disqus forum
# created for the purpose.
disqus_forum = None ...
Blogging on App Engine, part 5: Tagging
Posted by Nick Johnson | Filed under tech, app-engine, coding, bloggart
This is part of a series of articles on writing a blogging system on App Engine. An overview of what we're building is here.
Following on from our previous post, today we're going to deal with tagging. There are three components to adding tagging support to our blog:
- Adding tags to the model and the post/edit interface.
- Generating listing pages of posts with a given tag.
- Adding tags and links to the listing pages on individual posts.
We'll tackle these in order. First, adding tags to the model and to the add/edit post interface. Add the following property immediately after 'body' on the BlogPost class (in models.py):
tags = db.StringListProperty()
That's it. No, really. Thanks to our use of ModelForms, our admin interface now has support for adding and editing posts with tags. Try it, if you wish. One slight caveat: The interface expects tags to be separated by newlines, rather than by commas. That's something we could address with a custom widget, at a later stage.
Next, the listing pages. Nearly all the functionality required to generate listings of posts with a given tag is identical to that required to generate ...
Blogging on App Engine, part 4: Listings
Posted by Nick Johnson | Filed under coding, app-engine, tech, bloggart
As you may have surmised from previous posts in the series, the 'static serving' approach we're using can lead to regenerating a lot of pages at once. For a long lived blog, with lots of history, regenerating the archive pages could take a significant amount of time - potentially long enough that we could run into the 30 second request deadline when updating or adding a post. Fortunately, however, we have something custom-made for the purpose: the Task Queue API. Using the Task Queue API, we can take care of the essential updates immediately - the post page itself, for example - then queue up other updates, such as the archive pages, on the task queue for later execution. Using the task queue has the extra advantage that updates can be executed in parallel.
Even better, we can make use of a new library in version 1.2.5 of the SDK, called 'deferred'. deferred is a clone of Ruby's delayed::job library, and makes it easy to enqueue function and method calls on the App ...
Blogging on App Engine, part 3: Dependencies
Posted by Nick Johnson | Filed under tech, app-engine, coding, bloggart