Using the Channel API on App Engine for instant traffic analysis
Posted by Nick Johnson | Filed under python, channels, app-engine, javascript, prospective-search
In last week's post, we introduced Clio, a system for getting live insight into your site's traffic patterns, and we described how the Prospective Search API lets us filter the site's traffic to get just the records we care about.
This week, we'll cover the other part of the system: delivering results in real-time. For this, we'll be using the Channel API to stream new log entries to admin users in real-time. As with last week's post, where there's differences between our demo implementation and what you'd use in a real-world system, I'll point those out.
The admin interface
First up, we need to provide a simple admin interface to which we'll stream results. Here's the handler for that:
class IndexHandler(webapp.RequestHandler):
"""Serve up the Clio admin interface."""
def get(self):
client_id = os.urandom(16).encode('hex')
channel_key = channel.create_channel(client_id)
template_path = os.path.join(os.path.dirname(__file__),
'templates', 'index.html')
self.response.out.write(template.render(template_path, {
'config': config,
'client_id': client_id,
'channel_key': channel_key,
}))
The only thing of significance we do here relates to the Channel API. First, we generate a random client ID by getting some ...
Using the Prospective Search API on App Engine for instant traffic analysis
Posted by Nick Johnson | Filed under python, channels, app-engine, prospective-search
One of the really interesting new APIs released as part of App Engine recently is the Prospective Search API. Prospective search inverts the usual search paradigm, where you have a database of documents, and search queries match on those documents. In Prospective Search, you instead have a list of persistent search queries, and as new documents are created or updated, you match them against the queries. Twitter's live search interface is a good example of Prospective Search in action.
Today, in the first of a two post series, we'll be trying out the Prospective Search API with a sample application, Clio. Clio, named after muse of history, is designed to give administrators insight into the actual live traffic being served by their app. With it, you can see user request logs as they occur, and apply filters so you only see the hits that interest you - invaluable on a heavily trafficed site. Mystefied where people are getting to that 404 page from, and don't want to wait 12 hours for the analytics? Clio can help.
In this post, we'll go over the details of how to use the Prospective Search API to construct the server-side, query ...