Hiatus
Posted by Nick Johnson | Filed under travel, blogging, app-engine
I've been blogging regularly or semi-regularly about App Engine on this blog since mid 2008, and I've been on the App Engine team for even longer, and I've decided it's time for a well-earned break.
With that in mind, starting January 30th, I'm taking eight weeks' leave from my job at Google, during which I'll be cycling the length of New Zealand! Not just on a regular bike, either, but one one of these. With any luck, I'll come back from my break revitalised and ready to bring more new and interesting things to the App Engine community.
As a result, I won't be updating this blog during my vacation. Nor am I likely to answer (m)any questions on Stack Overflow or respond promptly to email directed my way. My most capable and excellent colleagues, Amy Unruh, Ikai Lan and Johan Euphrosine will continue to take good care of the community in my absence.
I'll be blogging and vlogging my experiences riding around New Zealand while I'm gone on my new blog, Laid Back Touring, so for anyone interested in following my exploits, that is the place to look ...
There's nothing quite like...
Posted by Nick Johnson | Filed under python, app-engine, theresnothingquitelike
Witness the product of an idle hour or two at work, and time spent pondering the meaninglessness of the cliche "There's nothing quite like...":
Powered by App Engine. Naturally.
Damn Cool Algorithms: Fountain Codes
Posted by Nick Johnson | Filed under bittorrent, lt-codes, damn-cool-algorithms, fountain-codes
That's right, it's time for another episode of the frustratingly infrequent Damn Cool Algorithms series! If you're not familiar with it, you might want to check out some of the previous posts.
Today's subject is Fountain Codes, otherwise known as "rateless codes". A fountain code is a way to take some data - a file, for example - and transform it into an effectively unlimited number of encoded chunks, such that you can reassemble the original file given any subset of those chunks, as long as you have a little more than the size of the original file. In other words, it lets you create a 'fountain' of encoded data; a receiver can reassemble the file by catching enough 'droplets', regardless of which ones they get and which ones they miss.
What makes this so remarkable is that it allows you to send a file over a lossy connection - such as, say, the internet - in a way that doesn't rely on you knowing the rate of packet loss, and doesn't require the receivers to communicate anything back to you about which packets they missed. You can see how this would be useful in a number of ...
Introducing iZac, the Android Bartender
Posted by Nick Johnson | Filed under iZac, android, barbot, adk, arduino
In a previous post, I alluded to a larger project I was working on in the physical/electronics/maker world, and now after much development, I'm pleased to present iZac, the Android Bartender!
A while ago I was inspired by Drink Making Unit 2.0 by Evil Mad Scientist Labs, and decided I wanted to build my own barbot. In this post I'll go over the high level details of what I did and why I did it.
One problem with any drink-dispensing machine is the difficulty of sourcing parts for interacting with food liquids. Pumps are expensive or impractical, valves are likewise problematic and difficult to obtain. My original intention was to use syringe pumps - effectively a DC motor hooked up to a threaded rod that pushes a syringe plunger in and out - and that was the reason behind my making the motor driver shield I wrote about previously. Unfortunately, syringe pumps turned out to be too fiddly and unreliable to use for the barbot, so I had to look for an alternate solution.
I ended up using a similar approach to that used by the DMU2.0, employing a battery powered aquarium pump to pressurize the ...
Migrating to Python 2.7, part 2: Webapp and templates
Posted by Nick Johnson | Filed under python, webapp2, python-27, jinja2, google-app-engine
In part 1 of the migration series, we discussed changes to your app to take advantage of the 2.7 runtime's support for multithreading. Today, we'll start looking at the changes to the included libraries, and how we'll have to modify our app to use them.
webapp2
First up is the replacement of App Engine's lightweight webapp framework with webapp2. Webapp2 is an external open-source project, which says it "follows the simplicity of webapp, but improves it in some ways: it adds better URI routing and exception handling, a full featured response object and a more flexible dispatching mechanism". Webapp2 does an excellent job of maintaining compatibility with webapp, which is why it was chosen as a replacement, but it also adds a number of useful improvements on it.
To see what's changed in concrete terms, let's assume we have a very straightforward app that looks something like this:
import os from google.appengine.ext import webapp from google.appengine.ext.webapp import template class BaseHandler(webapp.RequestHandler): def render_template(self, filename, **template_args): path = os.path.join(os.path.dirname(__file__), 'templates', filename) self.response.out.write(template.render(path, template_args)) class IndexHandler(BaseHandler ...
Grargh!
Posted by Nick Johnson | Filed under dog-ate-my-homework
I had about half of this week's post - part 2 in the "migrating to Python 2.7" series - written when my browser ate it. I foolishly hadn't made a backup, and I lack both the time and the energy to rewrite it right now.
Sorry to all of you who were waiting, and you can expect to see it mid next week, after GDD.
Migrating to Python 2.7, part 1: Threadsafe
Posted by Nick Johnson | Filed under python, threading, python-27, google-app-engine
With the recent experimental release of Python 2.7 support, many people are starting to move their apps from the 2.5 runtime to 2.7. In this series of posts, I'll go over the various considerations for migrating your app in detail, starting with the most immediately obvious - and arguably biggest impact - of them all: threadsafe and multithreaded Python apps.
As you're probably aware, the 2.7 runtime supports making your Python app multithreaded, meaning a single instance of the app may service multiple user requests at the same time. Due to the Global Interpreter Lock, a multithreaded Python app still has limited concurrency, but since most of the wallclock time of a typical App Engine app is spent waiting for RPCs - during which the GIL is not held - the parallelism, and corresponding improvements in utilization, can be substantial.
Moving to Python 2.7 and enabling multithreading is pretty straightforward. First, you have to update your app.yaml. Suppose the start of your app.yaml looks something like this:
application: myapp version: main runtime: python api_version: 1
To switch to Python 2.7, you change it to this (changes are highlighted):
application: myapp version: main runtime: python27 ...
Building a stackable motor driver shield for Arduino
Posted by Nick Johnson | Filed under busdriver-shield, i2c, avr, arduino
Hi, Hack a Day readers! If you're just looking for the source and schematics, they're here. Otherwise, read on!
Recently I've been working with embedded electronics a bit for a project I'm working on, using an Arduino. For the project, I need to control a number of low-power DC motors, and while there are plenty of solutions out there for driving one or two (or even three) like this, none of them let you drive as many motors as I need, and none of them are stackable, as they all require exclusive use of some IO lines. I could get several standalone driver boards like these, but I didn't much like the prospect of the ratsnest of wiring that would ensue. So, I decided to develop my own.
Before I go on, a brief aside about motor drivers. Motors can require a lot of power, so powering them directly from a microcontroller's IO pins is not practical. The obvious solution is to use a simple transistor, but in many cases, including mine, it's necessary to be able to drive the motor both forward and backwards, which isn't possible using a simple transistor ...
Using the Channel API on App Engine for instant traffic analysis
Posted by Nick Johnson | Filed under python, channels, app-engine, javascript, prospective-search
In last week's post, we introduced Clio, a system for getting live insight into your site's traffic patterns, and we described how the Prospective Search API lets us filter the site's traffic to get just the records we care about.
This week, we'll cover the other part of the system: delivering results in real-time. For this, we'll be using the Channel API to stream new log entries to admin users in real-time. As with last week's post, where there's differences between our demo implementation and what you'd use in a real-world system, I'll point those out.
The admin interface
First up, we need to provide a simple admin interface to which we'll stream results. Here's the handler for that:
class IndexHandler(webapp.RequestHandler):
"""Serve up the Clio admin interface."""
def get(self):
client_id = os.urandom(16).encode('hex')
channel_key = channel.create_channel(client_id)
template_path = os.path.join(os.path.dirname(__file__),
'templates', 'index.html')
self.response.out.write(template.render(template_path, {
'config': config,
'client_id': client_id,
'channel_key': channel_key,
}))
The only thing of significance we do here relates to the Channel API. First, we generate a random client ID by getting some ...
Using the Prospective Search API on App Engine for instant traffic analysis
Posted by Nick Johnson | Filed under python, channels, app-engine, prospective-search
One of the really interesting new APIs released as part of App Engine recently is the Prospective Search API. Prospective search inverts the usual search paradigm, where you have a database of documents, and search queries match on those documents. In Prospective Search, you instead have a list of persistent search queries, and as new documents are created or updated, you match them against the queries. Twitter's live search interface is a good example of Prospective Search in action.
Today, in the first of a two post series, we'll be trying out the Prospective Search API with a sample application, Clio. Clio, named after muse of history, is designed to give administrators insight into the actual live traffic being served by their app. With it, you can see user request logs as they occur, and apply filters so you only see the hits that interest you - invaluable on a heavily trafficed site. Mystefied where people are getting to that 404 page from, and don't want to wait 12 hours for the analytics? Clio can help.
In this post, we'll go over the details of how to use the Prospective Search API to construct the server-side, query ...
Newer Older