ReferenceProperty prefetching in App Engine
Posted by Nick Johnson | Filed under python, tech, app-engine, cookbook, coding
This post is a brief interlude in the webapps on App Engine series. Fear not, it'll be back!
Frequently, we need to do a datastore query for a set of records, then do something with a property referenced by each of those records. For example, supposing we are writing a blogging system, and we want to display a list of posts, along with their authors. We might do something like this:
class Author(db.Model):
name = db.StringProperty(required=True)
class Post(db.Model):
title = db.TextProperty(required=True)
body = db.TextProperty(required=True)
author = db.ReferenceProperty(Author, required=True)
posts = Post.all().order("-timestamp").fetch(20)
for post in posts:
print post.title
print post.author.name
On the surface, this looks fine. If we look closer, however - perhaps by using Guido's excellent AppStats tool, we'll notice that each iteration of the loop, we're performing a get request for the referenced author entity. This happens the first time we dereference any ReferenceProperty, even if we've previously dereferenced a separate ReferenceProperty that points to the same entity!
Obviously, this is less than ideal. We're doing a lot of RPCs, and incurring a lot of ...
Webapps on App Engine part 3: Request handlers
Posted by Nick Johnson | Filed under python, coding, app-engine, tech, framework
This is part of a series on writing a webapp framework for App Engine Python. For details, see the introductory post here.
Now that we've covered the background on request handling, it's time to tackle request handlers. Request handlers are the core and most obvious part of a web framework. They serve to simplify the writing of your app, and remove some of the boilerplate that you end up with if you write raw WSGI applications. Before we go any further, let's see what basic request handlers look like in a range of frameworks. Then we can discuss the pros and cons of each, and settle on one for ours.
def current_datetime(request):
now = datetime.datetime.now()
html = "It is now %s." % now
return HttpResponse(html)
App Engine's webapp framework:
class MainPage(webapp.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('Hello, webapp World!')
@expose('/display/')
def display(request, uid):
url = URL.query.get(uid)
if not url:
raise NotFound()
return render_template('display.html', url=url)
def hello1():
return "Hello World"
class HelloController(BaseController):
def index(self):
# Return a rendered template
#return render('/hello.mako')
# or ...
Webapps on App Engine part 2: Request & Response handling
Posted by Nick Johnson | Filed under python, tech, app-engine, coding, framework
This is part of a series on writing a webapp framework for App Engine Python. For details, see the introductory post here.
This post is mostly background on request and response encoding/decoding. If you're already fairly familiar with how this works in CGI and in higher level frameworks, you may want to skip this and wait for the next posting, on request handlers.
If you've ever written a CGI script, you'll be well aware of how much of a pain interpreting and decoding CGI environment variables and headers can be - so much so that the first action of many is to find or write a library to handle it for you! And if you've ever coded in a CGI-inspired language such as PHP, you're probably familiar with how much of a pain managing the combination of response headers and output content can likewise be.
As a result of this, one of the most basic tools a framework offers is some form of abstraction for request and response data. Typically, this takes care of collecting and parsing HTTP headers, parsing the query string (if any), decoding POSTed form data, and so forth. Better frameworks also ...
Webapps on App Engine part 1: Routing
Posted by Nick Johnson | Filed under python, coding, app-engine, tech, framework
This is part of a series on writing a webapp framework for App Engine Python. For details, see the introductory post here.
The first part of a framework you encounter when using one is, more often than not, the routing code. With that in mind, it's what we'll be tackling first. There are several approaches to handling request routing, and we'll go on a quick tour of the libraries and approaches before we decide on one and implement it.
The built-in App Engine webapp framework takes an extremely straightforward approach: The incoming request's URL is compared to a list of regular expressions in order, and the first one that matches has the corresponding handler executed. As an enhancement, any captured groups in the regular expression are passed to the request handler as additional arguments. You can see the code that does all this here - it's extremely straightforward and easy to follow.
The webapp module does one thing that I'm not a huge fan of: It ties the request routing in with handling the requests. The same function that finds the appropriate handler for a request also takes care of parsing the request and calling ...
Writing your own webapp framework for App Engine
Posted by Nick Johnson | Filed under python, coding, app-engine, tech, framework
Welcome back! I trust you all had a good holiday period? Mine was spent back in sunny New Zealand, seeing friends and family, visiting favorite restaurants, enjoying the sunshine, and learning to paraglide.
I would have started blogging again last week, but my first week back was made both exciting and frantically busy preparing for, then attending the BT Young Scientist Exhibition, where I gave tutorials on App Engine at the Google booth. But now, back to your regularly scheduled blogging...
Sometimes it seems like everyone has written their own blogging system, and everyone has written their own framework for webapps. That's not all these two things have in common, though: They're both excellent learning projects. Since you've already done the first, why not do the second? This will be the first of a series of posts covering how to write your own Python webapp framework. The framework, while targeted at App Engine, isn't exclusive to it.
As with the blogging project, it helps to set some goals before we get started. Here's our goals for this project:
- Lightweight. With cold startup time being a significant concern for many, it's essential to avoid creating ...
Merry Season!
Posted by Nick Johnson | Filed under app-engine
As you've probably guessed by now, I'm not posting over the holiday period. Expect new content, including an all new series of posts, in the new year, however!
In the meantime, enjoy your time with your families, if that's what you're doing. As for myself and my wife, Hayley, we're back in New Zealand, enjoying some quality time with our families.
OpenID on App Engine made easy with AEoid
Posted by Nick Johnson | Filed under python, coding, app-engine, tech
I'm pleased to present AEoid, a new App Engine library that aims to make user authentication with OpenID on App Engine simple.
AEoid is extremely easy to install, comprising a single piece of WSGI middleware, and its interface mirrors that of the App Engine Users API. Here's an example of it in action:
from aeoid import users
class TestData(db.Model):
user = users.UserProperty()
class TestHandler(webapp.RequestHandler):
def get(self):
user = users.get_current_user()
if not user:
self.redirect(users.create_login_url(self.request.url))
return
logging.warn("Logged in as %s (%s)", user.nickname(), user.user_id())
data = TestData(user=user)
data.put()
As you can see, the interface to AEoid is almost exactly identical to the App Engine Users API. There a few differences of note:
- Users are identified uniquely by their OpenID endpoint.
- You can't construct a User object without specifying an OpenID URL.
- Nicknames and email addresses are user-supplied, so they're not guaranteed unique or validated.
- is_current_user_admin() is not yet implemented.
- login: clauses in app.yaml are not affected by AEoid - they still authenticate using the regular Users API.
Installation
Installing AEoid is a simple matter of adding its WSGI middleware to your app ...
No post today
Posted by Nick Johnson | Filed under app-engine, tech
Unfortunately, I've been caught short today by a flood of important things at work, and didn't get a chance to write up my regular Wednesday blog post. In place of that, I offer you the winners of the App Engine USB drives:
- Jeff
- Rodrigo Moraes
- Olivier Deckmyn
If you are reading this, and you're one of the winners, please send me an email (nick AT notdot DOT net) with your name, address, and favorite Google color (Yellow, Green, Red, or Blue), and I'll get them out to you pronto.
As further tribute, I offer you this adorable picture of a kitten:
(Yes, I'm aware of how paradoxical the subject of this post is)
Damn Cool Algorithms: Log structured storage
Posted by Nick Johnson | Filed under tech, damn-cool-algorithms
Typically, if you're designing a storage system - such as a filesystem, or a database - one of your major concerns is how to store the data on disk. You have to take care of allocating space for the objects to be stored, as well as storing the indexing data; you have to worry about what happens when you want to extend an existing object (eg, appending to a file), and you have to take care of fragmentation, which happens when old objects are deleted, and new ones take their place. All of this adds up to a lot of complexity, and the solutions are often buggy or inefficient.
Log structured storage is a technique that takes care of all of these issues. It originated as Log Structured File Systems in the 1980s, but more recently it's seeing increasing use as a way to structure storage in database engines. In its original filesystem application, it suffers from some shortcomings that have precluded widespread adoption, but as we'll see, these are less of an issue for database engines, and Log Structured storage brings additional advantages for a database engine over and above easier storage management.
The basic organization of a ...
'Naked' domains on App Engine
Posted by Nick Johnson | Filed under naked-domains, app-engine, tech, dns
One topic that comes up frequently on the App Engine groups is that of 'naked' domains in App Engine, and how to handle them. A naked domain, for the uninitiated, is one without a service-specific subdomain. For example, "google.com" is naked, while "www.google.com" is not. This post provides an overview of why naked domains are a problem, and what you can do about them.
There are two separate factors that combine to make handling of naked domains a problem in App Engine. The first is the design of DNS, the system for resolving domain names to IP addresses. There are two different types of DNS record we're concerned about here: A records, which specify the IP address for a name, and CNAME records, which acts a "see also", specifying another name for a domain. For example, an A record might say "google.com has the IP 216.239.59.104", while a CNAME record might say "google.com is also known as www.l.google.com".
The problem arises with the way CNAME records work. An A record specifies the IP address only for a single record - for example, an A record on google.com specifies ...
Newer Older