A reminder
Posted by Nick Johnson | Filed under app-engine, blogging
Just a quick reminder: If you've got topics you'd like me to write about, there's a feedback widget on the right hand side of the page. Submit and upvote suggestions there! A lot of the posts on this blog were based on feedback from users, and it's always more useful to be writing about something people want to hear about.
Demystifying the App Engine request logs
Posted by Nick Johnson | Filed under python, app-engine, java, logs
People often ask about the App Engine request logs shown in the admin console: What do all the fields mean? How should they be interpreted? They're actually fairly easy to read once you know the format, so here's a quick-reference to help.
The basic format is based on the Apache combined log format, which is so widely used even outside Apache that it's become a de-facto standard of sorts for webserver request logs. In addition to that, App Engine adds a number of extra fields logging additional, App Engine specific stats. Suppose you're examining the following log entry:
107.10.42.191 - foobiz [10/May/2011:17:26:28 -0700] "GET /page.html HTTP/1.1" 500 2297 "http://www.example.com/home.html" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4,gzip(gfe),gzip(gfe),gzip(gfe)" "www.example.com" ms=364 cpu_ms=23 api_cpu_ms=0 cpm_usd=0.001059
Here's what the fields mean, in order:
- Client's IP address (107.10.42.191)
- The RFC1413 identity of the client (in practice, always '-')
- The userid determined ...
Promises
Posted by Nick Johnson | Filed under app-engine, blogging
One of the questions I got asked most at I/O was "when are you going to start blogging again"? I'm happy to be able to say that the answer is "really soon now". And yes, this time I really mean it.
I'm away on holiday for the next 10 days, but as soon as I get back to work, I'm back to blogging, on at least a weekly schedule - perhaps more often. I even have a couple of kick-ass blog posts lined up about some of the new features. If you have ideas of your own, post them to the issue tracker linked from the left.
In the meantime, the I/O session videos are well worth watching.
Using the SM130 RFID/NFC reader
Posted by Nick Johnson | Filed under python, sm130, rfid, nfc, arduino
Long time no blog, I know - and this post isn't even my usual. Don't worry, more posts are coming soon!
I've recently been playing around with Arduino, and with NFC tags and readers. Sparkfun (and their Aussie distributors, Little Bird Electronics sell a rather nifty NFC reader/writer, the SM130, by Sonmicro, along with a corresponding Arduino Shield for it. It has a fairly easy to use serial protocol, and supports both regular UART serial and I2C.
In order to use I2C, though, you need to flash the reader with a new firmware. Sonmicro will provide you the firmware for free, b ut the application to flash it to the device is Windows only, which puts a bit of a crimp in the plans of those of us who don't use that platform. I wanted to use the reader in I2C mode, so I decided to try and solve this not just for myself, but anyone else in the same situation.
First step was to obtain the firmware. Sonmicro promptly sent it to me when I asked, and even kindly agreed to allow me to redistribute it - so here it is, in a gist.
The first ...
Storage options on App Engine
Posted by Nick Johnson | Filed under app-engine, memcache, storage, datastore, task-queue
App Engine provides a number of ways for your app to store data. Some, such as the datastore, are well known, but others are less so, and all of them have different characteristics. This article is intended to enumerate the different options, and describe the pros and cons of each, so you can make more informed decisions about how to store your data.
Datastore
The best known, most widely used, and most versatile storage option is, of course, the datastore. The datastore is App Engine's non-relational database, and it provides robust, durable storage, as well as providing the most flexibility in how your data is stored, retrieved, and manipulated.
Pros
- Durable - data stored in the datastore is permanent.
- Read-write - apps can both read and write datastore data, and the datastore provides transaction mechanisms to enforce integrity.
- Globally consistent - all instances of an app have the same view of the datastore.
- Flexible - queries and indexing provide many ways to query and retrieve data.
Cons
- Latency - because the datastore stores data on disk and provides reliability guarantees, writes need to wait until data is confirmed to be stored before returning, and reads often have to fetch data from disk.
Memcache
Memcache ...
Breaking things down to Atoms
Posted by Nick Johnson | Filed under pubsubhubbub, app-engine, email, datastore, atom
Over the last few years, more and more sites have been providing RSS and Atom feeds. This has been a huge boon both for keeping up to date with content through feed readers, and for programmatically consuming data. There are, inevitably, a few holdouts, though. Notable amongst those holdouts -and particularly relevant to me at the moment - are property listing sites. Few, if any property listing sites provide any sort of Atom or RSS feed for listings, let alone a API of any kind.
To address this, I decided to put together a service for turning other types of notification into Atom feeds. I wanted to make the service general, so it can support many different formats, but the initial target will be email, since most property sites offer email notifications. The requirements for an email to Atom gateway are fairly straightforward:
- Incoming email should be converted to entries in an Atom feed in such a fashion as to be easily interpretable in a feed reader
- As much of the original message and metadata as possible should be preserved in the Atom feed entry
- It should be possible to access the original message if desired
In addition, I had a ...
Hello, developers.
Posted by Nick Johnson | Filed under app-engine, appstats, oldspice
Look at your app. Now back to me. Now back at your app. Now back to me.
Sadly, your app wasn't written by me. But if you used appstats, your app could be fast like mine.
Look down. Back up. Where are you? In your admin console, with the app your app could run like. What's in your hand? Back to me. It's a link from that site you love. Look again. Your site is now doing 20QPS.
Anything is possible when your app runs on App Engine and not PHP. I'm on a Python.
Anyone know a good voice actor?
Modeling relationships in App Engine
Posted by Nick Johnson | Filed under python, app-engine, datastore, relational-modelling
One source of difficulty for people who are used to relational databases - and certain ORMs in particular - is how to handle references and relationships on App Engine. There's two basic questions here: First, what does a relationship entail, in any database system? And second, how do we use them in App Engine?
The nature of relationships
Many ORMs expose multiple 'types' of relationships as first-class entities - one-to-one, one-to-many, and many-to-many. This obscures the fact that, in reality, they are all built on the same building block, that of references. A reference is simply a field of an entity that contains the key of another entity - for example, if a Pet references an Owner, that simply means the Pet has a field that contains the key of its owner.
All relationship types simply devolve to references. A one-to-many relationship is a reference in its simplest form - each Pet has one Owner, and therefore an Owner can have multiple pets, all pointing to it. The Owner is not modified; it relies on the individual Pets naming it as the owner. One-to-one relationships are one-to-many relationships with the additional constraint that there will be only one Pet referencing each Owner; this is ...
Under the hood with App Engine APIs
Posted by Nick Johnson | Filed under python, app-engine, coding, datastore
In the past, I've discussed various details of the way various App Engine APIs work under the hood. If you've used certain tools, such as Appstats, too, you probably already have a basic overview of how the App Engine APIs function. Today, we'll take a closer look at the interface between the App Engine runtime and its APIs and how it works, and learn what that means for the platform.
If you're interested in App Engine purely to write straightforward webapps, you can probably stop reading now. If you're interested in low-level optimisations, or in the platform itself, or you want to write a library or tool that tinkers with the innermost parts of App Engine, then read on!
The generic API interface
Ultimately, every API call comes down to a single generic interface, with 4 arguments: the service name (for example, 'datastore_v3' or 'memcache'), the method name (for example, 'Get' or 'RunQuery'), the request, and the response. The request and response components are both Protocol Buffers, a binary format widely used at Google for exchanging structured data between processes. The specific type of the request and response protocol buffers for an API call depend ...
An experiment in interactive blogging
Posted by Nick Johnson | Filed under blogging
First up, apologies for the recent inactivity here. I've had some pretty tough stuff to deal with in my personal life recently, and that's had to take priority over my blogging schedule. Rest assured that I do intend to return to a regular posting schedule, but in the short term, things may remain a little irregular. That said, there will be a post later this week, so it's not all doom and gloom.
Second, I'm now officially relocating to Sydney! Don't worry, I'll still be working on the App Engine team, but the Google office in Sydney has a substantial App Engine contingent, so I'll no longer be working alone in the Dublin office. It's also much, much closer to home for me, so I may get to see my family more than once a year or so. Finally, the weather is better. I expect to move early November, which will be another excuse for any raggedness in the blogging schedule.
Finally, the main subject of this post: I've decided to try a little experiment. I always have several post ideas in the pipeline, and readers often give me interesting new ...
Newer Older