Google I/O playlist, day 8: Next gen queries

This is the eighth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them. Don't worry, we're nearly done!

First up, apologies for the lack of posts the last few days. I spent the weekend at Hack Camp, so Friday was spent preparing for it, and Monday and Tuesday were spent recovering and catching up!

Today's session is Alfred Fuller's session, Next Gen Queries. If you're interested in the continued evolution of the datastore API, this is a must-watch talk. It's completely language-agnostic, too. If you're looking for better background knowledge, Brett's talk from I/O '09 is well worth watching first, though.

Alfred goes into a lot of detail about some of the exciting improvements you can expect in future iterations of the Datastore API, largely due to his work on extending and improving the ZigZag merge join strategy in App Engine, as well as better support for MultiQuery. As an extra bonus, check out 31:29 for details of an improvement you can take advantage of ...

Google I/O playlist, day 7: Data pipelines with Google App Engine

This is the seventh in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.

Today's session is Brett Slatkin's talk on Data Pipelines with Google App Engine. This is a really fascinating talk, and it's a must-watch for anyone wanting to do advanced, high-throughput processing on App Engine. Brett describes in detail, with working code, a couple of high level concepts that make it possible to do 'eventually consistent' processing on App Engine, with one of the goals being Materialized Views.

The examples are in Python, but the talk will be useful for Java developers too - it's the concepts that are really important here.

Google I/O playlist, day 6: Batch data processing with App Engine

This is the sixth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.

Today's video is Mike Aizatsky's Batch data processing with App Engine, where he describes the recently-released mapper framework for App Engine. I blogged previously about this framework, giving a detailed breakdown of the demo mapreduce used in this very talk!

The mapper API is initially being released for Python, so it'll be mostly of interest to Python users - but Java is coming soon, so it's well worth watching even if you only speak Java.

Google I/O playlist, day 5: Data migration in App Engine

This is the fifth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.

Today's video is Data migration in App Engine, Matthew Blain's talk on new improvements to the bulkloader. I've blogged about the new bulkloader previously, but Matthew's talk goes into a lot more detail.

Matt starts talking about the new configuration format at 6:38, if you want to skip the intro.

Google I/O playlist, day 4: The BigQuery and Prediction APIs

This is the fourth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.

Today's session isBigQuery and Prediction APIs. These are two awesome APIs that I described previously, and you can look forward to some forthcoming posts exploring how they work and what they can be used for.

This is another language-agnostic video - the APIs, by their nature, are pretty indifferent about what language you access them with. They both depend on Google Storage for their storage needs, so you should probably watch that talk first, though.

If you're only interested in one API or the other, the BigQuery talk starts at 6:15, and the Prediction API talk starts at 24:40. The whole talk is definitely worth watching, though.

Have something you'd particularly like to see demonstrated using the Prediction or BigQuery APIs in a future post? Leave a comment!

Google I/O playlist, day 3: What's hot in Java for App Engine

This is the third in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.

Today's video is "What's hot in Java for App Engine" by Don Schwarz and Toby Reyelts. It provides an overview of the first year of Java support on App Engine, focusing on a demo app that shows off a number of features of the Java runtime.

At first glance, this is definitely a talk for the Java programmers amongst us, and it certainly has a lot of content on those lines; the demo shows off to good advantage a number of the App Engine APIs. The secret hidden surprise, though, is something that will be of interest to nearly everyone: Details about the forthcoming channel API. The channel API implements the promised support for Comet on App Engine. For the juicy details, jump to 10:49, and keep watching up to 15:10. There's more details in Moishe's talk on building Real-time Webapps in App Engine, the video for which will be going up later today, and which ...

Google I/O playlist, day 2: Google Storage for Developers

This is the second in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.

Today's video is on Google Storage for Developers. Google Storage for Developers is a new API for storing, serving, and sharing large numbers of files, of nearly any size. While Google Storage isn't directly linked to App Engine, it's likely to be of interest to many App Engine developers. It's also the foundation for the BigQuery and Prediction APIs, which I discussed earlier.

The talk is language-agnostic, so both Python and Java developers will find it of interest. Currently, the only officially supported client library is the Python one, but the protocol is deliberately compatible with other similar services, so the Boto library works as-is, as will other libraries. Further, the API is simple enough that I expect Google-Storage-specific libraries will quickly spring up in most major languages.

Google I/O playlist, day 1: Appstats

With the full videos of the App Engine sessions at I/O now released, those who missed I/O, or, like me, didn't have time to catch all the sessions can now catch up on what they missed out on. The list is quite intimidating, though - there's a lot of content there, and most of us have a limited amount of time to spend watching each day.

With that in mind, over the next couple of weeks, I'm going to be posting session videos, one or two each day, in an order that's hopefully useful and makes sense to people. I'll also explicitly note who'll find each video of interest, and what other videos, if any, you should watch first.

First in the lineup is Guido's Appstats talk. This talk will be of interest to both Python and App Engine developers, at all levels of skill. There's some Python-specific code in the talk, but it mostly applies to both languages.

What's that, you say? You already know about appstats? Even if you do, there's almost certainly something you didn't know in Guido's talk. If you're already familiar ...

Exploring the new mapper API

One of the new features announced at this year's Google I/O is the new mapper library. This library makes it easy to perform bulk operations on your data, such as updating it, deleting it, or transforming/filtering/processing it in some fashion, using the 'map' (and soon, 'reduce') pattern. I'm happy to say that I'm deprecating my own bulkupdate library in favor of it.

The mapper API isn't just limited to mapping over datastore entities, either. You can map over lines in a text file in the blobstore, or over the contents of a zip file in the blobstore. It's even possible to write your own data sources - something we'll cover in a later post. Today, though, I'd like to dissect the demo that was presented at I/O. The demo uses a number of the mapper framework's more sophisticated features, so it's a good one to use to get an idea for how the framework works.

For basic usage, the Getting Started page in the mapper docs is the place to go. If you're interested in seeing something more complex in practice, read on...

The demo at I ...

Google I/O day 2: The Android push messaging API

As I mentioned in my earlier post about the keynote, one of the exciting new releases at I/O is the new Push messaging API for Android. With it, you can push notifications from any webapp to an Android device in real-time - no polling, no messy hacks.

Push messaging is particularly valuable on the Android, because of battery life concerns. Even a single app polling every 5 minutes can cause a substantial drain on the battery - mostly wasted on checking for changes that don't exist. Several Android APIs already use push for sync - including Contacts, CAlendar, and the GMail app. Implementing it yourself, and getting it right, is tough, though - and that's where the new API comes in.

Under the hood, this is implemented as a background service. It starts whenever the network is available, and maintains a connection with the server for new messages. In order to prevent NATs and other network hardware from destroying the connection, and to detect that if it happens, it uses heartbeat messages - but it's been extensively tested and optimized to minimise the frequency of these as much as possible.

To use it, an app on the Android device registers with ...