Google I/O playlist, day 7: Data pipelines with Google App Engine
Posted by Nick Johnson | Filed under app-engine, google-io, pipelines, task-queue, google-io-playlist
This is the seventh in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's session is Brett Slatkin's talk on Data Pipelines with Google App Engine. This is a really fascinating talk, and it's a must-watch for anyone wanting to do advanced, high-throughput processing on App Engine. Brett describes in detail, with working code, a couple of high level concepts that make it possible to do 'eventually consistent' processing on App Engine, with one of the goals being Materialized Views.
The examples are in Python, but the talk will be useful for Java developers too - it's the concepts that are really important here.
Google I/O playlist, day 6: Batch data processing with App Engine
Posted by Nick Johnson | Filed under app-engine, google-io, mapreduce, google-io-playlist
This is the sixth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's video is Mike Aizatsky's Batch data processing with App Engine, where he describes the recently-released mapper framework for App Engine. I blogged previously about this framework, giving a detailed breakdown of the demo mapreduce used in this very talk!
The mapper API is initially being released for Python, so it'll be mostly of interest to Python users - but Java is coming soon, so it's well worth watching even if you only speak Java.
Google I/O playlist, day 5: Data migration in App Engine
Posted by Nick Johnson | Filed under bulkloader, app-engine, google-io, google-io-playlist
This is the fifth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's video is Data migration in App Engine, Matthew Blain's talk on new improvements to the bulkloader. I've blogged about the new bulkloader previously, but Matthew's talk goes into a lot more detail.
Matt starts talking about the new configuration format at 6:38, if you want to skip the intro.
Google I/O playlist, day 4: The BigQuery and Prediction APIs
Posted by Nick Johnson | Filed under google-io, google-storage, prediction-api, bigquery, google-io-playlist
This is the fourth in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's session isBigQuery and Prediction APIs. These are two awesome APIs that I described previously, and you can look forward to some forthcoming posts exploring how they work and what they can be used for.
This is another language-agnostic video - the APIs, by their nature, are pretty indifferent about what language you access them with. They both depend on Google Storage for their storage needs, so you should probably watch that talk first, though.
If you're only interested in one API or the other, the BigQuery talk starts at 6:15, and the Prediction API talk starts at 24:40. The whole talk is definitely worth watching, though.
Have something you'd particularly like to see demonstrated using the Prediction or BigQuery APIs in a future post? Leave a comment!
Google I/O playlist, day 3: What's hot in Java for App Engine
Posted by Nick Johnson | Filed under channel-api, google-io, java, google-io-playlist
This is the third in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's video is "What's hot in Java for App Engine" by Don Schwarz and Toby Reyelts. It provides an overview of the first year of Java support on App Engine, focusing on a demo app that shows off a number of features of the Java runtime.
At first glance, this is definitely a talk for the Java programmers amongst us, and it certainly has a lot of content on those lines; the demo shows off to good advantage a number of the App Engine APIs. The secret hidden surprise, though, is something that will be of interest to nearly everyone: Details about the forthcoming channel API. The channel API implements the promised support for Comet on App Engine. For the juicy details, jump to 10:49, and keep watching up to 15:10. There's more details in Moishe's talk on building Real-time Webapps in App Engine, the video for which will be going up later today, and which ...
Google I/O playlist, day 2: Google Storage for Developers
Posted by Nick Johnson | Filed under google-io, google-storage, google-io-playlist
This is the second in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's video is on Google Storage for Developers. Google Storage for Developers is a new API for storing, serving, and sharing large numbers of files, of nearly any size. While Google Storage isn't directly linked to App Engine, it's likely to be of interest to many App Engine developers. It's also the foundation for the BigQuery and Prediction APIs, which I discussed earlier.
The talk is language-agnostic, so both Python and Java developers will find it of interest. Currently, the only officially supported client library is the Python one, but the protocol is deliberately compatible with other similar services, so the Boto library works as-is, as will other libraries. Further, the API is simple enough that I expect Google-Storage-specific libraries will quickly spring up in most major languages.
Google I/O playlist, day 1: Appstats
Posted by Nick Johnson | Filed under app-engine, google-io, appstats, google-io-playlist
With the full videos of the App Engine sessions at I/O now released, those who missed I/O, or, like me, didn't have time to catch all the sessions can now catch up on what they missed out on. The list is quite intimidating, though - there's a lot of content there, and most of us have a limited amount of time to spend watching each day.
With that in mind, over the next couple of weeks, I'm going to be posting session videos, one or two each day, in an order that's hopefully useful and makes sense to people. I'll also explicitly note who'll find each video of interest, and what other videos, if any, you should watch first.
First in the lineup is Guido's Appstats talk. This talk will be of interest to both Python and App Engine developers, at all levels of skill. There's some Python-specific code in the talk, but it mostly applies to both languages.
What's that, you say? You already know about appstats? Even if you do, there's almost certainly something you didn't know in Guido's talk. If you're already familiar ...
The wonders of HDMI
Posted by Nick Johnson | Filed under home-theater, tech, projector, hd
Recently, we embarked upon an update of our 'media center' / 'home theater' setup. Our projector had reached the end of its 2000 hour bulb life, and since it was a cheap projector to start, a replacement bulb would've cost nearly as much as the projector itself did a couple of years ago. Also, it's become extremely unreliable - 9 times out of 10, pressing the power button results in a brief flash of light from the bulb, followed by nothing. Plus, we promised ourselves that when the bulb ran out, we'd buy an HD projector, as they ought to be affordable by then.
Well, that time has come, and we've done the upgrade. Since we were upgrading to HD, it seemed necessary to get an AV receiver that supports HDMI to replace our 2-channel amplifier and let us switch audio and video at the same time. And if we have that, we may as well get a blu-ray player, too - after all, they're pretty cheap now.
Here's our new setup:
Going by the letters, we have:
- Our HTPC media box (Existing)
- Netgear XAVB1004 powerline networking switch (Existing)
- Philips BD3000/05 Blu-ray player (New)
- Yamaha RX-V367 ...
Using OpenID authentication on App Engine
Posted by Nick Johnson | Filed under python, openid, app-engine, coding, clickpass
With the release of SDK 1.3.4, preliminary support is available for native OpenID authentication in App Engine. Today, we'll demonstrate how to use the new OpenID support in your app.
Edit: There's now an official article on OpenID on App Engine!
The first step in setting up OpenID authentication is to change your app's authentication settings. Log in to the admin console, select your app, and go to "Application Settings". There, you can pull down the "Authentication Options" box, and select "(Experimental) Federated Login".
Once you've enabled OpenID authentication for your app, a few things change:
- URLs generated by create_login_url without a federated_identity parameter specified will redirect to the OpenID login page for Google Accounts.
- URLs that are protected by "login: required" in app.yaml or web.xml will result in a redirect to the path "/_ah/login_required", with a "continue" parameter of the page originally fetched. This allows you to provide your own openid login page.
- URLs generated by create_login_url with a federated_identity provider will redirect to the specified provider.
In order to make best use of this functionality, here's what we'll do:
- Provide an OpenID login page on /_ah/login_required ...
Exploring the new mapper API
Posted by Nick Johnson | Filed under app-engine, google-io, coding, mapreduce
One of the new features announced at this year's Google I/O is the new mapper library. This library makes it easy to perform bulk operations on your data, such as updating it, deleting it, or transforming/filtering/processing it in some fashion, using the 'map' (and soon, 'reduce') pattern. I'm happy to say that I'm deprecating my own bulkupdate library in favor of it.
The mapper API isn't just limited to mapping over datastore entities, either. You can map over lines in a text file in the blobstore, or over the contents of a zip file in the blobstore. It's even possible to write your own data sources - something we'll cover in a later post. Today, though, I'd like to dissect the demo that was presented at I/O. The demo uses a number of the mapper framework's more sophisticated features, so it's a good one to use to get an idea for how the framework works.
For basic usage, the Getting Started page in the mapper docs is the place to go. If you're interested in seeing something more complex in practice, read on...
The demo at I ...
Newer Older