Google I/O day 2: The Android push messaging API
Posted by Nick Johnson | Filed under push-messaging, android, google-io, app-engine
As I mentioned in my earlier post about the keynote, one of the exciting new releases at I/O is the new Push messaging API for Android. With it, you can push notifications from any webapp to an Android device in real-time - no polling, no messy hacks.
Push messaging is particularly valuable on the Android, because of battery life concerns. Even a single app polling every 5 minutes can cause a substantial drain on the battery - mostly wasted on checking for changes that don't exist. Several Android APIs already use push for sync - including Contacts, CAlendar, and the GMail app. Implementing it yourself, and getting it right, is tough, though - and that's where the new API comes in.
Under the hood, this is implemented as a background service. It starts whenever the network is available, and maintains a connection with the server for new messages. In order to prevent NATs and other network hardware from destroying the connection, and to detect that if it happens, it uses heartbeat messages - but it's been extensively tested and optimized to minimise the frequency of these as much as possible.
To use it, an app on the Android device registers with ...
Google I/O day 2: BigQuery and Prediction APIs
Posted by Nick Johnson | Filed under google-io, prediction-api, bigquery, google-storage
First up this morning on the App Engine track is the BigQuery and Prediction APIs talk.
BigQuery
First up is BigQuery. BigQuery is a new API that lets you make use of Google's infrastructure for performing queries and analysis over large collections of read only data. It's designed to scale to massive datasets, and integrates well with App Engine and other platforms.
To use it, you start by uploading your data to the new Google Storage service. Then, you import it into BigQuery tables, and you can run queries on those tables. Despite the fact that it handles billions of rows of data, there's no need to explicitly define indexes, or to shard your data.
The syntax used to query should be familiar: It's based on SQL, and is extremely flexible. Using the example of the database of all Wikipedia revisions, getting the 5 most edited titles is as simple as:
SELECT TOP(title, 5), COUNT(*) FROM [bigquery.test.001/tables/wikipedia] WHERE wp_namespace = 0;
The speed has to be seen to be believed - response times from under a second to a few seconds for hundreds of millions to tens of billions of rows - seemingly regardless ...
I/O Day 2
Posted by Nick Johnson | Filed under app-engine, google-io, android
It's day 2 at I/O!
The keynote was packed with Android stuff today - see the official blog post for the awesome details.
One in particular stands out for App Engine users: the new Cloud to Device messaging service. This makes it possible for an app in the cloud to send an Intent directly to an Android device, asynchronously. That means that you can send a notification to an android device, and have the device receive it immediately: no polling, no hacks.
Obviously, App Engine is a shoo-in for the cloud portion of this. Previously, I blogged about authenticating against App Engine from your Android app - look out for a blog post soon on how to do the reverse, and send notifications from your App Engine app back to your Android device.
The keynote's over, but there's still plenty of new App Engine content to come in the talks today. I'll be writing up short posts on the new APIs and features during the day.
I won't get to everything - I can only be in one talk at a time, and I have Office Hours to staff as well - but I'll attempt to cover ...
Help me name my distributed publishing system
Posted by Nick Johnson | Filed under censorship-resistant, dht, coding
A few days ago, I posted an outline for a distributed publishing system. I asked for name suggestions, and people have provided more than a few!
Here's the shortlist. Please vote on the one you think works best, or suggest your own.
I/O Day 1
Posted by Nick Johnson | Filed under app-engine, google-io, enterprise-app-engine
Piles of awesome new stuff at today's Keynote, including WebM - based on the on2 codec, and the Chrome Web Store. In this post, I'll focus on what's new and interesting in App Engine, though.
The big news for App Engine, of course, is the release of App Engine for Business. App Engine for Business lets businesses build and maintain apps using the App Engine architecture, but within a controlled environment just for their corporate apps. See the blog post for full details, including centralized administration, SLA, a new billing model, and coming soon, support for SQL databases!
Also, Bloggart was briefly featured on the big screen. Woohoo!
I'm sure some of you have been waiting for something exactly like this, and I'll be blogging about some of the details in the near future. If there's something you'd really like to see me write about, let me know in the comments.
I'll be updating this post with more news from I/O as the day goes on. If you're here at the event, don't forget to stop by the Office Hours and pay me a visit.
Edit: I've spent most ...
More I/O prep
Posted by Nick Johnson | Filed under app-engine, google-io
Once again, I'm afraid I have to tender my apologies for the lack of a decent blog post today, as I find myself suddenly swamped with last-minute I/O prep.
On Wednesday and Friday, though, I'll be providing recaps and details about some of the new App Engine features we're announcing at I/O, for those of you that couldn't make it. After that, I'll be going into more detail with howto posts and other material that goes into depth about some of the new stuff.
In the meantime, tell me what you're most excited about! Obviously it's hard to ask about the features that haven't been announced yet, but for those that are already out, let me know what you're interested in hearing about in the comments!
Behind the scenes with remote_api
Posted by Nick Johnson | Filed under python, remote_api, app-engine, internals
I've discussed remote_api in passing many times before on this blog, but never gone into detail about how it works, and the options you have for customizing it. Today, we'll remedy that, by taking a close look at its operation.
You may be wondering why anyone would want to customize remote_api - it seems like a fairly straightforward service, right? There are two main reasons you might want to do some degree of customization:
- You're providing a software-as-a-service solution, and need to provide remote_api access to your customers, but want to limit what they can do.
- You want to expose an API of your own via remote_api.
The first of these use-cases is particularly apt in the face of this nasty hack, which makes it possible to download a Python app's source if both the remote_api and deferred handlers are installed (and the user is an admin). You may want to use both of these libraries, but still keep your source to yourself. The second use-case is more complicated, and we'll only touch on it in passing.
How remote_api works
remote_api has two components, the client (otherwise known as the 'stub') and the server (otherwise known ...
No post today - preparing for I/O
Posted by Nick Johnson | Filed under app-engine, google-io
Unfortunately, a combination of preparation for I/O and jetlag leave me with little time to write a proper post today. For all of you who are going to come to I/O, though: It's going to be really, really awesome. For those of you that aren't: Sorry! At least you'll get to see the presentations on the web. :)
In the meantime, consider this an open thread for suggestions for future posts. Do you have something - App Engine related or otherwise - that you'd like to see me write about? Something I've written about in the past that you'd like to see me expand on? Let me know in the comments and I'll see what I can do.
App Engine Cookbook: On-demand Cron Jobs
Posted by Nick Johnson | Filed under python, tech, app-engine, cookbook, coding
Today's post is, by necessity, a brief one. I'm travelling to San Francisco for I/O at the moment, and my flight was delayed so much I missed my connection in Atlanta and had to stay the night; in fact, I'm writing and posting this from the plane, using the onboard WiFi!
In a previous post, I introduced a recipe for high concurrency counters, which used a technique that I believe deserves its own post, since it's a useful pattern on its own. That technique is what I'm calling "On-demand Cron Jobs"
It's not at all uncommon for apps to have a need to do periodic updates at intervals, where the individual updates are small, and may even shift in time. One example is deleting or modifying any entry that hasn't been modified in the last day. In apps that need to do this, it's not uncommon to see a cron job like the following:
cron: - description: Clean up old data url: /tasks/cleanup schedule: every 1 minute
This works, but it potentially consumes a significant amount of resources checking repeatedly if there's anything to clean up. Using the task queue ...
Building a censorship resistant publishing system
Posted by Nick Johnson | Filed under tech, censorship-resistant, dht, coding
I've had an idea related to censorship resistant publishing kicking around in my head for some time now, and it seems like it's about time I got it written down somewhere, for consideration and criticism. Part of my motivation is that I'm intending to snatch some spare time while I'm on the plane to the US this weekend (to attend I/O) to have a go at implementing a basic version of it.
In a nutshell, I have a design for what I believe would be a fairly robust censorship resistant publishing system, based on a DHT, and integrating fully with the web. Content published using this system would be available in exactly the same fashion as a regular website, which strikes me as a major advantage over many other proposals for similar systems.
The system consists of several layers, which I'll tackle in order:
- Document storage and retrieval
- Name resolution for documents within a 'site'
- External access
- Name resolution for sites
Document storage and retrieval
The lowest level of the system is also the simplest: That of storing and retrieving documents. This layer of the system acts more or less exactly like a regular ...
Newer Older