Storage options on App Engine
Posted by Nick Johnson | Filed under app-engine, memcache, storage, datastore, task-queue
App Engine provides a number of ways for your app to store data. Some, such as the datastore, are well known, but others are less so, and all of them have different characteristics. This article is intended to enumerate the different options, and describe the pros and cons of each, so you can make more informed decisions about how to store your data.
Datastore
The best known, most widely used, and most versatile storage option is, of course, the datastore. The datastore is App Engine's non-relational database, and it provides robust, durable storage, as well as providing the most flexibility in how your data is stored, retrieved, and manipulated.
Pros
- Durable - data stored in the datastore is permanent.
- Read-write - apps can both read and write datastore data, and the datastore provides transaction mechanisms to enforce integrity.
- Globally consistent - all instances of an app have the same view of the datastore.
- Flexible - queries and indexing provide many ways to query and retrieve data.
Cons
- Latency - because the datastore stores data on disk and provides reliability guarantees, writes need to wait until data is confirmed to be stored before returning, and reads often have to fetch data from disk.
Memcache
Memcache ...
Google I/O playlist, day 7: Data pipelines with Google App Engine
Posted by Nick Johnson | Filed under app-engine, google-io, pipelines, task-queue, google-io-playlist
This is the seventh in a series of posts providing a day-by-day playlist to help break up the Google I/O session videos - specifically the App Engine ones - into manageable chunks for those that haven't seen them.
Today's session is Brett Slatkin's talk on Data Pipelines with Google App Engine. This is a really fascinating talk, and it's a must-watch for anyone wanting to do advanced, high-throughput processing on App Engine. Brett describes in detail, with working code, a couple of high level concepts that make it possible to do 'eventually consistent' processing on App Engine, with one of the goals being Materialized Views.
The examples are in Python, but the talk will be useful for Java developers too - it's the concepts that are really important here.