Bulk updates with cursors
Posted by Nick Johnson | Filed under app-engine, python, coding, deferred, tech, cursors
Last week, I blogged about cursors, a new feature in version 1.3.1 of the App Engine SDK. Today, I'm going to demonstrate a practical use for them: Bulk datastore updates.
In both the Remote API and deferred articles, I used a (perhaps poorly named) 'mapper' class as an example of ways to use these libraries. In neither case was the class intended to be anything other than a sample use case for the library, but nevertheless, people have used the examples in production. The introduction of cursors provides a prime opportunity to introduce a more robust, yet simpler, version of the bulk updater concept.
First, let's define a few requirements for our bulk updater:
- Support for any query for which a cursor can be obtained
- Handles failure of individual updates gracefully
- Can fail the whole update process if enough errors are encountered
- Handles timeout errors, service unavailability, etc, transparently
- Can report completion to admins
As in the Remote API and Deferred articles, we'll implement the updater as an abstract class, which individual updater implementations should subclass. Here's the basic interface:
import logging import time from google.appengine.api import mail from google.appengine.ext ...