Under the hood with App Engine APIs
Posted by Nick Johnson | Filed under python, app-engine, coding, datastore
In the past, I've discussed various details of the way various App Engine APIs work under the hood. If you've used certain tools, such as Appstats, too, you probably already have a basic overview of how the App Engine APIs function. Today, we'll take a closer look at the interface between the App Engine runtime and its APIs and how it works, and learn what that means for the platform.
If you're interested in App Engine purely to write straightforward webapps, you can probably stop reading now. If you're interested in low-level optimisations, or in the platform itself, or you want to write a library or tool that tinkers with the innermost parts of App Engine, then read on!
The generic API interface
Ultimately, every API call comes down to a single generic interface, with 4 arguments: the service name (for example, 'datastore_v3' or 'memcache'), the method name (for example, 'Get' or 'RunQuery'), the request, and the response. The request and response components are both Protocol Buffers, a binary format widely used at Google for exchanging structured data between processes. The specific type of the request and response protocol buffers for an API call depend on the method being invoked. When an API call is made, the request Protocol Buffer is populated with the data being sent in the request, while the response Protocol Buffer is left empty, waiting to be filled by the data returned in the API call response.
API calls are made by passing the four values described above to a 'dispatch' function. In Python, this role is filled by the apiproxy_stub_map module. This module is responsible for maintaining a mapping between service name - the first of the parameters described above - and a 'stub' that handles that service. In the SDK, that mapping is established by creating 'local' or 'stub' implementations of the various APIs, and registering them with the module. In production, interfaces to the 'real' APIs are passed to this module during the startup process, before your application code is run. In this fashion, code that calls the APIs never has to care what particular implementation is backing a given API; nor does it have to know if the API call is being handled locally, or serialized and sent to another machine.
Once the dispatch function has found the appropriate stub for the API being invoked, it sends the API call to it. What happens here depends entirely on the API and the deployment environment, but in production what generally happens is this: the request Protocol Buffer is serialized into binary data, which is then sent to the server(s) responsible for handling this particular API. For example, datastore calls are serialized and sent to the Datastore service. The service then deserializes the request, executes it, creates a response object, serializes that, and sends it back to the stub that originated the call. Finally, the stub deserializes the response into the response Protocol Buffer kindly provided by the caller, and returns.
As an aside, you may be wondering why it's necessary to pass a response Protocol Buffer as part of the API call. This is because the Protocol Buffer format doesn't provide any way to distinguish between different types of Protocol Buffer; it's presumed that you know what type of message you're going to receive. Thus, it's necessary to provide a 'container' that understands how to deserialize the response when it's received.
Let's see an example of how this all works by making a low-level datastore call of our own - a straightforward get operation:
from google.appengine.datastore import datastore_pb
from google.appengine.api import apiproxy_stub_map
def do_get():
request = datastore_pb.GetRequest()
key = request.add_key()
key.set_app(os.environ['APPLICATION_ID'])
pathel = key.mutable_path().add_element()
pathel.set_type('TestKind')
pathel.set_name('test')
response = datastore_pb.GetResponse()
apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', request, response)
return str(response)
Verbose, right? Especially compared to the high-level way to do it - TestKind.get_by_key_name('test')! You should be able to see all the important components we described, though: we construct request and response Protocol Buffers, we populate the request PB with the relevant information, in this case the key of the entity we want to fetch, and then we call apiproxy_stub_map.MakeSyncCall to do the actual RPC. Once that call returns, the response object is populated, as we can see by looking at its string representation:
Entity {
entity <
key <
app: "deferredtest"
path <
Element {
type: "TestKind"
name: "test"
}
>
>
entity_group <
Element {
type: "TestKind"
name: "test"
}
>
property <
name: "test"
value <
stringValue: "foo"
>
multiple: false
>
>
}
Every RPC call for every API uses this same basic pattern internally - all that differs are the nature of the parameters in the request and response objects.
Asynchronous Calls
The process described above applies to synchronous API calls - those where we wait for the response before doing anything else. App Engine also supports asynchronous API calls, though. In an Asynchronous call, we send the API call to the stub, which returns immediately, without waiting for the response. Then, we either ask for the response at a later time (waiting for it if necessary), or we provide a callback function which will be invoked when the response is received.
As of the time of writing, only a few APIs support asynchronous invocations; notably the URLFetch API, where it is extremely useful for fetching multiple pages in parallel. The mechanics behind asynchronous APIs are the same for every API, though - it's simply a matter of library support for asynchronous calls. An API like urlfetch is easily adapted for asynchronous operation, but other, more complex APIs provide more challenges.
Let's see what it takes to convert our synchronous example above into an asynchronous one. The changes from the previous example are highlighted in bold:
from google.appengine.datastore import datastore_pb
from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore
def do_async_get():
request = datastore_pb.GetRequest()
key = request.add_key()
key.set_app(os.environ['APPLICATION_ID'])
pathel = key.mutable_path().add_element()
pathel.set_type('TestKind')
pathel.set_name('test')
response = datastore_pb.GetResponse()
rpc = datastore.CreateRPC()
rpc.make_call('Get', request, response)
return rpc, response
All that's changed here is that we're now constructing an RPC object - one specific to the datastore interface, in this case - and calling 'make_call' on that, instead of MakeAsyncCall. Then, we return the RPC object and the response PB.
Since this is an asynchronous call, the call hasn't actually completed when we return the RPC object. There's a number of ways we can handle the asynchronous response. We could have passed a callback function to the CreateRPC() method, for example, or we could call .check_success() on the RPC to have it wait until the call completes. We'll do the latter in this case, since it's simpler to demonstrate. Here's a simple test harness for our new function:
TestKind(key_name='test', test='foo').put()
self.response.headers['Content-Type'] = 'text/plain'
rpc, response = do_async_get()
self.response.out.write("RPC status is %s\n" % rpc.state)
rpc.check_success()
self.response.out.write("RPC status is %s\n" % rpc.state)
self.response.out.write(str(response))
And here's its output:
RPC status is 1
RPC status is 2
Entity {
entity <
key <
app: "deferredtest"
path <
Element {
type: "TestKind"
name: "test"
}
>
>
entity_group <
Element {
type: "TestKind"
name: "test"
}
>
property <
name: "test"
value <
stringValue: "foo"
>
multiple: false
>
>
}
The status constants are defined in google.appengine.api.apiproxy_rpc - in this case, 1 is 'running', and 2 is 'finishing', which demonstrates that the RPC is indeed executing asynchronously! The actual result is, of course, the same.
Now that you know how RPCs work at the lowest level, and how to make an asynchronous call, any number of possibilites are available to the determined programmer. Who'll be the first to write a new interface to App Engine's APIs that uses futures, similar to frameworks like Twisted?
Finally, don't forget to check out the feedback widget on the right, and vote for ideas for future posts that interest you - or suggest your own.
Previous Post Next Post