Thursday, May 20, 2010

asynctools - execute Google App Engine API calls in parallel

asynctools is stewarded by Vendasta Technologies; we're in the App Engine Pod at Google I/O.

Come by for a demo!

asynctools is a library allowing you to execute Google App Engine API calls in parallel. API calls can be mixed together and queued up and then all are kicked off in parallel. Callbacks can be used to operate over completed calls immediately, if desired.

Currently, wrappers for Query and UrlFetch API calls are available. More will come over time.

Additionally, CachedMultiTask provides a mechanism to transparently cache the results of the parallel API calls.

An example invocation kicking off a set of parallel data store queries is as follows:

# set up the async queries
runner = AsyncMultiTask()
for fbuid in facebook_user_ids:
query = db.GqlQuery("SELECT __key__ FROM Account WHERE facebook_id = :1", uid)
runner.append(QueryTask(query, limit=1, client_state=fbuid))

# kick off the work
runner.run()

# peel out the results
for task in runner:
task_result = task.get_result() # will raise any exception that occurred for the given query
print '%s: %s' % (task.client_state, task_result[0])