End Point

News

Welcome to End Point's blog

Ongoing observations by End Point people.

CouchDB pagination with couchdb-python

I've been working with couchdb-python v0.8 to meet some of a project's CouchDB needs and ran into an unfortunate shortcoming.

Retrieving database rows is as easy as:

for row in db.view(mapping_function):
    print row.key

however, all the rows will be loaded into memory. Now, for small databases this is not a problem but lucky me is dealing with a 2.5 million+ document database. Long story short: kaboom!

There is an excellent blog post by Marcus Brinkmann that details the memory issue and also provides a pagination solution. However, couchdb-python v0.9 (released 2013-04-25) does provide its own solution: the iterview method. If you've been able to manage small databases with v0.8 but are anticipating larger and larger databases, be sure to upgrade!

No comments: