CouchDB Externals in Python

I was on the CouchDB IRC channel last night getting some guidance on using externals in the latest couch (I got the source from svn last night). I thought I’d post my example code for you.

Here is the local.ini file

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings

; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.


[couchdb]
;max_document_size = 4294967296 ; bytes

[httpd]
;port = 5984
bind_address = 0.0.0.0

[log]
;level = info

[httpd_db_handlers]
_test = {couch_httpd_external, handle_external_req, <<"anystring">>}

[external]
anystring = /home/tim/git/couchish/couchish/external.py

[update_notification]
;unique notifier name=/home/tim/git/couchish/couchish/notifier.py

I’ve used anystring here to show that it is just a reference between the http handler and the external

The script is taken from the CouchDB wiki (with a couple of modifications) and basically echos queries back to you. Don’t forget to make sure it’s executable by the couchdb user (I had a problem where the couchdb user’s home directory didn’t exist and hence the simplejson egg couldn’t unpack). su - to the couchdb user and run the script to double check.

#! /usr/bin/env python

import sys
import simplejson

_logfile = file('/tmp/external.log', 'a')

def log(msg):

    _logfile.write('%s\n'%(msg,))
    _logfile.flush()

def requests():
    # `for line in sys.stdin` won't work here
    line = sys.stdin.readline()

    while line:
        data = simplejson.loads(line)

        log('data: %s'%data)
        yield data
        line = sys.stdin.readline()

def respond(code=200, data={}, headers={}):

    sys.stdout.write("%s\n" % simplejson.dumps({"code": code, "json": data, "headers": headers}))

    sys.stdout.flush()

def main():
    for req in requests():

        respond(data={"qs": req["query"]})

if __name__ == "__main__":

    main()

test this out by using curl to send a query to the /mydbname/_ test url

curl http://localhost:5984/mydbname/_test?query=foo

Refer to http://wiki.apache.org/couchdb/ExternalProcesses for more info about the data going in and coming out..

CouchDB Notification in Python

I’m playing with trying to get reference updates in CouchDB and as part of this I had to get notifications working.. Just to save anybody else the same pain here are the steps I took so far.

Here is the local.ini file

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes

[httpd]
;port = 5984
bind_address = 0.0.0.0

[log]
;level = info

[update_notification]
unique notifier name=/home/tim/git/couchish/couchish/notifier.py

The notifier gets called when couchdb starts and then couchdb streams updates to it line by line.

Here is the notifier script.. It has a few lines to check that things are ‘OK’.. you should make sure it’s executable by couchdb (it’s worth su’ing to couchdb and checking you can run the script and what happens if you <code>echo ‘foo’ | python {notifier script}</code>

#!/usr/bin/env python

import sys, time

import logging as log
log.basicConfig(level=log.INFO, filename='/tmp/couch-updater.log')


def notifications():
    simplejson_imported = False
    while True:
        line = sys.stdin.readline()
        log.info('read %s'%line)
        if not line:
            raise StopIteration()
        if not simplejson_imported:
            try:
                import simplejson
                simplejson_imported = True
            except ImportError:
                yield 'Bad Import'
        try:
            yield simplejson.loads(line)
        except Exception, e:
            yield 'Failed to Parse line : "%s"'%line

def main(args):
    for notification in notifications():
        log.info("notification: %r" % (notification,))
if __name__ == '__main__':
    try:
        args = sys.argv[1:]
        log.info("Started with args: %r"%(args,))
        main(args)
        log.info("Shutdown normally")
    except KeyboardInterrupt:
        log.info("Shutdown with Ctrl+C")
    except Exception, e:
        log.info("Shutdown by exception: %r"%(e,))

My next step is to get the script to remember the last seq no and to parse the list of individual updates to apply an referential referencing.

Starting Development

Well not really. Development is ongoing and this is just a start of an attempt to make notes on what I'm working with ..

Just a quick summary of what I'm up to ..

CouchDB Document Admin

Simple to setup schema, validation and forms that backend onto CouchDB

Restish

A minimal rest oriented wsgi 'framework'

Websites

A website for a photographer friend and for a photography company

Formish/Schemaish/Validatish/Convertish

Schema and Form tools

MakoCSS

a programmatic way of managing CSS resources

At the moment the CouchDB stuff is the most intersting - CouchDB could well be the storage of the future.. More about that later