Code Kuno

Download the following from http://hackage.haskell.org/.

I recently started an open source Python project called redisco. From its README:

Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm and its design and code are loosely based on Ohm and the Django ORM. It is built on top of redis-py. It includes container classes that allow easier access to Redis sets, lists, and sorted sets.

Why?

Short answer: I like Redis and I want to code in Python.

A bit longer answer: I’ve been hacking with Redis and Ohm since last month and I found it lighter than using SQL databases or Mongodb right from its installation to actually using it. It’s data structure approach appeals to me. Sets, lists, dicts, sorted set — using them feels more natural as compared to tables and joins. The Ruby gem Ohm provides a nice interface to Redis. I wanted to have something similar in Python. I looked around, I saw Redish which offers some nice abstractions, but after reading through its code looking specifically for model abstraction, I decided it didn’t have what I wanted. And with so much time in my hands, I rolled out my own. (Another reason for starting this project is that I wanted to learn more about and code more in Python.)

Simple Models

Redisco allows you to code model classes like you would in Django or AppEngine.

class Person(models.Model):
    name = models.Attribute(required=True)
    created_at = models.DateTimeField(auto_now_add=True)
    fave_colors = models.ListField(str)

Saving an object to Redis automatically creates indexes that are used when querying.

>>> p = Person(name="Ippo", fave_colors=["Red", "White"])
>>> p.save()
True

Objects can be queried via the Model.objects classmethod:

Person.objects.all()
Person.objects.filter(name='Ippo')
Person.objects.filter(name='Ippo').first()
Person.objects.all().order('name')
Person.objects.filter(fave_colors='Red')

Redisco has a limited support for queries involving ranges - it can only filter fields that are numeric, i.e. DateField, DateTimeField, IntegerField, and FloatField.

Person.objects.zfilter(created_at__lt=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__gte=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__in=(datetime(2010, 4, 20, 5, 2, 0), datetime(2010, 5, 1)))

Other limitations

redisco.models does not offer customizable managers and inheritance.

Abstractions to Python data structures

Lists, sets, sorted sets, dicts - this sort of things are easily persisted in Redis. But the main reason they are provided by redisco is that they are dependencies of the Model class. Making them available as well seems reasonable.

>>> from redisco.containers import List
>>> l = List('alpha')
>>> l.append('a')
>>> l.append('b')
>>> l.append('c')
>>> 'a' in l
True

Installation

The package is published in PyPI, so you can

pip install redisco

Or, you can clone the git repository:

git clone http://github.com/iamteem/redisco.git

Prerequisites: Redis 2.0.0 RC1 and the latest version of redis-py on Github.

Credits

Much credit goes to the Ohm team, whose work in Ohm is the primary inspiration and basis of the project.

Notes

Because redisco is relatively new, consider it alpha software and not production-ready. I’m currently testing it in a simple web toy that stores tweets from Twitter tracked using the tweetstream and using Hashes to store statistics.

Invitation

The project is MIT-licensed so anyone can do anything with it. Use it, fork it on Github, read the source code, and let me know your opinions and reactions. Thanks.

I thought I wanted to share some of the tools I use these days. In no particular order:

Tornado. A fast Python non-blocking web server. I’ve been hacking two small apps that will be rolled out in the next two weeks. Tornado is a good match for the two apps because the apps access external web services (i.e. Flickr API), and one app has Comet features via Ajax and long polling — both of these features are easily solved in Tornado. Since its code base is really small, I had no trouble looking around when I hit some roadblock.

Python 2.6. Recently, I updated my Python version to 2.6. Multiprocessing FTW.

pip and virtualenv. When experimenting with Python packages, this is THE way.

Homebrew. A package manager for OSX. Recently, I updated the my Macports system and several of the packages in it. The update broke my other tools (Ruby gems and others) because their dependencies (dylibs etc) are no longer where they expected them to be. I was done with Macports. So after looking around, I found Homebrew. It’s quite convenient and so far I have no issues with it. My impression is it’s pragmatic. E.g. doing brew install mercurial doesn’t install the mercurial package but recommends that you do brew install pip then pip install mercurial. It makes sense. Also, brew system is built in Ruby so I had no trouble looking at the package formulae.

Redis. A very fast key value store. Using redis is fun. It takes a much more (personally) enjoyable approach to storing data — using constructs such as lists and sets instead of rows and sql. I say enjoyable because with redis, the way your data is stored is close to the way you actually model and code your data. I bookmarked its command reference for easy access.

rvm. Ruby Version Manager. Playing around with multiple versions of Ruby without pain. Gemsets are really wonderful. My usual setup is one gemset per web project (be it Rails or sinatra).

Ruby 1.9. Saying goodbye to Ruby 1.8 is not so hard anymore since rvm. But I compiled Ruby 1.9 from source and it is now the default Ruby in my system.

Sinatra. Moving away from Rails is always an option. And could also a very liberating decision. With Sinatra, I write code my way (at least, I think that’s how it is). There are frameworks built on Sinatra (e.g. monk, padrino) so if ever you don’t want to use Rails for some reason, I’d suggest Sinatra.

Monk. A glue framework built on Sinatra.

Nodejs. Wohow. V8 JS not on the browser. I experimented on it a while back since I <3 programming javascript. I wish I can build some crazy service using node soon.

Mercurial. I use this when I’m in Python mode.

GHC. I’ve been reading on Haskell in the past two weeks, and I find the experience fun, exciting, and mind bending.

ZeroMQ. Fastest. Messaging. Ever. For pub/sub, I used ØMQ and its Python binding.

After removing macports and installing homebrew, I encountered a problem when I installed pycurl (a dependency of tornado). pycurl requires a newer version (v7.19) of libcurl than what my system (osx 10.5.8) has (v7.16). So I grabbed the curl source and installed it (./configure --with-ca-bundle=/usr/share/curl/curl-ca-bundle.crt && make && make install). With the newer curl installed in /usr/local, I tried installing pycurl again, and it failed. Checking my $PATH, I found out that /usr/local/bin is the last entry. I read the /etc/profile and the /usr/libexec/path_helper and attempted to reorganize the PATH but instead decided to remove /usr/local/bin from /etc/paths and manually add it in /etc/profile.

videolearning:

2 day class, 7 videos. Introduction to Python, that lets you work more quickly and integrate your systems more effectively.

When installing mysql gem in Ruby 1.9.1 (using rvm), the process keeps on failing even when specifying --with-mysql-dir=/usr/local/mysql etc. Apparently the solution is to add the /usr/local/mysql to $PATH before issuing the gem install command.

Fail fast, fail open.

http://github.com/lardawge/swfupload-rails-authentication

Rails 3 is going to require at least Ruby 1.8.7 so I decided to install Ruby Enterprise Edition on my Mac. I have OSX Leopard (10.5.8) and use macports. After installing rvm, I encountered the same problem whenever I executed my web server in my Rails app:

/usr/local/rvm/ree-1.8.7-2009.10/lib/ruby/1.8/openssl/ssl.rb:31: [BUG] Bus Error
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9.8.0], MBARI 0x8770, Ruby Enterprise Edition 2009.10

The culprit is that I have two openssl installations and the eventmachine gem isn’t compiled with the same openssl as the my REE install. This issue was reported here.

So the solution is either to recompile REE to use Macport’s openssl or recompile eventmachine to use the system’s openssl lib. I chose to recompile REE.

sudo rvm install ree --configure --enable-shared=true,--with-openssl-dir=/opt/local --debug