I recently started an open source Python project called redisco. From its README:
Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm and its design and code are loosely based on Ohm and the Django ORM. It is built on top of redis-py. It includes container classes that allow easier access to Redis sets, lists, and sorted sets.
Short answer: I like Redis and I want to code in Python.
A bit longer answer: I’ve been hacking with Redis and Ohm since last month and I found it lighter than using SQL databases or Mongodb right from its installation to actually using it. It’s data structure approach appeals to me. Sets, lists, dicts, sorted set — using them feels more natural as compared to tables and joins. The Ruby gem Ohm provides a nice interface to Redis. I wanted to have something similar in Python. I looked around, I saw Redish which offers some nice abstractions, but after reading through its code looking specifically for model abstraction, I decided it didn’t have what I wanted. And with so much time in my hands, I rolled out my own. (Another reason for starting this project is that I wanted to learn more about and code more in Python.)
Redisco allows you to code model classes like you would in Django or AppEngine.
class Person(models.Model): name = models.Attribute(required=True) created_at = models.DateTimeField(auto_now_add=True) fave_colors = models.ListField(str)
Saving an object to Redis automatically creates indexes that are used when querying.
>>> p = Person(name="Ippo", fave_colors=["Red", "White"]) >>> p.save() True
Objects can be queried via the Model.objects classmethod:
Person.objects.all() Person.objects.filter(name='Ippo') Person.objects.filter(name='Ippo').first() Person.objects.all().order('name') Person.objects.filter(fave_colors='Red')
Redisco has a limited support for queries involving ranges - it can only filter fields that are numeric, i.e. DateField, DateTimeField, IntegerField, and FloatField.
Person.objects.zfilter(created_at__lt=datetime(2010, 4, 20, 5, 2, 0)) Person.objects.zfilter(created_at__gte=datetime(2010, 4, 20, 5, 2, 0)) Person.objects.zfilter(created_at__in=(datetime(2010, 4, 20, 5, 2, 0), datetime(2010, 5, 1)))
redisco.models does not offer customizable managers and inheritance.
Abstractions to Python data structures
Lists, sets, sorted sets, dicts - this sort of things are easily persisted in Redis. But the main reason they are provided by redisco is that they are dependencies of the Model class. Making them available as well seems reasonable.
>>> from redisco.containers import List >>> l = List('alpha') >>> l.append('a') >>> l.append('b') >>> l.append('c') >>> 'a' in l True
The package is published in PyPI, so you can
pip install redisco
Or, you can clone the git repository:
git clone http://github.com/iamteem/redisco.git
Prerequisites: Redis 2.0.0 RC1 and the latest version of redis-py on Github.
Much credit goes to the Ohm team, whose work in Ohm is the primary inspiration and basis of the project.
Because redisco is relatively new, consider it alpha software and not production-ready. I’m currently testing it in a simple web toy that stores tweets from Twitter tracked using the tweetstream and using Hashes to store statistics.
The project is MIT-licensed so anyone can do anything with it. Use it, fork it on Github, read the source code, and let me know your opinions and reactions. Thanks.
I thought I wanted to share some of the tools I use these days. In no particular order:
Tornado. A fast Python non-blocking web server. I’ve been hacking two small apps that will be rolled out in the next two weeks. Tornado is a good match for the two apps because the apps access external web services (i.e. Flickr API), and one app has Comet features via Ajax and long polling — both of these features are easily solved in Tornado. Since its code base is really small, I had no trouble looking around when I hit some roadblock.
Python 2.6. Recently, I updated my Python version to 2.6. Multiprocessing FTW.
Homebrew. A package manager for OSX. Recently, I updated the my Macports system and several of the packages in it. The update broke my other tools (Ruby gems and others) because their dependencies (dylibs etc) are no longer where they expected them to be. I was done with Macports. So after looking around, I found Homebrew. It’s quite convenient and so far I have no issues with it. My impression is it’s pragmatic. E.g. doing
brew install mercurial doesn’t install the mercurial package but recommends that you do
brew install pip then
pip install mercurial. It makes sense. Also,
brew system is built in Ruby so I had no trouble looking at the package formulae.
Redis. A very fast key value store. Using redis is fun. It takes a much more (personally) enjoyable approach to storing data — using constructs such as lists and sets instead of rows and sql. I say enjoyable because with redis, the way your data is stored is close to the way you actually model and code your data. I bookmarked its command reference for easy access.
rvm. Ruby Version Manager. Playing around with multiple versions of Ruby without pain. Gemsets are really wonderful. My usual setup is one gemset per web project (be it Rails or sinatra).
Ruby 1.9. Saying goodbye to Ruby 1.8 is not so hard anymore since
rvm. But I compiled Ruby 1.9 from source and it is now the default Ruby in my system.
Sinatra. Moving away from Rails is always an option. And could also a very liberating decision. With Sinatra, I write code my way (at least, I think that’s how it is). There are frameworks built on Sinatra (e.g. monk, padrino) so if ever you don’t want to use Rails for some reason, I’d suggest Sinatra.
Monk. A glue framework built on Sinatra.
Mercurial. I use this when I’m in Python mode.
ZeroMQ. Fastest. Messaging. Ever. For pub/sub, I used ØMQ and its Python binding.
After removing macports and installing homebrew, I encountered a problem when I installed
pycurl (a dependency of tornado).
pycurl requires a newer version (v7.19) of
libcurl than what my system (osx 10.5.8) has (v7.16). So I grabbed the curl source and installed it (
./configure --with-ca-bundle=/usr/share/curl/curl-ca-bundle.crt && make && make install). With the newer
curl installed in
/usr/local, I tried installing pycurl again, and it failed. Checking my $PATH, I found out that
/usr/local/bin is the last entry. I read the
/etc/profile and the
/usr/libexec/path_helper and attempted to reorganize the PATH but instead decided to remove
/etc/paths and manually add it in
Fail fast, fail open.
Rails 3 is going to require at least Ruby 1.8.7 so I decided to install Ruby Enterprise Edition on my Mac. I have OSX Leopard (10.5.8) and use macports. After installing rvm, I encountered the same problem whenever I executed my web server in my Rails app:
/usr/local/rvm/ree-1.8.7-2009.10/lib/ruby/1.8/openssl/ssl.rb:31: [BUG] Bus Error ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9.8.0], MBARI 0x8770, Ruby Enterprise Edition 2009.10
The culprit is that I have two openssl installations and the eventmachine gem isn’t compiled with the same openssl as the my REE install. This issue was reported here.
So the solution is either to recompile REE to use Macport’s openssl or recompile eventmachine to use the system’s openssl lib. I chose to recompile REE.
sudo rvm install ree --configure --enable-shared=true,--with-openssl-dir=/opt/local --debug