Tim Medina's Software Development Blog

Introducing redisco - Python Containers and Simple Models for Redis

Sat, 29 May 2010 12:05:14 +0800

I recently started an open source Python project called redisco. From its README:

Redisco allows you to store objects in Redis. It is inspired by the Ruby library Ohm and its design and code are loosely based on Ohm and the Django ORM. It is built on top of redis-py. It includes container classes that allow easier access to Redis sets, lists, and sorted sets.

Why?

Short answer: I like Redis and I want to code in Python.

A bit longer answer: I’ve been hacking with Redis and Ohm since last month and I found it lighter than using SQL databases or Mongodb right from its installation to actually using it. It’s data structure approach appeals to me. Sets, lists, dicts, sorted set — using them feels more natural as compared to tables and joins. The Ruby gem Ohm provides a nice interface to Redis. I wanted to have something similar in Python. I looked around, I saw Redish which offers some nice abstractions, but after reading through its code looking specifically for model abstraction, I decided it didn’t have what I wanted. And with so much time in my hands, I rolled out my own. (Another reason for starting this project is that I wanted to learn more about and code more in Python.)

Simple Models

Redisco allows you to code model classes like you would in Django or AppEngine.

class Person(models.Model):
    name = models.Attribute(required=True)
    created_at = models.DateTimeField(auto_now_add=True)
    fave_colors = models.ListField(str)

Saving an object to Redis automatically creates indexes that are used when querying.

>>> p = Person(name="Ippo", fave_colors=["Red", "White"])
>>> p.save()
True

Objects can be queried via the Model.objects classmethod:

Person.objects.all()
Person.objects.filter(name='Ippo')
Person.objects.filter(name='Ippo').first()
Person.objects.all().order('name')
Person.objects.filter(fave_colors='Red')

Redisco has a limited support for queries involving ranges - it can only filter fields that are numeric, i.e. DateField, DateTimeField, IntegerField, and FloatField.

Person.objects.zfilter(created_at__lt=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__gte=datetime(2010, 4, 20, 5, 2, 0))
Person.objects.zfilter(created_at__in=(datetime(2010, 4, 20, 5, 2, 0), datetime(2010, 5, 1)))

Other limitations

redisco.models does not offer customizable managers and inheritance.

Abstractions to Python data structures

Lists, sets, sorted sets, dicts - this sort of things are easily persisted in Redis. But the main reason they are provided by redisco is that they are dependencies of the Model class. Making them available as well seems reasonable.

>>> from redisco.containers import List
>>> l = List('alpha')
>>> l.append('a')
>>> l.append('b')
>>> l.append('c')
>>> 'a' in l
True

Installation

The package is published in PyPI, so you can

pip install redisco

Or, you can clone the git repository:

git clone http://github.com/iamteem/redisco.git

Prerequisites: Redis 2.0.0 RC1 and the latest version of redis-py on Github.

Credits

Much credit goes to the Ohm team, whose work in Ohm is the primary inspiration and basis of the project.

Notes

Because redisco is relatively new, consider it alpha software and not production-ready. I’m currently testing it in a simple web toy that stores tweets from Twitter tracked using the tweetstream and using Hashes to store statistics.

Invitation

The project is MIT-licensed so anyone can do anything with it. Use it, fork it on Github, read the source code, and let me know your opinions and reactions. Thanks.

What's new in my toolbox

Tue, 06 Apr 2010 13:07:00 +0800

I thought I wanted to share some of the tools I use these days. In no particular order:

Tornado. A fast Python non-blocking web server. I’ve been hacking two small apps that will be rolled out in the next two weeks. Tornado is a good match for the two apps because the apps access external web services (i.e. Flickr API), and one app has Comet features via Ajax and long polling — both of these features are easily solved in Tornado. Since its code base is really small, I had no trouble looking around when I hit some roadblock.

Python 2.6. Recently, I updated my Python version to 2.6. Multiprocessing FTW.

pip and virtualenv. When experimenting with Python packages, this is THE way.

Homebrew. A package manager for OSX. Recently, I updated the my Macports system and several of the packages in it. The update broke my other tools (Ruby gems and others) because their dependencies (dylibs etc) are no longer where they expected them to be. I was done with Macports. So after looking around, I found Homebrew. It’s quite convenient and so far I have no issues with it. My impression is it’s pragmatic. E.g. doing brew install mercurial doesn’t install the mercurial package but recommends that you do brew install pip then pip install mercurial. It makes sense. Also, brew system is built in Ruby so I had no trouble looking at the package formulae.

Redis. A very fast key value store. Using redis is fun. It takes a much more (personally) enjoyable approach to storing data — using constructs such as lists and sets instead of rows and sql. I say enjoyable because with redis, the way your data is stored is close to the way you actually model and code your data. I bookmarked its command reference for easy access.

rvm. Ruby Version Manager. Playing around with multiple versions of Ruby without pain. Gemsets are really wonderful. My usual setup is one gemset per web project (be it Rails or sinatra).

Ruby 1.9. Saying goodbye to Ruby 1.8 is not so hard anymore since rvm. But I compiled Ruby 1.9 from source and it is now the default Ruby in my system.

Sinatra. Moving away from Rails is always an option. And could also a very liberating decision. With Sinatra, I write code my way (at least, I think that’s how it is). There are frameworks built on Sinatra (e.g. monk, padrino) so if ever you don’t want to use Rails for some reason, I’d suggest Sinatra.

Monk. A glue framework built on Sinatra.

Nodejs. Wohow. V8 JS not on the browser. I experimented on it a while back since I <3 programming javascript. I wish I can build some crazy service using node soon.

Mercurial. I use this when I’m in Python mode.

GHC. I’ve been reading on Haskell in the past two weeks, and I find the experience fun, exciting, and mind bending.

ZeroMQ. Fastest. Messaging. Ever. For pub/sub, I used ØMQ and its Python binding.

Note to self: /usr/local/bin in osx path

Fri, 02 Apr 2010 11:15:19 +0800

After removing macports and installing homebrew, I encountered a problem when I installed pycurl (a dependency of tornado). pycurl requires a newer version (v7.19) of libcurl than what my system (osx 10.5.8) has (v7.16). So I grabbed the curl source and installed it (./configure --with-ca-bundle=/usr/share/curl/curl-ca-bundle.crt && make && make install). With the newer curl installed in /usr/local, I tried installing pycurl again, and it failed. Checking my $PATH, I found out that /usr/local/bin is the last entry. I read the /etc/profile and the /usr/libexec/path_helper and attempted to reorganize the PATH but instead decided to remove /usr/local/bin from /etc/paths and manually add it in /etc/profile.

Google's Python Class

Wed, 24 Mar 2010 08:59:27 +0800

Google's Python Class:

videolearning:

2 day class, 7 videos. Introduction to Python, that lets you work more quickly and integrate your systems more effectively.

Note to self: Add mysql bin dir to path

Tue, 09 Mar 2010 01:25:28 +0800

When installing mysql gem in Ruby 1.9.1 (using rvm), the process keeps on failing even when specifying --with-mysql-dir=/usr/local/mysql etc. Apparently the solution is to add the /usr/local/mysql to $PATH before issuing the gem install command.

"Fail fast, fail open."

Mon, 01 Feb 2010 23:46:21 +0800

“Fail fast, fail open.”

SWFUpload + Rails == Invalid Authenticity Token

Tue, 19 Jan 2010 02:26:00 +0800

http://github.com/lardawge/swfupload-rails-authentication

openssl, eventmachine, and the ruby 1.8.7 (and ree, too) bus error

Wed, 06 Jan 2010 17:49:38 +0800

Rails 3 is going to require at least Ruby 1.8.7 so I decided to install Ruby Enterprise Edition on my Mac. I have OSX Leopard (10.5.8) and use macports. After installing rvm, I encountered the same problem whenever I executed my web server in my Rails app:

/usr/local/rvm/ree-1.8.7-2009.10/lib/ruby/1.8/openssl/ssl.rb:31: [BUG] Bus Error
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9.8.0], MBARI 0x8770, Ruby Enterprise Edition 2009.10

The culprit is that I have two openssl installations and the eventmachine gem isn’t compiled with the same openssl as the my REE install. This issue was reported here.

So the solution is either to recompile REE to use Macport’s openssl or recompile eventmachine to use the system’s openssl lib. I chose to recompile REE.

sudo rvm install ree --configure --enable-shared=true,--with-openssl-dir=/opt/local --debug

Symbian development on Linux and OS X

Sun, 04 Oct 2009 23:58:41 +0800

Symbian development on Linux and OS X

Adding Date And Time To Your Bash History

Fri, 04 Sep 2009 11:44:53 +0800

Adding Date And Time To Your Bash History:

export HISTTIMEFORMAT=”%h/%d - %H:%M:%S “

RailsAPI Textmate Bundle

Mon, 10 Aug 2009 11:43:23 +0800

RailsAPI Textmate Bundle:

Textmate bundle for viewing Rails documentation (using railsapi.com).

Moved to Tumblr

Mon, 10 Aug 2009 01:43:59 +0800

I migrated my stuff from Slicehost to tumblr. I started by backing up the contents of my slice to my laptop using rsync:

rsync -e ssh -avl --delete --stats --progress user@timmedina.com:/home/user backup

After everything was backed up, I deleted my slice, bought a .net domain, and configured my tumblr account to use custom domain names.

I’m still thinking of a way to organize my website. I’ll probably learn to create custom themes.

And I have to add commenting, too.

Tumblenote!

Tue, 04 Aug 2009 17:34:03 +0800

Tumblenote!

PL/Python cheatsheet with some tute links

Fri, 08 May 2009 11:42:00 +0800

PL/Python cheatsheet with some tute links

Postgresql Leopard Gotchas

Thu, 07 May 2009 10:12:00 +0800

Postgresql Leopard Gotchas:

After a fresh postgresql installation, initdb didn’t work until I reset the kern.sysv.shmall and kern.sysv.shmmax:

sudo sysctl -w kern.sysv.shmmax=134217728
sudo sysctl -w kern.sysv.shmall=65536

A Better ri

Fri, 24 Apr 2009 14:54:00 +0800

A Better ri

Debian Administration: Using GNU Screen

Wed, 22 Apr 2009 16:13:46 +0800

Debian Administration: Using GNU Screen :

screen
- Run a new screen session
screen -R
- Reattach to a previously detatched session
Ctrl-a c
- Create a new window.
Ctrl-a k
- Kill the current window - after confirmation
Ctrl-a Ctrl-a
- Switch to the other window
Ctrl-a S
- Split the current window in two.
Ctrl-a TAB
- Move between split sections of the screen.
Ctrl-a A
- Give the the current window a name.
Ctrl-a ”
- List all windows - move around to change the window with the arrow keys

Using Debugger in a Phusion Passenger Dev setup

Tue, 21 Apr 2009 10:16:36 +0800

Using Debugger in a Phusion Passenger Dev setup:

My setup: Rails 2.1.2, Passenger 2.2.1, ruby-debug 0.10.3

app-engine-patch -- ignoring mediagenerator

Mon, 20 Apr 2009 02:16:10 +0800

move the django admin media files from _generated_media/1/admin to some directory inside the PROJECT_DIR:

1. move the admin dir to desired location, say PROJECT_DIR/admin_media

2. modify app.yaml urls to point to that static_dir location, e.g. add

- url: /admin_media

static_dir: admin_media

3. modify the MEDIA_URL and ADMIN_MEDIA_PREFIX in settings.py

MEDIA_URL = “/media/”

ADMIN_MEDIA_PREFIX = “/admin_media/”

We need to override MEDIA_URL so that when we access the media_url path from the templates, we don’t get the altered media_url (settings_post.py alters this by appending the MEDIA_VERSION by interpolating the %s. ADMIN_MEDIA_PREFIX is also based on the MEDIA_URL, but we want it to use /admin_media.

Installed some stuff today

Mon, 20 Apr 2009 02:04:19 +0800

wxpython from http://www.wxpython.org/download.php Unicode py2.5 build

sqlalchemy 0.5.3 via easy_install

phusion passenger 2.2 and nginx (installed via passenger)