The Lantern Project

I've been having a bit of a love affair with various storage platforms over the last year or so. Some notable mentions of late are Tokyo Cabinet and redis. Both are incredibly cool, fast, trivial to interface and best of all can run across multiple machines. I'll post a few pre-built interfaces at the bottom of this post so jump there for their links.

Anyway, so all of that along with my new-found and seemingly unhealthy obsession with Erlang has driven me to, of course, to build my own...

Enter The Lantern Project.

No, there's no cool story-line behind its name, I wish there were. Anyway, so just a few of the "things" driving this currently are going to be (I don't trust anything with bullets, so I'll just delimit a list): RESTful, simple "string" storage (initially - see reasoning below), based on ETS/DETS (initially - see reasoning below) and finally distributable (that will be more or less the last "phase").

Why REST?

Well, for starters, I absolutely love it as an interface to CouchDB and other service-oriented sites as well (twitter, facebook {to an extent - stop inventing your own methods!}). Second, it's easy implement at the client level, we already know it scales, damn near anything can connect to it, etc...etc. This was a no-brainer, I really wish redis would have this, but, trade-offs, I understand...

Why just strings at first?

I had a fun time mulling over this, do you support more complex objects at first? At first, is what I kept coming back to. If this is a learning project, then let's start small and work our way up. Makes sense (to me), but what other data type could you really use to store "anything" (take binary data out of the picture for a moment). I think that by allowing folks to marshal/encode objects as they see fit (JSON,base64,XML,whatever) that makes for a pretty good first pass.

Why ETS/DETS?

Simple, it's in Erlang and I don't need to really "worry" about formats to get something functional out the door. If my reading is correct, they're based on Judy Arrays which gives you (I know, b-trees do it too) both constant O(1) and logarithmic O(log n)look-ups, NICE! They generally take up less memory, but you do end up utilizing more CPU. Meh, fair enough...

So there ye have it...if you want to peek at my (ahem) "progress" the github project is right here.

As promised, the gem links for redis/tokyo cabinet

• redis-rb


• rufus-tokyo

So what happened (the multi-key CouchDB question)?

So, if any of you have been following along with my fun ride upon the cozy cushions of CouchDB, you'll be happy to hear there is hope (well, some - I had to pick up a book on Erlang).

Anyhow, if you follow the link here you will see that there was a list of proposed fixes for my desire to pass in multiple keys to CouchDB instead of just one. Granted, yes it could get out of hand, you could pass in 1,000,000 keys. But so what! Either way you're going to have to query for them, whether it's a query after a query a billion times or if you can somehow lump them together.

So as you can see there were a few options:

1. We could execute our query with each key each time and lump them together. The downside of this is that I really don't want Ruby to have to lump all of those results together, I can only imagine the paging for it would get incredibly *blah*. Plus the memory usage for Ruby to merge all of those DM::Collection objects together would likely be the equivalent of a Chihuahua passing a peach seed. Not pretty.

2. We could use a full text search using Lucene, which CouchDB already has implemented. I suppose the only downside of that would be...??? I'll need to think of something, it's really not a bad idea, it just may come to me later. The extent of my full text search capabilities have been with MySQL and...oh, hold on I keep laughing, it sucked. bad. I really hope CouchDB's is better - I'm sure it is, I haven't seen anything so far that is crap!

3. We (read: Me {Bradford}) hack at the CouchDB internals and allow for views to accept multiple keys for comparison rather than one. I'd like to see this happen via something to the effect of:

A single key query URL
http://.../_view/foo/bars?key="mykey"

A Multi-key query URL
http://.../_view/foo/bars?key[]="mykey1"&key[]="mykey2"
OR
http://.../_view/foo/bars?keys=["mykey1","mykey2","mykey3"]

I'd like to keep it along the lines of form control arrays key[] this way you could just analyze the parameter itself to see what it was (an array or a string) and do the appropriate things with it. Whereas adding a parameter keys may need to be repeated throughout the code (Not likely, but the former makes sense to me most).

I'd like to extend a warm welcome to anyone willing to walk the lines of CouchDB's internals with me in order to implement such functionality, I promise to keep you in line as long as you return the favor. As I said, I'm JUST NOW learning Erlang and am horribly under-qualified, but hey, I really feel strongly about this functionality, so, what the hell, right?

Bring on the opinions!