The Lantern Project

I've been having a bit of a love affair with various storage platforms over the last year or so. Some notable mentions of late are Tokyo Cabinet and redis. Both are incredibly cool, fast, trivial to interface and best of all can run across multiple machines. I'll post a few pre-built interfaces at the bottom of this post so jump there for their links.

Anyway, so all of that along with my new-found and seemingly unhealthy obsession with Erlang has driven me to, of course, to build my own...

Enter The Lantern Project.

No, there's no cool story-line behind its name, I wish there were. Anyway, so just a few of the "things" driving this currently are going to be (I don't trust anything with bullets, so I'll just delimit a list): RESTful, simple "string" storage (initially - see reasoning below), based on ETS/DETS (initially - see reasoning below) and finally distributable (that will be more or less the last "phase").

Why REST?

Well, for starters, I absolutely love it as an interface to CouchDB and other service-oriented sites as well (twitter, facebook {to an extent - stop inventing your own methods!}). Second, it's easy implement at the client level, we already know it scales, damn near anything can connect to it, etc...etc. This was a no-brainer, I really wish redis would have this, but, trade-offs, I understand...

Why just strings at first?

I had a fun time mulling over this, do you support more complex objects at first? At first, is what I kept coming back to. If this is a learning project, then let's start small and work our way up. Makes sense (to me), but what other data type could you really use to store "anything" (take binary data out of the picture for a moment). I think that by allowing folks to marshal/encode objects as they see fit (JSON,base64,XML,whatever) that makes for a pretty good first pass.

Why ETS/DETS?

Simple, it's in Erlang and I don't need to really "worry" about formats to get something functional out the door. If my reading is correct, they're based on Judy Arrays which gives you (I know, b-trees do it too) both constant O(1) and logarithmic O(log n)look-ups, NICE! They generally take up less memory, but you do end up utilizing more CPU. Meh, fair enough...

So there ye have it...if you want to peek at my (ahem) "progress" the github project is right here.

As promised, the gem links for redis/tokyo cabinet

• redis-rb


• rufus-tokyo

Why Erlang will run "The Matrix"

Ok, so I had blogged and even somewhat complained about Erlang as an "every day" language. I'm here right now to say that I was dead wrong.

Erlang is simply amazing as I've gone from merely flipping through pages and references online and in books, to fiddling with gen_server off and on and to actually implementing a small facebook application based on Nick Gerakines' erlang_facebook module.

What I found funny about that last bit (the facebook app) is that after seeing an implementation of it in Ruby, it was actually easier to comprehend just what all was going on in the ERLANG version. Yes, I said it, the Erlang was easier to read.

Getting back on track...

After digging deep into getting my head around Erlang, something kind of crept up on me as I went along. Particularly when it came to message passing and hot code swapping...

You ready? Put on your tinfoil hat. Below is my list of 5 examples The Matrix could exist and why it could be written in Erlang.

1. You ever wonder why we can only really "use" less than 10% of our brains? Ever wonder what that other 90-some percent is doing? That's right, passing messages to other "processes" and handling other distributed tasks.

2. When it comes to the hot code loading, this is a good example of how a "Smith" would work.

3. Every wonder why we can’t utilize 100% of our brains? Maybe it’s because the other 90-ish percent is too busy doing distributed tasks in conjunction with other processes.

4. Anything that scales just by adding cores (or nice, warm, fresh humans) and has been sitting somewhat idle in the wings for the last 20 years or so has GOT to be hiding something. I mean c'mon, most of the "love languages" (Ruby, Python, PHP) look at a multi-core system and don't really know what to do with themselves.

5. (And this will be the last one as I'm losing momentum) The system can self-sustain for obscene periods of time, if one had the "available systems" (people) one could die (literally and figuratively) and it really wouldn't matter, you'd have another waiting to take it's place AND you've got around 90% of it's processing power at your disposal.

This has been fun() (I can't help myself, I'm sorry) I hope some of you who come across this add some more to this list in the comments, I think there's a lot of cracks to be made.

Why Erlang will run "The Matrix"

Ok, so I had blogged and even somewhat complained about Erlang as an "every day" language. I'm here right now to say that I was dead wrong.

Erlang is simply amazing as I've gone from merely flipping through pages and references online and in books, to fiddling with gen_server off and on and to actually implementing a small facebook application based on Nick Gerakines' erlang_facebook module.

What I found funny about that last bit (the facebook app) is that after seeing an implementation of it in Ruby, it was actually easier to comprehend just what all was going on in the ERLANG version. Yes, I said it, the Erlang was easier to read.

Getting back on track...

After digging deep into getting my head around Erlang, something kind of crept up on me as I went along. Particularly when it came to message passing and "hot" code swapping...

You ready? Put on your tinfoil hat. Below is my list of reasons that "The Matrix" could become a reality and why it would be implemented in Erlang.

1. You ever wonder why we can only really "use" less than 10% of our brains? Ever wonder what that other 90-some percent is doing? That's right, passing messages to other "processes" and handling other distributed tasks.

2. When it comes to the hot code loading, this (I think) is a prime example of how a "Smith" would work. Think about it, "Oh, someone just ate the blue pill, better inject myself without taking this heap offline!". Bahng!

3. This blog post is currently fueled by Jameson 18 yr. Irish Whiskey.

4. Anything that scales just by adding cores (or nice, warm, fresh humans) and has been sitting somewhat idle in the wings for the last 20 years or so has GOT to be hiding something. I mean c'mon, most of the "love languages" (your Ruby, Python, PHP) look at a multi-core system and don't really know what to do with themselves.

5. (And this will be the last one as I'm losing momentum) The system can self-sustain for obscene periods of time, if one had the crazy "available systems" (people) one could die (literally and figuratively) and it really wouldn't matter, you'd have another waiting to take it's place AND you've got around 90% of it's processing power at your disposal.

This has been fun() (I can't help myself, I'm sorry) I hope some of you who come across this add some more to this list in the comments, I think there's a lot of cracks to be made.