The Lantern Project

I've been having a bit of a love affair with various storage platforms over the last year or so. Some notable mentions of late are Tokyo Cabinet and redis. Both are incredibly cool, fast, trivial to interface and best of all can run across multiple machines. I'll post a few pre-built interfaces at the bottom of this post so jump there for their links.

Anyway, so all of that along with my new-found and seemingly unhealthy obsession with Erlang has driven me to, of course, to build my own...

Enter The Lantern Project.

No, there's no cool story-line behind its name, I wish there were. Anyway, so just a few of the "things" driving this currently are going to be (I don't trust anything with bullets, so I'll just delimit a list): RESTful, simple "string" storage (initially - see reasoning below), based on ETS/DETS (initially - see reasoning below) and finally distributable (that will be more or less the last "phase").

Why REST?

Well, for starters, I absolutely love it as an interface to CouchDB and other service-oriented sites as well (twitter, facebook {to an extent - stop inventing your own methods!}). Second, it's easy implement at the client level, we already know it scales, damn near anything can connect to it, etc...etc. This was a no-brainer, I really wish redis would have this, but, trade-offs, I understand...

Why just strings at first?

I had a fun time mulling over this, do you support more complex objects at first? At first, is what I kept coming back to. If this is a learning project, then let's start small and work our way up. Makes sense (to me), but what other data type could you really use to store "anything" (take binary data out of the picture for a moment). I think that by allowing folks to marshal/encode objects as they see fit (JSON,base64,XML,whatever) that makes for a pretty good first pass.

Why ETS/DETS?

Simple, it's in Erlang and I don't need to really "worry" about formats to get something functional out the door. If my reading is correct, they're based on Judy Arrays which gives you (I know, b-trees do it too) both constant O(1) and logarithmic O(log n)look-ups, NICE! They generally take up less memory, but you do end up utilizing more CPU. Meh, fair enough...

So there ye have it...if you want to peek at my (ahem) "progress" the github project is right here.

As promised, the gem links for redis/tokyo cabinet

• redis-rb


• rufus-tokyo

Erlang, you are holding out on me!

So I was playing around with list comprehensions today and found myself looking at permutations of strings when I noticed I wasn't getting "everything" back from the shell.

I'm sure there is a good reason for this, but I'm unclear as to what to search for in order to find an answer. The code I'm talking about is as follows:

[ [String1]++[String2] || String1<-"ABCDEFGHIJKLMNOPQRSTUVWXYZ", String2<-"ABCDEFGHIJKLMNOPQRSTUVWXYZ"].

which produces:

["AA","AB","AC","AD","AE","AF","AG","AH","AI","AJ","AK",
"AL","AM","AN","AO","AP","AQ","AR","AS","AT","AU","AV","AW",
"AX","AY","AZ","BA","BB",
[...]|...]

I'm sure it is just trying to save some horrendous output, but what if for some un-Godly reason I needed it all, how do I turn those [...]'s into more values?

Anyway, just thought this was interesting.

Why Erlang will run "The Matrix"

Ok, so I had blogged and even somewhat complained about Erlang as an "every day" language. I'm here right now to say that I was dead wrong.

Erlang is simply amazing as I've gone from merely flipping through pages and references online and in books, to fiddling with gen_server off and on and to actually implementing a small facebook application based on Nick Gerakines' erlang_facebook module.

What I found funny about that last bit (the facebook app) is that after seeing an implementation of it in Ruby, it was actually easier to comprehend just what all was going on in the ERLANG version. Yes, I said it, the Erlang was easier to read.

Getting back on track...

After digging deep into getting my head around Erlang, something kind of crept up on me as I went along. Particularly when it came to message passing and hot code swapping...

You ready? Put on your tinfoil hat. Below is my list of 5 examples The Matrix could exist and why it could be written in Erlang.

1. You ever wonder why we can only really "use" less than 10% of our brains? Ever wonder what that other 90-some percent is doing? That's right, passing messages to other "processes" and handling other distributed tasks.

2. When it comes to the hot code loading, this is a good example of how a "Smith" would work.

3. Every wonder why we can’t utilize 100% of our brains? Maybe it’s because the other 90-ish percent is too busy doing distributed tasks in conjunction with other processes.

4. Anything that scales just by adding cores (or nice, warm, fresh humans) and has been sitting somewhat idle in the wings for the last 20 years or so has GOT to be hiding something. I mean c'mon, most of the "love languages" (Ruby, Python, PHP) look at a multi-core system and don't really know what to do with themselves.

5. (And this will be the last one as I'm losing momentum) The system can self-sustain for obscene periods of time, if one had the "available systems" (people) one could die (literally and figuratively) and it really wouldn't matter, you'd have another waiting to take it's place AND you've got around 90% of it's processing power at your disposal.

This has been fun() (I can't help myself, I'm sorry) I hope some of you who come across this add some more to this list in the comments, I think there's a lot of cracks to be made.

Making RabbitMQ Stew

I just wanted to write this up real quick for anyone who may run into the same problem I did...

I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.

Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:

Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...

Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application

Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.

-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.

and

-mnesia dump_log_write_threshold Max, where Max is an integer which specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. It defaults to 100 log writes.

I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.

Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:

=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary

Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.

So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/

This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached

Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===

Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"

=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672


The short of it - whipe your mnesia directories and start over.

Who Knew?!?!

...that it would be more than two months since my last post.

...that Rails and Merb would get into bed with each other. (And may or may not come out producing something useful).

...that we would be on the verge of collapsing economically (ok, some did, but I didn't).

...that the Boston Bruins would lead the NHL in points almost half way through the year by winning 10 games straight! (they better knock this off, my Red Wings need some love)

...that the Tennessee Titans would be the 1st seed in the NFL playoffs.

...that I would grasp enough Erlang to seriously consider using it for an application.

...that I could come up with a post as "clever" as this with a horrid head cold.

Ok, well now that the busyness of Nov/Dec is over I can hopefully return to more sharing...Stay tuned.

Reia where scripting meets concurrency

Reia reminds me of that Reese's peanut butter cup commercial:

Hey, you got your Ruby/Python in my Erlang! Hey, you got your Erlang in my Ruby/Python!

And then there was this light that went off in someone's head - Why couldn't you bring the "stuff" people like about higher level languages down into something that, quite frankly just kicks ass, in Erlang?

To kick it off - and to quote the Reia wiki:

Reia (pronounced RAY-uh) is a Python/Ruby-like scripting language for the Erlang virtual machine (BEAM). Reia aims to expose all the features and functionality of Erlang in a language more familiar to programmers of scripting languages, while improving string handling, regular expressions, linking with external libraries, and other tasks which are generally considered outside the scope of Erlang. Reia is distributed under the MIT License.

Well, it's happening right now with Reia. I've been watching this project over my GitHub feed for maybe a month or so now and it's really cool seeing the leaps and bounds that Tony Arcieri is making with this little gem of a language. Granted, he still has a ways to go (and he openly admits this), I suggest giving it a try for fun() (that's my new favorite pun by the way).

Let me paste a few ideals from the Reia site to pique your interest a little more:
Concurrent programming is the future - very true, now if DB's would just catch up (oh, wait CouchDB!

You don't need an "everything is a..." language - he calls out SmallTalk and Ruby, but his argument is convincing, I think we've all been here before.

Don't build what you can steal - I really like that quote in regard to using a syntax of Python/Ruby and then laying it lovingly over the Erlang VM and OTP framework.

Judging a book by it's "sucks" result

A friend of mine asked me recently to go in on a little venture of his in the gaming industry. Unable to say "no" to what I believe is a fantastic idea, I went along with it...

Now there was another guy involved (technologically speaking) who wrote something that was a working prototype (awesome). It came down to a little discussion as to what WE should use for the front-end and storage mechanisms.

Well, having deployed a few real life applications in Ruby and having success with them, I only felt the need to display my affection and trust that I had with it and it's frameworks (yes, both Rails and Merb).

The problem I had wasn't that the guy wanted to use Python/Pylons or something in that realm. It was the argument. In fact, now that I've heard it a few times I can almost pinpoint exactly where the debate material is coming from. And that my friends, is simply fantastic. We're still not going along with the right tool for the right job, we're still shooting from the hip with what "we like".

Uh, but Brad, you just said you liked Ruby? You're right, I did. But given the requirements I was handed and based on past experiences it appeared to be the right tool for the right job.

I'm open to using new things - if I wasn't why would I ever give things like CouchDB and Erlang the time of day? They're sitting right on the edge of the programming paradigm shift from what most coders/programmers/developers are use to and it fascinates me.

Back to it.

I'm fine with using Python or whatever for a web interface. One more (hopefully) successful notch in my belt and slap proudly on a resume. But it just absolutely kills me when the keys of an argument are literally plucked from simply searching Google for "why sucks". What's even worse is when it's a predominantly blog rifling .

Please, please, please, I'm begging you the reader of this blog post; the rule here is simple, I'll even wrap it in a blockquote for you to save and tattoo across your chest in a Tupac-ish style:

Research first, execute later.

It's not the other way around, in fact, if you do it the other way around you're going to do your research the hard way; rewriting it in the layers you should have used in the first place.