Making RabbitMQ Stew

I just wanted to write this up real quick for anyone who may run into the same problem I did...

I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.

Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:

Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...

Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application

Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.

-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.

and

-mnesia dump_log_write_threshold Max, where Max is an integer which specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. It defaults to 100 log writes.

I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.

Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:

=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary

Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.

So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/

This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached

Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===

Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"

=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672


The short of it - whipe your mnesia directories and start over.

Reia where scripting meets concurrency

Reia reminds me of that Reese's peanut butter cup commercial:

Hey, you got your Ruby/Python in my Erlang! Hey, you got your Erlang in my Ruby/Python!

And then there was this light that went off in someone's head - Why couldn't you bring the "stuff" people like about higher level languages down into something that, quite frankly just kicks ass, in Erlang?

To kick it off - and to quote the Reia wiki:

Reia (pronounced RAY-uh) is a Python/Ruby-like scripting language for the Erlang virtual machine (BEAM). Reia aims to expose all the features and functionality of Erlang in a language more familiar to programmers of scripting languages, while improving string handling, regular expressions, linking with external libraries, and other tasks which are generally considered outside the scope of Erlang. Reia is distributed under the MIT License.

Well, it's happening right now with Reia. I've been watching this project over my GitHub feed for maybe a month or so now and it's really cool seeing the leaps and bounds that Tony Arcieri is making with this little gem of a language. Granted, he still has a ways to go (and he openly admits this), I suggest giving it a try for fun() (that's my new favorite pun by the way).

Let me paste a few ideals from the Reia site to pique your interest a little more:
Concurrent programming is the future - very true, now if DB's would just catch up (oh, wait CouchDB!

You don't need an "everything is a..." language - he calls out SmallTalk and Ruby, but his argument is convincing, I think we've all been here before.

Don't build what you can steal - I really like that quote in regard to using a syntax of Python/Ruby and then laying it lovingly over the Erlang VM and OTP framework.