Wednesday, January 7, 2009

Making RabbitMQ Stew

I just wanted to write this up real quick for anyone who may run into the same problem I did...

I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.

Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:

Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...

Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application

Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.


-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.

and

-mnesia dump_log_write_threshold Max, where Max is an integer which specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. It defaults to 100 log writes.


I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.

Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:

=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary

Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.

So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/

This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached

Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===

Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"

=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672


The short of it - whipe your mnesia directories and start over.

2 comments:

Jen said...

Thank you so much for posting this. I am very new to RabbitMQ and was getting the error you mentioned and cleaning out the mnesia directory worked like a charm. Thanks again.

BradfordW said...

Glad you found use of it Jen, this information sharing does seem to work after all!