Making RabbitMQ Stew

I just wanted to write this up real quick for anyone who may run into the same problem I did...

I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.

Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:

Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...

Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application

Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.

-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.

and

-mnesia dump_log_write_threshold Max, where Max is an integer which specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. It defaults to 100 log writes.

I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.

Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:

=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary

Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.

So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/

This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached

Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===

Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"

=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672


The short of it - whipe your mnesia directories and start over.

Judging a book by it's "sucks" result

A friend of mine asked me recently to go in on a little venture of his in the gaming industry. Unable to say "no" to what I believe is a fantastic idea, I went along with it...

Now there was another guy involved (technologically speaking) who wrote something that was a working prototype (awesome). It came down to a little discussion as to what WE should use for the front-end and storage mechanisms.

Well, having deployed a few real life applications in Ruby and having success with them, I only felt the need to display my affection and trust that I had with it and it's frameworks (yes, both Rails and Merb).

The problem I had wasn't that the guy wanted to use Python/Pylons or something in that realm. It was the argument. In fact, now that I've heard it a few times I can almost pinpoint exactly where the debate material is coming from. And that my friends, is simply fantastic. We're still not going along with the right tool for the right job, we're still shooting from the hip with what "we like".

Uh, but Brad, you just said you liked Ruby? You're right, I did. But given the requirements I was handed and based on past experiences it appeared to be the right tool for the right job.

I'm open to using new things - if I wasn't why would I ever give things like CouchDB and Erlang the time of day? They're sitting right on the edge of the programming paradigm shift from what most coders/programmers/developers are use to and it fascinates me.

Back to it.

I'm fine with using Python or whatever for a web interface. One more (hopefully) successful notch in my belt and slap proudly on a resume. But it just absolutely kills me when the keys of an argument are literally plucked from simply searching Google for "why sucks". What's even worse is when it's a predominantly blog rifling .

Please, please, please, I'm begging you the reader of this blog post; the rule here is simple, I'll even wrap it in a blockquote for you to save and tattoo across your chest in a Tupac-ish style:

Research first, execute later.

It's not the other way around, in fact, if you do it the other way around you're going to do your research the hard way; rewriting it in the layers you should have used in the first place.

Is your DataMapper CouchDB adapter PUT'ing where it shouldn't?

It sounds gross, sure...but this was actually a simple problem I foresee someone running into and potentially running around mad about for an hour if they're not too keen on how DataMapper/CouchDB work together. Even if you DO know how they work together, this was still a fun problem to track down.

So let's take an example model that's extremely simple:

class Person
include DataMapper::Resource
# CouchDB Specific
property :id, String, :key => true, :serial => true, :field => :_id
property :rev, String, :field => :_rev
#App Specific
property :name, String, :nullable => false, :length => 75, :key => true
end


Looks harmless, right? That's what I thought too! Well, if you begin your application and get to the point of actually creating a Person you'll be greeted with an error that kicks off with "EOFError: end of file reached ".

So what's happening here?

Basically in the adapter, there is a LOC in the create method where it checks to see if there is a key or not (Yea, the one we've specified above as :name).

This creates an assumption of using PUT instead of POST and that is where the issue stems from. So, to fix this problem, all you need to do is DROP the :key => true and things will run as they should.

Now you're probably asking "But, what if I NEED to have that key there, I NEED to relate my models with a foreign key". Not to sound harsh, but just to be clear, you may want to revisit why you're using CouchDB in the first place.

osCommerce and PHP5

The splendid little e-commerce application that could[n't]: osCommerce is a bit quirky running on the "bleeding edge" of PHP[5]. Just as a quick heads up to you AND any would-be script-kiddies it requires that both of the following are enabled - just as a caution, anyone concerned about their sites well-being and knows anything about PHP configs should get a bucket:

register_globals = On
register_long_arrays = On

Everyone done dry-heaving? Good! I was absolutely beside myself when I saw what was required to run this application. It's been around for a few years now and it is shocking that they still operate with such disregard for online purchasing security AND performance.

So anyway, if you're seeing issues in your admin area to the tune of tep_href_link NONSSL or really anything related to tep_href_link and then some path (check your query string) it's likely you need to enable those bountiful long arrays.

*Insert drawn out sigh here*

Can't we all just shopify?!?!