Making RabbitMQ Stew

I just wanted to write this up real quick for anyone who may run into the same problem I did...

I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.

Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:

Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...

Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application

Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.

-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.

and

-mnesia dump_log_write_threshold Max, where Max is an integer which specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. It defaults to 100 log writes.

I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.

Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:

=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary

Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.

So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/

This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached

Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===

Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"

=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672


The short of it - whipe your mnesia directories and start over.

Who Knew?!?!

...that it would be more than two months since my last post.

...that Rails and Merb would get into bed with each other. (And may or may not come out producing something useful).

...that we would be on the verge of collapsing economically (ok, some did, but I didn't).

...that the Boston Bruins would lead the NHL in points almost half way through the year by winning 10 games straight! (they better knock this off, my Red Wings need some love)

...that the Tennessee Titans would be the 1st seed in the NFL playoffs.

...that I would grasp enough Erlang to seriously consider using it for an application.

...that I could come up with a post as "clever" as this with a horrid head cold.

Ok, well now that the busyness of Nov/Dec is over I can hopefully return to more sharing...Stay tuned.

Reia where scripting meets concurrency

Reia reminds me of that Reese's peanut butter cup commercial:

Hey, you got your Ruby/Python in my Erlang! Hey, you got your Erlang in my Ruby/Python!

And then there was this light that went off in someone's head - Why couldn't you bring the "stuff" people like about higher level languages down into something that, quite frankly just kicks ass, in Erlang?

To kick it off - and to quote the Reia wiki:

Reia (pronounced RAY-uh) is a Python/Ruby-like scripting language for the Erlang virtual machine (BEAM). Reia aims to expose all the features and functionality of Erlang in a language more familiar to programmers of scripting languages, while improving string handling, regular expressions, linking with external libraries, and other tasks which are generally considered outside the scope of Erlang. Reia is distributed under the MIT License.

Well, it's happening right now with Reia. I've been watching this project over my GitHub feed for maybe a month or so now and it's really cool seeing the leaps and bounds that Tony Arcieri is making with this little gem of a language. Granted, he still has a ways to go (and he openly admits this), I suggest giving it a try for fun() (that's my new favorite pun by the way).

Let me paste a few ideals from the Reia site to pique your interest a little more:
Concurrent programming is the future - very true, now if DB's would just catch up (oh, wait CouchDB!

You don't need an "everything is a..." language - he calls out SmallTalk and Ruby, but his argument is convincing, I think we've all been here before.

Don't build what you can steal - I really like that quote in regard to using a syntax of Python/Ruby and then laying it lovingly over the Erlang VM and OTP framework.

Judging a book by it's "sucks" result

A friend of mine asked me recently to go in on a little venture of his in the gaming industry. Unable to say "no" to what I believe is a fantastic idea, I went along with it...

Now there was another guy involved (technologically speaking) who wrote something that was a working prototype (awesome). It came down to a little discussion as to what WE should use for the front-end and storage mechanisms.

Well, having deployed a few real life applications in Ruby and having success with them, I only felt the need to display my affection and trust that I had with it and it's frameworks (yes, both Rails and Merb).

The problem I had wasn't that the guy wanted to use Python/Pylons or something in that realm. It was the argument. In fact, now that I've heard it a few times I can almost pinpoint exactly where the debate material is coming from. And that my friends, is simply fantastic. We're still not going along with the right tool for the right job, we're still shooting from the hip with what "we like".

Uh, but Brad, you just said you liked Ruby? You're right, I did. But given the requirements I was handed and based on past experiences it appeared to be the right tool for the right job.

I'm open to using new things - if I wasn't why would I ever give things like CouchDB and Erlang the time of day? They're sitting right on the edge of the programming paradigm shift from what most coders/programmers/developers are use to and it fascinates me.

Back to it.

I'm fine with using Python or whatever for a web interface. One more (hopefully) successful notch in my belt and slap proudly on a resume. But it just absolutely kills me when the keys of an argument are literally plucked from simply searching Google for "why sucks". What's even worse is when it's a predominantly blog rifling .

Please, please, please, I'm begging you the reader of this blog post; the rule here is simple, I'll even wrap it in a blockquote for you to save and tattoo across your chest in a Tupac-ish style:

Research first, execute later.

It's not the other way around, in fact, if you do it the other way around you're going to do your research the hard way; rewriting it in the layers you should have used in the first place.

Is your DataMapper CouchDB adapter PUT'ing where it shouldn't?

It sounds gross, sure...but this was actually a simple problem I foresee someone running into and potentially running around mad about for an hour if they're not too keen on how DataMapper/CouchDB work together. Even if you DO know how they work together, this was still a fun problem to track down.

So let's take an example model that's extremely simple:

class Person
include DataMapper::Resource
# CouchDB Specific
property :id, String, :key => true, :serial => true, :field => :_id
property :rev, String, :field => :_rev
#App Specific
property :name, String, :nullable => false, :length => 75, :key => true
end


Looks harmless, right? That's what I thought too! Well, if you begin your application and get to the point of actually creating a Person you'll be greeted with an error that kicks off with "EOFError: end of file reached ".

So what's happening here?

Basically in the adapter, there is a LOC in the create method where it checks to see if there is a key or not (Yea, the one we've specified above as :name).

This creates an assumption of using PUT instead of POST and that is where the issue stems from. So, to fix this problem, all you need to do is DROP the :key => true and things will run as they should.

Now you're probably asking "But, what if I NEED to have that key there, I NEED to relate my models with a foreign key". Not to sound harsh, but just to be clear, you may want to revisit why you're using CouchDB in the first place.

Readability vs Optimization

I recently saw a post come across my Google Reader a few days ago asking what someone should code "for". His two options were Readability or Optimization. I'm not going to link the article for the simple reason it was incredibly brief and it was mostly just a question, the author never really jumped into going one way or the other or leading from one to the other.

So have no fear, I'll do it!

What makes sense? You want code you can read, yet you also want it to run incredibly fast. Interesting! Well, like any project the first one to market usually gets the worm. So cranking out code (read: code that "works") is typically a developer's (and many times the project managers) first concern. Do we have something out there to start drawing in the cash? Yes. So you've effectively coded for "readability" since we're not optimized. You've gotten your product on the market which is really what your first goal SHOULD be, unless of course there's some ungodly amount of volume that you KNEW of and should have optimized according for it in the first place.

Now, wait a minute...that sounds like common sense. If I'm developing something new that no one knows about, what's the point of optimization? I can just crank stuff out that "works" and polish up that while we gain customers/visitors/whomever. Correct!

On the flipside of the previous paragraph, our (I mean everyone's, not mine personally) good friend with a framework Ezra Zygmuntowicz had an interesting quote a while back where he said "Post-optimization is the route of all hosting bills". In the article I'm extrapolating on, one commenter said if one codes strictly for readability, optimization will come automatically because hardware continues to get faster. The author himself also said "Performance can be optimized at the hardware level", which blew my mind. Why are you even asking for hints on code optimization (for PHP) if you already made up your mind to throw hardware at it? I can't speak for him, but I'm pretty sure you're not going to be replacing your systems with the latest and greatest hardware every 18 months are you? Hmm, Nope!

So what do we need to do?

Well, we've already cranked our code to production, things may or may not be fine with the community accessing it, and I'm sure you've been paying close attention to which aspects of your system need work, help, or simply need to be scrapped and redone. Wait, you haven't? Ok, let's start there.

Pay attention to what your application is doing.

Get a decent logging or monitoring system and pay close attention to the needs of your app. Many times we get so caught up in producing that we never pay close attention to the simple things once it's out in the wild. EVEN MORE TIMES than not, people don't see what applications that may be similar in nature are doing or how they're reacting online to try and investigate why they are having these problems. Chances are, your application will have them too - you cannot assume that just because you wrote the code that it is exempt from bottlenecks. In fact, I'd say the vast majority of the time, you're going to hit the exact same problems as your competitors.

Do your research, if they're a fairly transparent company see what growing pains they were having and try to plan accordingly.

Now that we've addressed the issues surrounding our application AND applications which are similar in nature to ours we can now effectively move into the space where we begin optimizing our code. This is a rather vast term and your focus should (in my opinion) stem from your I/O bottlenecks first before jumping to your code. If all seems well in that area or you simply don't have the funding to do much with it, then sure, you're ready to start optimizing code.

So in essence the life cycle is as follows:
1 - Produce working code that meets your needs.

2 - Analyze how it and other I/O (disk/network/database) are affecting your application.

3 - Attempt to alleviate as much of that I/O overhead as you can within your boundaries (manpower, funding, etc...).

4 - Optimize your code.

I think that will get you through the day. I could be wrong, I flew through this post pretty damn fast. I feel a site optimization post coming soon, so get ready for some more Kool Aid to drink.

Ruby vs. Erlang vs. The People vs. Your Application

For the past six months or so I've been working on a somewhat under the radar service called Mumbl. Basically, mumbl is a micro-blogging service that solves the whole "Oh, we chose the wrong framework for the wrong platform and our db won't/can't scale" problems most of the bigger folks have. And for all I know they may well have already solved their issues as it seems to be getting better(?) as less and less downtime occurs. Good for them, the Mongolian Horde/money shovel approach to systems and development worked...for now.

Anyway, there's been a momentum gainer amongst Rubyists looking more and more at Erlang. Which, yea is great but I think one of the biggest problems with Ruby developers is that we (yes, I'm one too) don't just drink the punch if things "make sense", WE CHUG IT! We ask that it be hooked intravenously into our veins until it becomes a part of us on a molecular level.

So, having said that, why would someone that's vested deeply in Ruby, which preaches readability and quick, elegant solutions to common problems (whether it be via frameworks or gems) have any interest in Erlang? Erlang is based on ant-turd tokens, which aren't exactly easy to read. Erlang appears to also be fairly verbose in a sense that I can't see it "fitting" for your typical online applications and it certainly isn't the easiest thing to read. So please, if you need to undertake a project that would quite easily be completed using the rails/merbs/django's of the world out there - then utilize them! It's my opinion that Erlang is not a language for application that need constant adjustment, in this particular case applications for the online masses. Leave it to the back-side and you'll thank me on your wedding night!

This was touched on a little bit a while ago (the Erlang thing, not the back-side thing), I had to go dig up the post but you can see Damien Katz's blog entry about a few things he disliked about Erlang. See the comments of his posting for a real roller-coaster ride!

Anyway, getting back to the point here. I think we need to be careful when given the option of the punch, which really is true for anything. But something we, as developers should look at so closely is the separation of trendy with practical. I believe we can all agree that a readable/maintainable application layer with a concise entry point to a highly parallel back-end is probably the best way to go. Maybe not; well, let me try and convince you otherwise.

If you had to throw a jack of many trades developer into the mix of your application I described above how do you think he'd handle it? I think they would greatly appreciate the easy readability of the Ruby code over the Erlang. Once they ramped their way up gaining and understanding of the interfaces it would likely make their lives easier once they made it into the land of Erlang.

So please, let's keep it separate. And let's please, please stop creating half-assed Ruby implementations of things that really don't need it. This recent post on the register certainly struck a nerve with me. It was apparent they we are getting lazy and far too comfortable in our language to expand horizons and use the right tool for the right job. That tool doesn't need to take over completely, getting back to the punch-chugging I mentioned earlier. You need to find it's proper place in your application if it even fits at all, don't make a place for it. And certainly don't make a cheesy knock-off of it in Ruby so you get 1/2 points for utilizing some new (or trendy) methodology.

Ah...now that' feels better.