Judging a book by it's "sucks" result

A friend of mine asked me recently to go in on a little venture of his in the gaming industry. Unable to say "no" to what I believe is a fantastic idea, I went along with it...

Now there was another guy involved (technologically speaking) who wrote something that was a working prototype (awesome). It came down to a little discussion as to what WE should use for the front-end and storage mechanisms.

Well, having deployed a few real life applications in Ruby and having success with them, I only felt the need to display my affection and trust that I had with it and it's frameworks (yes, both Rails and Merb).

The problem I had wasn't that the guy wanted to use Python/Pylons or something in that realm. It was the argument. In fact, now that I've heard it a few times I can almost pinpoint exactly where the debate material is coming from. And that my friends, is simply fantastic. We're still not going along with the right tool for the right job, we're still shooting from the hip with what "we like".

Uh, but Brad, you just said you liked Ruby? You're right, I did. But given the requirements I was handed and based on past experiences it appeared to be the right tool for the right job.

I'm open to using new things - if I wasn't why would I ever give things like CouchDB and Erlang the time of day? They're sitting right on the edge of the programming paradigm shift from what most coders/programmers/developers are use to and it fascinates me.

Back to it.

I'm fine with using Python or whatever for a web interface. One more (hopefully) successful notch in my belt and slap proudly on a resume. But it just absolutely kills me when the keys of an argument are literally plucked from simply searching Google for "why sucks". What's even worse is when it's a predominantly blog rifling .

Please, please, please, I'm begging you the reader of this blog post; the rule here is simple, I'll even wrap it in a blockquote for you to save and tattoo across your chest in a Tupac-ish style:

Research first, execute later.

It's not the other way around, in fact, if you do it the other way around you're going to do your research the hard way; rewriting it in the layers you should have used in the first place.

Readability vs Optimization

I recently saw a post come across my Google Reader a few days ago asking what someone should code "for". His two options were Readability or Optimization. I'm not going to link the article for the simple reason it was incredibly brief and it was mostly just a question, the author never really jumped into going one way or the other or leading from one to the other.

So have no fear, I'll do it!

What makes sense? You want code you can read, yet you also want it to run incredibly fast. Interesting! Well, like any project the first one to market usually gets the worm. So cranking out code (read: code that "works") is typically a developer's (and many times the project managers) first concern. Do we have something out there to start drawing in the cash? Yes. So you've effectively coded for "readability" since we're not optimized. You've gotten your product on the market which is really what your first goal SHOULD be, unless of course there's some ungodly amount of volume that you KNEW of and should have optimized according for it in the first place.

Now, wait a minute...that sounds like common sense. If I'm developing something new that no one knows about, what's the point of optimization? I can just crank stuff out that "works" and polish up that while we gain customers/visitors/whomever. Correct!

On the flipside of the previous paragraph, our (I mean everyone's, not mine personally) good friend with a framework Ezra Zygmuntowicz had an interesting quote a while back where he said "Post-optimization is the route of all hosting bills". In the article I'm extrapolating on, one commenter said if one codes strictly for readability, optimization will come automatically because hardware continues to get faster. The author himself also said "Performance can be optimized at the hardware level", which blew my mind. Why are you even asking for hints on code optimization (for PHP) if you already made up your mind to throw hardware at it? I can't speak for him, but I'm pretty sure you're not going to be replacing your systems with the latest and greatest hardware every 18 months are you? Hmm, Nope!

So what do we need to do?

Well, we've already cranked our code to production, things may or may not be fine with the community accessing it, and I'm sure you've been paying close attention to which aspects of your system need work, help, or simply need to be scrapped and redone. Wait, you haven't? Ok, let's start there.

Pay attention to what your application is doing.

Get a decent logging or monitoring system and pay close attention to the needs of your app. Many times we get so caught up in producing that we never pay close attention to the simple things once it's out in the wild. EVEN MORE TIMES than not, people don't see what applications that may be similar in nature are doing or how they're reacting online to try and investigate why they are having these problems. Chances are, your application will have them too - you cannot assume that just because you wrote the code that it is exempt from bottlenecks. In fact, I'd say the vast majority of the time, you're going to hit the exact same problems as your competitors.

Do your research, if they're a fairly transparent company see what growing pains they were having and try to plan accordingly.

Now that we've addressed the issues surrounding our application AND applications which are similar in nature to ours we can now effectively move into the space where we begin optimizing our code. This is a rather vast term and your focus should (in my opinion) stem from your I/O bottlenecks first before jumping to your code. If all seems well in that area or you simply don't have the funding to do much with it, then sure, you're ready to start optimizing code.

So in essence the life cycle is as follows:
1 - Produce working code that meets your needs.

2 - Analyze how it and other I/O (disk/network/database) are affecting your application.

3 - Attempt to alleviate as much of that I/O overhead as you can within your boundaries (manpower, funding, etc...).

4 - Optimize your code.

I think that will get you through the day. I could be wrong, I flew through this post pretty damn fast. I feel a site optimization post coming soon, so get ready for some more Kool Aid to drink.

Merb does it again...

Remember in the old days when you'd create what I can only explain as a "temporary session" in Rails to notify someone ONCE on a page of something? It could have been a form validation error, a success, just anything that you needed to toss up to the user temporarily. Still with me?

Merb now has this wonderful "message" implementation which they've hooked onto the redirect method. So, by calling:

redirect '/some/path', "We're gonna need a bigger boat..."

You receive a variable "message" that contains your message (conventional, right?). It's basically marshalled and plopped into the querystring and then deserialized into the var. I really like this over the Rails method...but, following what the Pug says - I'm not pitting merb against rails.

.erawtaolB !lleh eht tahw ,naem I .noitatnemelpmi laivirt a hcus ekil smees siht nehw o/i pu gnitae ,snoisses pmt etaerc ot tnaw uoy od lleh eht yhw si gniyas m'I llA

It's late, I'm wiped out and that's the "best" I could do for humor.

DataMapper and CouchDB

It's likely that if you're a regular reader of this hapless place I call a blog then it's likely that you either:
a) have heard me spew off at the mouth about how incredibly awesome couchdb is
or
b) have not and are likely ready to completely disagree with the entire philosophy behind it and call me an idiot. Which is fine; that's fine!

Anyway to quote the wonderful couchdb page:

Apache CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API. Among other features, it provides robust, incremental replication with bi-directional conflict detection and resolution, and is queryable and indexable using a table-oriented view engine with JavaScript acting as the default view definition language.

So rather than having blobs of tables prone to locking or even rows that are prone to locking. Either way, no matter which RDBMS you use, you're likely to be trapped into the same issues.

A quick rant - it's not the language, it's the I/O. Be it database, disk (likely from a database) or the network; your language doesn't matter. An incredibly well-crafted application will still fall to pieces because of one of those three issues.

Back to it - so with couchdb, you store documents (think of them as rows, but they can be 3-dimensional), you can store just a standard field (text), arrays, hashes, marshal and base64 encode an object, you name it you can put it in there. And it's all linear and most importantly it's based on map/reduce.

So in the old country where you'd key in on those ever-so important relationships and eager-loading to save on the n+1 problem, you don't necessarily run into that with couchdb. Why? Because your map function will return all of the documents that are of interest to one another for your view. You're simply storing crap in all of the files, the more de-normalized the better! You can think of it sort of like this: a row could be considered a document. But within your document, you store all of the pertinent information instead of just storing an _id field with the appropriate row's id as a value. It works to get rid of those horrible join queries by de-normalizing the data.

It's really rather difficult for me to talk about as I'm still wading my way through it. I'm really not confident in my explanation about, but again I'm trying to find this as I go along as well. So feel free to post any "fixes" in the comments as seen fit.

Enter the DataMapper.

As of datamapper 0.9.2 they have added an adapter for couchdb. Some people would really like to just put some javascript together and link directly to couch to save some layers, but the fundamental problem with that (for now) is that you give up your databases security. A HUGE thing, I know. Even for people using it with ORM's in front of it, it's still problematic. You're prone to attacks because it's purely HTTP-based. And spoofing over HTTP is a joke. If you want to know why or how I know these things, just ask.

Anyways, the benefit of using an ORM in front of the database is two fold for me. First, I don't have to worry about some script-kiddie coming across, scraping my site and yanking my application out from under me. Yea, maybe that's selfish of me being an open source developer, but hey, I use open source to make; guess what? MONEY! Uh, where the hell was I? Oh yes. So the second thing aside from separating the databases secure-less protocol is simply the ease of use when tying it into the actual metal of my application. Validations, simple object creation, abstraction it's all there and makes my application a wonderful thing to work in.

Now, for every good there is certainly a bad. Currently DM's CouchDB adapter only works with version 0.7.2. There are a whole host of people trying to write ruby-based ORM's that work exclusively with CouchDB. That's great! Really, it is. But there is a long, long road to tread when you undertake such projects. Which is why I have chosen to stick by it and just alter the adapter to suit my needs and then when the time comes, release it back into the open. It's not a huge endeavor to alter the adapter as it stands now. Having hacked through the guts of it for the last week, I have come to greatly respect all of the hard, intelligent work that goes into DataMapper and Merb respectively. When comparing it to some of the internals of Rails, with all of the "magic", super calls, alias_method_chains and what not, one can truly appreciate readable, sensible code that you can literally walk through without having to do a stack trace! Excellent job, men!

Finally, this wouldn't be complete without me sharing a little nugget of information. This may not be the best way to go about handling this certain "problem" but it works well for me and maybe it will work well for you too (if you decide to drink the CouchDB punch)!

Let's talk about paging for a moment. How do you typically do it in applications like Rails, PHP, *.NET, anything really? Well, you typically need to perform two queries, right? One that gets the total count of records that you'll be acting upon and another to pull in that data a little at a time (via a limit and offset). Luckily for us, CouchDB actually provides you back the full list of documents which were queries on. Thus eliminating the count(*) query. So what I have done is modified the CouchDB adapter just slightly to return back an attr_reader which holds the total number of documents in your view. See the following code samples:
The first, is the override on how DataMapper's Collection works. My first thoughts were to just include the total rows with each object in the Collection. But that's sloppy, eats up memory and you wouldn't have access to it without instantiating one of the objects in the collection (redundant, yea?). So, let's go up a level and just attach it to the Collections themselves. Again, maybe there's a better way but hey, this is what I came up with.

module DataMapper
  class Collection < LazyArray
    attr_reader :_total_rows
    private
      
    def initialize(query, options = {}, &block)
      assert_kind_of 'query', query, Query

      unless block_given?
        raise ArgumentError, 'a block must be supplied
      end

      @query          = query
      @key_properties = model.key(repository.name)
      @_total_rows = options[:total_rows] || 0
      super()
      load_with(&block)
    end

  end
end

The second sample is where we pass in the total_rows attribute from the returned document's JSON store and pass it as an option to the Collection.initialize method as it is constructed. We grab the value, pass it up and it's now available in one spot for your collection as a read-only attribute. Bye-bye count query!

module DataMapper
  module Adapters
    class CouchDBAdapter < AbstractAdapter
      def read_many(query)
        doc = request do |http|
          http.request(build_request(query))
        end
        Collection.new(query, {:total_rows => doc['total_rows']}) do |collection|
          doc['rows'].each do |doc|
            collection.load(
              query.fields.map do |property|
                property.typecast(doc["value"][property.field.to_s])
              end
            )
          end
        end
      end
    end
  end
end

So please do yourself a favor, investigate couchdb, it's nowhere near perfect at the moment, but you owe it to yourself to expand your horizons and really marvel at the speed at which it performs. You know that "lag" in ActiveRecord/DataMapper when you're operating on rows in MySQL/Postgres or even Oracle? Guess what? It's simply not there!

In fact on a server of mine which only has 256Mb of RAM, I can write a record, pull it, build up and tear down a merb thread and respond to the browser in less than a second. The age of the created_at I store in the document reads "0 seconds ago". Now, right now you're thinking well Bradford, that's not really impressive...I mean what about concurrency? This web stuff is all about concurrency! You're right! But I forgot to mention that I was running siege on the same server running 20 concurrent connections reading the same stuff I was putting in and it was still just as fast. On cheap, un-optimized hardware it performs incredibly fast. So please, check it out - the guys who work on this thing are wonderful (irc.freenode.net #couchdb) and it's really exciting to be watching this project grow!

Next up - how mowing the grass and application development could maybe one day have a lot in common. Aside from both being done electronically.

Oh Noes, Your Business is Under Attack!

A somewhat interesting link came through my RSS reader this morning, it wasn't the well articulated basis of this gentleman's argument, nor was it the catchy title. What really made me think (and laugh) was the first post by a man named Will, who apparently works for the aforementioned company "waging war" upon Ruby/Rails.

I think it speaks volumes if you or the company you work for really has to dive into some random persons personal blog and begin defending yourself when all this guy is doing is giving an opinion. You're putting out a competing product (uh, sure!) and that's great...some will use it many will not. But it really makes me shake my head (knowing some people who work at Zend personally) when I see a Zend'er come out with the gloves up just clutching and grabbing for some traction on their products worth.

Really, if your product was worth a damn in the first place you'd likely have 9 other posts to this one touting the grace and power your product encapsulates. Instead, you put those gloves up because you know you have put out an inferior product and are trying not to be the last kid picked.

Finally, I must admit here that "Will" (IF that IS your REAL name...haha) does a good job of playing the diplomat, trying not to insult the blogger's intelligence and still keep him as a potential client (although, from the sounds of it, he's made his decision).

It's really great to have passion (see my last post) but sometimes you can't let that get in the way of what people outside your click think. Let them have it, if you're right, well then they'll all see in the end when you succeed.

Square pegs in round holes Pt. 1

Is everyone here? No? Fuck it, I'm going anyway...

I'm currently employed at a medical laboratory that was bought out a few months ago. In the beginning there were a few changes that needed to be made to the data-model (expected things). We (the people in "IT") were told that standard operating procedure at this "new" company was that everything server/application related would get swept into the corporate fruitcake and moved to North Carolina.

It was then apparent to these worker-bee's of the new corporate entity that this did not sit well with our group (which consists of me; a software developer, a network administrator and a help-desk technician). So stories were being floated down from people who, in the corporate hierarchy, are nothing more than someone's gopher about how we were unique and how everything would stay as is. Jobs weren't in jeopardy and life was good - not a care in the world. This same type of story was handed to other departments, sans servers of course.

Various meetings were held telling everyone to sit tight while things were jumbled about - you can insert your favorite corporate buzzwords here. I'm sure if you pictured it in your head you could just imagine this meeting. A southern gentleman who resembled Cotton Hill from King of the Hill standing in front of 80 people, fielding questions ever so diplomatically. The nervousness and longevity of his answers went completely unnoticed by my co-workers who resembled in my eyes a bunch of dogs at obedience class. *You can stop nodding now*

No one thought for a second to have a plan. They let these people waltz in and rip the rug right out from underneath them. Sad.

There's part of me that has some empathy for them but to take such a gamble on you and your families well-being based on some slick-talkin' stranger is simply ridiculous.

ESPN Radio and their advertisers

I have XM Radio and have had it for a number of years. I love it. I also love ESPN. So naturally having the two together is great. What's not so great is the advertising that ESPN hocks on it's xm channel (140). It's the same stuff every break: "Male enhancement in 3-5 seconds?!" or "Tired of seeing everyone around you living the good life?". You'd think a media juggernaut like ESPN could pull in some better schwag to push into our ears than this garbage.

I'm not sure how much longer I can take it; I mean you always have the ability to change the dial...but I'm American, that's a lot of work.

Thinking back on it - I think a solid work around would be the ability (like you have on a directv remote) to hit a "previous" button so that hopping between channels is easier.

This was a shitty rant...but what do I care - it's just a place for me to let the trash out of my head. It will be (I'm guessing) a good six months before anyone even see's this stuff.