So I have some downtime in the hospital with my son and don't really have much motivation to do anything at the moment...so, why not blog?
I've been hired recently by Vanderbilt University's Center for Human Genetic Research in which I'll be responsible for some super sweet SNP (single nucleotide polymorphisms or "snips") searching mechanisms and I will also be responsible for creating a service for sharing this information with other universities in what I can only think of as a "genetic research alliance" of sorts.
I am really looking forward to see what I learn and can apply to this job, hopefully I can put something together that helps the researchers here find something useful for humanity or even just my son!
What I've found amazing thus far is how much of an educational investment they (Vanderbilt) are willing to pump into you if you are up to it. They allow for "auditing" of classes which pertain to your field of work, so for me (from what I understand) I'll be auditing a biological statistics class and a genetics class come January (or whenever the next semester starts, but I think that's right).
This whole realm of genetic information has really put my mind at work in terms of storage (we are talking petabytes here) and retrieval. Something that's crossed my mind is the thought of consistent hashes almost like bookmarks in an individual's sequence. I'm sure there are flaws in thinking of it this way, but hey, it's a start. And if nothing else, I'll learn more from the failures anyways.
Which leads me to my next exciting topic. Since it is research work, there is of course the possibility of trying out some of the non-standard (read: "new fangled and non-relational") storage systems in order to see if they function better for searches over current methods. Sounds like a real drag, eh?
Hopefully I can regurgitate more than less in regard to the specifics of what I'm working on in the coming years. The amount of data and the structures within it are truly a rather amazing challenge the more I think about them.
Time for the rambling to end, it's almost time for T's late night bath - we're up to every 12 hours to wash the chemo off of him now, so that's one of the first steps towards getting him home.
Thursday, October 15, 2009
Wednesday, August 5, 2009
Benefit Reminder Aug 7th 5pm at Tin Roof Nashville
This Friday, August 7th from 5pm to 9pm at the Tin Roof in downtown Nashville, TN we will be holding my son's benefit concert.
Details are below, clipped from the electronic flier our wonderful organizer put together:
***CONFIRMED ARTIST FOR AUGUST 7TH!!!!!***
Richard Marx has so graciously offered to fly in for our event and sing!!! We are thrilled and can't wait to hear his awesome music!!! Thank you Richard!!!!
He has also invited two special guests that you just might know...Danielle Peck and Coley McCabe!!! Can't wait to hear them all on Friday!!!
***Confirmed Independent Artists***
Michael David
Erika Jo
Kelsie and Ryan
Joe Wiese
Clayton Quisenberry
Sarah Lenore
Brian Desveaux
Shanna Crooks
Gary Slayton
Holly Bakehorn
Devin Belle
Candice Crain
There will also be a silent auction full of awesome items such as gift certificates to local restaurants, clothing stores, Titans and Predators merchandise, autographed 8x10's, posters and t-shirts signed by the artists (big names!!) and so much more!
We would love to pack the Roof and raise a ton of money for this family...I can't even imagine what they are going through right now...please keep them in your thoughts and prayers. We hope to see you on the 7th!!!
*Even if you can't make it out to this event and would like to donate, please see the information below and you can give directly to the bank where there is a fund set up for my son.
Also if you want a support bracelet ($5) you can contact me at bradswinfrey over at yahoo dot com.
Details are below, clipped from the electronic flier our wonderful organizer put together:
***CONFIRMED ARTIST FOR AUGUST 7TH!!!!!***
Richard Marx has so graciously offered to fly in for our event and sing!!! We are thrilled and can't wait to hear his awesome music!!! Thank you Richard!!!!
He has also invited two special guests that you just might know...Danielle Peck and Coley McCabe!!! Can't wait to hear them all on Friday!!!
***Confirmed Independent Artists***
Michael David
Erika Jo
Kelsie and Ryan
Joe Wiese
Clayton Quisenberry
Sarah Lenore
Brian Desveaux
Shanna Crooks
Gary Slayton
Holly Bakehorn
Devin Belle
Candice Crain
There will also be a silent auction full of awesome items such as gift certificates to local restaurants, clothing stores, Titans and Predators merchandise, autographed 8x10's, posters and t-shirts signed by the artists (big names!!) and so much more!
We would love to pack the Roof and raise a ton of money for this family...I can't even imagine what they are going through right now...please keep them in your thoughts and prayers. We hope to see you on the 7th!!!
*Even if you can't make it out to this event and would like to donate, please see the information below and you can give directly to the bank where there is a fund set up for my son.
Also if you want a support bracelet ($5) you can contact me at bradswinfrey over at yahoo dot com.
Hardship
It's not often that I blog about life events (or at least it doesn't feel that way) but this one, or rather this series of events certainly requires some ventilation.
In May of '09, we were noticing that my (at the time) youngest son was walking a bit wobbly, not constantly however, so perhaps he was just growing into himself at the time as he had grown a great deal in height the last few months. I really wish that had been the case.
No. Instead, the only thing growing which, in turn was causing his wobbly gait was in fact a tumor. More specifically a cancerous tumor classified as Medulloblastoma.
I remember it very clearly, taking him in for his MRI and making a remark to my wife that we should take him out for a little present for the rough morning he had been through. Heh, well that was wishful thinking wasn't it? They brought Thomas in from his MRI (he was out from his anesthetic still) and began to wake him up a bit. About five minutes or so after he was awake again a team of two doctors came in to tell us that he had a tumor roughly about the size of a golf ball growing between his brain stem and his cerebellum. My mouth (from what my wife tells me) dropped. My wife, 8 months pregnant at the time bursts into tears. My son, the sweet little thing that he is tells his mother that "it's ok, what's wrong?". I calm her down while still talking to the doctors standing in our little 5x12" recovery room. They said that they specialize in these specific types of tumors and that they require immediate resection and typically treatment. But that certainly doesn't just make it all better or tolerable, it just means that you are putting your feet down on a path that leads you down a very, very long journey where you don't know what the end result is.
Shortly thereafter, we were taken up to the ER to wait while they prepared a room in the critical care unit at VCH. Numerous people were in and out of our room to explain what was about to go on, things we should be doing to help in the meantime...it was all quite overwhelming as one can imagine. We were then taken up to our new temporary home on the fifth floor where they monitored every little aspect of our little boy for the next 72 hours until his surgery.
I'd like to share this short story for a moment before I explain the surgery. The day prior to his surgery, my wife and everyone else who had come down to help out had left the confines of our room and floor to get some fresh air and try to re-energize for the long day ahead. I stayed back to be with my son for some quality time. We played with his x-men and star wars figures for a while, making the usual little jokes we have with one another. When all of the sudden he slowly stops playing, looks up at me with a look I couldn't even describe and says "Daddy, I'm scared..." in his then slightly delayed little voice. I immediately responded to him by saying "you have nothing to worry about buddy, you are going to get through this, because you're the toughest little guy I know". He smiled a little and flexed his little muscles like the Incredible Hulk. I excused myself from our little battle to walk over to the window where I began to feel control of every bit of sanity I had slip away. I did everything I could to hold back the overwhelming feelings of fear and uncertainty to keep him calm and to not show him that I was sad or fearful. I had to be strong for him; really for everyone in our family; even every one of our friends and their outstretching networks of people who had heard of what happened to my child.
I had to stop. HAD TO. I couldn't. A few short outbursts of mumbled crying mixed with what I can only describe as hysteria leaped out from my chest and through my eyes. I held it in so hard that for the rest of the day I had the worst pressure headache throughout my entire skull. Between that, the lack of nourishment from stress and the lack of sleep I had to remind myself. This was nothing compared to the awful things going on inside my little boys head right now. So it passed...and I endured.
Back to it...
It was now 4:30am Monday morning, my son's surgery was scheduled for 8:00am. At 7:30 that morning they came and got him ready for his long day and explained to me the risks of the surgery which I was about to allow these strangers to perform on my sons brain. HIS BRAIN! Stroke, loss of facial movements (smiling, blinking, etc...), loss of ability to swallow, balance issues, death. Sounds like a decent trade off right? I signed the agreement. Once I saw them ship him off in his little crib, head marked for where incisions would be made (thankfully they only needed one on the back of his head) I remember just being completely numb. Everything tasted the same that day, nothing was funny, no one was consoling enough, sleep wasn't possible, nothing was possible until I could see my child alive after his surgery. Which eventually did come. We received an update every hour, always positive from one of the nurses. When finally at about 4:30pm that same day they told us they were done and that he would be in the recovery room soon.
His surgeon came into the waiting room where we were and brought us into the consultation room to brief us on what had gone down. So I'll relay that information here:
A single incision was made to the back of my sons head. They then removed a piece of his skull in a square fashion in order to have room to operate. Once they were inside, they lifted the folds of the cerebellum and could see the tumor. From what they saw, it had not gotten tangled in his brain stem or invaded any other space in his brain. They were actually able to remove the entire tumor without actually cutting ANY of his brain tissue. Thank God.
When we were finally able to see Thomas he laid in his little crib looking very sweaty and exhausted. His face was incredibly swollen (I believe he was upside down for his surgery) and a little raw by one of his cheeks from the shifting around in surgery. But for the most part he looked the same as he ever did with the exception of the throbbing incision on the back of his head.
For the next week we got nothing out of him, not head shakes, not ughs or grunts just short bursts of sheer frustration and screaming. He was trapped, knowing full well what he wanted to do (seemingly) but unable to perform it. Another week went by, he'd have his eyes open more often, I had him reach out and pick a spiderman figure from my hand one day, he wouldn't do that for another two weeks. The only word he ever really spoke in the office was to his older brother, who brought him a Bumble Bee Mr. Potato Head toy to which he said "thanks". That again, would be the only thing he would say for about the next two weeks.
After that 2-3 week span he would surprise us each day. He had gone from not being able to hold his own head up or even being able to swallow to swallowing and actually LAUGHING. This may sound cliche, but the day I was able to get him to laugh seemed to really trigger a whole new list of things that he was able to do again. He began drinking in very small amounts (he had a feeding tube this whole time) and a week later after ripping out his feeding tube, he ate. And he would not stop eating from there on out until his 15th day of chemotherapy.
So here we are almost two months into this, he's begun his second cycle of his first treatment of chemotherapy (3 cycles in 1 treatment - each cycle is 21 days long). His hair has thinned significantly and he will most likely be completely bald soon. He has already fought off one infection. Just another testament to this little boy's strength. He's most often in good spirits while we are in the hospital, it was a blessing having him home for more than 28 hours (unlike the day he got his infection and we had to take him right back).
We recently welcomed our third child into this world on the 24th of July. So now we don't only have the privilege of taking care of a soon to be 3yr. old and a 5 year old but a newborn to keep us uh..."company" all night long (just kidding the wee one has been great). It's been a new hurdle every few weeks, for instance, just today (Aug. 7th) my oldest son started kindergarten. It will make things easier during the day, but not so much when he needs to be somewhere safe and loving after school. We'll take care of it. We have to!
Wrapping up, I can say that since the time my wife broke down when they told us our sons diagnosis, (with the exception of the choked back fear and anger in the hospital room that one day) I have not had an adequate means of ventilation. I thought perhaps this would help, but it really hasn't. I have not yet told my wife the story about him saying he was scared, I'm afraid it would tear her up too much (she doesn't read this blog, it bores her). Sorry, just had another support person drop off dinner for us. That's another thing, the support network of people we've had has been simply outstanding. Mowing the lawn (it's a big, very steep yard), cooking dinners, doing laundry, shuttling our oldest son places and so on, it's been a great, great help.
If there's one thing I've learned in all of this it's that God won't give me more than I can handle. I am intrigued to see just how far that limit goes as I believe it has been pushed much further than I would have expected. Once we get through this, I feel that my family from our newborn on up will be stronger collectively for it. Especially my little trooper.
I'll follow up this post with information regarding the benefit concert a friend of ours (who just today found out her father was diagnosed with brain cancer) has put together here in Nashville for our son to raise some money for his medical expenses (don't worry, we are insured, but insurance does not cover 100% obviously).
If you read all of this, I applaud you. It's tough to hold onto a though and go with it so I bet this was probably a choppy read...
Thanks again to everyone who has helped out through this and even those of you who just give me that pat on the back when it's clear things are rough. I can't say again how lucky I am to have such a great network of friends, family and even just online acquaintances.
Thank you all.
Brad
In May of '09, we were noticing that my (at the time) youngest son was walking a bit wobbly, not constantly however, so perhaps he was just growing into himself at the time as he had grown a great deal in height the last few months. I really wish that had been the case.
No. Instead, the only thing growing which, in turn was causing his wobbly gait was in fact a tumor. More specifically a cancerous tumor classified as Medulloblastoma.
I remember it very clearly, taking him in for his MRI and making a remark to my wife that we should take him out for a little present for the rough morning he had been through. Heh, well that was wishful thinking wasn't it? They brought Thomas in from his MRI (he was out from his anesthetic still) and began to wake him up a bit. About five minutes or so after he was awake again a team of two doctors came in to tell us that he had a tumor roughly about the size of a golf ball growing between his brain stem and his cerebellum. My mouth (from what my wife tells me) dropped. My wife, 8 months pregnant at the time bursts into tears. My son, the sweet little thing that he is tells his mother that "it's ok, what's wrong?". I calm her down while still talking to the doctors standing in our little 5x12" recovery room. They said that they specialize in these specific types of tumors and that they require immediate resection and typically treatment. But that certainly doesn't just make it all better or tolerable, it just means that you are putting your feet down on a path that leads you down a very, very long journey where you don't know what the end result is.
Shortly thereafter, we were taken up to the ER to wait while they prepared a room in the critical care unit at VCH. Numerous people were in and out of our room to explain what was about to go on, things we should be doing to help in the meantime...it was all quite overwhelming as one can imagine. We were then taken up to our new temporary home on the fifth floor where they monitored every little aspect of our little boy for the next 72 hours until his surgery.
I'd like to share this short story for a moment before I explain the surgery. The day prior to his surgery, my wife and everyone else who had come down to help out had left the confines of our room and floor to get some fresh air and try to re-energize for the long day ahead. I stayed back to be with my son for some quality time. We played with his x-men and star wars figures for a while, making the usual little jokes we have with one another. When all of the sudden he slowly stops playing, looks up at me with a look I couldn't even describe and says "Daddy, I'm scared..." in his then slightly delayed little voice. I immediately responded to him by saying "you have nothing to worry about buddy, you are going to get through this, because you're the toughest little guy I know". He smiled a little and flexed his little muscles like the Incredible Hulk. I excused myself from our little battle to walk over to the window where I began to feel control of every bit of sanity I had slip away. I did everything I could to hold back the overwhelming feelings of fear and uncertainty to keep him calm and to not show him that I was sad or fearful. I had to be strong for him; really for everyone in our family; even every one of our friends and their outstretching networks of people who had heard of what happened to my child.
I had to stop. HAD TO. I couldn't. A few short outbursts of mumbled crying mixed with what I can only describe as hysteria leaped out from my chest and through my eyes. I held it in so hard that for the rest of the day I had the worst pressure headache throughout my entire skull. Between that, the lack of nourishment from stress and the lack of sleep I had to remind myself. This was nothing compared to the awful things going on inside my little boys head right now. So it passed...and I endured.
Back to it...
It was now 4:30am Monday morning, my son's surgery was scheduled for 8:00am. At 7:30 that morning they came and got him ready for his long day and explained to me the risks of the surgery which I was about to allow these strangers to perform on my sons brain. HIS BRAIN! Stroke, loss of facial movements (smiling, blinking, etc...), loss of ability to swallow, balance issues, death. Sounds like a decent trade off right? I signed the agreement. Once I saw them ship him off in his little crib, head marked for where incisions would be made (thankfully they only needed one on the back of his head) I remember just being completely numb. Everything tasted the same that day, nothing was funny, no one was consoling enough, sleep wasn't possible, nothing was possible until I could see my child alive after his surgery. Which eventually did come. We received an update every hour, always positive from one of the nurses. When finally at about 4:30pm that same day they told us they were done and that he would be in the recovery room soon.
His surgeon came into the waiting room where we were and brought us into the consultation room to brief us on what had gone down. So I'll relay that information here:
A single incision was made to the back of my sons head. They then removed a piece of his skull in a square fashion in order to have room to operate. Once they were inside, they lifted the folds of the cerebellum and could see the tumor. From what they saw, it had not gotten tangled in his brain stem or invaded any other space in his brain. They were actually able to remove the entire tumor without actually cutting ANY of his brain tissue. Thank God.
When we were finally able to see Thomas he laid in his little crib looking very sweaty and exhausted. His face was incredibly swollen (I believe he was upside down for his surgery) and a little raw by one of his cheeks from the shifting around in surgery. But for the most part he looked the same as he ever did with the exception of the throbbing incision on the back of his head.
For the next week we got nothing out of him, not head shakes, not ughs or grunts just short bursts of sheer frustration and screaming. He was trapped, knowing full well what he wanted to do (seemingly) but unable to perform it. Another week went by, he'd have his eyes open more often, I had him reach out and pick a spiderman figure from my hand one day, he wouldn't do that for another two weeks. The only word he ever really spoke in the office was to his older brother, who brought him a Bumble Bee Mr. Potato Head toy to which he said "thanks". That again, would be the only thing he would say for about the next two weeks.
After that 2-3 week span he would surprise us each day. He had gone from not being able to hold his own head up or even being able to swallow to swallowing and actually LAUGHING. This may sound cliche, but the day I was able to get him to laugh seemed to really trigger a whole new list of things that he was able to do again. He began drinking in very small amounts (he had a feeding tube this whole time) and a week later after ripping out his feeding tube, he ate. And he would not stop eating from there on out until his 15th day of chemotherapy.
So here we are almost two months into this, he's begun his second cycle of his first treatment of chemotherapy (3 cycles in 1 treatment - each cycle is 21 days long). His hair has thinned significantly and he will most likely be completely bald soon. He has already fought off one infection. Just another testament to this little boy's strength. He's most often in good spirits while we are in the hospital, it was a blessing having him home for more than 28 hours (unlike the day he got his infection and we had to take him right back).
We recently welcomed our third child into this world on the 24th of July. So now we don't only have the privilege of taking care of a soon to be 3yr. old and a 5 year old but a newborn to keep us uh..."company" all night long (just kidding the wee one has been great). It's been a new hurdle every few weeks, for instance, just today (Aug. 7th) my oldest son started kindergarten. It will make things easier during the day, but not so much when he needs to be somewhere safe and loving after school. We'll take care of it. We have to!
Wrapping up, I can say that since the time my wife broke down when they told us our sons diagnosis, (with the exception of the choked back fear and anger in the hospital room that one day) I have not had an adequate means of ventilation. I thought perhaps this would help, but it really hasn't. I have not yet told my wife the story about him saying he was scared, I'm afraid it would tear her up too much (she doesn't read this blog, it bores her). Sorry, just had another support person drop off dinner for us. That's another thing, the support network of people we've had has been simply outstanding. Mowing the lawn (it's a big, very steep yard), cooking dinners, doing laundry, shuttling our oldest son places and so on, it's been a great, great help.
If there's one thing I've learned in all of this it's that God won't give me more than I can handle. I am intrigued to see just how far that limit goes as I believe it has been pushed much further than I would have expected. Once we get through this, I feel that my family from our newborn on up will be stronger collectively for it. Especially my little trooper.
I'll follow up this post with information regarding the benefit concert a friend of ours (who just today found out her father was diagnosed with brain cancer) has put together here in Nashville for our son to raise some money for his medical expenses (don't worry, we are insured, but insurance does not cover 100% obviously).
If you read all of this, I applaud you. It's tough to hold onto a though and go with it so I bet this was probably a choppy read...
Thanks again to everyone who has helped out through this and even those of you who just give me that pat on the back when it's clear things are rough. I can't say again how lucky I am to have such a great network of friends, family and even just online acquaintances.
Thank you all.
Brad
Friday, March 6, 2009
The Lantern Project
I've been having a bit of a love affair with various storage platforms over the last year or so. Some notable mentions of late are Tokyo Cabinet and redis. Both are incredibly cool, fast, trivial to interface and best of all can run across multiple machines. I'll post a few pre-built interfaces at the bottom of this post so jump there for their links.
Anyway, so all of that along with my new-found and seemingly unhealthy obsession with Erlang has driven me to, of course, to build my own...
Enter The Lantern Project.
No, there's no cool story-line behind its name, I wish there were. Anyway, so just a few of the "things" driving this currently are going to be (I don't trust anything with bullets, so I'll just delimit a list): RESTful, simple "string" storage (initially - see reasoning below), based on ETS/DETS (initially - see reasoning below) and finally distributable (that will be more or less the last "phase").
Well, for starters, I absolutely love it as an interface to CouchDB and other service-oriented sites as well (twitter, facebook {to an extent - stop inventing your own methods!}). Second, it's easy implement at the client level, we already know it scales, damn near anything can connect to it, etc...etc. This was a no-brainer, I really wish redis would have this, but, trade-offs, I understand...
I had a fun time mulling over this, do you support more complex objects at first? At first, is what I kept coming back to. If this is a learning project, then let's start small and work our way up. Makes sense (to me), but what other data type could you really use to store "anything" (take binary data out of the picture for a moment). I think that by allowing folks to marshal/encode objects as they see fit (JSON,base64,XML,whatever) that makes for a pretty good first pass.
Simple, it's in Erlang and I don't need to really "worry" about formats to get something functional out the door. If my reading is correct, they're based on Judy Arrays which gives you (I know, b-trees do it too) both constant O(1) and logarithmic O(log n)look-ups, NICE! They generally take up less memory, but you do end up utilizing more CPU. Meh, fair enough...
So there ye have it...if you want to peek at my (ahem) "progress" the github project is right here.
Anyway, so all of that along with my new-found and seemingly unhealthy obsession with Erlang has driven me to, of course, to build my own...
Enter The Lantern Project.
No, there's no cool story-line behind its name, I wish there were. Anyway, so just a few of the "things" driving this currently are going to be (I don't trust anything with bullets, so I'll just delimit a list): RESTful, simple "string" storage (initially - see reasoning below), based on ETS/DETS (initially - see reasoning below) and finally distributable (that will be more or less the last "phase").
Why REST?
Well, for starters, I absolutely love it as an interface to CouchDB and other service-oriented sites as well (twitter, facebook {to an extent - stop inventing your own methods!}). Second, it's easy implement at the client level, we already know it scales, damn near anything can connect to it, etc...etc. This was a no-brainer, I really wish redis would have this, but, trade-offs, I understand...
Why just strings at first?
I had a fun time mulling over this, do you support more complex objects at first? At first, is what I kept coming back to. If this is a learning project, then let's start small and work our way up. Makes sense (to me), but what other data type could you really use to store "anything" (take binary data out of the picture for a moment). I think that by allowing folks to marshal/encode objects as they see fit (JSON,base64,XML,whatever) that makes for a pretty good first pass.
Why ETS/DETS?
Simple, it's in Erlang and I don't need to really "worry" about formats to get something functional out the door. If my reading is correct, they're based on Judy Arrays which gives you (I know, b-trees do it too) both constant O(1) and logarithmic O(log n)look-ups, NICE! They generally take up less memory, but you do end up utilizing more CPU. Meh, fair enough...
So there ye have it...if you want to peek at my (ahem) "progress" the github project is right here.
As promised, the gem links for redis/tokyo cabinet
Labels:
biting off more than you can chew,
erlang,
lantern,
storage
Friday, February 20, 2009
Erlang, you are holding out on me!
So I was playing around with list comprehensions today and found myself looking at permutations of strings when I noticed I wasn't getting "everything" back from the shell.
I'm sure there is a good reason for this, but I'm unclear as to what to search for in order to find an answer. The code I'm talking about is as follows:
which produces:
I'm sure it is just trying to save some horrendous output, but what if for some un-Godly reason I needed it all, how do I turn those [...]'s into more values?
Anyway, just thought this was interesting.
I'm sure there is a good reason for this, but I'm unclear as to what to search for in order to find an answer. The code I'm talking about is as follows:
[ [String1]++[String2] || String1<-"ABCDEFGHIJKLMNOPQRSTUVWXYZ", String2<-"ABCDEFGHIJKLMNOPQRSTUVWXYZ"].
which produces:
["AA","AB","AC","AD","AE","AF","AG","AH","AI","AJ","AK",
"AL","AM","AN","AO","AP","AQ","AR","AS","AT","AU","AV","AW",
"AX","AY","AZ","BA","BB",
[...]|...]
I'm sure it is just trying to save some horrendous output, but what if for some un-Godly reason I needed it all, how do I turn those [...]'s into more values?
Anyway, just thought this was interesting.
Tuesday, February 17, 2009
Why Erlang will run "The Matrix"
Ok, so I had blogged and even somewhat complained about Erlang as an "every day" language. I'm here right now to say that I was dead wrong.
Erlang is simply amazing as I've gone from merely flipping through pages and references online and in books, to fiddling with gen_server off and on and to actually implementing a small facebook application based on Nick Gerakines' erlang_facebook module.
What I found funny about that last bit (the facebook app) is that after seeing an implementation of it in Ruby, it was actually easier to comprehend just what all was going on in the ERLANG version. Yes, I said it, the Erlang was easier to read.
Getting back on track...
After digging deep into getting my head around Erlang, something kind of crept up on me as I went along. Particularly when it came to message passing and hot code swapping...
You ready? Put on your tinfoil hat. Below is my list of 5 examples The Matrix could exist and why it could be written in Erlang.
1. You ever wonder why we can only really "use" less than 10% of our brains? Ever wonder what that other 90-some percent is doing? That's right, passing messages to other "processes" and handling other distributed tasks.
2. When it comes to the hot code loading, this is a good example of how a "Smith" would work.
3. Every wonder why we can’t utilize 100% of our brains? Maybe it’s because the other 90-ish percent is too busy doing distributed tasks in conjunction with other processes.
4. Anything that scales just by adding cores (or nice, warm, fresh humans) and has been sitting somewhat idle in the wings for the last 20 years or so has GOT to be hiding something. I mean c'mon, most of the "love languages" (Ruby, Python, PHP) look at a multi-core system and don't really know what to do with themselves.
5. (And this will be the last one as I'm losing momentum) The system can self-sustain for obscene periods of time, if one had the "available systems" (people) one could die (literally and figuratively) and it really wouldn't matter, you'd have another waiting to take it's place AND you've got around 90% of it's processing power at your disposal.
This has been fun() (I can't help myself, I'm sorry) I hope some of you who come across this add some more to this list in the comments, I think there's a lot of cracks to be made.
Erlang is simply amazing as I've gone from merely flipping through pages and references online and in books, to fiddling with gen_server off and on and to actually implementing a small facebook application based on Nick Gerakines' erlang_facebook module.
What I found funny about that last bit (the facebook app) is that after seeing an implementation of it in Ruby, it was actually easier to comprehend just what all was going on in the ERLANG version. Yes, I said it, the Erlang was easier to read.
Getting back on track...
After digging deep into getting my head around Erlang, something kind of crept up on me as I went along. Particularly when it came to message passing and hot code swapping...
You ready? Put on your tinfoil hat. Below is my list of 5 examples The Matrix could exist and why it could be written in Erlang.
1. You ever wonder why we can only really "use" less than 10% of our brains? Ever wonder what that other 90-some percent is doing? That's right, passing messages to other "processes" and handling other distributed tasks.
2. When it comes to the hot code loading, this is a good example of how a "Smith" would work.
3. Every wonder why we can’t utilize 100% of our brains? Maybe it’s because the other 90-ish percent is too busy doing distributed tasks in conjunction with other processes.
4. Anything that scales just by adding cores (or nice, warm, fresh humans) and has been sitting somewhat idle in the wings for the last 20 years or so has GOT to be hiding something. I mean c'mon, most of the "love languages" (Ruby, Python, PHP) look at a multi-core system and don't really know what to do with themselves.
5. (And this will be the last one as I'm losing momentum) The system can self-sustain for obscene periods of time, if one had the "available systems" (people) one could die (literally and figuratively) and it really wouldn't matter, you'd have another waiting to take it's place AND you've got around 90% of it's processing power at your disposal.
This has been fun() (I can't help myself, I'm sorry) I hope some of you who come across this add some more to this list in the comments, I think there's a lot of cracks to be made.
Wednesday, January 7, 2009
Making RabbitMQ Stew
I just wanted to write this up real quick for anyone who may run into the same problem I did...
I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.
Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:
Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...
Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application
Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.
and
I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.
Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:
=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary
Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.
So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/
This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached
Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===
Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"
=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672
The short of it - whipe your mnesia directories and start over.
I'm working on a project where I need to load some work from a queue for indexing into a Xapian Database (coming from CouchDB). So, RabbitMQ seemed like a good fit long-term for such a task.
Anyway, After getting RabbitMQ installed, I went to start'er up and noticed that it kept failing. So, off to the logs we go...and that is where I found the following:
Mnesia('rabbit@[myhostname]'): ** WARNING ** Mnesia is overloaded: {dump_log, ...
Hmm, well as much as I'd love to be pegged for resources I knew this wasn't the case. So after some googling I had found some suggestions that would squelch the warnings. What really seemed to do the trick was starting up rabbitmq-server with the following parameters: -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 (this was suggested from the Million User Comet Application
Now, since I know people don't really enjoy reading the documentation on things, I'll make it easier on you. From the mnesia documentation here is a quick description for the above-mentioned parameters.
-mnesia dc_dump_limit Number. Controls how often disc_copies tables are dumped from memory. Tables are dumped when filesize(Log) > (filesize(Tab)/Dc_dump_limit). Lower values reduces cpu overhead but increases disk space and startup times. The default is 4.
and
-mnesia dump_log_write_threshold Max, where Max is an integer which specifies the maximum number of writes allowed to the transaction log before a new dump of the log is performed. It defaults to 100 log writes.
I'd like to note that I can omit dump_log_write_threshold and still get rabbit up and running, so it may or may not help you out.
Now that I had begun passing these parameters to rabbitmq-server I was now getting a new error in the logs:
=INFO REPORT==== 7-Jan-2009::11:52:45 ===
application: rabbit
exited: {{timeout_waiting_for_tables,
[user,user_vhost,vhost,rabbit_config,listener,durable_routes,
route,reverse_route,durable_exchanges,exchange,
durable_queues,amqqueue]},
{rabbit,start,[normal,[]]}}
type: temporary
Oy, alright - so the tables don't seem to either be there or they just refuse to be created. Let's see if I can drop the mnedia tables that we created when starting the server and start over again.
So, I...
cd /var/lib/rabbitmq/mnesia/
an "ls" revealed two rabbit directories: rabbit/ and rabbit_20090107031256/
rm -rf rabbit/
rm -rf rabbit_20090107031256/
This path...is now clear! Let's start rabbitmq-server up again, but this time, without all the voodoo.
rabbitmq-server -detached
Starts up fine and now if we check our logs, we should see something like:
=INFO REPORT==== 7-Jan-2009::12:40:48 ===
Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"
=INFO REPORT==== 7-Jan-2009::12:40:48 ===
started TCP listener on 0.0.0.0:5672
The short of it - whipe your mnesia directories and start over.
Friday, January 2, 2009
Who Knew?!?!
...that it would be more than two months since my last post.
...that Rails and Merb would get into bed with each other. (And may or may not come out producing something useful).
...that we would be on the verge of collapsing economically (ok, some did, but I didn't).
...that the Boston Bruins would lead the NHL in points almost half way through the year by winning 10 games straight! (they better knock this off, my Red Wings need some love)
...that the Tennessee Titans would be the 1st seed in the NFL playoffs.
...that I would grasp enough Erlang to seriously consider using it for an application.
...that I could come up with a post as "clever" as this with a horrid head cold.
Ok, well now that the busyness of Nov/Dec is over I can hopefully return to more sharing...Stay tuned.
...that Rails and Merb would get into bed with each other. (And may or may not come out producing something useful).
...that we would be on the verge of collapsing economically (ok, some did, but I didn't).
...that the Boston Bruins would lead the NHL in points almost half way through the year by winning 10 games straight! (they better knock this off, my Red Wings need some love)
...that the Tennessee Titans would be the 1st seed in the NFL playoffs.
...that I would grasp enough Erlang to seriously consider using it for an application.
...that I could come up with a post as "clever" as this with a horrid head cold.
Ok, well now that the busyness of Nov/Dec is over I can hopefully return to more sharing...Stay tuned.
Wednesday, September 24, 2008
Reia where scripting meets concurrency
Reia reminds me of that Reese's peanut butter cup commercial:
And then there was this light that went off in someone's head - Why couldn't you bring the "stuff" people like about higher level languages down into something that, quite frankly just kicks ass, in Erlang?
To kick it off - and to quote the Reia wiki:
Well, it's happening right now with Reia. I've been watching this project over my GitHub feed for maybe a month or so now and it's really cool seeing the leaps and bounds that Tony Arcieri is making with this little gem of a language. Granted, he still has a ways to go (and he openly admits this), I suggest giving it a try for fun() (that's my new favorite pun by the way).
Let me paste a few ideals from the Reia site to pique your interest a little more:
Concurrent programming is the future - very true, now if DB's would just catch up (oh, wait CouchDB!
You don't need an "everything is a..." language - he calls out SmallTalk and Ruby, but his argument is convincing, I think we've all been here before.
Don't build what you can steal - I really like that quote in regard to using a syntax of Python/Ruby and then laying it lovingly over the Erlang VM and OTP framework.
Hey, you got your Ruby/Python in my Erlang! Hey, you got your Erlang in my Ruby/Python!
And then there was this light that went off in someone's head - Why couldn't you bring the "stuff" people like about higher level languages down into something that, quite frankly just kicks ass, in Erlang?
To kick it off - and to quote the Reia wiki:
Reia (pronounced RAY-uh) is a Python/Ruby-like scripting language for the Erlang virtual machine (BEAM). Reia aims to expose all the features and functionality of Erlang in a language more familiar to programmers of scripting languages, while improving string handling, regular expressions, linking with external libraries, and other tasks which are generally considered outside the scope of Erlang. Reia is distributed under the MIT License.
Well, it's happening right now with Reia. I've been watching this project over my GitHub feed for maybe a month or so now and it's really cool seeing the leaps and bounds that Tony Arcieri is making with this little gem of a language. Granted, he still has a ways to go (and he openly admits this), I suggest giving it a try for fun() (that's my new favorite pun by the way).
Let me paste a few ideals from the Reia site to pique your interest a little more:
Concurrent programming is the future - very true, now if DB's would just catch up (oh, wait CouchDB!
You don't need an "everything is a..." language - he calls out SmallTalk and Ruby, but his argument is convincing, I think we've all been here before.
Don't build what you can steal - I really like that quote in regard to using a syntax of Python/Ruby and then laying it lovingly over the Erlang VM and OTP framework.
Monday, September 15, 2008
Judging a book by it's "sucks" result
A friend of mine asked me recently to go in on a little venture of his in the gaming industry. Unable to say "no" to what I believe is a fantastic idea, I went along with it...
Now there was another guy involved (technologically speaking) who wrote something that was a working prototype (awesome). It came down to a little discussion as to what WE should use for the front-end and storage mechanisms.
Well, having deployed a few real life applications in Ruby and having success with them, I only felt the need to display my affection and trust that I had with it and it's frameworks (yes, both Rails and Merb).
The problem I had wasn't that the guy wanted to use Python/Pylons or something in that realm. It was the argument. In fact, now that I've heard it a few times I can almost pinpoint exactly where the debate material is coming from. And that my friends, is simply fantastic. We're still not going along with the right tool for the right job, we're still shooting from the hip with what "we like".
Uh, but Brad, you just said you liked Ruby? You're right, I did. But given the requirements I was handed and based on past experiences it appeared to be the right tool for the right job.
I'm open to using new things - if I wasn't why would I ever give things like CouchDB and Erlang the time of day? They're sitting right on the edge of the programming paradigm shift from what most coders/programmers/developers are use to and it fascinates me.
Back to it.
I'm fine with using Python or whatever for a web interface. One more (hopefully) successful notch in my belt and slap proudly on a resume. But it just absolutely kills me when the keys of an argument are literally plucked from simply searching Google for "why sucks". What's even worse is when it's a predominantly blog rifling .
Please, please, please, I'm begging you the reader of this blog post; the rule here is simple, I'll even wrap it in a blockquote for you to save and tattoo across your chest in a Tupac-ish style:
It's not the other way around, in fact, if you do it the other way around you're going to do your research the hard way; rewriting it in the layers you should have used in the first place.
Now there was another guy involved (technologically speaking) who wrote something that was a working prototype (awesome). It came down to a little discussion as to what WE should use for the front-end and storage mechanisms.
Well, having deployed a few real life applications in Ruby and having success with them, I only felt the need to display my affection and trust that I had with it and it's frameworks (yes, both Rails and Merb).
The problem I had wasn't that the guy wanted to use Python/Pylons or something in that realm. It was the argument. In fact, now that I've heard it a few times I can almost pinpoint exactly where the debate material is coming from. And that my friends, is simply fantastic. We're still not going along with the right tool for the right job, we're still shooting from the hip with what "we like".
Uh, but Brad, you just said you liked Ruby? You're right, I did. But given the requirements I was handed and based on past experiences it appeared to be the right tool for the right job.
I'm open to using new things - if I wasn't why would I ever give things like CouchDB and Erlang the time of day? They're sitting right on the edge of the programming paradigm shift from what most coders/programmers/developers are use to and it fascinates me.
Back to it.
I'm fine with using Python or whatever for a web interface. One more (hopefully) successful notch in my belt and slap proudly on a resume. But it just absolutely kills me when the keys of an argument are literally plucked from simply searching Google for "why
Please, please, please, I'm begging you the reader of this blog post; the rule here is simple, I'll even wrap it in a blockquote for you to save and tattoo across your chest in a Tupac-ish style:
Research first, execute later.
It's not the other way around, in fact, if you do it the other way around you're going to do your research the hard way; rewriting it in the layers you should have used in the first place.
Monday, September 8, 2008
Is your DataMapper CouchDB adapter PUT'ing where it shouldn't?
It sounds gross, sure...but this was actually a simple problem I foresee someone running into and potentially running around mad about for an hour if they're not too keen on how DataMapper/CouchDB work together. Even if you DO know how they work together, this was still a fun problem to track down.
So let's take an example model that's extremely simple:
Looks harmless, right? That's what I thought too! Well, if you begin your application and get to the point of actually creating a Person you'll be greeted with an error that kicks off with "EOFError: end of file reached ".
So what's happening here?
Basically in the adapter, there is a LOC in the create method where it checks to see if there is a key or not (Yea, the one we've specified above as :name).
This creates an assumption of using PUT instead of POST and that is where the issue stems from. So, to fix this problem, all you need to do is DROP the :key => true and things will run as they should.
Now you're probably asking "But, what if I NEED to have that key there, I NEED to relate my models with a foreign key". Not to sound harsh, but just to be clear, you may want to revisit why you're using CouchDB in the first place.
So let's take an example model that's extremely simple:
class Person
include DataMapper::Resource
# CouchDB Specific
property :id, String, :key => true, :serial => true, :field => :_id
property :rev, String, :field => :_rev
#App Specific
property :name, String, :nullable => false, :length => 75, :key => true
end
Looks harmless, right? That's what I thought too! Well, if you begin your application and get to the point of actually creating a Person you'll be greeted with an error that kicks off with "EOFError: end of file reached ".
So what's happening here?
Basically in the adapter, there is a LOC in the create method where it checks to see if there is a key or not (Yea, the one we've specified above as :name).
This creates an assumption of using PUT instead of POST and that is where the issue stems from. So, to fix this problem, all you need to do is DROP the :key => true and things will run as they should.
Now you're probably asking "But, what if I NEED to have that key there, I NEED to relate my models with a foreign key". Not to sound harsh, but just to be clear, you may want to revisit why you're using CouchDB in the first place.
Labels:
configuration,
couchdb,
datamapper,
thank god that's over
Readability vs Optimization
I recently saw a post come across my Google Reader a few days ago asking what someone should code "for". His two options were Readability or Optimization. I'm not going to link the article for the simple reason it was incredibly brief and it was mostly just a question, the author never really jumped into going one way or the other or leading from one to the other.
So have no fear, I'll do it!
What makes sense? You want code you can read, yet you also want it to run incredibly fast. Interesting! Well, like any project the first one to market usually gets the worm. So cranking out code (read: code that "works") is typically a developer's (and many times the project managers) first concern. Do we have something out there to start drawing in the cash? Yes. So you've effectively coded for "readability" since we're not optimized. You've gotten your product on the market which is really what your first goal SHOULD be, unless of course there's some ungodly amount of volume that you KNEW of and should have optimized according for it in the first place.
Now, wait a minute...that sounds like common sense. If I'm developing something new that no one knows about, what's the point of optimization? I can just crank stuff out that "works" and polish up that while we gain customers/visitors/whomever. Correct!
On the flipside of the previous paragraph, our (I mean everyone's, not mine personally) good friend with a framework Ezra Zygmuntowicz had an interesting quote a while back where he said "Post-optimization is the route of all hosting bills". In the article I'm extrapolating on, one commenter said if one codes strictly for readability, optimization will come automatically because hardware continues to get faster. The author himself also said "Performance can be optimized at the hardware level", which blew my mind. Why are you even asking for hints on code optimization (for PHP) if you already made up your mind to throw hardware at it? I can't speak for him, but I'm pretty sure you're not going to be replacing your systems with the latest and greatest hardware every 18 months are you? Hmm, Nope!
Well, we've already cranked our code to production, things may or may not be fine with the community accessing it, and I'm sure you've been paying close attention to which aspects of your system need work, help, or simply need to be scrapped and redone. Wait, you haven't? Ok, let's start there.
Get a decent logging or monitoring system and pay close attention to the needs of your app. Many times we get so caught up in producing that we never pay close attention to the simple things once it's out in the wild. EVEN MORE TIMES than not, people don't see what applications that may be similar in nature are doing or how they're reacting online to try and investigate why they are having these problems. Chances are, your application will have them too - you cannot assume that just because you wrote the code that it is exempt from bottlenecks. In fact, I'd say the vast majority of the time, you're going to hit the exact same problems as your competitors.
Do your research, if they're a fairly transparent company see what growing pains they were having and try to plan accordingly.
Now that we've addressed the issues surrounding our application AND applications which are similar in nature to ours we can now effectively move into the space where we begin optimizing our code. This is a rather vast term and your focus should (in my opinion) stem from your I/O bottlenecks first before jumping to your code. If all seems well in that area or you simply don't have the funding to do much with it, then sure, you're ready to start optimizing code.
So in essence the life cycle is as follows:
1 - Produce working code that meets your needs.
2 - Analyze how it and other I/O (disk/network/database) are affecting your application.
3 - Attempt to alleviate as much of that I/O overhead as you can within your boundaries (manpower, funding, etc...).
4 - Optimize your code.
I think that will get you through the day. I could be wrong, I flew through this post pretty damn fast. I feel a site optimization post coming soon, so get ready for some more Kool Aid to drink.
So have no fear, I'll do it!
What makes sense? You want code you can read, yet you also want it to run incredibly fast. Interesting! Well, like any project the first one to market usually gets the worm. So cranking out code (read: code that "works") is typically a developer's (and many times the project managers) first concern. Do we have something out there to start drawing in the cash? Yes. So you've effectively coded for "readability" since we're not optimized. You've gotten your product on the market which is really what your first goal SHOULD be, unless of course there's some ungodly amount of volume that you KNEW of and should have optimized according for it in the first place.
Now, wait a minute...that sounds like common sense. If I'm developing something new that no one knows about, what's the point of optimization? I can just crank stuff out that "works" and polish up that while we gain customers/visitors/whomever. Correct!
On the flipside of the previous paragraph, our (I mean everyone's, not mine personally) good friend with a framework Ezra Zygmuntowicz had an interesting quote a while back where he said "Post-optimization is the route of all hosting bills". In the article I'm extrapolating on, one commenter said if one codes strictly for readability, optimization will come automatically because hardware continues to get faster. The author himself also said "Performance can be optimized at the hardware level", which blew my mind. Why are you even asking for hints on code optimization (for PHP) if you already made up your mind to throw hardware at it? I can't speak for him, but I'm pretty sure you're not going to be replacing your systems with the latest and greatest hardware every 18 months are you? Hmm, Nope!
So what do we need to do?
Well, we've already cranked our code to production, things may or may not be fine with the community accessing it, and I'm sure you've been paying close attention to which aspects of your system need work, help, or simply need to be scrapped and redone. Wait, you haven't? Ok, let's start there.
Pay attention to what your application is doing.
Get a decent logging or monitoring system and pay close attention to the needs of your app. Many times we get so caught up in producing that we never pay close attention to the simple things once it's out in the wild. EVEN MORE TIMES than not, people don't see what applications that may be similar in nature are doing or how they're reacting online to try and investigate why they are having these problems. Chances are, your application will have them too - you cannot assume that just because you wrote the code that it is exempt from bottlenecks. In fact, I'd say the vast majority of the time, you're going to hit the exact same problems as your competitors.
Do your research, if they're a fairly transparent company see what growing pains they were having and try to plan accordingly.
Now that we've addressed the issues surrounding our application AND applications which are similar in nature to ours we can now effectively move into the space where we begin optimizing our code. This is a rather vast term and your focus should (in my opinion) stem from your I/O bottlenecks first before jumping to your code. If all seems well in that area or you simply don't have the funding to do much with it, then sure, you're ready to start optimizing code.
So in essence the life cycle is as follows:
1 - Produce working code that meets your needs.
2 - Analyze how it and other I/O (disk/network/database) are affecting your application.
3 - Attempt to alleviate as much of that I/O overhead as you can within your boundaries (manpower, funding, etc...).
4 - Optimize your code.
I think that will get you through the day. I could be wrong, I flew through this post pretty damn fast. I feel a site optimization post coming soon, so get ready for some more Kool Aid to drink.
Labels:
code,
info,
performance,
problemsolving,
rants,
the architecture is a lie
Wednesday, August 20, 2008
Ruby vs. Erlang vs. The People vs. Your Application
For the past six months or so I've been working on a somewhat under the radar service called Mumbl. Basically, mumbl is a micro-blogging service that solves the whole "Oh, we chose the wrong framework for the wrong platform and our db won't/can't scale" problems most of the bigger folks have. And for all I know they may well have already solved their issues as it seems to be getting better(?) as less and less downtime occurs. Good for them, the Mongolian Horde/money shovel approach to systems and development worked...for now.
Anyway, there's been a momentum gainer amongst Rubyists looking more and more at Erlang. Which, yea is great but I think one of the biggest problems with Ruby developers is that we (yes, I'm one too) don't just drink the punch if things "make sense", WE CHUG IT! We ask that it be hooked intravenously into our veins until it becomes a part of us on a molecular level.
So, having said that, why would someone that's vested deeply in Ruby, which preaches readability and quick, elegant solutions to common problems (whether it be via frameworks or gems) have any interest in Erlang? Erlang is based on ant-turd tokens, which aren't exactly easy to read. Erlang appears to also be fairly verbose in a sense that I can't see it "fitting" for your typical online applications and it certainly isn't the easiest thing to read. So please, if you need to undertake a project that would quite easily be completed using the rails/merbs/django's of the world out there - then utilize them! It's my opinion that Erlang is not a language for application that need constant adjustment, in this particular case applications for the online masses. Leave it to the back-side and you'll thank me on your wedding night!
This was touched on a little bit a while ago (the Erlang thing, not the back-side thing), I had to go dig up the post but you can see Damien Katz's blog entry about a few things he disliked about Erlang. See the comments of his posting for a real roller-coaster ride!
Anyway, getting back to the point here. I think we need to be careful when given the option of the punch, which really is true for anything. But something we, as developers should look at so closely is the separation of trendy with practical. I believe we can all agree that a readable/maintainable application layer with a concise entry point to a highly parallel back-end is probably the best way to go. Maybe not; well, let me try and convince you otherwise.
If you had to throw a jack of many trades developer into the mix of your application I described above how do you think he'd handle it? I think they would greatly appreciate the easy readability of the Ruby code over the Erlang. Once they ramped their way up gaining and understanding of the interfaces it would likely make their lives easier once they made it into the land of Erlang.
So please, let's keep it separate. And let's please, please stop creating half-assed Ruby implementations of things that really don't need it. This recent post on the register certainly struck a nerve with me. It was apparent they we are getting lazy and far too comfortable in our language to expand horizons and use the right tool for the right job. That tool doesn't need to take over completely, getting back to the punch-chugging I mentioned earlier. You need to find it's proper place in your application if it even fits at all, don't make a place for it. And certainly don't make a cheesy knock-off of it in Ruby so you get 1/2 points for utilizing some new (or trendy) methodology.
Ah...now that' feels better.
Anyway, there's been a momentum gainer amongst Rubyists looking more and more at Erlang. Which, yea is great but I think one of the biggest problems with Ruby developers is that we (yes, I'm one too) don't just drink the punch if things "make sense", WE CHUG IT! We ask that it be hooked intravenously into our veins until it becomes a part of us on a molecular level.
So, having said that, why would someone that's vested deeply in Ruby, which preaches readability and quick, elegant solutions to common problems (whether it be via frameworks or gems) have any interest in Erlang? Erlang is based on ant-turd tokens, which aren't exactly easy to read. Erlang appears to also be fairly verbose in a sense that I can't see it "fitting" for your typical online applications and it certainly isn't the easiest thing to read. So please, if you need to undertake a project that would quite easily be completed using the rails/merbs/django's of the world out there - then utilize them! It's my opinion that Erlang is not a language for application that need constant adjustment, in this particular case applications for the online masses. Leave it to the back-side and you'll thank me on your wedding night!
This was touched on a little bit a while ago (the Erlang thing, not the back-side thing), I had to go dig up the post but you can see Damien Katz's blog entry about a few things he disliked about Erlang. See the comments of his posting for a real roller-coaster ride!
Anyway, getting back to the point here. I think we need to be careful when given the option of the punch, which really is true for anything. But something we, as developers should look at so closely is the separation of trendy with practical. I believe we can all agree that a readable/maintainable application layer with a concise entry point to a highly parallel back-end is probably the best way to go. Maybe not; well, let me try and convince you otherwise.
If you had to throw a jack of many trades developer into the mix of your application I described above how do you think he'd handle it? I think they would greatly appreciate the easy readability of the Ruby code over the Erlang. Once they ramped their way up gaining and understanding of the interfaces it would likely make their lives easier once they made it into the land of Erlang.
So please, let's keep it separate. And let's please, please stop creating half-assed Ruby implementations of things that really don't need it. This recent post on the register certainly struck a nerve with me. It was apparent they we are getting lazy and far too comfortable in our language to expand horizons and use the right tool for the right job. That tool doesn't need to take over completely, getting back to the punch-chugging I mentioned earlier. You need to find it's proper place in your application if it even fits at all, don't make a place for it. And certainly don't make a cheesy knock-off of it in Ruby so you get 1/2 points for utilizing some new (or trendy) methodology.
Ah...now that' feels better.
Wednesday, August 6, 2008
Timezones in Gentoo
I'm on Central time...and there wasn't a clear way (within the gentoo docs) for getting myself on the correct timezone. SO, for anyone in my timezone that needs it...this is the way to do it:
rm /etc/localtime && ln -s /usr/share/zoneinfo/US/Central /etc/localtime
For anyone else, after zoneinfo// - make sense?
40 minutes...and sometimes I wonder about myself.
rm /etc/localtime && ln -s /usr/share/zoneinfo/US/Central /etc/localtime
For anyone else, after zoneinfo/
40 minutes...and sometimes I wonder about myself.
Monday, August 4, 2008
SFTP and blogger
Just a hint for anyone setting up a hosted blog via the blogger platform.
You will likely need to modify your /etc/ssh/sshd_config file...or else you will be unable to publish to your sftp site.
The change you need to make is:
It's likely already set to "no".
I was getting this error while trying to publish to a hosted blog to my domain and was seeing an error in my ssh log that read:
So yea, I hope that helps anyone who needs it.
You will likely need to modify your /etc/ssh/sshd_config file...or else you will be unable to publish to your sftp site.
The change you need to make is:
PasswordAuthentication yes
It's likely already set to "no".
I was getting this error while trying to publish to a hosted blog to my domain and was seeing an error in my ssh log that read:
Did not receive identification string from [google ip addr]
So yea, I hope that helps anyone who needs it.
Labels:
info,
merb,
thank god that's over,
the architecture is a lie
Saturday, August 2, 2008
Merb does it again...
Remember in the old days when you'd create what I can only explain as a "temporary session" in Rails to notify someone ONCE on a page of something? It could have been a form validation error, a success, just anything that you needed to toss up to the user temporarily. Still with me?
Merb now has this wonderful "message" implementation which they've hooked onto the redirect method. So, by calling:
You receive a variable "message" that contains your message (conventional, right?). It's basically marshalled and plopped into the querystring and then deserialized into the var. I really like this over the Rails method...but, following what the Pug says - I'm not pitting merb against rails.
.erawtaolB !lleh eht tahw ,naem I .noitatnemelpmi laivirt a hcus ekil smees siht nehw o/i pu gnitae ,snoisses pmt etaerc ot tnaw uoy od lleh eht yhw si gniyas m'I llA
It's late, I'm wiped out and that's the "best" I could do for humor.
Merb now has this wonderful "message" implementation which they've hooked onto the redirect method. So, by calling:
redirect '/some/path', "We're gonna need a bigger boat..."
You receive a variable "message" that contains your message (conventional, right?). It's basically marshalled and plopped into the querystring and then deserialized into the var. I really like this over the Rails method...but, following what the Pug says - I'm not pitting merb against rails.
.erawtaolB !lleh eht tahw ,naem I .noitatnemelpmi laivirt a hcus ekil smees siht nehw o/i pu gnitae ,snoisses pmt etaerc ot tnaw uoy od lleh eht yhw si gniyas m'I llA
It's late, I'm wiped out and that's the "best" I could do for humor.
Friday, July 25, 2008
CouchDBXapr
Ok, so I've been working on a Xapian implementation for CouchDB to feed off of in Ruby. Just a quick note for anyone who is attempting the same.
1. On OSX, everything works great, the bindings install to their proper places. All is well with the world.
2. However, in production mode, on a Gentoo server, running make check reveals that we are unable to find our libxapian.so file.
SO, in order to remedy this and to save anyone else looking for this fine obscure error. Run make install...
You'll then notice any of your Ruby scripts are unable to process anything that even mentions the word 'xapian'.
Then, simply do the following:
cd /usr/local/lib
cp libxapian.so.15 /usr/lib64/
et voila!
You are now ready for some fulltext goodness and if you're like me are now ready for some much needed rest.
1. On OSX, everything works great, the bindings install to their proper places. All is well with the world.
2. However, in production mode, on a Gentoo server, running make check reveals that we are unable to find our libxapian.so file.
SO, in order to remedy this and to save anyone else looking for this fine obscure error. Run make install...
You'll then notice any of your Ruby scripts are unable to process anything that even mentions the word 'xapian'.
Then, simply do the following:
cd /usr/local/lib
cp libxapian.so.15 /usr/lib64/
et voila!
You are now ready for some fulltext goodness and if you're like me are now ready for some much needed rest.
Thursday, July 10, 2008
Integrating with UrlZen
For my twitter-like application (mumbl - not yet released) I've decided to give users the ability to shorten their long URL's for their mumbls as they're typing them using UrlZen.
Here's the Ruby implementation - I'll post a follow-up containing the jQuery portion. basically to get around XSS I call a URL on my system which then calls up UrlZen
Here's the Ruby implementation - I'll post a follow-up containing the jQuery portion. basically to get around XSS I call a URL on my system which then calls up UrlZen
require 'mechanize'
agent = WWW::Mechanize.new
@url = params[:url]
@new_url = agent.get("http://urlzen.com/shorten/?autocopy=true&url=#{@url}").search("//p[@class='short_url']//strong").inner_html
Sunday, July 6, 2008
So what happened (the multi-key CouchDB question)?
So, if any of you have been following along with my fun ride upon the cozy cushions of CouchDB, you'll be happy to hear there is hope (well, some - I had to pick up a book on Erlang).
Anyhow, if you follow the link here you will see that there was a list of proposed fixes for my desire to pass in multiple keys to CouchDB instead of just one. Granted, yes it could get out of hand, you could pass in 1,000,000 keys. But so what! Either way you're going to have to query for them, whether it's a query after a query a billion times or if you can somehow lump them together.
So as you can see there were a few options:
1. We could execute our query with each key each time and lump them together. The downside of this is that I really don't want Ruby to have to lump all of those results together, I can only imagine the paging for it would get incredibly *blah*. Plus the memory usage for Ruby to merge all of those DM::Collection objects together would likely be the equivalent of a Chihuahua passing a peach seed. Not pretty.
2. We could use a full text search using Lucene, which CouchDB already has implemented. I suppose the only downside of that would be...??? I'll need to think of something, it's really not a bad idea, it just may come to me later. The extent of my full text search capabilities have been with MySQL and...oh, hold on I keep laughing, it sucked. bad. I really hope CouchDB's is better - I'm sure it is, I haven't seen anything so far that is crap!
3. We (read: Me {Bradford}) hack at the CouchDB internals and allow for views to accept multiple keys for comparison rather than one. I'd like to see this happen via something to the effect of:
I'd like to keep it along the lines of form control arrays key[] this way you could just analyze the parameter itself to see what it was (an array or a string) and do the appropriate things with it. Whereas adding a parameter keys may need to be repeated throughout the code (Not likely, but the former makes sense to me most).
I'd like to extend a warm welcome to anyone willing to walk the lines of CouchDB's internals with me in order to implement such functionality, I promise to keep you in line as long as you return the favor. As I said, I'm JUST NOW learning Erlang and am horribly under-qualified, but hey, I really feel strongly about this functionality, so, what the hell, right?
Bring on the opinions!
Anyhow, if you follow the link here you will see that there was a list of proposed fixes for my desire to pass in multiple keys to CouchDB instead of just one. Granted, yes it could get out of hand, you could pass in 1,000,000 keys. But so what! Either way you're going to have to query for them, whether it's a query after a query a billion times or if you can somehow lump them together.
So as you can see there were a few options:
1. We could execute our query with each key each time and lump them together. The downside of this is that I really don't want Ruby to have to lump all of those results together, I can only imagine the paging for it would get incredibly *blah*. Plus the memory usage for Ruby to merge all of those DM::Collection objects together would likely be the equivalent of a Chihuahua passing a peach seed. Not pretty.
2. We could use a full text search using Lucene, which CouchDB already has implemented. I suppose the only downside of that would be...??? I'll need to think of something, it's really not a bad idea, it just may come to me later. The extent of my full text search capabilities have been with MySQL and...oh, hold on I keep laughing, it sucked. bad. I really hope CouchDB's is better - I'm sure it is, I haven't seen anything so far that is crap!
3. We (read: Me {Bradford}) hack at the CouchDB internals and allow for views to accept multiple keys for comparison rather than one. I'd like to see this happen via something to the effect of:
A single key query URL
http://.../_view/foo/bars?key="mykey"
A Multi-key query URL
http://.../_view/foo/bars?key[]="mykey1"&key[]="mykey2"
OR
http://.../_view/foo/bars?keys=["mykey1","mykey2","mykey3"]
I'd like to keep it along the lines of form control arrays key[] this way you could just analyze the parameter itself to see what it was (an array or a string) and do the appropriate things with it. Whereas adding a parameter keys may need to be repeated throughout the code (Not likely, but the former makes sense to me most).
I'd like to extend a warm welcome to anyone willing to walk the lines of CouchDB's internals with me in order to implement such functionality, I promise to keep you in line as long as you return the favor. As I said, I'm JUST NOW learning Erlang and am horribly under-qualified, but hey, I really feel strongly about this functionality, so, what the hell, right?
Bring on the opinions!
Thursday, July 3, 2008
It's all about document design with CouchDB
Hello everyone, been stalking the mailing list for a while and thought this might be worthy of a post as I was asked to solve it, yet, I couldn't!
Let's take the classic blog example document with the following fields/values:
"_id": "1f2fc3955b91aed5e7369f0b0ba8214e",
"_rev": "1226709986",
"Author": "Bradford",
"Type": "Post",
"Body": "Just mentioning this for a sample blog post.",
"PostedDate": "2008-07-02T23:22:12-04:00",
"Subject": "My Fine Blog Post",
"Tags": ["octopus","hockey","squidward","bradford","recreation"]
Next, I'd like to find each blog post that contains ANY of the following tags ["octopus","hockey"]. Now, generally speaking this isn't so bad. We could write a simple view:
We would get back each one of our tags as a key, yea? Only if we supplied one at a time. So how does one go about supplying a range, array (not sure what we'd call it here) of keys to be searched on? http://...?key=["octopus","hockey"] maybe? I'm unsure of the plan of attack for such a thing. Maybe I'm just going about it in the wrong direction. Any thoughts?
Let's take the classic blog example document with the following fields/values:
"_id": "1f2fc3955b91aed5e7369f0b0ba8214e",
"_rev": "1226709986",
"Author": "Bradford",
"Type": "Post",
"Body": "Just mentioning this for a sample blog post.",
"PostedDate": "2008-07-02T23:22:12-04:00",
"Subject": "My Fine Blog Post",
"Tags": ["octopus","hockey","squidward","bradford","recreation"]
Next, I'd like to find each blog post that contains ANY of the following tags ["octopus","hockey"]. Now, generally speaking this isn't so bad. We could write a simple view:
function (doc) {
if (doc.Type == 'Post') {
for (var i = 0;i < doc.tags.length; i++) {
emit(doc.tags[i],doc);
}
}
}
We would get back each one of our tags as a key, yea? Only if we supplied one at a time. So how does one go about supplying a range, array (not sure what we'd call it here) of keys to be searched on? http://...?key=["octopus","hockey"] maybe? I'm unsure of the plan of attack for such a thing. Maybe I'm just going about it in the wrong direction. Any thoughts?
Labels:
couchdb,
problemsolving,
the architecture is a lie
Subscribe to:
Posts (Atom)