I have mentioned elsewhere on this blog that I am currently doing a thesis in which I am using Hadoop and MapReduce. One thing I have always liked whether in school or in life in general is the applicability of what I am learning in the "real world". Once you get past why I really need to learn and know this stuff, I usually can put my mind into it and get motivated to learn. In high school, I took Advance Math, not out of love for Math but it was the most convinient way to take Computer Science. Looking back into it now, I wish someone had really explained to me why I needed to pay more attention. I really did not foresee how some of the stuff we were doing could possibly come in handy later on in life.
Anyway back to my thesis: This afternoon, I bumped into an article at GigaOm by Gary Orenstein entitled "Digging Deeper Into Data With Hadoop". In the article, the author gave some kind of an overview of Hadoop and how its being used by some of the top companies. To me, this made my work very relevant because coincidentally, I had just finished writting a MapReduce program that processes a server log to find out query terms and their frequency. According to Gary, Yahoo is using Hadoop to build a database for its Search Assist feature. I was smiling when I read that before the use of Hadoop, the task used to take 26 days to complete! Thanks to Hadoop, it now takes only 20 minutes! I guess thats a cluster with thousands of nodes. I am also encourage that even Microsoft is using Hadoop although its not clear if its limited to Powerset.
I am glad I found this article because it gives my work relevancy and applicability to real world situations. Although I am working on a relatively small cluster with only 12 nodes, I know all this is not in vain!
1 week ago