Before I began what I thought to have been an easy task setting up Eclipse so that I could write MapReduce programs and not have to run the mapreduce programs manually, I came found a website that had a statement mentioning how getting Hadoop running could be a whole day's job. They were marketing a Hadoop live cd and just assumed it was a clever marketing ploy. Unfortunately with my limited Linux environment experience, the author was right! (Another live cd, virtual image)
If you got to this article searching for a tutorial, this post will likely disappoint you. I know the feeling. I scoured endlessly on the internet for step by step instructions but I just found many unanswered questions.
I had Hadoop setup before on both my laptop and desktop following Michael Noll's instructions, running in a pseudo-distributed mode. The problem was getting Eclipse set up to run mapreduce jobs - a very cool and time saving feature. After following many how-to's, I ended up with a bad Hadoop installation. This was the bad part. I had replaced the hadoop-site.xml file with a different one. This new file had the hadoop temporary directory under Ubuntu's temporary directory. It worked until I rebooted. I found out later the hard way that by rebooting, the directories or files needed by hadoop were being deleted. As a result, you would have a good working install soon after you format the namenode but have all sorts of problems once you reboot.
The other major problem was that Eclipse was having all sorts of access errors. What I found out was that following Noll's instructions, he advises to create a separate hadoop account. This was no problem using ssh but was a problem trying to connect through eclipse. What I eventually did was just forget about a separate hadoop account and it eliminated all the problems I was having.
I wish all the frustration had not prevented me from taking better notes so that my experience could be of help to someone. The trick though was to use Eclipse 3.3 . The most recent versions had too many options which became too complicated for someone who is still yet to fully master Linux.
1 week ago
Post a Comment