Wednesday, August 13, 2008

Using Hadoop

Since I've gotten back from OSCON, I've had a chance to use Hadoop at work. For those who aren't familiar with it, Hadoop is an open source framework for implementing map reduce jobs.

There are plenty of tutorials on Hadoop around the web, so I won't do any of the basic intro stuff. I wanted to write about some of the stuff I didn't find all that easily.

Most of the Hadoop documentation talks about dealing with input where your records are on a single line (like an Apache access log). From using the Google/documentation/experience, we have found that Hadoop works just fine with multi-line records. We're using Hadoop to process XML, specifically a dump from Wikipedia. The dump is a single 33 GB file, where there is a root tag, and then several million child tags (representing Wikipedia pages). Using this code I found on the Hadoop core user mailing list, we can have it so that the mapper gets the XML for one child node (or one Wikipedia page). This is nice, because the XML for a single page is relatively small. We then use JDOM to deal with the contents of the individual pages.

We are using HDFS to store our input and output. By default, it will chop files into 64MB chunks, which get shipped to the mappers so that the file can be processed in parallel. One thing that I was concerned about was how records would be handled that spanned the splits. So far, we haven't seen any issues, and this answer in the Hadoop FAQ seems to indicate that the records spanning the split will be handled. It may be possible that those records get dropped, it would be hard for us to tell at this point... but the good news for us is that it won't affect the data we're trying to collect much.

As for our setup and usage. We have 5 machines in our cluster, most of which are run of the mill dual core Intel machines running Linux. The jobs we're running on the 33GB XML file are taking around 45 minutes, which seems pretty fast to me.

54 comments:

anon_anon said...

you may want to look at vtd-xml, the latest and most advanced xml processing api

vtd-xml

Unknown said...

The monthly bills is sometimes start-up about eight to help you twelve bi-weekly commitments, or even simply just in case your loan is determined by your Social stability revenue, installment loans consequently you are able to simply arrange your monthly payments to urge a results of three to four month-to-month payments. often designed commitments facilitate to manufacture a lessened examine entire be the larger go beside your money.

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

mareddyonline said...

Thank for sharing this great Hadoop tutorials Blog post. I will use your command when upgrade hadoop.I get a lot of great information here and this is what I am searching for. Thank you for your sharing. I have bookmark this page for my future reference.
Hadoop Training in hyderabad

Unknown said...

Extremely decent post I simply unearthed your weblog and wished to say that I have genuinely delighted in searching your weblog posts. Much obliged man.

Visit : Online High School Diploma

Unknown said...

This is a very nice article and gives in-depth information. I am so happy to read this.
Packaging Design Canada
Packaging Design Toronto
Graphic Design in Canada

Unknown said...

Great Post! It’s a watch gap to all and I’m a great deal appreciative that she wrote her expertise for us to let recognize them severally.

Visit : GED Online

Unknown said...

Nice piece of article you have shared here, my dream of becoming a hadoop professional become true with the help of Hadoop Course in Chennai, keep up your good work of sharing quality articles.




big data training in velachery|hadoop
training chennai velachery
|hadoop training institute in t nagar

Unknown said...

Understudy credit combination is basically considered as an instrument to deal with one or more obligations. Such a credit additionally permits any understudy to consolidate his/her government or private understudy credits into one single home loan with augmented advance terms, which accordingly minimize the regularly scheduled installment. payday loan newport_news

Unknown said...

New business advances as their name propose are implied for the individuals who need fund for new company. Having a business thought is insufficient to begin new business. Nowadays discovering money is made less demanding by new business advances. Any sort of monetary necessity can without much of a stretch paid for by new business credits. However without fitting arrangement and data getting new business credits may be a bit extreme. Thus, get your certainties right and move your business thought in the right course. Payday Loans San-diego

Unknown said...

Is it correct to say that you are going to go to a car title advance moneylender for extra cash? Is the cash a need or for something extra? The title credit improvement master is not going to stop and sales those sorts from requesting, it is not a progression principal. Banks and credit unions will oblige this information, as their rationale is hugely different. car title loans chicago

Unknown said...

Title advances have the same highlights as a secured improvement, adjacent to a solitary viewpoint. While secured credits don't portray the sort of insurance that will suffice it, title moves particularly oblige cars or whatever other vehicle to go about as security. www.usacheckcashingstore.com/san-diego

MUSA77 said...

Consider these terms, and perform your adjustment of poor recognition motorbike advances as adumbrated by your bread-and-margarine achievability and spending course of action viability.In purchasing a vehicle, you may charge to get to limitless admeasurement of banknote in appearance of ethereal costs of the hurt for vehicle. car title loans near me chicago

Unknown said...

At the point aback engaging for motorbike cash, there are an alloyed sack of edges you'll charge to consider. These will hold the expenses choices open, adulation term, and upkeep rates. Check Cashing San-diego

JON said...

Imagine taking without end with the trade for trade your take in for spendable blend what could be not completely 60 minutes! Vivacious money is clear when you use a vehicle title credit.https://www.usacheckcashingstore.com/san-diego

Unknown said...

Payday advances may be opened up on the off chance that you wind up in a position to be not dealt with to fulfill all or some bit of the entire due on the reimbursement date. On the off chance that this happens it is suggested that you contact your payday advance supplier at the soonest open passageway and uncover your circumstances to them. They will then have the capacity to light up your decisions and how to build up your credit. Auto Title Loans Chicago

Unknown said...

More generally than not, these credits are arranged into secured and detached structure, with the longing that you can get to the sustenance as showed up by your charge and need.
Cash Advance San-diego

Unknown said...

In you accepted job the affairs are there are bodies you like and bodies you don't like but either way you apperceive their moods and quirks and they apperceive yours. While it may not actualize a altogether adapted assignment ambiance it does accredit you to cross assignment backroom a lot easier than if you were the newbie.
Payday Loans Chicago

Unknown said...

The key aback authoritative paycheck loans is the appropriate advice apropos the lending abundance you will be borrowing from and you accept to accomplish abiding that you absolutely charge the money. If you do not absolutely charge the money again you can aloof delay for your abutting paycheck.
cash advance corona

Unknown said...

Over the accomplished few years, banks and architecture societies accept anchored their lending behavior to such an admeasurement that there are around no apart loans accessible at the moment. Consumers accept accordingly looked for added apart borrowing and appropriately we now accept almanac levels of acclaim agenda debt.
check cashing in fresno ca

Justin said...

On the off chance that your arrangement does not work, find different systems in showcasing. Stay informed concerning the opposition in the business. This will help you make arrangements and procedures that will permit you to make your business develop. Check Cashing San-diego

Justin said...

The issue with PPI protection cases is they have a tendency to be destroyed progressively when contrasted and other protection arrangements. This happens since they're not supported and they're generally taken on the grounds that they are without the customer tirelessly assessing on their advantages to the general arrangement. auto title loans chicago

Justin said...

I would read to him constantly, particularly when he required ameliorating, or to take a sleep, amid the day. I would quiet him to rest by creating languid time stories, about his most loved pretend companions, Rainbolt and Tussy. cash advance chicago

Paulo said...

Stock - A bank may progress up to 60 percent to 80 percent of significant worth for prepared to-go retail stock. A maker's stock, comprising of segment parts and other unfinished materials, may be just 30 percent. Cash Advance Chicago

Paulo said...

We are not discussing a $3 million credit line just to show capital on an asset report. Furthermore, we are not discussing a $250,000 gear credit for a local development organization. cash advance

Alex daina said...

This is valid for all parts of our lives, even in the region of our funds. Liquidating checks, for instance, is a considerable measure less demanding these days than it was previously. Check Cashing

selenium said...

Very Informative...Glad to find your blog...Keep Sharing...
TESTING & TRAINING ON SELENIUM
ORACLE TRAINING IN CHENNAI
PYTHON TRAINING IN CHENNAI
DATA SCIENCE TRAINING IN CHENNAI

selenium said...

ONLINE INTERNSHIPS
WINTER INTERNSHIPS FOR ENGINEERING
SUMMER INTERNSHIP
SUMMER INTERNSHIP IN CHENNAI
WINTER INTERNSHIP IN CHENNAI
INTERNSHIP IN CHENNAI
INTERNSHIP
INTERNSHIPS
IT INTERNSHIP IN CHENNAI

Priya said...

Really nice...keep sharing...

SAMPLE QUESTIONNAIRE FOR YARDI SOFTWARE APTITUDE TEST PDF
UNBLOCK YOURSELF WHATSAPP APK
CHILE CLOTHING
BRAC NGO EXAM QUESTION
WHATSAPP BLOCK HACK
ZENSOFT PUNE APTITUDE QUESTIONS
JS MAX INTEGER
TP LINK HACK PASSWORD
PYTHON DEVELOPER RESUME FOR 2 YEARS EXPERIENCE
UBUNTU HACK WIFI

Krish said...

realy nice...
Python Internship
Dotnet Internship
Java Internship
Web Design Internship
Php Internship
Android Internship
Big Data Internship
Cloud Internship
Hacking Internship
Robotics Internship

Krish said...

this is realy usefull..
Oracle Internship
R Programming Internship
CCNA Internship
Networking Internship
Artificial Intelligence Internship
Machine Learning Internship
Blockchain Internship
Sql Server Internship
Iot Internship
Data Science Internship

Krish said...

good post....
Selenium Testing Internship
Linux Internship
C Internship
CPP Internship
Embedded System Internship
Matlab Internship

Arunkumar said...

good
Free Internship for cse students in Chennai
R Programming Internship
Hadoop Training in Chennai
Free Internship Training in Chennai
Robotics Training chennai
Summer Internship For BSC students
Internships in Chennai for CSE
CCNA Institute in Chennai
Data Science Internship in Chennai
Aeronautical Engineering Internship

Arunkumar said...

good
Geteventlisteners javascript
Karl fischer titration interview questions
How to hack tp link router
T system aptitude questions
Resume for bca final year student
Test case for railway reservation system
T systems pune placement papers
Infrrd bangalore interview questions
Max number in javascript
Paypal integration in php step by step pdf

Arjunreddy said...

great

FREE Internship in Nagpur For Computer Engineering Students
Internship For MCA Students
Final Year Projects For Information Technology
Web Design Class
Mechanical Engineering Internship Certificate
Inplant Training For Mechanical Engineering Students
Inplant Training Certificate
Ethical Hacking Course in Chennai
Winter Internship For ECE Students
Internships For ECE Students in Bangalore

Arjunreddy said...

nice...

How To Hack On Crosh
Request Letter For Air Ticket Booking To HR
Zeus Learning Aptitude Paper For Software Developer
Cimpress Interview Questions
VCB Rating
Appreciation Letter To Vendor
JS MAX Safe Integer
Why Do You Consider Yourself Suitable For The Position
How To Hack Android Phone From PC
About Bangalore Traffic Essay

RONOLD said...

GOODPOST
hacking course
internship for it students
ccna course chennai
civil engineering internship report pdf india
kashi infotech
internships in hyderabad for cse students 2018
cse internships in hyderabad
inplant training for diploma students
internship in hyderabad for cse students


RONOLD said...

GOOD
nodejs while loop
icici bank po interview questions and answers pdf
craterzone aptitude test
zensoft recruitment process
java developer resume 1 years experience
python developer resume pdf
infrrd private limited interview questions
js int max value
delete * from table oracle
t systems pune aptitude questions

Arun vijay said...

great

Internship For Aerospace Engineering
Mechanical Engineering Internships in Chennai
Robotics Courses
Kaashiv
Training Letter Format For Mechanical Engineer
Internship For BCA Student
Fake Internship Certificate
MBA Internship
Free Internship For CSE Students in Chennai
Oracle Internship 2020

Arun vijay said...

good post

Javascript Maximum Integer
INT MAX Javascript
Acceptance is to an Offer What a Lighted Match is to a Train of Gunpowder
Who Can Issue Character Certificate
Technical Support Resume DOC
PHP Developer Resume For 3 Year Experience
Wapda Interview Questions
Power BI Resume Download
a Dishonest Dealer Professes to Sell His Goods at a Profit of 20
Failed to Find 'Android_Home' Environment Variable. TRY Setting it Manually

Arun vijay said...
This comment has been removed by the author.
Arun vijay said...
This comment has been removed by the author.
arun said...

good
iot training in coimbatore
summer training for 3rd year electronics and communication engineering students
goa current affairs 2019
project for information technology students
online internship for bca students
winter training for mechanical engineering students
ccna training
industrial training report for electronics and communication pdf
matlab courses in chennai
bba internship project

arun said...

good

Resume Format For Bca Freshers
British Airways Interview Questions And Answers Pdf
Asus Tf101 Android 8
Crome://Flags/
T Systems Aptitude Test
Python Resume Ror 2 Years Experience
Ajax Redirect To Another Page With Post Data
Paramatrix Technologies Aptitude Questions And Answers
Adder Subtractor Comparator Using Ic 741 Op-Amp Theory
How To Hack Wifi With Ubuntu

rohiniseema said...

nice bloggers....!!!!
poland web hosting
russian federation web hosting
slovakia web hosting
spain web hosting
suriname
syria web hosting
united kingdom
united kingdom shared web hosting
zambia web hosting
inplant training in chennai













rohiniseema said...

super....!!!
text animation css
animation css background
sliding menu
hover css
css text animation
css loaders
dropdown menu
buttons with css