Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re^3: Hadoop and perl

by spazm (Monk)
on Sep 09, 2010 at 17:05 UTC ( #859535=note: print w/replies, xml ) Need Help??

in reply to Re^2: Hadoop and perl
in thread Hadoop and perl

I've added Hadoop::Streaming to cpan. This module aims to simplify the Streaming interface to Hadoop, by automating the input/output mapping. I found the initial module on github, then extended it with new features and tests and pushed it to CPAN.

Please take a look and let me know how to make it more useful, especially to the new-to-hadoop and new-to-hadoop-streaming user! Specifically I've added new documentation to the root level Hadoop::Streaming, and I'd love feedback.

Thank you,

package My::Hadoop::Example::Wordcount; use Moose::Role; sub map { my ($self,$line) = @_; my @words = split( /\W+/, $line); $self->emit( $_ => 1 ) for @words; } sub reduce { my ( $self, $key, $value_iterator) = @_; my $sum = 0; while( $value_iterator->has_next() ) { my $value = $value_iterator->next(); $sum += $value; } $self->emit( $key, $sum ); } sub combine { my ( $self, $key, $value_iterator) = @_; my $sum = 0; while( $value_iterator->has_next() ) { my $value = $value_iterator->next(); $sum += $value; } $self->emit( $key, $sum ); } package My::Hadoop::Example::Wordcount::Mapper; use Moose; with Hadoop::Streaming::Mapper, My::Hadoop::Example::Wordcount; package My::Hadoop::Example::Combiner::Wordcount::Mapper; use Moose; with Hadoop::Streaming::Combiner, My::Hadoop::Example::Wordcount; package My::Hadoop::Example::Wordcount::Reducer; use Moose; with Hadoop::Streaming::Reducer, My::Hadoop::Example::Wordcount; 1;
Driver files:


#!/usr/bin/perl use My::Hadoop::Example; My::Hadoop::Example::Mapper->run();
#!/usr/bin/perl use My::Hadoop::Example; My::Hadoop::Example::Combiner->run();
#!/usr/bin/perl use My::Hadoop::Example; My::Hadoop::Example::Reducer->run();
Hadoop streaming jar command:
hadoop \ jar $streaming_jar_name \ -D"my hadoop example" \ -input my_input_file \ -output my_output_hdfs_path \ -mapper my_mapper \ -combiner my_combiner \ -reducer my_reducer

Replies are listed 'Best First'.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://859535]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (5)
As of 2018-03-22 17:56 GMT
Find Nodes?
    Voting Booth?
    When I think of a mole I think of:

    Results (282 votes). Check out past polls.