A common technique in MapReduce is to input a group of records, calculate a value from that group, and emit each record with the new value attached. While this is easy to do in raw MR jobs, the solution in Cascading is not very obvious. This tutorial introduces a new operation to Cascading called BufferedSum. BufferedSum allows us to calculate values from a group of tuples and emit the group value to individual tuples in a scalable way.

Describing the operation of BufferedSum is clearer when discussed in concrete terms, so let’s work with an example.

Example

When dealing with large amounts of documents in Hadoop, its common to have each input file to contain many documents. Our input file in this case will contain two documents:

a.txt\thello world world
b.txt\tgoodbye goodbye world

Lets say we want to calculate tf-idf for these documents. One of the first values we need is the count of the occurrence particular term within each document.

First, we will split each line into (document_id, body) pairs:

  pipe = new Each(pipe, 
      new Fields("line"), 
      new RegexSplitter(new Fields("document_id", "body"), "\t"));

From there we “tokenize” the document and extract each term:

  pipe = new Each(pipe, // tokenize words by space
      new Fields("body"),
      new RegexSplitGenerator(new Fields("term"), "\\s+"), 
      new Fields("document_id", "term"));

Now our tuple stream is the following:

a.txt hello
a.txt world
a.txt world
b.txt goodbye
b.txt goodbye
b.txt world

Count of term in document_id

We now have (document_id, term) and we want to calculate (document_id, term, term_count_in_document). With Cascading, this is easy, simply group by document_id and term and use the Count() function:

  // count how many times `term` appears in `document_id`
  pipe = new GroupBy(pipe, new Fields("document_id", "term"));
  pipe = new Every(pipe, 
      new Fields("term"), 
      new Count(new Fields("term_count_in_document")), 
      new Fields("document_id", "term", "term_count_in_document"));

Calculating total_terms_in_document

So far, so good. Up to this point Cascading has provided everything we need. However, next we want to get the total terms within each document and keep the tuples we have calculated thus far. Put another way, we have an input of (document_id, term, term_count_in_document) and we want to emit (document_id, term, term_count_in_document, total_terms_in_document)

Our first instinct might be to use GroupBy() and Count() like before. But there is a catch: Every operations emit the operator result with the group tuple (see the Each and Every Pipes in the Cascading User Guide).

This means if we group by document_id and Sum() the total_terms_in_document we will emit (document_id, total_terms_in_document). The number in total_terms_in_document will be accurate, but we lose our term and term_count_in_document.

If we try to save our other fields by grouping on all three of them (document_id, term, term_count_in_document) then we’ve “over-grouped” and every “group” is a single tuple (the input tuple) and we won’t get the count of terms in the document as a whole. BufferedSum was created to solve this problem.

BufferedSum

BufferedSum takes as its input three things:

• The name of the Field to output
• The name of the Field to sum
• The other Fields to “pull through” the operation

Here is how we can use BufferedSum to achieve the desired effect:

// input: (document_id, term, term_count_in_document)
// emits: (document_id, term, term_count_in_document, total_terms_in_document) 
pipe = new GroupBy(pipe, new Fields("document_id"));
pipe = new Every(pipe, 
    new BufferedSum(new Fields("total_terms_in_document"), 
                    new Fields("term_count_in_document"),
                    new Fields("document_id", "term", "term_count_in_document")), 
    Fields.SWAP);

Note: the output selector Fields.SWAP is critical due to Cascading tuple selection.

Memory considerations

One thing to be careful of when using BufferedSum is to try and keep your groups small enough to fit in memory. However, this is not a requirement. BufferedSum uses Cascading’s SpillableTupleList which will spill to the HDFS if it grows too large. That said, spilling is an expensive operation and should be avoided if possible.

Summary

BufferedSum is a widely useful operation when dealing with sums in Cascading. In Part 2 we will use BufferedSum and Cascading to finish calculating tf-idf.

The Code

package com.xcombinator.cascading.operations.buffers;
import cascading.flow.FlowProcess;
import cascading.operation.BaseOperation;
import cascading.operation.Buffer;
import cascading.operation.BufferCall;
import cascading.tuple.Fields;
import cascading.tuple.Tuple;
import cascading.tuple.TupleEntry;
import cascading.tuple.SpillableTupleList;
 
import java.util.Iterator;
 
/**
 * BufferedSum sums a value for every Tuple in a Group and emits every input
 * Tuple with the sum appended.
 * <p/>
 * 
 * 
 * EXAMPLE:
 *
 * {@code 
 *
 * // input: (document_id, term, term_count_in_document)
 * // emits: (document_id, term, term_count_in_document, total_terms_in_document) 
 *
 *     pipe = new GroupBy(pipe, new Fields("document_id"));
 *     pipe = new Every(pipe, 
 *         new BufferedSum(new Fields("total_terms_in_document"), 
 *                        new Fields("term_count_in_document"),
 *                        new Fields("document_id", "term", "term_count_in_document")), 
 *         Fields.SWAP);
 * }
 *
 * @see BufferedSum
 * 
 */
 
public class BufferedSum extends BaseOperation implements Buffer
  {
  private Double sum;
  private SpillableTupleList list;
  private Fields extrasSelector;
  private Fields fieldToSum;
 
  /**
   * Returns a BufferedSum Buffer Operation. 
   *
   * @param emittedSumFieldName a {@link Fields} naming the field to emit the sum value
   * @param fieldToSum          a {@link Fields} naming the field to sum
   * @param extrasSelector      a {@link Fields} naming the other fields to "pull through". These fields *must* be of the same order and size as the input Tuple
   */
  public BufferedSum( Fields emittedSumFieldName, Fields fieldToSum, Fields extrasSelector )
    {
    super( extrasSelector.append( emittedSumFieldName ) );
    this.extrasSelector = extrasSelector;
    this.fieldToSum = fieldToSum;
    }
 
  public void operate( FlowProcess flowProcess, BufferCall bufferCall )
    {
    Iterator<TupleEntry> iterator = bufferCall.getArgumentsIterator();
    sum = 0.0D;
    list = new SpillableTupleList( 10000 );
 
    while( iterator.hasNext() )
      {
      TupleEntry arguments = iterator.next(); // must be called
      sum += arguments.getDouble( this.fieldToSum.get(0) );
      list.add( arguments.getTuple() );
      }
 
    for( Tuple tuple : list )
      {
      bufferCall.getOutputCollector().add( tuple.append( new Tuple( sum ) ) );
      }
 
    }
  }

Share:

However, sometimes you may have code for an existing MapReduce job or want to drop directly to Hadoop for efficiency. Even if you’re using raw MapReduce jobs, Cascading can still be useful in planning the overall data pipeline.

The code below is an example of how to use a raw MapReduce job in a Cascade. The main thing to take away is that we are creating intermediate sinks and sources and relying on Cascading to schedule the flows in the correct order.

NOTE: this code below depends on commit f0dd84cd which is a patch to MapReduceFlow.java that allows you to specifically set the Taps for a MapReduceFlow. I’ve contacted Chris about integrating this into the trunk.

Also note this patch applies to the branch wip-1.1 and later.

package com.xcombinator.hadoopjobs.mapreducetest;
 
import cascading.cascade.*;
import cascading.flow.Flow;
import cascading.flow.FlowConnector;
import cascading.flow.MapReduceFlow;
import cascading.operation.aggregator.Count;
import cascading.operation.regex.*;
import cascading.pipe.*;
import cascading.scheme.*;
import cascading.tap.*;
import cascading.tuple.Fields;
import cascading.operation.Identity;
import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.mapred.lib.IdentityMapper;
import org.apache.hadoop.mapred.lib.IdentityReducer;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;
import cascading.operation.Debug;
 
import org.apache.hadoop.mapred.KeyValueTextInputFormat;
 
/**
 * An example file to use a raw MapReduce job in cascading
 */
public class Main extends Configured implements Tool
  {
  private static final Logger LOG = Logger.getLogger( Main.class );
 
  public int run(String[] args)
  {
    JobConf conf = new JobConf(getConf(), this.getClass());
    Properties properties = new Properties();
    FlowConnector.setApplicationJarClass(properties, this.getClass());
 
    CascadeConnector cascadeConnector = new CascadeConnector();
    FlowConnector flowConnector = new FlowConnector(properties);
 
    String inputPath  = args[0];
    String outputPath = args[1];
    String intermediatePath1 = args[1] + "-mr-input";
    String intermediatePath2 = args[1] + "-mr-output";
 
    Scheme textLineScheme = new TextLine();
 
    Tap sourceTap = new Hfs(textLineScheme, inputPath);
    Tap intermediateTap1 = new Hfs(new TextLine(new Fields("line")), intermediatePath1);
    Tap intermediateTap2 = new Hfs(textLineScheme, intermediatePath2);
    Tap sinkTap   = new Hfs(textLineScheme, outputPath);
 
    // create our first flow, sink to the intermediateTap
    Pipe wsPipe = new Each("wordsplit", 
        new Fields("line"), 
        new RegexSplitGenerator(new Fields("word"), "\\s+"), 
        new Fields("word"));
 
    Flow parsedLogFlow = flowConnector.connect(sourceTap, intermediateTap1, wsPipe);
 
    // Create a pipe and set our mr job for it 
    Pipe importPipe = new Pipe("mr pipe");
 
    JobConf mrconf = new JobConf();
    mrconf.setJobName("custom mr");
 
    mrconf.setOutputKeyClass(LongWritable.class);
    mrconf.setOutputValueClass(Text.class);
 
    // the IdentityMapper, in this case, will actually output the key, which is
    // a long of offset in bytes. Not what we'd usually want, but we'll leave
    // it in for now.
    mrconf.setMapperClass(IdentityMapper.class);
    mrconf.setReducerClass(IdentityReducer.class);
 
    // note that your input is straight text-lines. This means in a real mr job
    // you'd most likely need to split the line by some convention
    TextInputFormat format = new TextInputFormat();
    format.configure(mrconf);
 
    // NOTE: this is both here and in the MapReduceFlow below
    FileInputFormat.setInputPaths(mrconf, intermediateTap1.getPath());  
    FileOutputFormat.setOutputPath(mrconf, intermediateTap2.getPath()); // likewise
 
    // create our second flow, this one is for the mrjob. Notice source and sink taps
    Flow mrFlow = new MapReduceFlow("mrflow", 
      mrconf, intermediateTap1, intermediateTap2, false, true);
 
    // create our third "regular" cascading pipe
    Pipe countPipe = new Pipe("count");
 
    // b/c our IdentityMapper is emitting long of offset in the line, just
    // strip that out. You wouldn't have to do this if you had a smarter Mapper
    // class.
    countPipe = new Each(countPipe, 
        new Fields("line"), 
        new RegexParser(new Fields("word"), ".*?\\t(.*)"), 
        new Fields("word"));
 
    countPipe = new GroupBy(countPipe, new Fields("word"));
    countPipe = new Every(countPipe, new Count(), new Fields("count", "word"));
 
    // create the flow for the last count pipe
    Flow countFlow = flowConnector.connect(intermediateTap2, sinkTap, countPipe);
 
    Cascade cascade = cascadeConnector.connect(parsedLogFlow, mrFlow, countFlow);
    cascade.complete();
 
    // if you want to get rid of the intermediate files you 
    // could do something like the following here:
    // Path tmp = tap.getPath();
    // FileSystem fs = tmp.getFileSystem(conf);
    // fs.delete(tmp, true);
 
    return 0;
  }
 
  public static void main(String[] args) throws Exception 
  {
    int res = ToolRunner.run(new Configuration(), new Main(), args);
    System.exit(res);
  }
 
  }

Share:

Summary

Setting up a scalable Hadoop cluster isn’t easy, but PoolParty makes it easier
and manageable.

By the time we’re done with this tutorial you’ll have a Hadoop cluster consisting of one master node and two slaves. The slaves are formatted with HDFS and process MapReduce jobs that are delegated to them from the master.

The whole cluster is monitored by Ganglia.

Benefits of PoolParty

The nodes are very interdependent. By that I mean that each node needs to have 2 or 3 configuration files that are based on the other currently running nodes in the cluster. As nodes are joining and leaving the cluster each of these files on every node needs to be updated. PoolParty handles this process for you more-or-less automatically. The benefit is that you don’t have roll your own methods to do this every time you want to setup a cluster.

In PoolParty plugins are first-class citizens. This means you can write your own plugins and they are every bit as powerful as the resources that make up PoolParty core itself. This makes it easy to break up server functionality into modules of code . PoolParty, in a sense, gives you object-oriented server configurations. You can, for instance, take a Ganglia object, call a few methods and PoolParty takes care of executing the required commands to deploy a configured Ganglia cluster.

Architecture

PoolParty is built around the notion of pools and clouds . A pool is simply a collection of clouds. A cloud is a homogeneous set of nodes. i.e. every node in a cloud is configured the same way . Obviously nodes in a cloud will have different sets of working data as they run, but the idea is any node in a cloud could be substituted for any other node in that same cloud.

PoolParty itself is designed to be fully distributed and masterless. There is no required concept of “master” and “slave” in PoolParty itself. That said, many pieces of software, such as Hadoop, do have this concept and PoolParty can be configured to take advantage of that.

We’ll be setting up our pool as two clouds hadoop_master and hadoop_slave. Obviously, hadoop_slave will be a cloud (cluster) of nodes configured to be Hadoop slaves. hadoop_master will also be a cloud of masters. In our example we’re only going to use 1 node as the master. But you could relatively easily configure everything to have more than one master.

Software involved

• Hadoop
• Hive
• Ganglia
• PoolParty

Prerequisites

This tutorial assumes that:

1. You have Amazon EC2 java tools installed. See EC2: Getting Started with the Command Line Tools
2. You have the proper EC2 environment variables setup. See Setting up EC2 on the PoolParty website. For instance, a typical PoolParty install would have these variables in $HOME/.ec2/keys_and_secrets.sh.
3. You have PoolParty installed from source. In theory, you should be able to install the gem. However, today you should probably install from source. Make sure you have git://github.com/auser/poolparty.git checked out and then follow the “Installing” directions on the PoolParty wiki. You only need to complete the two sections Dependencies required to build gem locally and Instructions . This will install all the development dependency gems and then make sure you have all of the submodules. NOTE PoolParty deploys ruby gem versions based on the versions on your local machine. So make sure you have the most recent versions of the required gems installed locally.
4. You have the jashmenn/poolparty-examples repository. git clone git://github.com/jashmenn/poolparty-examples.git /path/to/poolparty-examples
5. You have the jashmenn/poolparty-extensions repository. Note that this directory must be a sibling directory to the poolparty-examples directory. git clone git://github.com/jashmenn/poolparty-extensions.git /path/to/poolparty-extensions

EC2 Security

Now that we have the code issue complete, we now need to deal with Amazon’s security. (See here if you are unclear on how EC2 security works.)

Setup Keypairs

Every cloud in PoolParty must have its own unique keypair. Thats important enough it’s worth repeating: every cloud in PoolParty must have its own unique keypair .

So run the following commands:

ec2-add-keypair cloud_hadoop_slave > ~/.ssh/cloud_hadoop_slave
ec2-add-keypair cloud_hadoop_master > ~/.ssh/cloud_hadoop_master
chmod 600 ~/.ssh/cloud_hadoop_*

Security Groups

You’ll also want to create a security group for our pool .

ec2-add-group hadoop_pool -d "the pool of hadoop masters and slaves"

NOTICE: Hadoop has a crazy number of ports that it requires. The ports below will work but may not be the most secure configuration. If you understand this better than I please recommend better settings. Otherwise proceed knowing that these ports are probably a little too open.

We also need to open a number of ports for this security group:

<code>ec2-authorize -p 22 hadoop_pool               # ssh
ec2-authorize -p 8642 hadoop_pool             # poolparty internal daemons
ec2-authorize -P icmp -t -1:-1 hadoop_pool    # if you want to ping (optional, i guess)
ec2-authorize -p 80 hadoop_pool               # apache
 
ec2-authorize -p 8649 -P udp hadoop_pool      # ganglia UDP
ec2-authorize hadoop_pool -o hadoop_pool -u xxxxxxxxxxxx # xxxxxxxxxxxx is your amazon account id. ugly but true
</code>

Start your cloud

NOTE : There are a number of configurations that rely on the whole cloud being booted. This means that the first time you run cloud-start you may see a few shell errors. This is okay as long as it goes away after subsequent configures. The idea is that all nodes need to be started before the whole configuration will work properly.

cd /path/to/poolparty-examples/hadoop
cloud-list # sanity check, no instances should show up, no exceptions should be raised
cloud-start -vd

Tons of information will fly by. Be patient, this could take upwards of 15 minutes.

Everything done? Good. Now you’re going to need to configure a second time. Now that all the nodes are booted they can be configured to talk to each other properly.

cloud-configure -vd

Again, tons of output should fly by. Wait for it to finish.

Now what we want to do is actually run our hadoop sample job. Open up the hadoop/clouds.rb and find the lines that look like this:

hadoop do
  configure_master
  prep_example_job
  # run_example_job
end

Uncomment the run_example_job line and configure, but this time we only need to configure master.

cloud-configure -vd -c hadoop_master

This should work, but there is a chance the hdfs wont be started in time to load the sample job. If that happens, just configure one more time.
You know it worked if you see output like the following (it wont be at the bottom):

[Fri, 26 Jun 2009 20:09:50 +0000] DEBUG: STDERR: 09/06/26 20:09:11 INFO input.FileInputFormat: Total input paths to process : 3
09/06/26 20:09:12 INFO mapred.JobClient: Running job: job_200906262006_0001
09/06/26 20:09:13 INFO mapred.JobClient:  map 0% reduce 0%
09/06/26 20:09:32 INFO mapred.JobClient:  map 66% reduce 0%
09/06/26 20:09:38 INFO mapred.JobClient:  map 100% reduce 0%
09/06/26 20:09:47 INFO mapred.JobClient:  map 100% reduce 100%
09/06/26 20:09:49 INFO mapred.JobClient: Job complete: job_200906262006_0001
09/06/26 20:09:49 INFO mapred.JobClient: Counters: 17

Congradulations! You now have a scalable Hadoop cluster at your disposal!

What to do when something goes wrong

• Checkout the PoolParty IRC channel, we’re always around and ready to help #poolpartyrb.

This plugin was based on a number of helpful sites on the web. Checkout the following links:

Hadoop

• Michael Noll’s Haddop Tutorial)

Hive

• Apache’s Hive website

Ganglia

• IBM’s Ganglia Tutorial

References

• PoolParty Documentation

Share: