Tuesday, August 6, 2013

How to enable snappy compression using cascading

The following code enables snappy compression for the output of a cascading flow:
Properties properties = new Properties();
// set path to main class
AppProps.setApplicationJarClass(properties, Main.class);

// compress mapreduce output
properties.put("mapred.output.compress", "true");

// set compression codec
properties.put("mapred.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");

FlowConnector flowConnector =  new HadoopFlowConnector(properties);

No comments:

Post a Comment