How to install the sun 6 / oracle 7 jdk on ubuntu

https://github.com/flexiondotorg/oab-java6 provides a script for building a java package for sun jdk 6 and oracle jdk 7 for ubuntu.

wget https://github.com/flexiondotorg/oab-java6/raw/0.3.0/oab-java.sh -O oab-java.sh
chmod +x oab-java.sh
sudo ./oab-java.sh

sudo apt-get install sun-java6-jdk

How to cache your git credentials

The following command tells git to cache your password for 5 minutes:

git config --global credential.helper 'cache --timeout=300'

See also http://git-scm.com/docs/git-credential-cache.


https://confluence.atlassian.com/display/STASH/Permanently+authenticating+with+Git+repositories

OS X

Follow these steps to use Git with credential caching on OS X:

1. Download the binary git-credential-osxkeychain.
2. Run the command below to ensure the binary is executable:
   

   chmod a+x git-credential-osxkeychain

3. Put it in the directory /usr/local/bin.
4. Run the command below:
   

   git config --global credential.helper osxkeychain

Use avro-tools to inspect avro files in hdfs

Since version 1.7.5 avro-tools include support for hdfs tools. To read an avro file in hdfs use the following:

hadoop jar avro-tools-1.7.5.jar  tojson hdfs://<hostname>:/path/to/file.avro | less

Working with data

A very important insight from http://radar.oreilly.com/2012/07/data-jujitsu.html I want to share with you:

 It’s impossible to overstress this: 80% of the work in any data project is in cleaning the data.

How to check the integrity of a bunch of gzip files

gunzip -t *.gz

will report broken gzip archives without unpacking them.

How to serve files over http from the shell

This blog published a short bash script for serving files over http. Because I like this solution so much I copy it for you here:

while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; cat <filename>; } | nc -l 8080; done

How to enable snappy compression using cascading

The following code enables snappy compression for the output of a cascading flow:

Properties properties = new Properties();
// set path to main class
AppProps.setApplicationJarClass(properties, Main.class);

// compress mapreduce output
properties.put("mapred.output.compress", "true");

// set compression codec
properties.put("mapred.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");

FlowConnector flowConnector =  new HadoopFlowConnector(properties);