How to increase Xmx for the hadoop client applications

Sometimes you need more memory for hadoop client application tools like hive, beeline or pig.

export HADOOP_CLIENT_OPTS=-Xmx2G

Find out the total size of directories with the 'du' command

To get a sorted list of folder sizes run:

du -sm * | sort -n

How to access the web interface of a remote hadoop cluster over a SOCKS proxy

You want to use the web interface of a hadoop cluster but you only have ssh access to it? SOCKS is the solution:

Use ssh to open a SOCKS proxy:

ssh -f -N -D 7070 user@remote-host

Click here for an explanation of the command.

After that you can configure firefox to use this proxy:

1. Go to the manual proxy settings and add localhost:7070 as SOCKS v5 host
2. Go to about:config and set network.proxy.socks_remote_dns to true to use DNS resolution over the proxy (thanks to Aczire for this!).

Thats all!

How to rsync a folder over ssh

Another short shell snippet:

rsync -avze ssh source user@host:target-folder

For an explanation of the command click here.

How to write unit tests for your Hadoop MapRecude jobs

Simple answer: use MRUnit.

You will need a classifier to include it in your maven project:

  <dependency>
   <groupId>org.apache.mrunit</groupId>
   <artifactId>mrunit</artifactId>
   <version>1.0.0</version>
   <classifier>hadoop2</classifier>
   <scope>test</scope>
  </dependency>

Time based tests with joda time

In unit tests it is helpful to set a fixed date and time. With joda time you can do this by simply calling

DateTimeUtils.setCurrentMillisFixed(new DateTime(2013, 3, 2, 0, 0).getMillis());

How to launch eclipse with a specific jdk on mac os

Edit /Eclipse.app/Contents/MacOS/eclipse.ini. Add the following before -vmargs:

-vm 
/path/to/java/home

The linebreak after -vm matters! To find out the path to java home run

/usr/libexec/java_home
or 
/usr/libexec/java_home -v 1.7