How to rsync a folder over ssh

Another short shell snippet:

rsync -avze ssh source user@host:target-folder

For an explanation of the command click here.

How to write unit tests for your Hadoop MapRecude jobs

Simple answer: use MRUnit.

You will need a classifier to include it in your maven project:

  <dependency>
   <groupId>org.apache.mrunit</groupId>
   <artifactId>mrunit</artifactId>
   <version>1.0.0</version>
   <classifier>hadoop2</classifier>
   <scope>test</scope>
  </dependency>

Time based tests with joda time

In unit tests it is helpful to set a fixed date and time. With joda time you can do this by simply calling

DateTimeUtils.setCurrentMillisFixed(new DateTime(2013, 3, 2, 0, 0).getMillis());

How to launch eclipse with a specific jdk on mac os

Edit /Eclipse.app/Contents/MacOS/eclipse.ini. Add the following before -vmargs:

-vm 
/path/to/java/home

The linebreak after -vm matters! To find out the path to java home run

/usr/libexec/java_home
or 
/usr/libexec/java_home -v 1.7

How to install the sun 6 / oracle 7 jdk on ubuntu

https://github.com/flexiondotorg/oab-java6 provides a script for building a java package for sun jdk 6 and oracle jdk 7 for ubuntu.

wget https://github.com/flexiondotorg/oab-java6/raw/0.3.0/oab-java.sh -O oab-java.sh
chmod +x oab-java.sh
sudo ./oab-java.sh

sudo apt-get install sun-java6-jdk

How to cache your git credentials

The following command tells git to cache your password for 5 minutes:

git config --global credential.helper 'cache --timeout=300'

See also http://git-scm.com/docs/git-credential-cache.


https://confluence.atlassian.com/display/STASH/Permanently+authenticating+with+Git+repositories

OS X

Follow these steps to use Git with credential caching on OS X:

1. Download the binary git-credential-osxkeychain.
2. Run the command below to ensure the binary is executable:
   

   chmod a+x git-credential-osxkeychain

3. Put it in the directory /usr/local/bin.
4. Run the command below:
   

   git config --global credential.helper osxkeychain

Use avro-tools to inspect avro files in hdfs

Since version 1.7.5 avro-tools include support for hdfs tools. To read an avro file in hdfs use the following:

hadoop jar avro-tools-1.7.5.jar  tojson hdfs://<hostname>:/path/to/file.avro | less