Working with data

A very important insight from http://radar.oreilly.com/2012/07/data-jujitsu.html I want to share with you:

 It’s impossible to overstress this: 80% of the work in any data project is in cleaning the data.

How to check the integrity of a bunch of gzip files

gunzip -t *.gz

will report broken gzip archives without unpacking them.

How to serve files over http from the shell

This blog published a short bash script for serving files over http. Because I like this solution so much I copy it for you here:

while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; cat <filename>; } | nc -l 8080; done

How to enable snappy compression using cascading

The following code enables snappy compression for the output of a cascading flow:

Properties properties = new Properties();
// set path to main class
AppProps.setApplicationJarClass(properties, Main.class);

// compress mapreduce output
properties.put("mapred.output.compress", "true");

// set compression codec
properties.put("mapred.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");

FlowConnector flowConnector =  new HadoopFlowConnector(properties);

How to create an executable jar file with maven

Executable jars may be created using the maven shade plugin. The shade plugin allows for the creation of 'uber-jars'. This are self contained jar files which contain the application code as well as all dependencies.
The shade plugin is configured in the build-section of your pom.xml:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.1</version>
        <configuration>
          <!-- put your configurations here -->
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

To make the jar file executable, is has to contain an MANIFEST.MF which names the class containing the main function. Just add the following lines to the plugin config:

<configuration>
  <transformers>
    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
      <mainClass>path.to.MainClass</mainClass>
    </transformer>
  </transformers>
</configuration>

See also: http://maven.apache.org/plugins/maven-shade-plugin/examples/executable-jar.html

Big Data Priciples - Part I

Your Input Data Is Immutable

Data storage systems are used to store information. This information may change in the course of time. You have two basic options to reflect these changes in your data storage systems:

• Update the information
• Store the new information in addition to the existing information

Consider the following example:

In a social network a user has a list of followers. This list may be modified by two events:

• follow event - A new follower is added to the list of followers
• unfollow event - A follower chooses to unfollow the user.

One possibility to store this information is to always store and update a list of current followers for each user. Each time a new follower is added or removed you update this list in your storage system.

The second possibility is to store all follow and unfollow events. The current list of followers for a user is derived from this information.

scenario	solution one	solution one
Get the current list of followers for the user 'arthur'.	Read the list of current followers.	Derive the list of current followers from the sequence of follow and unfollow events.
Get the number of followers for the user 'arthor'.	Compute the length of the list of current followers.	Compute the length of the list of current followers.
Get all users that have been following the user 'arthur' two years ago	-	Derive the list of followers from the sequence of follow and unfollow events.
Get a list of all users the user 'ford' has been following on 2000-01-01.	-	Derive the list of followed users from the sequence of follow and unfollow events.

As you can see solution one offers a simple and efficient solution for answering the questions you may have had in mind while implementing the solution. But there are many possible questions you cannot answer with this data model.

Solutions two requires much more storage and also additional computation efforts to answer simple questions. This is a high price to pay, but you get a great reward: Since you do not lose information due to data updates you have the possibilites to answer that arise later in time. Questions that you did not even think of when you where implementing your application.

Solution two provides another big advantage: Since you never update your raw data the danger of data corruption due to an application error is much less!

List all used ports on a unix machine

netstat -a -t --numeric-ports -p