Welcome to Mining Big Data Assignment 1 Documentation

This Documentation consists of the answers to the assignment questions and the code used to solve the assignment. You can find the readme file in the particular exercise folders for more information.

Private Repository till grades are released

Due to academic integrity policies, The source code will be submitted in the group's tab. And all the information about the code & application can be found in the documentation and report submitted.

You can find the code in the exercise3, exercise4 folder under both MacOS & Cloudera folders for the particular exercise.

The instructions to run the program is given [here](#how-to-run-the-program)

Exercise 3


Run the given program in both standalone and pseudo-distributed mode and record the outputs.

The program should be able to run in both modes and the outputs should be the same.


Exercise 4


Part 1

Program to count the number of words with the specific number of letters uses a Mapper to convert the input to key-value pairs of <length, 1> and Reducer will reduce the key-value pairs to <SameLength, n>. The input string is sanitized against delimiter to get the proper words. Dataflow is as follows: Count Word

Part 2

Program to count the number of words (unique) with the specific number of letters, we have decided to submit two solutions as both have their significant impact on the computation. Unit test cases are written with the help of mockito and mrunit library.

ChainMapper is not supported by MrUnit

ChainMapper is not supported by MrUnit, so the unit testing is done for the mapper and reducers. But not the MapReduce together. The alternative way could be running the job under a "test" prefixed name would be a good choice.

a. ChainMapper

Using ChainMapper, the program will first convert the input to key-value pairs of <word, 1> and then the second mapper will get the unique words and convert them to <word, 1>. The third mapper will get the length of the word and replace the key-value pairs as <WordLength, 1>. The Reducer will reduce the key-value pairs to <WordLength, n>. The computation is slower compared to the other solution. For the firstInputFile, it takes around ~8 ms to complete the computation. But the resource consumption is lower. (datanode, resourcemanager, yarn, nodemanager.) Dataflow is as follows: Unique Word Count Using ChainMapper

b. Two MapReduce Jobs

Considering working with Two MapReduce Jobs instead of a single job, the program will first convert the input to key-value pairs of <word, 1> and reducer will get the count of the words. The second job will map the unique words and convert them to <word, 1> and reduce the key-value pairs to <WordLength, n>. Compared to the other solution, it is faster. For the firstInputFile, it takes around ~2 ms to complete the computation. But the resource consumption is higher. (datanode, resourcemanager, yarn, nodemanager.) Dataflow is as follows: Unique Word Count Using MapReduce

How to run the program.

For MacOS, you can find the installation guide here

For Windows, you can find the installation guide here

Folder Structure

The repository consists of two different OS folders as both of team members used different setup for running the assignment problems, For Mac OS with M1 Chip we have used Docker and VM Cloudera was done by the Assignment Handout instructions. Exercise 3 & 4 and the report latex file will found on the root.

Assign1
├── README.md
├── package.json
├── .gitignore
├── docs
│   ├── docs
│   └── mkdocs.yml
├── MacOS
│   ├── Exercise4
│   │   ├── src
│   │   │   ├── com.mbdassign.wordfreq
│   │   │   │   ├── CustomUniqueWordCount.java # Custom Unique Word Count Using Chain Mapper
│   │   │   │   ├── CustomWordCount.java
│   │   │   │   ├── UniqueWordsTwoJobs.java # Custom Unique Words Using Two MapReduceJobs
│   │   │   ├── test.mbdassign.wordfreq # Unit tests with MRUnit Library
│   │   │   │   ├── CustomUniqueWordCountMRUnit.java 
│   │   │   │   ├── CustomWordCountMRUnit.java
│   │   │   │   ├── UniqueWordsTwoJobsMRUnit.java 
│   │   ├── input # input folders
│   │   ├── output # output folders (Artifacts)
│   ├── Exercise3
│   │   ├── src
│   │   │   ├── com.basicsetup.hadoop
│   │   │   │   ├── WordCount.java
│   │   ├── input # input folders
│   │   ├── output # output folders (Artifacts)
├── VM Cloudera
│   ├── Exercise 3
│   │   ├── 
│   │   │   ├── WordCount
│   │   │   │   ├── WordCount.java
│   │   ├── input # input folders
│   │   ├── output # output folders 
│   │   ├── Jars # SetupJars
│   ├── Exercise 4
│   │   ├── src
│   │   │   ├── FreqCounter
│   │   │   │   ├── CustomUniqueWordCount.java # Custom Unique Word Count Using Chain Mapper
│   │   │   │   ├── CustomWordCount.java
│   │   │   │   ├── UniqueWordsTwoJobs.java # Custom Unique Words Using Two MapReduceJobs
│   │   ├── input # input folders
│   │   ├── output # output folders 
│   │   ├── Jars # SetupJars
└── Report.pdf

Cloudera Instructions

For the Cloudera Setup, we use VirtualBox. For the setting up the environment we have followed the steps given in the handout.

Running the program

The given source code is compiled and the jar file can be used to run the program.

hdfs dfs -mkdir /input
hdfs dfs -put input/* /input
hdfs dfs -ls /input
hdfs dfs -cat /input/input.txt
hadoop jar WordCount.jar com.basicsetup.hadoop.WordCount /input/input.txt /output
hdfs dfs -ls /output
hdfs dfs -cat /output/part-r-00000
hdfs dfs -get /output

MacOS Instructions

Requirements

The best way to install Docker on MacOS is Homebrew Cask.

brew cask install docker # Install Docker
open /Applications/Docker.app # Start Docker

Using docker-hadoop

To run Hadoop on Arm64 (Apple M1 Chip) and Intel Native we have used docker-hadoop

Spinning up the environment for Standalone mode

Open terminal and clone the docker-hadoop. Then run the following commands

git clone https://github.com/wxw-matt/docker-hadoop # Clone the docker-hadoop
cd hadoop-standalone  # Go to the docker-hadoop folder
docker build -t wxwmatt/hadoop-standalone:2.1.1-hadoop3.3.1-java8 .  # Build the image
Running the Hadoop Standalone mode

Once the image is built, you can get in the container by running the following command

docker container run --name standalone -it {$imageID} /bin/bash # Run the container

To get the imageID

You can use the command docker images to get the imageID.

The commands given below will run the Hadoop Standalone mode.

mkdir -p /hadoop/input/ # Create the input folder
cp input/* /hadoop/input/ # Copy the input files to the input folder
hadoop jar {$jarFileName}.jar {$className} /hadoop/input/ /hadoop/output/ # Run the jar file
cat /hadoop/output/part-r-00000 # Print the output
Spinning up the environment for Pseudo-distributed mode

Open terminal and clone the docker-hadoop. Then run the following commands

git clone https://github.com/wxw-matt/docker-hadoop # Clone the docker-hadoop
cd docker-hadoop # Go to the docker-hadoop folder
docker-compose up # Spin up the environment and wait for the environment to spin up.
Running the Hadoop Job (Pseduo Distributed Mode)

Once the environment is up and running, we can run the Hadoop Job. We can do the following inside the namenode container of the environment. The namenode container will be mounting the /app folder of the environment from the host machine. The /app folder will contain the jars and the input files for the program.

docker exec -it namenode /bin/bash
hdfs dfs -mkdir /input
hdfs dfs -put /app/input/* /input
hdfs dfs -ls /input
hdfs dfs -cat /input/input.txt
hadoop jar /app/WordCount.jar com.basicsetup.hadoop.WordCount /input/input.txt /output
hdfs dfs -ls /output
hdfs dfs -cat /output/part-r-00000
hdfs dfs -get /output
exit

The output of the program will be stored in the /output folder in the container, to get the output from the container, we can use the following command.

docker cp namenode:/app/output .

Your local machine will have the /output folder with the output of the program.

Special Thanks for Matt ❤️ for the work on Docker Hadoop.