MapReduce real world example on e-commerce transactions data is described here using Python streaming. The example does not require Hadoop installation. However, if you have Hadoop already installed it will run just fine on it. Python programming language is used because it is easy to read and understand. A real world e-commerce transactions dataset from a UK based retailer is used. The best way to learn with this example is to use an Ubuntu machine with Python 2 or 3 installed on it.

Outline

1. The dataset consists of real world e-commerece data from UK based retailer
2. The dataset is provided by Kaggle
3. It contains 5.42k records (which is not small)
4. Our goal is to find out country wise total sales
5. Mapper multiplies quantity and unit price
6. Mapper emits key-value pair as country, sales
7. Reducer sums-up all pairs for same country
8. Final output is country, sales for all countries

The Data

Download:Link to Kaggle DatasetSource: The dataset has real-life transaction data from a UK retailer. Format: CSV Size: 43.4 MB (5,42,000 records) Columns:

1. InvoiceNo
2. StockCode
3. Description
4. Quantity
5. InvoiceDate
6. UnitPrice
7. CustomerID
8. Country

The Problem

In this MapReduce real world example, we calculate total sales for each country from given dataset.

The Approach

Firstly, our data doesn’t have a Total column so it is to be computed using Quantity and UnitPrice columns as Total = Quantity * UnitPrice.

What Mapper Does

1. Read the data
2. Convert data into proper format
3. Calculate total
4. Print output as key-value pair CountryName:Total

What Reducer Does

1. Read input from mapper
2. Check for existing country key in the disctionary
3. Add total to existing total value
4. Print all key-value pairs

See this article on how to run this code

Python Code for Mapper (MapReduce Real World Example)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/env python
import sys

# Get input lines from stdin
for line in sys.stdin:
# Remove spaces from beginning and end of the line

line = line.strip()

# Split it into tokens

tokens = line.split(',')

#Get country, price and quantity values
try:
  country = tokens\[7\]
  price = float(tokens\[5\])
  qty = int(tokens\[3\])
  print '%s\\t%s' % (country, (price\*qty))
except ValueError: 
  pass

Python Code for Reducer (MapReduce Real World Example)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/usr/bin/env python
import sys

# Create a dictionary to map countries to totals
countrySales = {}

# Get input from stdin
for line in sys.stdin:
  #Remove spaces from beginning and end of the line
  line = line.strip()

# parse the input from mapper.py
country, total = line.split('\\t', 1)

# convert total (currently a string) to float
try:
  total = float(total)
except ValueError:
  pass

#update dictionary
try:
  countrySales\[country\] = countrySales\[country\] + total
except:
  countrySales\[country\] = total
 
# Write the tuples to stdout
for country in countrySales.keys():
  print '%s\\t%s'% (country, countrySales\[country\])

Output

Conclusions

• Mapper picks-up a record and emits country and total for that record
• Mapper repeats this process for all 5.42k records
• Now, we have 5.42k key value pairs
• Reducer’s role is to combine these pairs until all keys are unique!

If you have questions, please feel free to comment below.

Functional programming in Python is possible with the use of lambda map reduce and filter functions. This article briefly describe use of each these functions.

Lambda : Lambda specifies an anonymous function. It is used to declare a function with no name; When you want to use function only once. But why would you declare a function if you don’t want to reuse the code? Read on you’ll see. Syntax: lambda arg1, arg2 : expression

lambda x : x*x

This lambda expression with just one argument x which returns square of x.

Map : It takes two arguments, the first argument is name of a function and second argument is a sequence. map() applies function f to all elements in the sequence and returns a new sequence. Syntax: map (func, sequence)

1
2
3
4

list = [1, 2, 3]
map (lambda x : x*x, list)

#output: [1, 4, 9]

This code also demonstrates use of lambda. Instead writing a square function we substituted it with a lambda expression. map() applies it to all elements in the list and returns a new list with each element square of original element.

Reduce: reduce() continuously applies a function to a sequence and returns one value. In the following example we sum all elements in the original list. Syntax: reduce (func, sequence)

1

#output: 6

Filter: It filters all values in a sequence for which given function returns True. Syntax: filter (booleanFunc, sequence)

1
2

filter (lambda x : x%2, list)
#output: [1, 3]

Above example returns all odd integers in the list. Remember 2%2=0 is treated as boolean value False.