MapReduce real world example on e-commerce transactions data is described here using Python streaming. The example does not require Hadoop installation. However, if you have Hadoop already installed it will run just fine on it. Python programming language is used because it is easy to read and understand. A real world e-commerce transactions dataset from a UK based retailer is used. The best way to learn with this example is to use an Ubuntu machine with Python 2 or 3 installed on it.
- The dataset consists of real world e-commerece data from UK based retailer
- The dataset is provided by Kaggle
- It contains 5.42k records (which is not small)
- Our goal is to find out country wise total sales
- Mapper multiplies quantity and unit price
- Mapper emits key-value pair as country, sales
- Reducer sums-up all pairs for same country
- Final output is country, sales for all countries
Download:Link to Kaggle DatasetSource: The dataset has real-life transaction data from a UK retailer. Format: CSV Size: 43.4 MB (5,42,000 records) Columns:
In this MapReduce real world example, we calculate total sales for each country from given dataset.
Firstly, our data doesn’t have a Total column so it is to be computed using Quantity and UnitPrice columns as Total = Quantity * UnitPrice.
- Read the data
- Convert data into proper format
- Calculate total
- Print output as key-value pair CountryName:Total
- Read input from mapper
- Check for existing country key in the disctionary
- Add total to existing total value
- Print all key-value pairs
|United Arab Emirates||1877.08|
- Mapper picks-up a record and emits country and total for that record
- Mapper repeats this process for all 5.42k records
- Now, we have 5.42k key value pairs
- Reducer’s role is to combine these pairs until all keys are unique!
If you have questions, please feel free to comment below.