The objective of this post is to present an intuitive overview of features pandas DataFrame object. Minimum temperature data from 1901 to 2017 provided by data.gov.in is used as an example. Table of Contents What is pandas? Installing pandas Running this example on Kaggle Creating a DataFrame from Excel or CSV Glancing at the data […]
Configure Anaconda on Emacs
Perhaps my quest for an ultimate IDE ends with Emacs. My goal was to use Emacs as full-flagged Python IDE. This post describes how to setup Anaconda on Emacs. My Setup: OS: Trisquel 8.0 Emacs: GNU Emacs 25.3.2 Quick Key Guide (See full guide) : C-x = Ctrl + x M-x = Alt + x […]
Blockchain Applications Beyond Cryptocurrencies
Cryptocurrencies are changing the way people buy or sell anonymously. But they are plagued by hype, regulatory issues and possible abuse. As a result of so much buzz around the term, many think cryptocurrency and blockchain are same things. That is not true; Cryptocurrencies is just one of the many blockchain applications. Blockchain is the […]
Creating Your First Blockchain with Python
In this tutorial we’ll learn how to create a very basic Blockchain with Python. We will create a Blockchain with just 30 lines of code! The aim is to introduce you to Blockchain programming without getting into inessential details. You should already know fundamentals of Blockchain, if not then you may want to read this […]
A Beginner’s Guide to Bitcoin Mining : Learn Blockchain
The temptation to earn Bitcoins is understandable as Bitcoin is valued as world’s most expensive currency at current rates. But who mints Bitcoins? How are they distributed? and most importantly how can you earn them? these are the questions we will answer in this beginner’s guide to bitcoin mining. But let me give you a […]
Proof of Work Explained Simply : Learn Blockchain
Proof of Work (PoW) is a consensus algorithm used in the original Bitcoin implementation. In a Blockchain system new transactions are periodically added by packaging these transactions in a block. This block is then added to the Blockchain. Please read What is Blockchain Technology if you don’t already know. Background Users send each other digital […]
What is Blockchain Technology?
What is blockchain technology? is perhaps the most buzzing question in tech right now. A blockchain as the name suggests is a chain of blocks. Each block contains some information; a blockhain can store complete information about a financial transaction, a contract or a medical record. Important property of blockchain is that the data stored […]
MapReduce Real World Example in Python : Learn Data Science
MapReduce is a great approach to problem solving. It is very popular too, but MapReduce examples other than word-count are scarce on the web. This article describes MapReduce problem solving that is beyond word-count.
Arduino Serial Communication Basics : Learn Arduino
Arduino uses asynchronous serial communication to send-receive data to and from other devices. Arduino Uno supports serial communication via on-board UART port and Tx/Rx pins. Generally this transmission happens at 9600 bits per second which is termed as baud rate.
Arduino Pushbutton Example : Learn Arduino
Arduino pushbutton example shows how to read a pushbutton with Arduino Uno. Pushbuttons (also spelled push-buttons) are widely used in calculators, phones and appliances. It closes the circuit when pushed and keeps it close until it is pressed. As soon as you release the button the circuit is open again. Here, we are going to […]
Displaying Hello World on LCD : Learn Arduino
In this article we’ll learn displaying hello world on LCD; Interfacing 16×2 LCD with Arduino Uno and display some text on it. Writing Hello World pleases the gods of any new programming language that you want to learn. But in case of Arduino it is the LED blinking program that is generally written first. Any […]
LED Blinking on Arduino : Learn Arduino
LED Blinking on Arduino Uno should be your first Arduino project. Arduino Uno has an on-board or built-in LED, in this project we will see how to blink it. Parts you will need: An Arduino Uno Steps: Connect Arduino with your computer. Download, install and open Arduino IDE. Within IDE chooose File Menu > Examples […]
An Introduction to Arduino Platform : Learn Arduino
Arduino is an open-source hardware prototyping platform. It is widely used today in electronics projects because it is easy to learn, simple in design, well documented and cheaper. We call it platform because it is both hardware circuit as well as piece of software, the IDE. It also has its own programming language. All these […]
Reclaiming Your Privacy on Android
A major threat to your privacy emanates from your smart phone. These devices have become central medium of social interaction for everyone. Android is the only open source platform among the popular smart phone operating systems. But even with Android your privacy is under attack as large corporations have figured out that data is the […]
Merge Sort : Why Sorting Lists in Halves is Interesting?
Searching and sorting are two basic problems which occur in various more complex computing problems. Time complexity of sorting algorithms cannot get better than O (nlogn). Many popular algorithms have polynomial time complexity O (n2). Undoubtedly sorted lists are interesting because we can solve many problems using it: from binary search to Kruskal’s algorithm. This […]
Zeno’s Paradoxes
Zeno was a Greek philosopher who lived circa 490 to 430 BC. Zeno’s paradoxes paradoxes have puzzled us for more than 2500 years now; three of them are presented here for you to ponder upon. 1. Achilles and the tortoise In a race, the quickest runner can never overtake the slowest, since the pursuer must […]
How Credit Card Numbers Work?
How credit card numbers work? It is interesting to see how quickly payment gateway web page can check whether given card number is valid. There is a pattern that every card number adheres to. Such patterns can be matched using JavaScript. A credit card number is made-up of four different components: MII (digit 1), IIN […]
Apache Log Visualization with Matplotlib : Learn Data Science
This post discusses Apache log visualization with Matplotlib library. First, download the data file used in this example We will require numpy and matplotlib In [1]: import numpy as np import matplotlib.pyplot as plt numpy.loadtext() can directly load a text file in an array requests-idevji.txt contains only hour on which request was made, this is achieved […]
NLTK Example : Detecting Geographic Setting of Sherlock Holmes Stories
As a young adult nothing thrilled me more than Jeremy Brett’s performance as Sherlock Holmes. “You know my methods, apply them!” he would say. So let’s try to play Sherlock ourselves. We use Natural Language Tool Kit or NLTK to guess setting of a Sherlock story in terms of its geographic location. In this NLTK […]
Building a Movie Recommendation Service with Apache Spark
In this tutorial I’ll show you building a movie recommendation service with Apache Spark. Two users are alike if they rated a product similarly. For example, if Alice rated a book 3/5 and Bob also rated the same book 3.3/5 they are very much alike. Now if Bob buys another book and rates it 4/5 […]
GraphFrames PySpark Example : Learn Data Science
In this post, GraphFrames PySpark example is discussed with shortest path problem. GraphFrames is a Spark package that allows DataFrame-based graphs in Saprk. Spark version 1.6.2 is considered for all examples. Including the package with PySaprk shell : pyspark –packages graphframes:graphframes:0.1.0-spark1.6pyspark –packages graphframes:graphframes:0.1.0-spark1.6 Code: from pyspark import SparkContext from pyspark.sql import SQLContext sc = SparkContext […]
Logistic Regression with Spark : Learn Data Science
Logistic regression with Spark is achieved using MLlib. Logistic regression returns binary class labels that is “0” or “1”. In this example, we consider a data set that consists only one variable “study hours” and class label is whether the student passed (1) or not passed (0). from pyspark import SparkContext from pyspark import SparkContext […]
k-Means Clustering Spark Tutorial : Learn Data Science
k-Means clustering with Spark is easy to understand. MLlib comes bundled with k-Means implementation (KMeans) which can be imported from pyspark.mllib.clustering package. Here is a very simple example of clustering data with height and weight attributes. Arguments to KMeans.train: k is the number of desired clusters maxIterations is the maximum number of iterations to run. […]
Apriori Algorithm for Generating Frequent Itemsets
Apriori Algorithm is used in finding frequent itemsets. Identifying associations between items in a dataset of transactions can be useful in various data mining tasks. For example, a supermarket can make better shelf arrangement if they know which items are purchased together frequently. The challenge is that given a dataset D having T transactions each […]
Functional Programming in Python with Lambda Map Reduce and Filter
Functional programming in Python is possible with the use of lambda map reduce and filter functions. This article briefly describe use of each these functions. Lambda : Lambda specifies an anonymous function. It is used to declare a function with no name; When you want to use function only once. But why would you declare […]
Data Mining : Intuitive Partitioning of Data or 3-4-5 Rule
Intuitive partitioning or natural partitioning is used in data discretization. Data discretization is the process of converting continuous values of an attribute into categorical data or partitions or intervals. Discretization helps reducing data size by reducing number of possible values. Instead of storing every observation we can only store partition range in which each observation […]
Why Logarithms are Beautiful?
Binary logarithm or log2 n is the power to which the number 2 must be raised to obtain value n. Binary logarithm (and others) has numerous applications in computer science. Let’s take analysis of algorithms for example. All algorithms have a running time, also called time complexity of algorithms.
k-means Clustering Algorithm with Python : Learn Data Science
k-means clustering algorithm is used to group samples (items) in k clusters; k is specified by the user. The method works by calculating mean distance between cluster centroids and samples, hence the name k-means clustering. Euclidean distance is used as distance measure. See references for more information on the algorithm. This is a article describes k-means […]
Commonly Used HDFS Commands : Learn Data Science
Hadoop Distributed File System or HDFS is the underlying storage for all Hadoop applications. HDFS can be manipulated using APIs such as Java API or REST API but using HDFS shell is the most commonly used option. Below is a list of ten commonly used HDFS commands. 1. Invoking the file system: HDFS Shell supports […]
Python MapReduce with Hadoop Streaming in Hortonworks Sandbox
Hortonworks sandbox for Hadoop Data Platform (HDP) is a quick and easy personal desktop environment to get started on learning, developing, testing and trying out new features. It saves the user from installation and configuration of Hadoop and other tools. This article explains how to run Python MapReduce word count example using Hadoop Streaming.
MapReduce Streaming Example : Running Your First Job on Hadoop
Step by step tutorial to run Python Map Reduce code using Hadoop Streaming
Extracting Text from PDF Using Apache Tika – Learn NLP
Most NLP applications need to look beyond text and HTML documents as information is contained in PDF, ePub or other formats. Apache Tika is a toolkit that extracts meta data and text from documents. There is a REST based Python library for Tika.
Pangat : Finding CS Parallels in Non-Computing Environments
Computer science theories, metaphors and parallels can be found in non-computing environments if observed closely.
Improving fastText Classifier
This post describes how to improve fastText classifier using various techniques.
Tutorial: Text Classification With Python Using fastText
We start by training the classifier with training data. It contains questions from cooking.stackexchange.com and their associated tags on the site. Let’s build a classifier that automatically recognize a topic of the question and assign a label to it.
Map Reduce Word Count With Python : Learn Data Science
This tutorial jumps on to hands-on coding to help anyone get up and running with Map Reduce. No Hadoop installation is required.
A Note on Vernacular Chatbots
If bots can actually speak on phone having real conversation with the user in vernacular languages, people who cannot read and write can also shop online.
Artificial Intelligence and Human Values
We create technology. An algorithm cannot be neutral when it is designed by biased humans. Can an AI model be indiscriminating?
Flea Markets : Anonymous and Virtual
E-commerce fails to deliver an exploratory shopping experience where a customer does not know what is she looking for. Online stores are too organized where products are arranged in categories and sub-categories.
Getting Started with fastText : Learn NLP
fastText is a text representation and classification library from Facebook Research developed by FAIR lab. Classification of text documents is an important natural language processing (NLP) task. It is originally written in C++ but can be accessed using Python interface. It is massively fast. See references for two defining papers.
Programming Computers to Read Stories
Can a computer read the stories, the way humans do? Of course computers can read from files much faster and accurately but second part of the question is more important. When we read a story we understand it we read the feelings of the protagonist, challenge here is to make computers do the same.
Most Inspiring Satya Nadella Quotes
Nadella talks about culture, empathy, philosophy and trust apart from technology in his Hit Refresh: The Quest to Rediscover Microsoft’s Soul and Imagine a Better Future for Everyone. Six most powerful thoughts from the book that inspired me and will surely help you stay grounded.
Your Next Insurance Claim Surveyor Can Be A Drone
There is a great potential for drones in insurance industry across the verticals. Advancements in robotics and AI can bring exciting scenarios for insurance companies and consumers.
Hello World in 10 Different Programming Languages
This is my first blog post here on iDevji, a customary Hello World!