intro-to-data-science/intro_to_machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A hands-on Machine Learning Introduction in Python with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Machine Learning\n",
    "\n",
    "\n",
    "Machine learning is the process of **extracting knowledge from data** automatically.\n",
    "\n",
    "The goals usually include making predictions on new, unseen data or simply understanding given data better by finding patterns.\n",
    "\n",
    "Central to machine learning is the concept of **automating decision making** from data **without the user specifying explicit rules** how this decision should be made.\n",
    "\n",
    "<img src=\"raw/what_is_machine_learning.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples\n",
    "\n",
    "<img src=\"raw/examples.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Types of Machine Learning\n",
    "\n",
    "<img src=\"raw/3_types_of_machine_learning.png\" width=\"100%\">\n",
    "\n",
    "- **Supervised** (focus of this notebook): Each entry in the dataset comes with a \"label\". Examples are a list of emails where spam mail is already marked as such or a sample of handwritten digits. The goal is to use the historic data to make predictions.\n",
    "\n",
    "- **Unsupervised**: There is no desired output associated with a data entry. In a sense, one can think of unsupervised learning as a means of discovering labels from the data itself. A popular example is the clustering of customer data.\n",
    "\n",
    "- **Reinforcement**: Conceptually, this can be seen as \"learning by doing\". Some kind of \"reward function\" tells how good a predicted outcome is. For example, chess computers are typically programmed with this approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Types of Supervised Learning\n",
    "\n",
    "<img src=\"raw/classification_vs_regression.png\" width=\"100%\">\n",
    "\n",
    "- **In classification, the label is discrete**, such as \"spam\" or \"no spam\" for emails.\n",
    "Furthermore, labels are nominal (e.g., colors of something), not ordinal (e.g., T-shirt sizes in S, M, or L).\n",
    "\n",
    "\n",
    "- **In regression, the labels are continuous**. For example, given a person's age, education, and position, infer his/her salary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study: Iris flower classification\n",
    "\n",
    "<img src=\"raw/iris_data.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python for scientific computing\n",
    "\n",
    "Python itself does not come with any scientific algorithms implemented it. However, over time, many open source libraries emerged that are useful to build machine learning applications.\n",
    "\n",
    "Among the popular ones are numpy (numerical computations, linear algebra), pandas (data processing), matplotlib (visualisations), and scikit-learn (machine learning algorithms).\n",
    "\n",
    "First, import the libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following line is needed so that this Jupyter notebook creates the visiualizations in the notebook and not in a new window. This has nothing to do with Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standard Python can do basic arithmetic operations ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 1\n",
    "b = 2\n",
    "c = a + b\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and provides some simple **data structures**, such as a list of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l = [a, b, c, 4]\n",
    "l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy provides a data structure called an **n-dimensional array**. This may sound fancy at first but when used with only 1 or 2 dimensions, it basically represents vectors and matrices. Arrays allow for much faster computations as they use very low level functions modern computers provide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create an array, use the **array()** function from the imported **np** module and provide it with a list of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = np.array([1, 2, 3])\n",
    "v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A vector can be multiplied with a scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2 = v1 * 3\n",
    "v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a matrix, just use a list of (row) list of values instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1 = np.array([\n",
    "    [1, 2, 3],\n",
    "    [4, 5, 6],\n",
    "])\n",
    "m1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use numpy to multiply a matrix with a vector to obtain a new vector ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v3 = np.dot(m1, v1)\n",
    "v3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... or simply transpose it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rules from maths still apply and it makes a difference if a vector is multiplied from the left or the right by a matrix. The following operation will fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(v1, m1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to retrieve only a slice (= subset) of an array's data, we can \"index\" into it. For example, the first row of the matrix is ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1[0, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... while the second column is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To acces the lowest element in the right column, two indices can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1[1, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy also provides various other functions and constants, such as sinus or pi. To further illustrate the concept of **vectorization**, let us calculate the sinus curve over a range of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-3*np.pi, 3*np.pi, 100)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = np.sin(x)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With matplotlib's **plot()** function we can visualize the sinus curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us quickly generate some random data and draw a scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.random.normal(42, 3, 100)\n",
    "y = np.random.gamma(7, 1, 100)\n",
    "plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case study: importing the Iris data\n",
    "\n",
    "The sklearn library provides several sample datasets, among which is also the Iris dataset.\n",
    "\n",
    "As a table, the dataset would look like:\n",
    "<img src=\"raw/iris.png\" width=\"100%\">\n",
    "\n",
    "However, the data object imported from sklearn is organized slightly different. In particular, the so-called **features** are seperated from the **labels**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Python's **dir()** function we can inspect the data object, i.e. find out what **attributes** it has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(iris)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "iris.data provides us with a Numpy array, where the first dimension equals the number of observed flowers (**instances**) and the second dimension lists the various features of a flower."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find out what the four features are, we can list them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can also print the flowers' labels (a.k.a. targets):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three flower classes are encoded with integers. Let's show the corresponding names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study: Simple visualizations\n",
    "\n",
    "Since the data is four dimensional, we cannot visualize all features together. Instead, we can plot the distribution of the flower classes by a single feature using histograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_index = 2\n",
    "colors = ['blue', 'red', 'green']\n",
    "\n",
    "for label, color in zip(range(len(iris.target_names)), colors):\n",
    "    plt.hist(iris.data[iris.target==label, feature_index], \n",
    "             label=iris.target_names[label],\n",
    "             color=color)\n",
    "\n",
    "plt.xlabel(iris.feature_names[feature_index])\n",
    "plt.legend(loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, we can draw scatter plots of two features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_feature_index = 1\n",
    "second_feature_index = 0\n",
    "\n",
    "colors = ['blue', 'red', 'green']\n",
    "\n",
    "for label, color in zip(range(len(iris.target_names)), colors):\n",
    "    plt.scatter(iris.data[iris.target==label, first_feature_index], \n",
    "                iris.data[iris.target==label, second_feature_index],\n",
    "                label=iris.target_names[label],\n",
    "                c=color)\n",
    "\n",
    "plt.xlabel(iris.feature_names[first_feature_index])\n",
    "plt.ylabel(iris.feature_names[second_feature_index])\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the higher level library pandas, one can easily create a so-called **scatterplot matrix**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
    "pd.tools.plotting.scatter_matrix(iris_df, figsize=(8, 8));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept of Generalization\n",
    "\n",
    "The goal of a supervised machine learning model is to make predictions on new (i.e., previously unseen) data.\n",
    "\n",
    "In a real-world application, we are not interested in marking an already labeled email as spam or not. Instead, we want to make the user's life easier by automatically classifying new incoming mail.\n",
    "\n",
    "In order to get an idea of how good a model generalizes, a best practice is to split the available data into a training and a test set. Only the former is used to train the model. Then predictions are made on the test data and the predictions can be compared with the actual labels.\n",
    "\n",
    "Common splits are 75/25 or 60/40.\n",
    "\n",
    "<img src=\"raw/generalization.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study: Train/Test split for the Iris data\n",
    "\n",
    "It is common practice to refer to the feature matrix as X and the vector of labels as y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = iris.data, iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A naive splitting approach could be to use array slicing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = X[0:100, :], X[100:150, :], y[0:100], y[100:150]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, this would lead to unbalanced label distributions. For example, the test set would only be made up of flowers of the same type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.bincount(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn provides a function that not only randomizes the split but also ensures that the resulting label distribution is proportionate to the overall distribution (called **stratification**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, stratify=y)\n",
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.bincount(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A simple classification model: k-Nearest Neighbors\n",
    "\n",
    "To predict the label for any observation, just determine the k \"nearest\" observations in the training set (e.g., by Euclidean distance) and use a simple majority vote.\n",
    "\n",
    "<img src=\"raw/knn.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study: train and predict the Iris data\n",
    "\n",
    "sklearn provides a uniform interface for all its classification models. They all have a **fit()** and a **predict()** method that abstract away the actual machine learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "\n",
    "knn.fit(X_train, y_train)\n",
    "\n",
    "y_pred = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us list the labels predicted for the test set ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and compare them with the actual labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy can show us the indices where the predictions are wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.where(y_pred != y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can calculate the fraction of correctly predicted flowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(y_pred == y_test) / len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to mention that we can also \"predict\" the training set. Surprisingly, the model does not get the training set 100% correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = knn.predict(X_train)\n",
    "np.sum(y_train_pred == y_train) / len(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A visualization reveals that the misclassified flowers are right \"at the borderline\" between two neighboring clusters of flower classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_feature_index = 3\n",
    "second_feature_index = 0\n",
    "\n",
    "correct_idx = np.where(y_pred == y_test)[0]\n",
    "incorrect_idx = np.where(y_pred != y_test)[0]\n",
    "\n",
    "colors = [\"darkblue\", \"darkgreen\", \"gray\"]\n",
    "\n",
    "for n, color in enumerate(colors):\n",
    "    idx = np.where(y_test == n)[0]\n",
    "    plt.scatter(X_test[idx, first_feature_index], X_test[idx, second_feature_index], color=color,\n",
    "                label=iris.target_names[n])\n",
    "\n",
    "plt.scatter(X_test[incorrect_idx, first_feature_index], X_test[incorrect_idx, second_feature_index],\n",
    "            color=\"darkred\", label='misclassified')\n",
    "\n",
    "plt.xlabel('sepal width [cm]')\n",
    "plt.ylabel('petal length [cm]')\n",
    "plt.legend(loc='best')\n",
    "plt.title(\"Iris Classification results\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, the number of neighbors is to chosen before the model is trained. Therefore, it is possible to \"optimize\" it. This process is referred to as **hyper-parameter** tuning. For the Iris dataset this does not make much of a difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(1, 31):\n",
    "    knn = KNeighborsClassifier(n_neighbors=i)\n",
    "    knn.fit(X_train, y_train)\n",
    "    y_pred = knn.predict(X_test)\n",
    "    correct = np.sum(y_pred == y_test) / len(y_test)\n",
    "    print(i, correct)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Literature\n",
    "\n",
    "Depending on the programming language one chooses, the following books are recommended.\n",
    "\n",
    "- Python\n",
    "\n",
    "<img src=\"raw/python_general.png\">\n",
    "\n",
    "<img src=\"raw/python_ml.png\">\n",
    "\n",
    "- R\n",
    "\n",
    "<img src=\"raw/r.png\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}