alexander/intro-to-data-science
1
1
Fork 0
Code Issues Pull requests Releases Activity
intro-to-data-science/02_a_first_example.ipynb

1186 lines
170 KiB
Text
Raw Normal View History

Add chapter-02 - A first Example
2021-05-25 01:33:04 +02:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Chapter 2: A first Example - Classifying Flowers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The purpose of this notebook is to look at a first example of a typical data science application, namely **statistical learning**, which is often referred to by its more well-known name **machine learning**. To do so, we look at a very popular example involving the classification of flowers. Albeit simplistic and almost boring in its kind, the example is a rather good one to look at from a beginner's point of view as it does not involve too many decision variables. That makes understanding technicalities and visualizing the data set a lot easier."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What is Machine Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's at first review a couple of generic definitions to get started.\n",
"\n",
"Machine learning is the process of **extracting knowledge from data** in an automated fashion.\n",
"\n",
"Typical use cases regard making predictions on new and unseen data or simply understanding a given dataset better by finding patterns.\n",
"\n",
"Central to machine learning is the idea of **automating** the **decision making** from data **without** the user specifying **explicit rules** how these decisions should be made.\n",
"\n",
"That is in direct opposition to what we learned in the \"Expressing Logic\" section in Chapter 0, where we learned how to implement decision criterions \"by hand\" with the `if` statement."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./static/what_is_machine_learning.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example Applications"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"static/examples.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Types of Machine Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Concete machine learning algorithms are commonly classified into three broad categories that may overlap as well:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"static/3_types_of_machine_learning.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- **Supervised** (focus of the example in this notebook): Each entry in the dataset comes with a **label**. Examples are a list of emails where spam mail is already marked as such or a sample of handwritten digits. The goal is to use the historic data to make predictions.\n",
"\n",
"- **Unsupervised**: There is no desired output associated with a data entry. In a sense, one can think of unsupervised learning as a means of discovering labels from the data itself. A popular example is the clustering of customer data.\n",
"\n",
"- **Reinforcement**: Conceptually, this can be seen as \"learning by doing\". Some kind of **reward function** tells how good a predicted outcome is. A rather recent and extremely popular example for his approach is the Alpha Go machine."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Types of Supervised Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Algorithms from the supervised learning category are often broken down further into classification and regression:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"static/classification_vs_regression.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- In **classification** tasks, the labels are *discrete*, such as \"spam\" or \"no spam\" for emails. Often, labels are nominal (e.g., colors of something), or ordinal (e.g., T-shirt sizes in S, M, or L).\n",
"- In **regression**, the labels are *continuous*. For example, given a person's age, education, and position, infer his/her salary."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example: Iris Flower Classification"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the example, we are given measurments regarding the size of various parts of the so-called Iris flower kind. A concrete flower always belongs to one of three distinct special Iris classes. This example application is about classifying a given flower into one of the three classes by only looking at the measurements."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"static/iris_data.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Importing the Data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `sklearn` library provides several sample datasets, among which is also the Iris dataset.\n",
"\n",
"In a tabular visualization, the dataset could be portrayed somewhat like this:\n",
"\n",
"<img src=\"static/iris.png\" width=\"50%\">\n",
"\n",
"However, the data object imported from `sklearn` is organized slightly different. In particular, the so-called **features** are separated from the **labels**."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_iris"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"iris = load_iris()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using Python's `dir()` function we can inspect the data object, i.e. find out what **attributes** it has."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['DESCR',\n",
" 'data',\n",
" 'feature_names',\n",
" 'filename',\n",
" 'frame',\n",
" 'target',\n",
" 'target_names']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dir(iris)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`iris.data` provides us with a `numpy.ndarray`, where the first dimension equals the number of observed flowers (i.e., the **instances**) and the second dimension lists the various features of a flower."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[5.1, 3.5, 1.4, 0.2],\n",
" [4.9, 3. , 1.4, 0.2],\n",
" [4.7, 3.2, 1.3, 0.2],\n",
" [4.6, 3.1, 1.5, 0.2],\n",
" [5. , 3.6, 1.4, 0.2],\n",
" [5.4, 3.9, 1.7, 0.4],\n",
" [4.6, 3.4, 1.4, 0.3],\n",
" [5. , 3.4, 1.5, 0.2],\n",
" [4.4, 2.9, 1.4, 0.2],\n",
" [4.9, 3.1, 1.5, 0.1],\n",
" [5.4, 3.7, 1.5, 0.2],\n",
" [4.8, 3.4, 1.6, 0.2],\n",
" [4.8, 3. , 1.4, 0.1],\n",
" [4.3, 3. , 1.1, 0.1],\n",
" [5.8, 4. , 1.2, 0.2],\n",
" [5.7, 4.4, 1.5, 0.4],\n",
" [5.4, 3.9, 1.3, 0.4],\n",
" [5.1, 3.5, 1.4, 0.3],\n",
" [5.7, 3.8, 1.7, 0.3],\n",
" [5.1, 3.8, 1.5, 0.3],\n",
" [5.4, 3.4, 1.7, 0.2],\n",
" [5.1, 3.7, 1.5, 0.4],\n",
" [4.6, 3.6, 1. , 0.2],\n",
" [5.1, 3.3, 1.7, 0.5],\n",
" [4.8, 3.4, 1.9, 0.2],\n",
" [5. , 3. , 1.6, 0.2],\n",
" [5. , 3.4, 1.6, 0.4],\n",
" [5.2, 3.5, 1.5, 0.2],\n",
" [5.2, 3.4, 1.4, 0.2],\n",
" [4.7, 3.2, 1.6, 0.2],\n",
" [4.8, 3.1, 1.6, 0.2],\n",
" [5.4, 3.4, 1.5, 0.4],\n",
" [5.2, 4.1, 1.5, 0.1],\n",
" [5.5, 4.2, 1.4, 0.2],\n",
" [4.9, 3.1, 1.5, 0.2],\n",
" [5. , 3.2, 1.2, 0.2],\n",
" [5.5, 3.5, 1.3, 0.2],\n",
" [4.9, 3.6, 1.4, 0.1],\n",
" [4.4, 3. , 1.3, 0.2],\n",
" [5.1, 3.4, 1.5, 0.2],\n",
" [5. , 3.5, 1.3, 0.3],\n",
" [4.5, 2.3, 1.3, 0.3],\n",
" [4.4, 3.2, 1.3, 0.2],\n",
" [5. , 3.5, 1.6, 0.6],\n",
" [5.1, 3.8, 1.9, 0.4],\n",
" [4.8, 3. , 1.4, 0.3],\n",
" [5.1, 3.8, 1.6, 0.2],\n",
" [4.6, 3.2, 1.4, 0.2],\n",
" [5.3, 3.7, 1.5, 0.2],\n",
" [5. , 3.3, 1.4, 0.2],\n",
" [7. , 3.2, 4.7, 1.4],\n",
" [6.4, 3.2, 4.5, 1.5],\n",
" [6.9, 3.1, 4.9, 1.5],\n",
" [5.5, 2.3, 4. , 1.3],\n",
" [6.5, 2.8, 4.6, 1.5],\n",
" [5.7, 2.8, 4.5, 1.3],\n",
" [6.3, 3.3, 4.7, 1.6],\n",
" [4.9, 2.4, 3.3, 1. ],\n",
" [6.6, 2.9, 4.6, 1.3],\n",
" [5.2, 2.7, 3.9, 1.4],\n",
" [5. , 2. , 3.5, 1. ],\n",
" [5.9, 3. , 4.2, 1.5],\n",
" [6. , 2.2, 4. , 1. ],\n",
" [6.1, 2.9, 4.7, 1.4],\n",
" [5.6, 2.9, 3.6, 1.3],\n",
" [6.7, 3.1, 4.4, 1.4],\n",
" [5.6, 3. , 4.5, 1.5],\n",
" [5.8, 2.7, 4.1, 1. ],\n",
" [6.2, 2.2, 4.5, 1.5],\n",
" [5.6, 2.5, 3.9, 1.1],\n",
" [5.9, 3.2, 4.8, 1.8],\n",
" [6.1, 2.8, 4. , 1.3],\n",
" [6.3, 2.5, 4.9, 1.5],\n",
" [6.1, 2.8, 4.7, 1.2],\n",
" [6.4, 2.9, 4.3, 1.3],\n",
" [6.6, 3. , 4.4, 1.4],\n",
" [6.8, 2.8, 4.8, 1.4],\n",
" [6.7, 3. , 5. , 1.7],\n",
" [6. , 2.9, 4.5, 1.5],\n",
" [5.7, 2.6, 3.5, 1. ],\n",
" [5.5, 2.4, 3.8, 1.1],\n",
" [5.5, 2.4, 3.7, 1. ],\n",
" [5.8, 2.7, 3.9, 1.2],\n",
" [6. , 2.7, 5.1, 1.6],\n",
" [5.4, 3. , 4.5, 1.5],\n",
" [6. , 3.4, 4.5, 1.6],\n",
" [6.7, 3.1, 4.7, 1.5],\n",
" [6.3, 2.3, 4.4, 1.3],\n",
" [5.6, 3. , 4.1, 1.3],\n",
" [5.5, 2.5, 4. , 1.3],\n",
" [5.5, 2.6, 4.4, 1.2],\n",
" [6.1, 3. , 4.6, 1.4],\n",
" [5.8, 2.6, 4. , 1.2],\n",
" [5. , 2.3, 3.3, 1. ],\n",
" [5.6, 2.7, 4.2, 1.3],\n",
" [5.7, 3. , 4.2, 1.2],\n",
" [5.7, 2.9, 4.2, 1.3],\n",
" [6.2, 2.9, 4.3, 1.3],\n",
" [5.1, 2.5, 3. , 1.1],\n",
" [5.7, 2.8, 4.1, 1.3],\n",
" [6.3, 3.3, 6. , 2.5],\n",
" [5.8, 2.7, 5.1, 1.9],\n",
" [7.1, 3. , 5.9, 2.1],\n",
" [6.3, 2.9, 5.6, 1.8],\n",
" [6.5, 3. , 5.8, 2.2],\n",
" [7.6, 3. , 6.6, 2.1],\n",
" [4.9, 2.5, 4.5, 1.7],\n",
" [7.3, 2.9, 6.3, 1.8],\n",
" [6.7, 2.5, 5.8, 1.8],\n",
" [7.2, 3.6, 6.1, 2.5],\n",
" [6.5, 3.2, 5.1, 2. ],\n",
" [6.4, 2.7, 5.3, 1.9],\n",
" [6.8, 3. , 5.5, 2.1],\n",
" [5.7, 2.5, 5. , 2. ],\n",
" [5.8, 2.8, 5.1, 2.4],\n",
" [6.4, 3.2, 5.3, 2.3],\n",
" [6.5, 3. , 5.5, 1.8],\n",
" [7.7, 3.8, 6.7, 2.2],\n",
" [7.7, 2.6, 6.9, 2.3],\n",
" [6. , 2.2, 5. , 1.5],\n",
" [6.9, 3.2, 5.7, 2.3],\n",
" [5.6, 2.8, 4.9, 2. ],\n",
" [7.7, 2.8, 6.7, 2. ],\n",
" [6.3, 2.7, 4.9, 1.8],\n",
" [6.7, 3.3, 5.7, 2.1],\n",
" [7.2, 3.2, 6. , 1.8],\n",
" [6.2, 2.8, 4.8, 1.8],\n",
" [6.1, 3. , 4.9, 1.8],\n",
" [6.4, 2.8, 5.6, 2.1],\n",
" [7.2, 3. , 5.8, 1.6],\n",
" [7.4, 2.8, 6.1, 1.9],\n",
" [7.9, 3.8, 6.4, 2. ],\n",
" [6.4, 2.8, 5.6, 2.2],\n",
" [6.3, 2.8, 5.1, 1.5],\n",
" [6.1, 2.6, 5.6, 1.4],\n",
" [7.7, 3. , 6.1, 2.3],\n",
" [6.3, 3.4, 5.6, 2.4],\n",
" [6.4, 3.1, 5.5, 1.8],\n",
" [6. , 3. , 4.8, 1.8],\n",
" [6.9, 3.1, 5.4, 2.1],\n",
" [6.7, 3.1, 5.6, 2.4],\n",
" [6.9, 3.1, 5.1, 2.3],\n",
" [5.8, 2.7, 5.1, 1.9],\n",
" [6.8, 3.2, 5.9, 2.3],\n",
" [6.7, 3.3, 5.7, 2.5],\n",
" [6.7, 3. , 5.2, 2.3],\n",
" [6.3, 2.5, 5. , 1.9],\n",
" [6.5, 3. , 5.2, 2. ],\n",
" [6.2, 3.4, 5.4, 2.3],\n",
" [5.9, 3. , 5.1, 1.8]])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris.data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find out what the four features are, we can list them:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['sepal length (cm)',\n",
" 'sepal width (cm)',\n",
" 'petal length (cm)',\n",
" 'petal width (cm)']"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris.feature_names"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, we can also print the flowers' labels (a.k.a. targets):"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
" 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
" 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
" 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
" 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris.target"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The three flower classes are encoded with integers. Let's show the corresponding names:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['setosa', 'versicolor', 'virginica'], dtype='<U10')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"iris.target_names"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Simple Visualizations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the data is four-dimensional, we cannot visualize all features together. Instead, we can plot the distribution of the flower classes by a single feature using histograms."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEGCAYAAAB8Ys7jAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZfklEQVR4nO3de3RV9bnu8e8LiQaUCkrqiYQI3S0oN4lEqbqViiBsZWAdQmmPWi5tqSIXa6vFnh7BtqftHqdDtuMcsEVFrCKXHerYuw53BdylHnetkqTBQBBRjBpAw6UiIHfe88ea0CRAsrLWTFZ+4fmMsUbWmmte3pmEh5nfmvOd5u6IiEh42mW6ABERSY0CXEQkUApwEZFAKcBFRAKlABcRCVRWS26sa9eu3qNHj5bcpIhI8EpLS3e4e2796S0a4D169KCkpKQlNykiEjwze/9U0zWEIiISKAW4iEigFOAiIoFq0TFwEWlbDh8+THV1NQcOHMh0KW1CTk4O+fn5ZGdnJzW/AlxEUlZdXU2nTp3o0aMHZpbpcoLm7uzcuZPq6mp69uyZ1DIaQhGRlB04cIALLrhA4R0DM+OCCy5o0l8zCnARSYvCOz5N/V4qwEVEAqUAF5HYmMX7iNvChQvZunVr/CvOkDMywDPxiyMimacAFxFpRfbt28fNN9/MZZddRr9+/Vi6dCmlpaUMGTKEQYMGMWLECLZt20ZxcTElJSXcfvvtDBw4kP379/Pyyy9TWFhI//79mTRpEgcPHgRg5syZ9OnThwEDBvCDH/wAgN///vcMHjyYwsJChg0bxscff5zJ3U5w9xZ7DBo0yFsDaPwhIo2rrKys8zqZf1tNeSSjuLjYv/3tb594/cknn/hVV13lNTU17u6+ZMkSnzhxoru7DxkyxNesWePu7vv37/f8/HzfuHGju7vfeeedPmfOHN+xY4f36tXLjx075u7uf/vb39zdfdeuXSemPf74437fffel9k1rRP3vqbs7UOKnyFSdBy4iQevfvz/f//73+eEPf8ioUaPo0qUL69atY/jw4QAcPXqUvLy8k5bbuHEjPXv2pFevXgCMHz+euXPnMnXqVHJycvjWt77FqFGjGDVqFJA4533cuHFs27aNQ4cOJX2udnPSEIqIBK1Xr16UlZXRv39/fvzjH7N8+XL69u1LeXk55eXlVFRUsGLFiqTXl5WVxRtvvMGYMWN44YUXGDlyJADTpk1j6tSpVFRU8Jvf/KZVXH2qABeRoG3dupWOHTtyxx13cP/99/P666+zfft2XnvtNSBxuf/69esB6NSpE3v27AGgd+/eVFVV8c477wDwzDPPMGTIEPbu3cvu3bu56aabmDNnDmvXrgVg9+7ddOvWDYCnn366pXfzlDSEIiKxcW/5bVZUVHD//ffTrl07srOzeeyxx8jKymL69Ons3r2bI0eOcO+999K3b18mTJjAXXfdRYcOHXjttdd46qmnGDt2LEeOHOGKK67grrvuYteuXdxyyy0cOHAAd+eRRx4BYPbs2YwdO5YuXbowdOhQ3nvvvZbf2XrMW/A7XlRU5K3hhg7JnCaYiV9EkdBs2LCBSy+9NNNltCmn+p6aWam7F9WfV0MoIiKBUoCLiARKAS4iEigFuIhIoBoNcDNbYGY1Zrau1rT/bWZvmdmbZva8mXVu1ipFROQkyRyBLwRG1pu2Eujn7gOAt4EHY65LREQa0WiAu/srwK5601a4+5Ho5V+A/GaoTURC09r7ySbhoYceYtWqVU1ebvXq1Scuu28pcVzIMwlYGsN6RERaxPFmUO3anXwM+5Of/KRFajhy5AhZWelFcFpLm9n/AI4AixqYZzIwGaCgoCCdzYlIE9jDTT+C9VlhXcE2c+ZMunfvzj333AMkrpY899xzcXeWLVvGwYMHufXWW3n44YepqqpixIgRDB48mNLSUl588UVmzZpFSUkJZsakSZP43ve+x4QJExg1ahRjxoxhzZo1zJgxg3379nH22Wfz8ssvk52dzd13301JSQlZWVk88sgjXH/99XXq2rVrF5MmTWLz5s107NiR+fPnM2DAAGbPns27777L5s2bKSgoYPHixWntf8pnoZjZBGAUcLs3cDmnu8939yJ3L8rNzU11cyIiJxk3bhzLli078XrZsmXk5uayadMm3njjDcrLyyktLeWVV14BYNOmTUyZMoX169ezY8cOtmzZwrp166ioqGDixIl11n3o0CHGjRvHo48+ytq1a1m1ahUdOnRg7ty5mBkVFRUsXryY8ePHn9TYatasWRQWFvLmm2/y85//nG9+85sn3qusrGTVqlVphzekeARuZiOBB4Ah7v5Z2lWIiKSgsLCQmpoatm7dyvbt2+nSpcuJ7oOFhYUA7N27l02bNlFQUMDFF1/Ml7/8ZQC+8IUvsHnzZqZNm8bNN9/MjTfeWGfdGzduJC8vjyuuuAKAz33ucwC8+uqrTJs2DYBLLrmEiy++mLfffrvOsq+++irLly8HYOjQoezcuZNPP/0UgNGjR9OhQ4dY9r/RADezxcBXgK5mVg3MInHWydnAyuguyn9x97tiqUhEpAnGjh1LcXExH330EePGjeP999/nwQcf5Lvf/W6d+aqqqjjnnHNOvO7SpQtr167lpZde4te//jXLli1jwYIFzV5v7RrSlcxZKN9w9zx3z3b3fHd/0t2/6O7d3X1g9FB4i0hGjBs3jiVLllBcXMzYsWMZMWIECxYsYO/evQBs2bKFmpqak5bbsWMHx44d47bbbuNnP/sZZWVldd7v3bs327ZtY82aNQDs2bOHI0eOcO2117JoUeJjv7fffpsPPviA3r1711m29jyrV6+ma9euJ47g46R2siISnwy08ezbty979uyhW7du5OXlkZeXx4YNG7jqqqsAOPfcc3n22Wdp3759neW2bNnCxIkTOXbsGAC/+MUv6rx/1llnsXTpUqZNm8b+/fvp0KEDq1atYsqUKdx9993079+frKwsFi5cyNlnn11n2dmzZzNp0iQGDBhAx44dm61/uNrJnobayUroWuIsFLWTjZ/ayYqInAEU4CIigVKAi4gESgEuIhIoBbiISKAU4CIigdJ54CISm1ROXWxIKs21tm7dyvTp0ykuLm7ScjfddBPPPfccnTt3Pu08Dz30ENdddx3Dhg1rcl3NQQEuIm3KRRdddMrwbqx964svvtjouluq1WyyNIQiIsGaOXMmc+fOPfF69uzZ/OpXv6Jfv34ALFy4kNGjRzN06FBuuOEGPvvsM772ta/Rp08fbr31VgYPHszxiwt79OjBjh07qKqq4tJLL+U73/kOffv25cYbb2T//v0ATJgw4cR/DmvWrOHqq6/msssu48orr2TPnj1UVVVx7bXXcvnll3P55Zfz5z//uVn3XwEuIsE6VTvZwYMH15mnrKyM4uJi/vSnPzFv3jy6dOlCZWUlP/3pTyktLT3lejdt2sQ999zD+vXr6dy584nOgsedrtXs5z//eVauXElZWRlLly5l+vTp8e90LRpCEZFgnaqdbPfu3evMM3z4cM4//3wg0eZ1xowZAPTr148BAwaccr09e/Zk4MCBAAwaNIiqqqo675+u1ey+ffuYOnUq5eXltG/f/qQ2s3FTgItI0Oq3k60vlfattZtTtW/f/sQQSmPmzJnDhRdeyNq1azl27Bg5OTlN3nZTaAhFRIJWv51sQ6655poTQy6VlZVUVFSktM3TtZrdvXs3eXl5tGvXjmeeeYajR4+mtP5k6QhcRGKTiXtq1m8nW3+4o7YpU6Ywfvx4+vTpwyWXXELfvn0577zzmrzNhlrN3nbbbfz2t79l5MiRsd684VTUTvY01E5WQqd2sic7evQohw8fJicnh3fffZdhw4axceNGzjrrrEyXdkJT2snqCFxEzhifffYZ119/PYcPH8bdmTdvXqsK76ZSgIvIGaNTp060hlGAuOhDTBFJS0sOw7Z1Tf1eKsBFJGU5OTns3LlTIR4Dd2fnzp1NOvVQQygikrL8/Hyqq6vZvn17pktpE3JycsjPz096fgW4iKQsOzubnj17ZrqMM1ajQyhmtsDMasxsXa1p55vZSjP
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"feature_index = 2\n",
"colors = ['blue', 'red', 'green']\n",
"\n",
"for label, color in zip(range(len(iris.target_names)), colors):\n",
" plt.hist(\n",
" iris.data[iris.target==label, feature_index], \n",
" label=iris.target_names[label],\n",
" color=color,\n",
" )\n",
"\n",
"plt.xlabel(iris.feature_names[feature_index])\n",
"plt.legend(loc='upper right')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Also, we can draw scatter plots of two features."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEGCAYAAACO8lkDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzn0lEQVR4nO3de3xcdZn48c+TSUobLqVCd60tTcoqRdq09MKlVuSSoixW1B/U4kYoF4kkAltZUNyubUWj66oUdWkxKoImy2WL7gKCSruichFoa9vQQgUhKW3RXpBQaAtN+/z+OCftzGRmMudkzpkzZ5736zWvzJw5l+85J5lv5jzPeb6iqhhjjDHJKordAGOMMdFjnYMxxpg+rHMwxhjTh3UOxhhj+rDOwRhjTB+VxW6AV0cffbTW1tYWuxnGGFNSVq5cuV1Vh+c7f8l1DrW1taxYsaLYzTDGmJIiIl1e5g/0spKIfE5E1onIMyJyp4gMTnv/EBG5W0ReEJEnRaQ2yPYYY4zJT2Cdg4iMBK4BpqrqeCABXJg22+XA31T13cAi4BtBtccYY0z+gg5IVwJDRKQSqAa2pL3/UeAO9/lSoF5EJOA2GWOM6UdgMQdV3Swi3wI2AruBX6vqr9NmGwm87M7fIyLdwFHA9uSZRKQRaAQYPXp0n23t3buXTZs2sWfPnoLvRzkaPHgwo0aNoqqqqthNMcYUSWCdg4gMw/lmMAZ4DfhvEfmUqrZ5XZeqtgKtAFOnTu1TDGrTpk0cfvjh1NbWYl88BkZV2bFjB5s2bWLMmDHFbo4xpkiCvKw0A3hJVbep6l7gZ8D70ubZDBwD4F56Ggrs8LqhPXv2cNRRR1nHUAAiwlFHHWXfwkyg2jvaqb25loovV1B7cy3tHe3FbpJJE2TnsBE4VUSq3ThCPfBs2jz3AXPc5xcA/6c+y8Rax1A4dixNkNo72mm8v5Gu7i4Upau7i8b7G62DiJjAOgdVfRInyLwK6HC31SoiN4rIee5sPwKOEpEXgGuBG4JqjzEmGuYtn8euvbtSpu3au4t5y+cVqUUmk0BvglPVBcCCtMnzk97fA8wKsg1RdPvtt/PBD36Qd73rXcVuijGh29i90dN0UxxWW6kIbr/9drZsSc/qNaY8jB7aN+Mw13RTHGXZObS3Q20tVFQ4P9sLcKnzzTff5MMf/jATJ05k/Pjx3H333axcuZLTTz+dKVOm8KEPfYhXXnmFpUuXsmLFChoaGjjxxBPZvXs3y5cvZ9KkSdTV1XHZZZfx1ltvAXDDDTdwwgknMGHCBK677joA7r//fk455RQmTZrEjBkz+Otf/zrwxhsTopb6FqqrqlOmVVdV01LfUqQWmYxUtaQeU6ZM0XTr16/vMy2btjbV6mpVOPiornamD8TSpUv105/+9IHXr732mk6bNk23bt2qqqp33XWXXnrppaqqevrpp+vTTz+tqqq7d+/WUaNG6YYNG1RV9aKLLtJFixbp9u3b9bjjjtP9+/erqurf/vY3VVV99dVXD0z7wQ9+oNdee+3AGp6Fl2NqjFdta9u0ZlGNykLRmkU12rZ2gH+Apl/ACvXwWVtyhfcGat482JUaC2PXLmd6Q4P/9dbV1fEv//IvfOELX2DmzJkMGzaMZ555hrPPPhuAffv2MWLEiD7LbdiwgTFjxnDccccBMGfOHG655RauuuoqBg8ezOWXX87MmTOZOXMm4NzTMXv2bF555RXefvttuxfBlKSGugYa6gbwB2cCV3aXlTZmiXllm56v4447jlWrVlFXV8e//du/ce+99zJu3DhWr17N6tWr6ejo4Ne/Tr9BPLvKykqeeuopLrjgAh544AHOOeccAK6++mquuuoqOjo6+P73v2/3IxhjAlF2nUOG6hs5p+dry5YtVFdX86lPfYrrr7+eJ598km3btvHEE08ATomPdevWAXD44Yezc+dOAMaOHUtnZycvvPACAD/96U85/fTTeeONN+ju7ubcc89l0aJFrFmzBoDu7m5GjhwJwB133JHeDGOMKYiyu6zU0gKNjamXlqqrnekD0dHRwfXXX09FRQVVVVUsWbKEyspKrrnmGrq7u+np6WHu3LmMGzeOSy65hCuvvJIhQ4bwxBNP8OMf/5hZs2bR09PDSSedxJVXXsmrr77KRz/6Ufbs2YOqctNNNwGwcOFCZs2axbBhwzjrrLN46aWXBtZwY4zJxEuAIgqPgQakVZ3gc02Nqojzc6DB6Dh6evXTgQcMwwhKWuAzmuy8hA8LSPevoWFgwee427FrBzt276Cr2xk4qre8AVCwIGJvCYXeO2VLdRvGOzsvpaHsYg6mf5t3bkbTSlwVurxBGCUUrExDNNl5KQ3WOZg+3t73dsbphSxvEEYJBSvTEE12XkqDdQ6mj0GJQRmnF7K8QRglFKxMQzTZeSkN1jmYPkYePrJP2e5ClzcIo4SClWmIJjsvpcE6B9PHUdVHcdSQo6gZWoMg1AytofUjrQUNFjbUNdD6kdbAtzFn4hwSkgAgIQnmTJxjQc8iC+Pcm4GT9MBj1E2dOlVXrFiRMu3ZZ5/lve99b5FaFIz58+fzgQ98gBkzZnha7pFHHuFb3/oWDzzwwIC2H4djmp4VA85/qPZBZMqRiKxU1an5zm/fHIpIVdm/f3/G92688UbPHYMfPT09gW+jWCwrxhj/yrNzKHDN7htuuIFbbrnlwOuFCxfyrW99i29+85ucdNJJTJgwgQULnDGPOjs7GTt2LBdffDHjx4/n5Zdf5pJLLmH8+PHU1dWxaNEiAC655BKWLl0KwNNPP8373vc+Jk6cyMknn8zOnTvZs2cPl156KXV1dUyaNInf/OY3fdr16quv8rGPfYwJEyZw6qmnsnbt2gPtu+iii5g+fToXXXTRgPY9yiwrxhj/yq9zaG936md0dTkVu7u6nNcD6CBmz57NPffcc+D1Pffcw/Dhw3n++ed56qmnWL16NStXruR3v/sdAM8//zzNzc2sW7eO7du3s3nzZp555hk6Ojq49NJLU9b99ttvM3v2bL7zne+wZs0ali1bxpAhQ7jlllsQETo6OrjzzjuZM2dOnyJ8CxYsYNKkSaxdu5avfe1rXHzxxQfeW79+PcuWLePOO+/0vd9RZ1kxxvgXWOcgImNFZHXS43URmZs2zxki0p00z/wsqyucXDW7fZo0aRJbt25ly5YtrFmzhmHDhh2owjpp0iQmT57Mc889x/PPPw9ATU0Np556KgDHHnssL774IldffTW//OUvOeKII1LWvWHDBkaMGMFJJ50EwBFHHEFlZSWPPvoon/rUpwA4/vjjqamp4U9/+lPKso8++uiBbwZnnXUWO3bs4PXXXwfgvPPOY8iQIb73uRRYVowx/gVWPkNVNwAnAohIAtgM/DzDrL9X1ZlBtaOPgGp2z5o1i6VLl/KXv/yF2bNn09XVxRe/+EU+85nPpMzX2dnJoYceeuD1sGHDWLNmDb/61a+49dZbueeee7jtttsG1JZ8JLchrnqDzvOWz2Nj90ZGDx1NS32LBaONyUNYl5XqgT+raldI28suoJrds2fP5q677mLp0qXMmjWLD33oQ9x222288cYbAGzevJmtW7f2WW779u3s37+f888/n69+9ausWrUq5f2xY8fyyiuv8PTTTwOwc+dOenp6OO2002h3L4X96U9/YuPGjYwdOzZl2eR5HnnkEY4++ug+30zirqGugc65nexfsJ/OuZ3WMRiTp7AK710IZLu4PU1E1gBbgOtUdV36DCLSCDQCjB7owAsB1eweN24cO3fuZOTIkYwYMYIRI0bw7LPPMm3aNAAOO+ww2traSCQSKctt3ryZSy+99EDW0te//vWU9wcNGsTdd9/N1Vdfze7duxkyZAjLli2jubmZpqYm6urqqKys5Pbbb+eQQw5JWXbhwoVcdtllTJgwgerqahv/wRiTt8DvcxCRQTgf/ONU9a9p7x0B7FfVN0TkXOA7qvqeXOsryH0O7e1OjGHjRucbQ0uLlWlNE4f7HIwxB3m9zyGMbw7/CKxK7xgAVPX1pOcPishiETlaVbcH2iKr2W2MMTmFEXP4JFkuKYnIO8U
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"first_feature_index = 1\n",
"second_feature_index = 0\n",
"\n",
"colors = ['blue', 'red', 'green']\n",
"\n",
"for label, color in zip(range(len(iris.target_names)), colors):\n",
" plt.scatter(\n",
" iris.data[iris.target==label, first_feature_index], \n",
" iris.data[iris.target==label, second_feature_index],\n",
" label=iris.target_names[label],\n",
" c=color,\n",
" )\n",
"\n",
"plt.xlabel(iris.feature_names[first_feature_index])\n",
"plt.ylabel(iris.feature_names[second_feature_index])\n",
"plt.legend(loc='upper left')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the higher level library `pandas`, one can easily create a so-called **scatterplot matrix**."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeYAAAHjCAYAAAD2Xrx8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAEAAElEQVR4nOz9d3gkyXnYj3+qJ+dBzmFzzuFu73YvkjySd0fymEmRlEiRFC1ZsmVLlvSzv1awJdGWZVmSbQUrWFZgFMkjeQx3PJIXeGnDbc67yHEGwOTY3fX7owezAHYADDKw25/nwQPMoLrr7a6uqq633iCklJiYmJiYmJisDpSVFsDExMTExMTkFubEbGJiYmJisoowJ2YTExMTE5NVhDkxm5iYmJiYrCLMidnExMTExGQVYU7MJiYmJiYmqwjrSgsAUF1dLdvb21daDJN50tnZidl+axez/dYuZtutbU6ePBmWUtZM/X5VTMzt7e2cOHFipcUwmScHDx4022+BnOgcpS+S5siGKmp9zmWt22y/1cHlwRiXB+Lsbg6wvsZb1jFm2y0+qZzKC1dC2K0KD26uwWpZOsWyEKKr1PerYmI2MbmbGUvmeOlaGIC8Jnn/geYVlshkuZFS8uyFITRdMhzP8NkyJ2aTxefN7giXB+MA1Aec7GgMLLsM5h6zickK47Jb8DqMd+Qan2OFpTFZCYQQVHuNtjefgZVl/P5blFttstyYK2YTkxXGabPwsXvbiKbz1PnNQflu5f0HmgknstSaE/OKsrnOR+UROzZFIeC2rYgM5sQ8hfZff6ascp2ff3yJJTG5m3DZLbjslpUWw2QFsVsVGoOulRbDBFZspTyOqco2MTExMTFZRZgrZpM1RSav8bVTfcQyeR7f1UBLpfu2MpFUjq+d6kMCT+1rotJjv61MXyTNt8/043FYed/+5hVdrWq65Ftn+umLpHlkay3bGvwrJovJ0hPL5Pn6qT7yms679zbNuKd8smuM//nDa/icNn7ryR2c6YtwsT/GvtYg922oXkap7x6GYxmePt2Pw6bw1L4mfM7b1dk3Qwm+f2GIKo+d9+xrIqfq/Pa3LjAcz/LpY+s4tuk2D6g5Ya6YTdYUvWNphmIZ0jmNSwOxkmVuhBJE03li6TzXhxMly1zqj5HKaYTiWXrGUksp8qyMpXJ0hJPkVJ1zvdEVlcVk6ekIJRlN5ohnVK4OxWcs+6PLw6RyGkOxDCe7RjjTEyGn6pzuiSyPsHchV4biJLIqI4kcXSOlx4bz/TEyeY2+iDEeXeiP0j2aIpPXeP7S8IJlMFfMy0C5+9Zg7l3PRnOFi2qfg3gmP+3Kcn21lze7IwBsqPGULLO1wcfFgRg+p5XmipXd16tw22mtdNMfSbOjyVwt3+m0V3sIuGzkNZ1NtdO7ReU1nSMbKjnbF8HvtHGgrQpFUbjQH2N3U3D5BL7L2Fzn42J/DLtVoa3K0MjpuiSr6kXN2vYGPz2jKao8dur8TircdhqDLsKJLA9tWdhqGcyJ2WSN4bRZ+Pi9bTOWqfDY+fSx9TOWyao6YAx+qi4XTb75YFEE7zN9l+8aAi4bnzq6bsYyo8kcXzreg6rp/O57dhW3bB7dVsej2+qWQ8y7ljq/k597cEPxs65Lvnyih4FohnvXV3FkQxUba71srN1YLGO3KvzRh/YumgymKtvkrqQjlETTJcmsxlA0s9LimJhMonfMUIuquqR7dGW3Wu52kjmVgcIYcT1UemtssZlxxSyEOAJ8DDgGNABp4DzwDPAPUkpzQ8xkTbKnJchgLIPXYaWtqrS628RkpdhU6+PqUIK8prOj0dzeWEl8Thv7WoN0hpPcu65yWeqcdmIWQnwX6AeeBn4XGAacwGbgYeBpIcR/l1J+czkENTFZTGp8Dj42i0rcxGSlcNktZmjWVcRDW2phy/LVN9OK+eNSyvCU7xLAqcLPHwohTHt9ExMTExOTRWTaPeapk7IQwi+EqBz/KVXGxGS10BdJ07vCblAmJtMxEE3TY+4drwrSOY1rQ3HSOW2lRSkyq1W2EOLngN8GMsC4+aoEZjZ7NTFZIW6EEnzzdD8AT+xuYFOdb4UlMjG5RfdIin8+1QvA23bUrUj2IpNbfPVUL+F4dlVtb5XjLvUrwE5zdWyyVkhk1OLf8aw6Q0kTk+UnMeGZTGZXzyrtbmV8vEisorGinIn5BmDqXEzWDDubAiSzKhLY3WSsRsKJLF0jSTbV+fCXCLFnYrJcbK33EcvkyWs6e1uCpHMaFwdiNAVd1AecKy3eHc/VoTg5VWd7gx9FETyxu4HLg3G2NawezVo5E/NvAK8IIV4HsuNfSil/acmkMjFZABZFcN/GW3aJui756sneQhjP+KpRV5ncnSiK4N71VcXPX3+zl85wCptF8Olj63HazCxjS8WNUIJnzg4AkNN09rdW0FLpLhlzfyUpZ2L+C+CHwDlAX1pxTEyWBinHf69slC8Tk6ncejZv/W2yNEzs/6t5LChnYrZJKf/NkksyD8zcySbloCiC9x1ooiOUZGu9Eawhq2oc7xjD47Cwr7Wi5HHxTJ6TXWM0BFxsqZ+/mqtnNMX14QQ7mvzU+kxV5d1EJJXjzZ4IrZVuNtR4uTQQYyiawaIILBbBofZKHttRz4X+GE0VLjMn9wI53xdlJJnjUHsFbvvt09uGGi/NFS7SeY2dy2x0J6XkZNcYOVXn0LpKbJbpA2+WMzF/VwjxWeBbTFZljy5cVBOT5aHW55w0Kb5+c5STXWOAkUSivfr26F8/uhLixnCC0yJCrc9BRYn0kbOh6ZKnT/eR1yRdI0l+5v6ZYySb3Fk8e2GIvkiasz1R3re/ie+dHyScyDKazLG5zofNonCovZLDyxRR6k5mMJrhuYtDgJEe9rEd9beVuTIUp3csDcDpngj3TNhSWGquDMV56ZphQ21RxIx1lzMxf6Tw+zcmfGe6S5msacZXJkIw7Z6eq/C9VRHYrPMLK68Uzp/XVHM1dBcy3uZ2q4LTbsFmEcbzVFgtucz95EXDYVWwKAJNl9PeV7ft1pRXakW9lMyl7lklk1Kar/gmdxxb632c641S4bFTO02i+oe31NBa6abaa8frmF8nFkLwoUMt9I6lWVdiVW5yZ/PYjno21SWIZ1R+cj3M/RuqcTusOKwKEsxnYhGp8Nj58KEWIuk8G2tKp9NsrXLzgYPN5FSd9dOUKcWZngidI0kOr6ukITC/NLFzqXvWZYAQ4heEEMEJnyuEED9fxnFvF0L8uPAzIIR4Txmym5gsCyc6x4im83SGk9wMJ0uWsVoUttT7qPKWnrjLxee0sa3Bb1rb3oXYrQqba328emOEm6EkJ7rG2FLvo73aY07KS0Ct38nmOh+KIqYt01zhntOknMyq/PDyMDdDSX50ObQg+cqtuxz93GeklJHxD1LKMeAzsx0kpfyelPIhKeVDQDfwgzLqMjFZFqoLk61VEVS4Tb9mk6VDUQSVBfuEat/c7RRMVha7VcHvMsaIau/ytF85+jmLEELIgm25EMIClC2dEGI9MCSlXJ5EliYmZbCrOUCd34HDZiHgKj0x94ymeOFqiIaAk0e21iLE9G/hJncGoXiWH1wawu+08diOOqwzWM7OhQ8ebCGUyFI3zbaJyey8dC1E50iK+zZUsWEOK96FYrMo/NQ9rYwmc9T7l8eropyn7nvAl4QQjwohHgW+UPiuXN4LfH0+wpmYLCW1fue0kzLAGx2jhOJZzvZGCSWy05YzuXM41T3GYDTD1aE43YuYZMJuVWgKuhZtor/biGfynOgcIxzP8sqNkWWv32mz0Bh0zagiX0zKeUp+DSPAyL8o/DwP/Ls51PEkcFvOZiHEZ4UQJ4QQJ0KhhentTUyWgvZqIxpQhds24wRucufQVuVGCPA4LNSYq9tVg9tupa6wWl1XdefvzZdjla0Df174mRNCiHogJ6W87RVHSvmXwF8CHDx4cPWGYDG5K8jlND71/07QG0nxsXva+PSx9ficNoQAr9OKVTFXOncDW+v9tFa6sVkUbBaFH14e4tJAnEPtlWxt8PH1U32ouuQ9exuLRoE/ujzMxYEYB9sqir6
"text/plain": [
"<Figure size 576x576 with 16 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
"\n",
"pd.plotting.scatter_matrix(iris_df, figsize=(8, 8));"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Concept of Generalization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The goal of a supervised machine learning model is to make predictions on *new* (i.e., previously unseen) data.\n",
"\n",
"In a real-world application, we are not interested in marking an already labeled email as spam or not. Instead, we want to make the user's life easier by automatically classifying new incoming mail.\n",
"\n",
"In order to get an idea of how good a model **generalizes**, a best practice is to *split* the available data into a **training** and a **test** set. Only the former is used to train the model. Then, predictions are made on the test data and the predictions can be compared with the actual labels.\n",
"\n",
"Common splits are 75/25 or 60/40."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./static/generalization.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Train/Test Split for the Iris data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is common practice to refer to the feature matrix as `X` and the vector of labels as `y`."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"X, y = iris.data, iris.target"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A naive splitting approach could be to use array slicing."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = X[0:100, :], X[100:150, :], y[0:100], y[100:150]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, this would lead to unbalanced label distributions. For example, the test set would only be made up of flowers of the same type."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
" 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
" 2, 2, 2, 2, 2, 2])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_test"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0, 0, 50])"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.bincount(y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`sklearn` provides a function that not only randomizes the split but also ensures that the resulting label distribution is proportionate to the overall distribution, a concept called **stratification**."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 0, 2, 2, 1, 0, 1, 1, 1, 0, 0, 2, 2, 0, 0, 2, 0, 1, 0, 0, 2, 2,\n",
" 0, 2, 1, 0, 2, 2, 2, 1, 0, 1, 1, 2, 0, 1, 2, 1, 2, 1, 2, 1, 0, 1,\n",
" 0])"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, test_size=0.3, stratify=y)\n",
"\n",
"y_test"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([15, 15, 15])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.bincount(y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A simple Classification Model: k-Nearest Neighbors"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To predict the label for any observation, just determine the k \"nearest\" observations in the training set (e.g., by Euclidean distance) and use a simple majority vote."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"./static/knn.png\" width=\"60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Training and Predicting with the Iris data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`sklearn` provides a uniform interface for all its classification models. They all have a `.fit()` and a `.predict()` method that abstract away the actual machine learning algorithm."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.neighbors import KNeighborsClassifier"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"knn = KNeighborsClassifier(n_neighbors=5)\n",
"\n",
"knn.fit(X_train, y_train)\n",
"\n",
"y_pred = knn.predict(X_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us list the labels predicted for the test set ..."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 0, 2, 2, 1, 0, 1, 1, 1, 0, 0, 2, 1, 0, 0, 2, 0, 2, 0, 0, 2, 2,\n",
" 0, 2, 1, 0, 2, 1, 2, 1, 0, 1, 1, 2, 0, 1, 2, 1, 2, 1, 2, 1, 0, 1,\n",
" 0])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_pred"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"... and compare them with the actual labels."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 0, 2, 2, 1, 0, 1, 1, 1, 0, 0, 2, 2, 0, 0, 2, 0, 1, 0, 0, 2, 2,\n",
" 0, 2, 1, 0, 2, 2, 2, 1, 0, 1, 1, 2, 0, 1, 2, 1, 2, 1, 2, 1, 0, 1,\n",
" 0])"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_test"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`numpy` shows us the indices where the predictions are wrong."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(array([12, 17, 27]),)"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.where(y_pred != y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, we can calculate the fraction of correctly predicted flowers."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.9333333333333333"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.sum(y_pred == y_test) / len(y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is important to mention that we can also \"predict\" the training set. Somehow surprisingly, the model does not get the training set 100% correct."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.9523809523809523"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_train_pred = knn.predict(X_train)\n",
"\n",
"np.sum(y_train_pred == y_train) / len(y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A visualization reveals that the misclassified flowers are right \"at the borderline\" between two neighboring clusters of flower classes."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1FUlEQVR4nO3de3xU1dXA/d8KASEIEQQVwSRYFQok4Sog1BviFbC+woM2orFqVLzU13rBxhfRxzy1FcW7GK2FalQUxar1AmJtpYpyEQhEUMSEmxUETYCAEFjvH+ckTsJMMpnkzHV9P5/5MLPPOfvsnIRZs8/es7aoKsYYY0xdSZFugDHGmOhkAcIYY4xfFiCMMcb4ZQHCGGOMXxYgjDHG+GUBwhhjjF8WIEyzEZEcEZnrQb25IrKguev1qf8dEbnM5/W9IvK9iPxXRNJEZKeItPDgvDtF5NjmrjdcRORDEbky0u0w3rEAYYImIqUickag7apapKpnhlj3WSLybxHZISJbReRfIjIm9NYGT1XPUdWZbjvSgN8DvVT1KFVdr6qHqur+ppzD35upW++6ptQbLbwO4iYyLECYZiEiyU04dizwCvA3oBtwJDAZGN08rWuUNGCbqm6JwLmbXVN+L8ZYgDAhcT8x/kdEponINmCK76dIcUwTkS0iUiEixSLSx089AjwI/K+qPqOq5ap6QFX/papXBTj3wyKywa13iYj8ymfbiSKy2N32nYg86Ja3FpHnRWSbiPwoIotE5Eh324cicqXbO5oHHO3e/pkhIhkiotVvtCLSUUT+KiKbReQHEXndLe8gIm+5vZ8f3Ofd3G0FwK+Ax9x6H3PLVUSOc5+nisjf3OPLROROEUnyudYLRGSqW/c3InJOPb+bUhG5XURWALtEJFlEhojIx+7PvlxETq3zu1zn9t6+EZEct3yKiDzvs1+ta+FT/ktgOjDU/fl+dMvPFZESt95NInJLoDab6GQBwjTFYGAdzif+gjrbzgROBk4AUoH/Abb5qaMHcAwwuxHnXQT0BToCLwCviEhrd9vDwMOq2h74BfCyW36Z245jgMOBa4DdvpWq6vvAOcBm9/ZPrp9zPwekAL2BI4BpbnkS8FcgHacXsht4zK03H/gIuN6t93o/9T7qtu9Y4BTgUuByn+2DgTVAJ+DPwF/c4BrIxcB5wGE4v59/APfiXLNbgFdFpLOItAUeAc5R1XbAScCyeuo9iKp+gXM9P3F/vsPcTX8Brnbr7QN80Jh6TeRZgDBNsVlVH1XVKlXdXWfbPqAd0BMQVf1CVb/1U8fh7r/+tvmlqs+r6jb3vA8Ah+AEmurzHicinVR1p6ou9Ck/HDhOVfer6hJVrQj2nAAi0gUngFyjqj+o6j5V/Zfbpm2q+qqqVqrqDpyAeUqQ9bYALgLuUNUdqloKPABM8NmtTFWfdsdCZgJdcN74A3lEVTe4v5dLgLdV9W23dzYPWAyc6+57AOgjIm1U9VtVXRXcFWnQPqCXiLR3r9fSZqrXhIkFCNMUGwJtUNUPcD5BPw5sEZFCEWnvZ9fqXkWXYE8qIreIyBciUu7ezkjF+WQNcAVOr2W1extplFv+HPAe8JJ7e+jPItIy2HO6jgG2q+oPftqUIiJPubeHKoB/A4dJcLOfOgEtgTKfsjKgq8/r/1Y/UdVK9+mh9dTp+7tJB8a5t5d+dK/ZcKCLqu4CxuP0AL4VkX+ISM8g2hyMC3GCUJk4kw6GNlO9JkwsQJimqDcVsKo+oqoDgF44b9q3+tltDc6b2YXBnNAdb7gN55ZVB/d2Rjkg7jm/UtWLcW7//AmYLSJt3U/7d6tqL5zbKKNwbuM0xgago4gc5mfb73F6MYPd21snVzfZ/be+a/U9zqftdJ+yNGBTI9vny/d8G4DnVPUwn0dbVb0PQFXfU9WROEF6NfC0e9wunNtp1Y4K8ny49S5S1fNxfhev8/PtPhMjLEAYT4jIIBEZ7H5K3wXswbmVUYs6+eZvBv4/EblcRNqLSJKIDBeRQj9VtwOqgK1AsohMBmp6JiJyiYh0VtUDwI9u8QEROU1EMt1P9BU4b8gHtac+7i2yd4An3EHpliJSHQja4Yw7/CgiHYG76hz+Hc74gr969+O8eRaISDsRSXevyfP+9g/B88BocaYStxBnwP5UEekmIkeKyPnuWMRPwE5+vi7LgJPF+S5IKnBHPef4DugmIq0ARKSVON+LSVXVfTjXvFHX20SeBQjjlfY4n0R/wLldsg2439+Oqjob5zbHb4HNOG829wJ/97P7e8C7wJduvXuofTvlbGCViOzEGbC+yL0PfxTOQHgF8AXwL5zbTo01ASe4rAa2ADe55Q8BbXB6AwvdNvp6GBjrzkJ6xE+9N+AE0nXAApzB92dDaN9BVHUDcD7wB5zAugGnN5fkPm7Gue7bccZNrnWPmwfMAlYAS4C36jnNB8Aq4L8i8r1bNgEodW+5XQPkNMfPY8JHbMEgY4wx/lgPwhhjjF8WIIwxxvhlAcIYY4xfFiCMMcb4FXOJvDp16qQZGRmRboYxxsSUJUuWfK+qnRtzTMwFiIyMDBYvXhzpZhhjTEwRkbKG96rNbjEZY4zxywKEMcYYvyxAGGOM8SvmxiD82bdvHxs3bmTPnj2RbooBWrduTbdu3WjZsrHJUo0x0SQuAsTGjRtp164dGRkZ1L+GivGaqrJt2zY2btxI9+7dI90cY0wTxEWA2LNnjwWHKCEiHH744WzdujXSTTEmKhUXFzN//nzKy8tJTU1lxIgRZGZmRrpZfsVFgAAsOEQR+10Y419xcTFvvvkm+/btA6C8vJw333wTICqDhA1SG2NMmMyfP78mOFTbt28f8+fPj1CL6mcBIgJmzJjB5s2bI90MY0yYlZeXN6o80ixARIAFCGMSU2pqaqPKIy0hA0RRUQkZGYUkJU0lI6OQoqKSJte5a9cuzjvvPLKzs+nTpw+zZs1iyZIlnHLKKQwYMICzzjqLb7/9ltmzZ7N48WJycnLo27cvu3fvZv78+fTr14/MzEx++9vf8tNPPwEwadIkevXqRVZWFrfccgsAb775JoMHD6Zfv36cccYZfPfdd01uuzEmPEaMGHHQ9O+WLVsyYsSICLWoAaoaU48BAwZoXSUlJQeVBfL886s0JWWawv01j5SUafr886uCrsOf2bNn65VXXlnz+scff9ShQ4fqli1bVFX1pZde0ssvv1xVVU855RRdtGiRqqru3r1bu3XrpmvWrFFV1QkTJui0adP0+++/1xNOOEEPHDigqqo//PCDqqpu3769puzpp5/Wm2++uUnt9kpjfifGJJIVK1botGnTdMqUKTpt2jRdsWJFWM4LLNZGvt/GzSymYOXnL6CysqpWWWVlFfn5C8jJ6RVyvZmZmfz+97/n9ttvZ9SoUXTo0IGVK1cycuRIAPbv30+XLl0OOm7NmjV0796dE044AYDLLruMxx9/nOuvv57WrVtzxRVXMGrUKEaNGgU43/kYP3483377LXv37rXvGhgTYzIzM6NyxpI/CXeLaf36ikaVB+uEE05g6dKlZGZmcuedd/Lqq6/Su3dvli1bxrJlyyguLmbu3LlB15ecnMxnn33G2LFjeeuttzj77LMBuOGGG7j++uspLi7mqaeesm+PG2M8k3ABIi2tfaPKg7V582ZSUlK45JJLuPXWW/n000/ZunUrn3zyCeBMZVu1ahUA7dq1Y8eOHQD06NGD0tJS1q5dC8Bzzz3HKaecws6dOykvL+fcc89l2rRpLF++HHBmO3Tt2hWAmTNnNqnNxhhTn4S7xVRQMJy8vLm1bjOlpCRTUDC8SfUWFxdz6623kpSURMuWLXnyySdJTk7mxhtvpLy8nKqqKm666SZ69+5Nbm4u11xzDW3atOGTTz7hr3/9K+PGjaOqqopBgwZxzTXXsH37ds4//3z27NmDqvLggw8CMGXKFMaNG0eHDh04/fTT+eabb5rUbmOMCUScsYvYMXDgQK27YNAXX3zBL3/5y6DrKCoqIT9/AevXV5CW1p6CguFNGn8wB2vs78QY4y0RWaKqAxtzjGc9CBHpAczyKToWmKyqD/nscyrwd6D
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"first_feature_index = 3\n",
"second_feature_index = 0\n",
"\n",
"correct_idx = np.where(y_pred == y_test)[0]\n",
"incorrect_idx = np.where(y_pred != y_test)[0]\n",
"\n",
"colors = [\"darkblue\", \"darkgreen\", \"gray\"]\n",
"\n",
"for n, color in enumerate(colors):\n",
" idx = np.where(y_test == n)[0]\n",
" plt.scatter(\n",
" X_test[idx, first_feature_index],\n",
" X_test[idx, second_feature_index],\n",
" color=color,\n",
" label=iris.target_names[n],\n",
" )\n",
"\n",
"plt.scatter(\n",
" X_test[incorrect_idx, first_feature_index],\n",
" X_test[incorrect_idx, second_feature_index],\n",
" color=\"darkred\",\n",
" label='misclassified',\n",
")\n",
"\n",
"plt.xlabel('sepal width [cm]')\n",
"plt.ylabel('petal length [cm]')\n",
"plt.legend(loc='best')\n",
"plt.title(\"Iris Classification results\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In practice, the number of neighbors must be chosen before the model is trained. Therefore, it is possible to \"optimize\" it. This process is referred to as **hyper-parameter tuning**. For the Iris dataset this does not make much of a difference."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 0.9555555555555556\n",
"2 0.9333333333333333\n",
"3 0.9333333333333333\n",
"4 0.9333333333333333\n",
"5 0.9333333333333333\n",
"6 0.9333333333333333\n",
"7 0.9111111111111111\n",
"8 0.9111111111111111\n",
"9 0.9111111111111111\n",
"10 0.9333333333333333\n",
"11 0.9555555555555556\n",
"12 0.9555555555555556\n",
"13 0.9333333333333333\n",
"14 0.9111111111111111\n",
"15 0.9333333333333333\n",
"16 0.9111111111111111\n",
"17 0.9333333333333333\n",
"18 0.9111111111111111\n",
"19 0.9333333333333333\n",
"20 0.9333333333333333\n",
"21 0.9333333333333333\n",
"22 0.9333333333333333\n",
"23 0.9111111111111111\n",
"24 0.9555555555555556\n",
"25 0.9111111111111111\n",
"26 0.9333333333333333\n",
"27 0.9111111111111111\n",
"28 0.9333333333333333\n",
"29 0.9555555555555556\n",
"30 0.9111111111111111\n"
]
}
],
"source": [
"for i in range(1, 31):\n",
" knn = KNeighborsClassifier(n_neighbors=i)\n",
" knn.fit(X_train, y_train)\n",
" y_pred = knn.predict(X_test)\n",
" correct = np.sum(y_pred == y_test) / len(y_test)\n",
" print(i, correct)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Further Resources on Machine Learning"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Depending on the programming language one chooses, the following books are recommended:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Python Machine Learning](https://www.amazon.de/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1787125939/ref=sr_1_1?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&keywords=python+machine+learning&qid=1575545025&sr=8-1) by Sebastian Raschka\n",
"\n",
"<img src=\"static/python_ml_book.png\">"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [An Introduction to Statistical Learning](http://faculty.marshall.usc.edu/gareth-james/ISL/)\n",
"\n",
"<img src=\"static/r_ml_book.png\">"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.9"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": false,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in a new issue Copy permalink