Prediction

# Prediction
### Applied Machine Learning with R <a href='https://therbootcamp.github.io'> The R Bootcamp @ AMLD </a> <a href='https://therbootcamp.github.io/AML_2021AMLD/'> </a>  <a href='https://therbootcamp.github.io'> </a>  <a href='mailto:therbootcamp@gmail.com'> </a>  <a href='https://www.linkedin.com/company/basel-r-bootcamp/'> </a>
### November 2021

---

<div class="my-footer">
 
 
 <img src="https://raw.githubusercontent.com/therbootcamp/therbootcamp.github.io/master/_sessions/_image/by-sa.png" height=14 style="vertical-align: middle"/>
 
 <a href="https://therbootcamp.github.io/">
 
 
 www.therbootcamp.com
 
 
 </a>
 <a href="https://therbootcamp.github.io/">
 
 Applied Machine Learning with R @ AMLD | November 2021
 
 </a>
 
 </div>

---

# Predict hold-out data

<ul>
 <li class="m1">Model performance must be evaluated as true prediction on an <high>unseen data set</high>.</li>
 <li class="m2">The unseen data set can be <high>naturally</high> occurring, e.g., using 2019 stock prizes to evaluate a model fit using 2018 stock prizes.</li>
 <li class="m3">More commonly unseen data is created by <high>splitting the available data</high> into a training set and a test set.</li>
</ul>

]

]

---

# Overfitting

<ul>
 <li class="m1">Occurs when a model <high>fits data too closely</high> and therefore <high>fails to reliably predict</high> future observations.</li> 
 <li class="m2">In other words, overfitting occurs when a model <high>'mistakes' random noise for a predictable signal</high>.</li> 
 <li class="m3">More <high>complex models</high> are more <high>prone to overfitting</high>.</li>
</ul>

]

]

---

# Overfitting

---

# Training

<ul>
 <li class="m1">Training a model means to <high>fit the model</high> to data by finding the parameter combination that <high>minizes some error function</high>, e.g., mean squared error (MSE).</li> 
</ul>

---

# Test

<ul style="margin-bottom:-20px">
 <li class="m1">To test a model means to <high>evaluate the prediction error</high> for a fitted model, i.e., for a <high>fixed parameter combination</high>.</li> 
</ul>

---

<h1>Regression</h1>

<high><h1>Decision Trees</h1></high>

<h1>Random Forests</h1>

---

# CART

<ul>
 <li class="m1">CART is short for <high>Classification and Regression Trees</high>, which are often just called <high>Decision trees</high>.</li> 
 <li class="m2">In <a href="https://en.wikipedia.org/wiki/Decision_tree">decision trees</a>, the criterion is modeled as a <high>sequence of logical TRUE or FALSE questions</high>.</li> 
</ul>

]

<img src="image/tree.png">

]

---

# Classificiation trees

<ul>
 <li class="m1">Classification trees (and regression trees) are created using a relatively simple <high>three-step algorithm</high>.</li> 
 <li class="m2">Algorithm:
 
 <ul class="level">
 <li>1 - <high>Split</high> nodes to maximize purity gain (e.g., Gini gain).</li> 
 <li>2 - <high>Repeat</high> until pre-defined threshold (e.g., <mono>minsplit</mono>) splits are no longer possible.</li> 
 <li>3 - <high>Prune</high> tree to reasonable size.</li>
 </ul>
 </li>
</ul>

]

<img src="image/tree.png">

]

---

# Node splitting

<ul>
 <li class="m1">Classification trees attempt to <high>minize node impurity</high> using, e.g., the <high>Gini coefficient</high>.</li>
</ul>

`$$\large Gini(S) = 1 - \sum_j^kp_j^2$$`

<ul>
 <li class="m2">Nodes are <high>split</high> using the variable and split value that <high>maximizes Gini gain</high>.</li>
</ul>

`$$Gini \; gain = Gini(S) - Gini(A,S)$$`

with

`$$Gini(A, S) = \sum \frac{n_i}{n}Gini(S_i)$$`

]

]

---

# Pruning trees

<ul>
 <li class="m1">Classification trees are <high>pruned</high> back such that every split has a purity gain of at least <high><mono>cp</mono></high>, with <mono>cp</mono> often set to <mono>.01</mono>.</li>
 <li class="m2">Minimize:</li>
</ul>

$$
\large
`\begin{split}
Loss = & Impurity\,+\\
&cp*(n\:terminal\:nodes)\\
\end{split}`
$$

]

]
---

# Regression trees

<ul>
 <li class="m1">Trees can also be used to perform regression tasks. Instead of impurity, regression trees attempt to <high>minimize within-node variance</high>.</li> 
</ul>

`$$\large SSE = \sum_{i \in S_1}(y_i - \bar{y}_1)^2+\sum_{i \in S_2}(y_i - \bar{y}_2)^2$$`

<ul>
 <li class="m2">Algorithm:
 
 <ul class="level">
 <li>1 - <high>Split</high> nodes to maximize homogeneity gain.</li> 
 <li>2 - <high>Repeat</high> until pre-defined threshold (e.g., <mono>minsplit</mono>) splits are no longe possible.</li> 
 <li>3 - <high>Prune</high> tree to reasonable size.</li>
 </ul>
 </li>
</ul>

]

]

---

<h1>Regression</h1>

<h1>Decision Trees</h1>

<high><h1>Random Forests</h1></high>

---

# Random Forest

<ul>
 <li class="m1">In <a href="https://en.wikipedia.org/wiki/Random_forest">Random Forest</a>, the criterion is modeled as the <high>aggregate prediction of a large number of decision trees</high> each based on different features.</li> 
 <li class="m2">Algorithm:
 
 <ul class="level">
 <li>1 - <high>Repeat</high> n times</li>
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 - <high>Resample</high> data 
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2 - <high>Grow</high> non-pruned decision tree 
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Each split <high>consider only m 
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;features</high>
 <li>2 - <high>Average</high> fitted values.</li> 
 </ul>
 </li>
</ul>

]

]

---

# Random Forest

<ul>
 <li class="m1">Random forests make use of important machine learning elements, <high>resampling</high> and <high>averaging</high> that together are also referred to as <high>bagging</high>.</li>
</ul>

<table style="cellspacing:0; cellpadding:0; border:none;">
 <col width="30%">
 <col width="70%">
<tr>
 <td bgcolor="white">
 Element
 </td>
 <td bgcolor="white">
 Description
 </td> 
</tr>
<tr>
 <td bgcolor="white">
 Resampling
 </td>
 <td bgcolor="white">
 Creates new data sets that vary in their composition thereby <high>deemphasizing idiosyncracies</high> of the available data. 
 </td> 
</tr>
<tr>
 <td bgcolor="white">
 Averaging
 </td>
 <td bgcolor="white">
 Combining predictions typically <high>evens out idiosyncracies</high> of the models created from single data sets. 
 </td> 
</tr>
</table>
]

]

---

<img src="https://www.tidymodels.org/images/tidymodels.png" width=240px> 
from <a href="https://www.tidymodels.org/packages/">tidymodels.org</a>

---

# Fitting <mono>tidymodels</mono>

<ul>
 <li class="m1">Split the data.</li> 
 <li class="m2">Change the model.</li> 
 <li class="m3">Predict.</li> 
</ul>

]

]

---

# Split data into training and test

<ul>
 <li class="m1">Use <mono>initial_split()</mono> to <high>split a dataset</high> into separate training and test datasets.</li>
</ul>

<table style="cellspacing:0; cellpadding:0; border:none;">
 <col width="30%">
 <col width="70%">
<tr>
 <td bgcolor="white">
 Argument
 </td>
 <td bgcolor="white">
 Description
 </td> 
</tr>
<tr>
 <td bgcolor="white">
 <mono>data</mono>
 </td>
 <td bgcolor="white">
 The dataset. Used to create a <high>balanced split</high>. 
 </td> 
</tr>
<tr>
 <td bgcolor="white">
 <mono>prop</mono>
 </td>
 <td bgcolor="white">
 The <high>proportion of data</high> going into the training set. Often .8, <mono>.75</mono> (default) or <mono>.5</mono>. 
 </td> 
</tr>
<tr>
 <td bgcolor="white">
 <mono>strata</mono>
 </td>
 <td bgcolor="white">
 The criterion or another stratification variable. Used to create a <high>balanced split</high>. 
 </td> 
</tr>
</table>

]

```r
# Get initial split
baselers_split <-
 initial_split(baselers,
 prop = .8,
 strata = income)

# Create training data
baselers_train <- training(baselers_split)

# Create test data
baselers_test <- testing(baselers_split)
```

]

---

# Fit to training

<ul> 
 <li class="m2">To fit training rather than test simply replace <highm>data</highm> in the <mono>recipe</mono>.</li>
</ul>

]

```r
# fit regression model to training

recipe <- 
 recipe(income ~ ., data = baselers_train) %>% 
 step_dummy(all_nominal_predictors())

lm_model <- 
 linear_reg() %>% 
 set_engine("lm") %>% 
 set_mode("regression")

lm_workflow <- 
 workflow() %>% 
 add_recipe(recipe) %>% 
 add_model(lm_model)

income_lm <- fit(lm_workflow,
 data = baselers_train)
```

]

---

# Fit Decision tree

<ul>
 <li class="m1">Fit <high>decision trees</high> in <mono>tidymodels</mono> using <mono>decision_tree()</mono> in the model definition.</li>
 <li class="m2">Set the engine to <mono>rpart</mono>.</li>
</ul>

]

```r
# fit decision tree to training

recipe <- 
 recipe(income ~ ., data = baselers_train) %>% 
 step_other(all_nominal_predictors(),
 threshold = 0.005)

dt_model <- 
 decision_tree() %>% 
 set_engine("rpart") %>% 
 set_mode("regression") # or classification

dt_workflow <- 
 workflow() %>% 
 add_recipe(recipe) %>% 
 add_model(dt_model)

income_dt <- fit(dt_workflow,
 data = baselers_train)
```

]

---

# Fit Random Forest

<ul>
 <li class="m1">Fit <high>random forests</high> in <mono>tidymodels</mono> using <mono>rand_forest()</mono> in the model definition.</li>
 <li class="m2">Set the engine to, e.g., <mono>ranger</mono>.</li>
</ul>

]

```r
# fit decision tree to training

recipe <- 
 recipe(income ~ ., data = baselers_train) %>% 
 step_other(all_nominal_predictors(),
 threshold = 0.005)

rf_model <- 
 rand_forest() %>% 
 set_engine("ranger") %>% 
 set_mode("regression") # or classification

rf_workflow <- 
 workflow() %>% 
 add_recipe(recipe) %>% 
 add_model(rf_model)

income_rf <- fit(rf_workflow,
 data = baselers_train)
```

]

---

# Random forest engines

<ul>
 <li class="m1">Show all available engines with <mono>show_engines("rand_forest")</mono>.</li>
</ul>

]

```r
# show possible engines for random forest
show_engines("rand_forest")
```

```
# A tibble: 6 x 2
 engine mode 
 <chr> <chr> 
1 ranger classification
2 ranger regression 
3 randomForest classification
4 randomForest regression 
5 spark classification
6 spark regression 
```

]

---

# <mono>predict(, newdata)</mono>

<ul>
 <li class="m1">To <high>test model predictions</high> with <mono>tidymodels</mono>, supply new data frame <mono>newdata</mono> to the <mono>predict()</mono> functionbaseler_train.</li>
</ul>

]

```r
# generate out-of-sample predictions
lm_pred <-
 income_lm %>% 
 predict(baselers_test) %>% 
 bind_cols(baselers_test %>% select(income))

metrics(lm_pred, 
        truth = income,
        estimate = .pred)
```

```
# A tibble: 3 x 3
 .metric .estimator .estimate
 <chr> <chr> <dbl>
1 rmse standard 1024. 
2 rsq standard 0.846
3 mae standard 821. 
```

]

---
class: middle, center

<h1><a href=https://therbootcamp.github.io/AML_2021AMLD/_sessions/Prediction/Prediction_practical.html>Practical</a></h1>