Fitting

# Fitting
### Applied Machine Learning with R <a href='https://therbootcamp.github.io'> The R Bootcamp @ AMLD </a> <a href='https://therbootcamp.github.io/AML_2020AMLD/'> </a>  <a href='https://therbootcamp.github.io'> </a>  <a href='mailto:therbootcamp@gmail.com'> </a>  <a href='https://www.linkedin.com/company/basel-r-bootcamp/'> </a>
### January 2020

---

<div class="my-footer">
 
 
 <img src="https://raw.githubusercontent.com/therbootcamp/therbootcamp.github.io/master/_sessions/_image/by-sa.png" height=14 style="vertical-align: middle"/>
 
 <a href="https://therbootcamp.github.io/">
 
 
 www.therbootcamp.com
 
 
 </a>
 <a href="https://therbootcamp.github.io/">
 
 Applied Machine Learning with R @ AMLD | January 2020
 
 </a>
 
 </div>

---

# Fitting

<ul>
 <li class="m1">Models are actually <high>families of models</high>, with every parameter combination specifying a different model.</li>
 <li class="m2">To fit a model means to <high>identify</high> from the family of models <high>the specific model that fits the data best</high>.</li>
</ul>

]

<img src="image/curvefits.png" height=480px> 
adapted from <a href="https://www.explainxkcd.com/wiki/index.php/2048:_Curve-Fitting">explainxkcd.com</a>

]

---

# Which of these models is better? Why?

---

# Which of these models is better? Why?

---

# Loss function

<ul>
 <li class="m1">Possible <high>the most important concept</high> in statistics and machine learning.</li>
 <li class="m2">The loss function defines some <high>summary of the errors committed by the model</high>.</li>
</ul>

`$$\Large Loss = f(Error)$$`

<table style="cellspacing:0; cellpadding:0; border:none;">
<tr>
 <td>
 Purpose
 </td>
 <td>
 Description
 </td>
</tr>
<tr>
 <td bgcolor="white">
 Fitting
 </td>
 <td bgcolor="white">
 Find parameters that minimize loss function.
 </td>
</tr>
<tr>
 <td>
 Evaluation
 </td>
 <td>
 Calculate loss function for fitted model.
 </td>
</tr>
</table>

]

]

---

<high><h1>Regression</h1></high>

<h1>Decision Trees</h1>

<h1>Random Forests</h1>

---

# Regression

In [regression](https://en.wikipedia.org/wiki/Regression_analysis), the criterion `$Y$` is modeled as the <high>sum</high> of <high>features</high> `$X_1, X_2, ...$` <high>times weights</high> `$\beta_1, \beta_2, ...$` plus `$\beta_0$` the so-called the intercept.

`$$\large \hat{Y} =  \beta_{0} + \beta_{1} \times X_1 + \beta_{2} \times X2 + ...$$`

The weight `$\beta_{i}$` indiciates the <high>amount of change</high> in `$\hat{Y}$` for a change of 1 in `$X_{i}$`.

Ceteris paribus, the <high>more extreme</high> `$\beta_{i}$`, the <high>more important</high> `$X_{i}$` for the prediction of `$Y$` (Note: the scale of `$X_{i}$` matters too!).

If `$\beta_{i} = 0$`, then `$X_{i}$` <high>does not help</high> predicting `$Y$`

]

]

---

# Regression loss

<ul style="margin-bottom:-20px">
 <li class="m1">Mean Squared Error (MSE)
 
 <ul class="level">
 <li>Average <high>squared distance</high> between predictions and true values.</li>
 </ul>
 </li>
</ul>

`$$MSE = \frac{1}{n}\sum_{i \in 1,...,n}(Y_{i} - \hat{Y}_{i})^{2}$$`

<ul>
 <li class="m2">Mean Absolute Error (MAE)
 
 <ul class="level">
 <li>Average <high>absolute distance</high> between predictions and true values.</li>
 </ul>
 </li>
</ul>

$$ MAE = \frac{1}{n}\sum_{i \in 1,...,n} \lvert Y_{i} - \hat{Y}_{i} \rvert$$

]

]

---

# Fitting

<ul style="margin-bottom:-20px">
 <li class="m1">Analytically
 
 <ul class="level">
 <li>In rare cases, the parameters can be <high>directly calculated</high>, e.g., using the normal equation.</li>
 </ul>
 </li>
</ul>

`$$\boldsymbol \beta = (\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y$$`

<ul>
 <li class="m2">Numerically
 
 <ul class="level">
 <li>In most cases, parameters need to be found using a <high>directed trial and error</high>, e.g., gradient descent.</li>
 </ul>
 </li>
</ul>

`$$\boldsymbol \beta_{n+1} = \boldsymbol \beta_{n}+\gamma \nabla F(\boldsymbol \beta_{n})$$`

]

<img src="image/gradient.png" height=420px> 
adapted from <a href="https://me.me/i/machine-learning-gradient-descent-machine-learning-machine-learning-behind-the-ea8fe9fc64054eda89232d7ffc9ba60e">me.me</a>

]

---

# Fitting

`$$\boldsymbol \beta = (\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y$$`

`$$\boldsymbol \beta_{n+1} = \boldsymbol \beta_{n}+\gamma \nabla F(\boldsymbol \beta_{n})$$`

]

<br2>

<img src="image/gradient1.gif" height=250px> 
adapted from <a href="https://dunglai.github.io/2017/12/21/gradient-descent/
">dunglai.github.io</a> 
<img src="image/gradient2.gif" height=250px> 
adapted from <a href="https://dunglai.github.io/2017/12/21/gradient-descent/
">dunglai.github.io</a>

]

---

# 2 types of supervised problems

<ul style="margin-bottom:-20px">
 <li class="m1">Regression
 
 <ul class="level">
 <li>Regression problems involve the <high>prediction of a quantitative feature</high>.</li>
 <li>E.g., predicting the cholesterol level as a function of age</high>.</li>
 </ul>
 </li> 
 <li class="m2">Classification
 
 <ul class="level">
 <li>Classification problems involve the <high>prediction of a categorical feature</high>.</li>
 <li>E.g., predicting the type of chest pain as a function of age</high>.</li>
 </ul>
 </li>
</ul>

]

]

---

# Logistic regression

<ul style="margin-bottom:-20px">
 <li class="m1">In <a href="https://en.wikipedia.org/wiki/Logistic_regression">logistic regression</a>, the class criterion <mono>Y &isin; (0,1)</mono> is modeled also as the <high>sum of feature times weights</high>, but with the prediction being transformed using a <high>logistic link function</high>.</li>
</ul>

`$$\large \hat{Y} =  Logistic(\beta_{0} + \beta_{1} \times X_1 + ...)$$`

<ul style="margin-bottom:-20px">
 <li class="m2">The logistic function <high>maps predictions to the range of 0 and 1</high>, the two class values..</li>
</ul>

$$ Logistic(x) = \frac{1}{1+exp(-x)}$$

]

]

---

# Logistic regression

`$$\large \hat{Y} =  Logistic(\beta_{0} + \beta_{1} \times X_1 + ...)$$`

<ul style="margin-bottom:-20px">
 <li class="m2">The logistic function <high>maps predictions to the range of 0 and 1</high>, the two class values..</li>
</ul>

$$ Logistic(x) = \frac{1}{1+exp(-x)}$$

]

]

---

# Classification loss

<ul style="margin-bottom:-20px">
 <li class="m1">Distance
 
 <ul class="level">
 <li>Logloss is <high>used to fit the parameters</high>, alternative distance measures are MSE and MAE.</li>
 </ul>
 </li>
</ul>

`$$\small LogLoss = -\frac{1}{n}\sum_{i}^{n}(log(\hat{y})y+log(1-\hat{y})(1-y))$$`
`$$\small MSE = \frac{1}{n}\sum_{i}^{n}(y-\hat{y})^2, \: MAE = \frac{1}{n}\sum_{i}^{n} \lvert y-\hat{y} \rvert$$`

<ul> 
 <li class="m2">Overlap
 
 <ul class="level">
 <li>Does the <high>predicted class match the actual class</high>. Often preferred for <high>ease of interpretation</high>..</li>
 </ul>
 </li>
</ul>

`$$\small Loss_{01}=\frac{1}{n}\sum_i^n I(y \neq \lfloor \hat{y} \rceil)$$`

]

]

---

# Confusion matrix

<ul style="margin-bottom:-20px">
 <li class="m1">The confusion matrix <high>tabulates prediction matches and mismatches</high> as a function of the true class.</li>
 <li class="m2">The confusion matrix permits specification of a number of <high>helpful performance metrics</high>.</li>
</ul>

Confusion matrix

<table style="cellspacing:0; cellpadding:0; border:none;">
<col width=20%>
<col width=40%>
<col width=40%>
<tr>
 <td>
 </td>
 <td>
 <eq>y = 1</eq>
 </td>
 <td>
 <eq>y = 0</eq>
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <eq>y&#770; = 1</eq>
 </td>
 <td bgcolor="white">
 True positive (TP)
 </td>
 <td bgcolor="white">
 False positive (FP)
 </td>
</tr>
<tr>
 <td>
 <eq>y&#770; = 0</eq>
 </td>
 <td>
 False negative (FN)
 </td>
 <td>
 True negative (TN)
 </td>
</tr>
</table>

]

Accuracy: Of all cases, what percent of predictions are correct?

`$$\small Acc. = \frac{TP + TN}{ TP + TN + FN + FP} = 1-Loss_{01}$$`

Sensitivity: Of the truly Positive cases, what percent of predictions are correct?

`$$\small Sensitivity = \frac{TP}{ TP +FN }$$`

Specificity: Of the truly Negative cases, what percent of predictions are correct?

`$$\small Specificity = \frac{TN}{ TN + FP }$$`

]

---

# Confusion matrix

Confusion matrix

<table style="cellspacing:0; cellpadding:0; border:none;">
<col width=20%>
<col width=40%>
<col width=40%>
<tr>
 <td>
 </td>
 <td>
 <eq>Default</eq>
 </td>
 <td>
 <eq>Repay</eq>
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <eq>"Default"</eq>
 </td>
 <td bgcolor="white">
 TP = 3
 </td>
 <td bgcolor="white">
 FP = 1
 </td>
</tr>
<tr>
 <td>
 <eq>"Repay"</eq>
 </td>
 <td>
 FN = 1
 </td>
 <td>
 TN = 2
 </td>
</tr>
</table>

]

Accuracy: Of all cases, what percent of predictions are correct?

`$$\small Acc. = \frac{TP + TN}{ TP + TN + FN + FP} = 1-Loss_{01}$$`

Sensitivity: Of the truly Positive cases, what percent of predictions are correct?

`$$\small Sensitivity = \frac{TP}{ TP +FN }$$`

Specificity: Of the truly Negative cases, what percent of predictions are correct?

`$$\small Specificity = \frac{TN}{ TN + FP }$$`

]

---
class: center,  middle

# Fitting regression models with `caret`

---

# `caret`s fitting functions

<table style="cellspacing:0; cellpadding:0; border:none;">
<tr>
 <td>
 Function
 </td>
 <td>
 Description
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>trainControl()</mono>
 </td>
 <td bgcolor="white">
 Choose <high>settings</high> for how fitting should be carried out.
 </td>
</tr>
<tr>
 <td>
 <mono>train()</mono>
 </td>
 <td>
 Specify the model and <high>find 'best' parameters</high>.
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>postResample()</mono>
 </td>
 <td bgcolor="white">
 <high>Evaluate</high> model performance (fitting or prediction) for regression.
 </td>
</tr>
<tr>
 <td>
 <mono>confusionMatrix()</mono>
 </td>
 <td bgcolor="white">
 <high>Evaluate</high> model performance (fitting or prediction) for classification.
 </td>
</tr>
</table>

]

```r
# Step 1: Define control parameters
#   trainControl()

ctrl <- trainControl(...)

# Step 2: Train and explore model
#   train()

mod <- train(...)
summary(mod)
mod$finalModel # see final model

# Step 3: Assess fit
#   predict(), postResample(), fon

fit <- predict(mod)
postResample(fit, truth)
confusionMatrix(fit, truth)
```

]

---

# `trainControl()`

<ul>
 <li class="m1"><mono>trainControl()</mono> controls <high>how <mono>caret</mono> fits an ML model</high>.</li>
 <li class="m2">For now, we set <mono>method = "none"</mono> to keep things simple. More in the session on optimization.</li>
</ul>

```r
# Fit the model without any 
# advanced parameter tuning methods
ctrl <- trainControl(method = "none")

# show help file
?trainControl
```

]

]

---

# `train()`

<table style="cellspacing:0; cellpadding:0; border:none;">
<tr>
 <td>
 Argument
 </td>
 <td>
 Description
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>form</mono>
 </td>
 <td bgcolor="white">
 Formula specifying features and criterion.
 </td>
</tr>
<tr>
 <td>
 <mono>data</mono>
 </td>
 <td>
 Training data.
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>method</mono>
 </td>
 <td bgcolor="white">
 The model (algorithm). 
 </td>
</tr>
<tr>
 <td>
 <mono>trControl</mono>
 </td>
 <td bgcolor="white">
 Control parameters for fitting.
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>tuneGrid</mono>, <mono>preProcess</mono>
 </td>
 <td bgcolor="white">
 Cool stuff for later.
 </td>
</tr>
</table>

]

```r
# Fit a regression model predicting Price

income_mod <- 
 train(form = income ~ ., # Formula
 data = baselers, # Training data
 method = "glm", # Regression
 trControl = ctrl) # Control Param's
income_mod
```

```
Generalized Linear Model

1000 samples
  19 predictor

No pre-processing
Resampling: None 
```

]

---

# `train()`

]

```r
# Fit a random forest predicting Price

income_mod <- 
 train(form = income ~ .,# Formula
 data = baselers, # Training data
 method = "rf", # Random Forest
 trControl = ctrl) # Control Param's
income_mod
```

```
Random Forest

1000 samples
  19 predictor

No pre-processing
Resampling: None 
```

]

---

# `train()`

<ul>
 <li class="m1"><mono>train()</mono> is the <high>fitting workhorse</high> of <mono>caret</mono>, offering you <high>200+ models</high> by merely changing the <high>method</high> argument!.</li>
 <li class="m2">Find all 200+ models <a href="http://topepo.github.io/caret/available-models.html">here</a>.</li>
</ul>

]

]

---

# `train()`

<ul style="margin-bottom:-20px">
 <li class="m1">The criterion must be the right type:
 
 <ul class="level">
 <li><high><mono>numeric</mono></high> criterion &rarr; <high>Regression</high> </li>
 <li><high><mono>factor</mono></high> criterion &rarr; <high>Classification</high> </li>
 </ul>
 </li>
</ul>

```
# A tibble: 5 x 5
 Default Age Gender Cards Education
 <dbl> <dbl> <chr> <dbl> <dbl>
1 0 45 M 3 11
2 1 36 F 2 14
3 0 76 F 5 12
4 1 25 M 2 17
5 1 36 F 3 12
```

]

```r
# Will be a regression task

loan_mod <- train(form = Default ~ .,
 data = Loans,
 method = "glm",
 trControl = ctrl)

# Will be a classification task

load_mod <- train(form = factor(Default) ~ .,
 data = Loans,
 method = "glm",
 trControl = ctrl)
```

]

---

# <mono>mod$finalModel</mono>

<ul>
 <li class="m1">The <mono>train()</mono> function returns a <mono>list</mono> with an object called <mono>finalModel</mono> - this is your <high>fitted machine learning model</high>!</li>
 <li class="m2"><high>Access</high> the model with <mono>mod$finalModel<mono> and <high>explore</high> the object with generic functions.</li>
</ul>

<table style="cellspacing:0; cellpadding:0; border:none;">
<tr>
 <td>
 Function
 </td>
 <td>
 Description
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>summary()</mono>
 </td>
 <td bgcolor="white">
 <high>Overview</high> of the most important results.
 </td>
</tr>
<tr>
 <td bgcolor="white">
 <mono>names()</mono>
 </td>
 <td bgcolor="white">
 See all <high>named elements</high> you can access with $.
 </td>
</tr>
</table>

]

```r
# Create a regression object
income_mod <- 
 train(form = income ~ age + height,
 data = baselers) # Training data

# Look at all named outputs
names(income_mod$finalModel)
```

```
[1] "coefficients"  "residuals"     "fitted.values"
[4] "effects"       "R"             "rank"         
 [ reached getOption("max.print") -- omitted 28 entries ]
```

```r
# Access specific outputs
income_mod$finalModel$coefficients
```

```
(Intercept)         age      height 
    177.084     151.786       3.466 
```

]

---

# `predict()`

<ul>
 <li class="m1">The <mono>predict()</mono> function <high>produces predictions</high> from a model. Simply put model object as the first argument.</li>
</ul>

```r
# Get fitted values
glm_fits <- predict(object = income_mod)
glm_fits[1:8]
```

```
    1     2     3     4     5     6     7     8 
 5508  6960  6982  8645  5325 10648  8663  4592 
```

]

]

---

# `postResample()`

<ul>
 <li class="m1">The <nono>postResample()</nono> function <high>gives a simple summary</high> of a models' performance in a <high>regression task</high>. Simply put the predicted values and the true values inside the function.</li>
</ul>

```r
# evaluate
postResample(glm_fits,
             baselers$income)
```

```
    RMSE Rsquared      MAE 
1173.079    0.821  937.113 
```

]

]

---

## `confusionMatrix()`

<ul>
 <li class="m1">The <nono>confusionMatrix()</nono> does the same for a models' performance in a <high>classification task</high>. Simply put the predicted values and the true values inside the function..</li>
</ul>

```r
# eyecor to factor
baselers$eyecor <- factor(baselers$eyecor)

# run glm model for classification
eyecor_mod <- 
 train(form = eyecor ~ age + height,
 data = baselers, 
 method = "glm", 
 trControl = ctrl)

# evaluate
confusionMatrix(predict(eyecor_mod), 
                baselers$eyecor)
```

]

```
Confusion Matrix and Statistics

Reference
Prediction no yes
 no 0 0
 yes 353 647
 
 Accuracy : 0.647 
 95% CI : (0.616, 0.677)
 No Information Rate : 0.647 
 P-Value [Acc > NIR] : 0.514 
 
 Kappa : 0 
 
 Mcnemar's Test P-Value : <2e-16 
 
 Sensitivity : 0.000 
 Specificity : 1.000 
 Pos Pred Value : NaN 
 Neg Pred Value : 0.647 
 Prevalence : 0.353 
 Detection Rate : 0.000 
 Detection Prevalence : 0.000 
 Balanced Accuracy : 0.500 
 
 'Positive' Class : no 
 
```

]

---

<h1><a href=https://therbootcamp.github.io/AML_2020AMLD/_sessions/Fitting/Fitting_practical.html>Practical</a></h1>