Fitting

• Models are actually families of models, with every parameter combination specifying a different model.
• To fit a model means to identify from the family of models the specific model that fits the data best.

• Possible the most important concept in statistics and machine learning.
• The loss function defines some summary of the errors committed by the model.

$$\Large Loss = f(Error)$$

Purpose	Description
Fitting	Find parameters that minimize loss function.
Evaluation	Calculate loss function for fitted model.

• Possible the most important concept in statistics and machine learning.
• The loss function defines some summary of the errors committed by the model.

$$\Large Loss = f(Error)$$

Purpose	Description
Fitting	Find parameters that minimize loss function.
Evaluation	Calculate loss function for fitted model.

In regression, the criterion $Y$ is modeled as the sum of features $X_1, X_2, ...$ times weights $\beta_1, \beta_2, ...$ plus $\beta_0$ the so-called the intercept.

$$\large \hat{Y} = \beta_{0} + \beta_{1} \times X_1 + \beta_{2} \times X2 + ...$$

The weight $\beta_{i}$ indiciates the amount of change in $\hat{Y}$ for a change of 1 in $X_{i}$.

Ceteris paribus, the more extreme $\beta_{i}$, the more important $X_{i}$ for the prediction of $Y$ (Note: the scale of $X_{i}$ matters too!).

If $\beta_{i} = 0$, then $X_{i}$ does not help predicting $Y$

• Mean Squared Error (MSE)
  
  
  • Average squared distance between predictions and true values.

` $$MSE = \frac{1}{n}\sum_{i \in 1,...,n}(Y_{i} - \hat{Y}_{i})^{2}$$ `

• Mean Absolute Error (MAE)
  
  
  • Average absolute distance between predictions and true values.

$$MAE = \frac{1}{n}\sum_{i \in 1,...,n} \lvert Y_{i} - \hat{Y}_{i} \rvert$$

• Regression
  
  
  • Regression problems involve the prediction of a quantitative feature.
  • E.g., predicting the cholesterol level as a function of age.


• Classification
  
  
  • Classification problems involve the prediction of a categorical feature.
  • E.g., predicting the type of chest pain as a function of age.

• In logistic regression, the class criterion Y ∈ (0,1) is modeled also as the sum of feature times weights, but with the prediction being transformed using a logistic link function.

$$\large \hat{Y} = Logistic(\beta_{0} + \beta_{1} \times X_1 + ...)$$

• The logistic function maps predictions to the range of 0 and 1, the two class values..

$$Logistic(x) = \frac{1}{1+exp(-x)}$$

• In logistic regression, the class criterion Y ∈ (0,1) is modeled also as the sum of feature times weights, but with the prediction being transformed using a logistic link function.

$$\large \hat{Y} = Logistic(\beta_{0} + \beta_{1} \times X_1 + ...)$$

• The logistic function maps predictions to the range of 0 and 1, the two class values..

$$Logistic(x) = \frac{1}{1+exp(-x)}$$

• Distance
  
  
  • Logloss is used to fit the parameters, alternative distance measures are MSE and MAE.

$$\small LogLoss = -\frac{1}{n}\sum_{i}^{n}(log(\hat{y})y+log(1-\hat{y})(1-y))$$

• Overlap
  
  
  • Does the predicted class match the actual class. Often preferred for ease of interpretation..

$$\small Loss_{01} = 1-Accuracy = \frac{1}{n}\sum_i^n I(y \neq \lfloor \hat{y} \rceil)$$

Fitting tidymodels

• Define the recipe.


• Define the model.


• Define the workflow.


• Fit the workflow


• Assess model performance.

• The recipe specifies two things:


• The criterion and the features, i.e., the formula to use.


• How the features should be pre-processed before the model fitting.


• To set up a recipe:


• Initialize it with recipe(), wherein the formula and data are specified.


• Add pre-processing steps, using step_*() functions and dplyr-like selectors.

# set up recipe for regression model
lm_recipe <- 
  recipe(income ~ ., data = baselers) %>% 
  step_dummy(all_nominal_predictors())
lm_recipe

Data Recipe
Inputs:
      role #variables
   outcome          1
 predictor         19
Operations:
Dummy variables from all_nominal_predictors()

• The recipe specifies two things:


• The criterion and the features, i.e., the formula to use.


• How the features should be pre-processed before the model fitting.


• To set up a recipe:


• Initialize it with recipe(), wherein the formula and data are specified.


• Add pre-processing steps, using step_*() functions and dplyr-like selectors.

# set up recipe for logistic regression
# model
logistic_recipe <- 
  recipe(eyecor ~., data = baselers) %>% 
  step_dummy(all_nominal_predictors())
logistic_recipe

Data Recipe
Inputs:
      role #variables
   outcome          1
 predictor         19
Operations:
Dummy variables from all_nominal_predictors()

• The model specifies:


• Which model (e.g. linear regression) to use.


• Which engine (underlying model-fitting algorithm) to use.


• The problem mode, i.e., regression vs. classification.


• To set up a model:


• Specify the model, e.g., using linear_reg() or logistic_reg().


• Specify the engine using set_engine().


• Specify the problem mode using set_mode().

# set up model for regression model
lm_model <- 
  linear_reg() %>% 
  set_engine("lm") %>% 
  set_mode("regression")
lm_model

Linear Regression Model Specification (regression)
Computational engine: lm

• The model specifies:


• Which model (e.g. linear regression) to use.


• Which engine (underlying model-fitting algorithm) to use.


• The problem mode, i.e., regression vs. classification.


• To set up a model:


• Specify the model, e.g., using linear_reg() or logistic_reg().


• Specify the engine using set_engine().


• Specify the problem mode using set_mode().

# set up model for logistic regression
# model
logistic_model <- 
  logistic_reg() %>% 
  set_engine("glm") %>% 
  set_mode("classification")
logistic_model

Logistic Regression Model Specification (classification)
Computational engine: glm

• A workflow combines the recipe and model and facilitates fitting the model. To set up a workflow:


• Initialize it using the workflow() function.


• Add a recipe using add_recipe().


• Add a model using add_model().

# set up workflow for regression model
lm_workflow <- 
  workflow() %>% 
  add_recipe(lm_recipe) %>% 
  add_model(lm_model)
lm_workflow

══ Workflow ══════════════════════════════════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: linear_reg()
── Preprocessor ──────────────────────────────────────────────────────────────────────────────────────────────
1 Recipe Step
• step_dummy()
── Model ─────────────────────────────────────────────────────────────────────────────────────────────────────
Linear Regression Model Specification (regression)
Computational engine: lm

• A workflow combines the recipe and model and facilitates fitting the model. To set up a workflow:


• Initialize it using the workflow() function.


• Add a recipe using add_recipe().


• Add a model using add_model().

# set up workflow for logistic regression
# model
logistic_workflow <- 
  workflow() %>% 
  add_recipe(logistic_recipe) %>% 
  add_model(logistic_model)
logistic_workflow

══ Workflow ══════════════════════════════════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: logistic_reg()
── Preprocessor ──────────────────────────────────────────────────────────────────────────────────────────────
1 Recipe Step
• step_dummy()
── Model ─────────────────────────────────────────────────────────────────────────────────────────────────────
Logistic Regression Model Specification (classification)
Computational engine: glm

• A workflow is fitted using the fit() function.


• Applies the recipe with the pre-processing steps.


• Run the specified algorithm (i.e., model).

# fit the workflow
income_lm <- fit(lm_workflow,
                 data = baselers)
tidy(income_lm)

# A tibble: 25 × 5
   term           estimate std.error statistic   p.value
   <chr>             <dbl>     <dbl>     <dbl>     <dbl>
 1 (Intercept) -192.         631.     -0.304   7.61e-  1
 2 id             0.000895     0.113   0.00792 9.94e-  1
 3 age          115.           2.88   40.1     2.23e-208
 4 height         4.95         3.02    1.64    1.02e-  1
 5 weight         1.01         3.27    0.307   7.59e-  1
 6 children     -48.9         31.9    -1.54    1.25e-  1
 7 happiness   -156.          31.1    -5.02    6.00e-  7
 8 fitness        6.94        17.9     0.389   6.97e-  1
 9 food           2.50         0.142  17.6     2.33e- 60
10 alcohol       26.1          2.47   10.6     8.05e- 25
# … with 15 more rows

• A workflow is fitted using the fit() function.


• Applies the recipe with the pre-processing steps.


• Run the specified algorithm (i.e., model).

# fit the logistic regression workflow
eyecor_glm <- fit(logistic_workflow,
                  data = baselers)
tidy(eyecor_glm)

# A tibble: 25 × 5
   term          estimate std.error statistic p.value
   <chr>            <dbl>     <dbl>     <dbl>   <dbl>
 1 (Intercept) -3.04      1.32         -2.31   0.0211
 2 id           0.0000834 0.000236      0.354  0.723 
 3 age          0.00734   0.00973       0.755  0.451 
 4 height       0.00572   0.00630       0.907  0.364 
 5 weight       0.00446   0.00678       0.658  0.510 
 6 income      -0.0000395 0.0000666    -0.593  0.553 
 7 children     0.0329    0.0665        0.495  0.621 
 8 happiness    0.0386    0.0653        0.591  0.554 
 9 fitness     -0.0419    0.0372       -1.13   0.261 
10 food        -0.0000755 0.000339     -0.222  0.824 
# … with 15 more rows

• Use predict() to obtain model predictions on specified data.


• Use metrics() to obtain performance metrics, suited for the current problem mode.

# generate predictions
lm_pred <-
  income_lm %>% 
  predict(baselers) %>% 
  bind_cols(baselers %>% select(income))
metrics(lm_pred,
        truth = income,
        estimate = .pred)

# A tibble: 3 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rmse    standard    1008.   
2 rsq     standard       0.868
3 mae     standard     792.

• Use predict() to obtain model predictions on specified data.


• Use metrics() to obtain performance metrics, suited for the current problem mode.

# generate predictions logistic regression
logistic_pred <- 
  predict(eyecor_glm, baselers,
          type = "prob") %>% 
  bind_cols(predict(eyecor_glm, baselers)) %>% 
  bind_cols(baselers %>% select(eyecor))
metrics(logistic_pred,
        truth = eyecor,
        estimate = .pred_class,
        .pred_yes)

# A tibble: 4 × 3
  .metric     .estimator .estimate
  <chr>       <chr>          <dbl>
1 accuracy    binary        0.647 
2 kap         binary        0.0566
3 mn_log_loss binary        0.634 
4 roc_auc     binary        0.605

• Use roc_curve() to obtain sensitivity and specificity for every unique value of the predicted probabilities.


• Sensitivity = Of the truly positive cases, what proportion is classified as positive.


• Specificity = Of the truly negative cases, what proportion is classified as negative.


• Use autoplot() to plot the ROC-curve based on the different combinations of sensitivity and specificity.

# ROC curve for logistic model
logistic_pred %>% 
  roc_curve(truth = eyecor, .pred_yes) %>% 
  autoplot()