We’re pleased to announce the release of tune 0.1.1. tune is a tidy interface for optimizing model tuning parameters.
You can install it from CRAN with:
install.packages("tune")
You can see a full list of changes in the release notes. The release was originally motivated by dplyr 1.0.0 changes although there are a lot of nice, new features to talk about.
Better autoplot()
The previous plot method produces what we refer to as a marginal plot; each predictor was plotted against performance. That is probably the best that we can do for non-regular grids (which tends to be the default in tune). Here’s an example using the Chicago train data:
library(tidymodels)
data(Chicago, package = "modeldata")
# Time-series resampling
set.seed(7898)
data_folds <-
rolling_origin(
Chicago,
initial = 364 * 15,
assess = 7 * 4,
skip = 7 * 4,
cumulative = FALSE
)
svm_mod <-
svm_rbf(cost = tune(), rbf_sigma = tune("kernel parameter")) %>%
set_mode("regression") %>%
set_engine("kernlab")
ctrl <- control_grid(save_pred = TRUE)
set.seed(2893)
non_regular_grid <-
svm_mod %>%
tune_grid(
ridership ~ Harlem + Archer_35th,
resamples = data_folds,
grid = 36,
control = ctrl
)
autoplot(non_regular_grid, metric = "rmse") +
ggtitle("old method, irregular grid")
Not bad but it could be improved in a few ways:
-
Both tuning parameters are generated on log scales. The data are shown above in the natural units and the data at the low end of the scale gets smashed together.
-
We could show its parameter label (e.g. “Radial Basis Function sigma”) when no parameter ID is given.
What happens when a regular (i.e. factorial) grid is used?
grid <-
svm_mod %>%
parameters() %>%
grid_regular(levels = c(3, 12))
grid
## # A tibble: 36 x 2
## cost `kernel parameter`
## <dbl> <dbl>
## 1 0.000977 0.0000000001
## 2 0.177 0.0000000001
## 3 32 0.0000000001
## 4 0.000977 0.000000000811
## 5 0.177 0.000000000811
## 6 32 0.000000000811
## 7 0.000977 0.00000000658
## 8 0.177 0.00000000658
## 9 32 0.00000000658
## 10 0.000977 0.0000000534
## # … with 26 more rows
set.seed(2893)
regular_grid <-
svm_mod %>%
tune_grid(
ridership ~ Harlem + Archer_35th,
resamples = data_folds,
grid = grid,
control = ctrl
)
autoplot(regular_grid, metric = "rmse") +
ggtitle("old method, regular grid")
This visualization also could be improved, since there might be a pattern in one parameter for each value of the other.
The new version of tune creates improved versions of both of these plots:
autoplot(non_regular_grid, metric = "rmse") +
ggtitle("new method, irregular grid")
This tells a completely different story than the previous version where the parameters were in their natural units.
The regular grid results are also much better and tell a cleaner story:
autoplot(regular_grid, metric = "rmse") +
ggtitle("new method, regular grid")
Extra arguments can be passed when a numeric grouping column is used;l these are given to format(). To avoid scientific notation:
autoplot(regular_grid, metric = "rmse", digits = 3, scientific = FALSE) +
ggtitle("Formatting for coloring column")
A ggplot2 coord for plotting observed and predicted values
One helpful visualization of the fit of a regression model is to plot the true outcome value against the predictions. These should be on the same scale. Let’s look at such a plot:
best_values <- select_best(regular_grid, metric = "rmse")
best_values
## # A tibble: 1 x 3
## cost `kernel parameter` .config
## <dbl> <dbl> <chr>
## 1 32 0.0152 Model30
holdout_predictions <-
regular_grid %>%
collect_predictions(parameters = best_values)
holdout_predictions
## # A tibble: 224 x 7
## id .pred .row cost `kernel parameter` ridership .config
## <chr> <dbl> <int> <dbl> <dbl> <dbl> <chr>
## 1 Slice1 17.8 5461 32 0.0152 19.6 Model30
## 2 Slice1 18.9 5462 32 0.0152 20.0 Model30
## 3 Slice1 17.5 5463 32 0.0152 20.4 Model30
## 4 Slice1 8.88 5464 32 0.0152 20.4 Model30
## 5 Slice1 3.03 5465 32 0.0152 20.1 Model30
## 6 Slice1 6.07 5466 32 0.0152 4.78 Model30
## 7 Slice1 4.48 5467 32 0.0152 3.26 Model30
## 8 Slice1 12.6 5468 32 0.0152 19.3 Model30
## 9 Slice1 15.6 5469 32 0.0152 19.3 Model30
## 10 Slice1 15.4 5470 32 0.0152 19.9 Model30
## # … with 214 more rows
ggplot(holdout_predictions, aes(x = ridership, y = .pred)) +
geom_abline(lty = 2) +
geom_point(alpha = 0.3)
This is very helpful but there are a few possible improvements. The new version of tune has coord_obs_pred() that produces a square plot with the same axes:
ggplot(holdout_predictions, aes(x = ridership, y = .pred)) +
geom_abline(lty = 2) +
geom_point(alpha = 0.3) +
coord_obs_pred()
Tuning engine parameters
Bruna Wundervald (from Maynooth University) gave a
great presentation that used tidymodels packages. She ran into the problem that, if you wanted to tune parameters that were specific to the engine, you’d have to go through a lot of trouble to do so. This used to work well in a previous version of tune. Unfortunately, we accidentally broke it, but now you can once again tune engine specific parameters. One feature in this version of tune, along with the new 0.0.8 version of the dials package, is that we have added dials parameter objects for every parameter that users might tuned with the existing engines that we support (this was not as difficult as it sounds).
To demonstrate, we’ll use the time series data above, but this time we’ll optimize the ranger parameters that Bruna was interested in.
Since parsnip has a pre-defined list of models and engines, we’ve gone ahead and set up the infrastructure for tuning most engine-specific values. For example, in the above example we could tune two regularization parameters specific to ranger.
rf_mod <-
rand_forest(min_n = tune()) %>%
set_mode("regression") %>%
set_engine("ranger",
regularization.factor = tune(),
regularization.usedepth = tune())
# Are there dials objects to work with these?
rf_param <- parameters(rf_mod)
rf_param
## Collection of 3 parameters for tuning
##
## id parameter type object class
## min_n min_n nparam[+]
## regularization.factor regularization.factor nparam[+]
## regularization.usedepth regularization.usedepth dparam[+]
There are parameter objects for these (and they keep their original names). You can adjust the ranges and values for these parameters using the update() function as you would for others. To see their underlying functions:
rf_param$object[[2]]
## Gain Penalization (quantitative)
## Range: [0, 1]
From here, the standard tools in the tune package can be used. You can use one of the grid_*() functions to create a grid of values, let tune_grid() create a set for you, or use Bayesian optimization to find appropriate values sequentially.
The new autoplot() method can also be used to produce nice visualizations of the relationship between performance and the parameters.
set.seed(4976)
ranger_params <-
rf_mod %>%
tune_grid(
ridership ~ Harlem + Archer_35th,
resamples = data_folds,
grid = 10,
control = ctrl
)
autoplot(ranger_params, metric = "rmse") +
theme(legend.position = "top")
Note that the tuning parameter labels (i.e. “Gain Penalization” instead of “regularization.factor”) are used.
I’m sure that we missed someone’s favorite engine-specific parameter so please put in a GitHub issue for dials to let us know.
.config columns
When model tuning is conducted, the tune package now saves a new column in the output called .config. This column is a qualitative identification column for unique tuning parameter combinations. It often reflects what is being tuned. A value of .config = "Recipe1_Model3" indicates that the first recipe tuning parameter set is being evaluated in conjunction with the third set of model parameters. Here’s an example from the random forest model that we just fit:
ranger_params %>%
collect_metrics() %>%
# get the unique tuning parameter combinations:
select(min_n, regularization.factor, regularization.usedepth, .config) %>%
distinct()
## # A tibble: 10 x 4
## min_n regularization.factor regularization.usedepth .config
## <int> <dbl> <lgl> <chr>
## 1 7 0.288 TRUE Model01
## 2 20 0.322 FALSE Model02
## 3 25 0.703 FALSE Model03
## 4 14 0.0781 FALSE Model04
## 5 36 0.882 TRUE Model05
## 6 29 0.192 FALSE Model06
## 7 35 0.404 TRUE Model07
## 8 24 0.687 TRUE Model08
## 9 3 0.580 FALSE Model09
## 10 10 0.968 TRUE Model10
Other changes
-
conf_mat_resampled()is a new function that computes the average confusion matrix across resampling statistics for a single model. -
show_best()and theselect_*()functions will now use the first metric in the metric set if no metric is supplied. -
filter_parameters()can trim the.metricscolumn of unwanted results (as well as columns.predictionsand.extracts) fromtune_*objects. -
If a grid is given, parameters do not need to be finalized to be used in the
tune_*()functions.
Acknowledgements
Thanks to everyone who contributed code or filed issues since the last version: @cimentadaj, @connor-french, @cwchang-nelson, @DavisVaughan, @dcossyleon, @EmilHvitfeldt, @jg43b, @JHucker, @juliasilge, @karaesmen, @kbzsl, @kylegilde, @LucyMcGowan, @mdancho84, @py9mrg, @realauggieheschmeyer, @robyjos, @rorynolan, @simonpcouch, @ThomasWolf0701, and @UnclAlDeveloper.