Third post from the short series about black-box explainers implemented in the DALEX package. Learn more about DALEX at SER (Warsaw, April 2018), eRum (Budapest, May 2018), WhyR (Wroclaw, June 2018) or UseR (Brisbane, July 2018).
Two weeks ago I wrote about single variable conditional responses and last week I wrote about decompositions of a single prediction.
Sometimes we would like to know the general structure of a model, or at least know which variables are the most influential. There is a lot of different approaches to this problem proposed in literature. A nice, simple, and model agnostic approach is described in this article (Fisher, Rudin, and Dominici 2018). To see how important is variable X let’s permute it’s values and measure the drop in model accuracy.
This procedure is implemented in the DALEX package in the variable_dropout() function. There are some tweaks (for large datasets you do not need to permute all rows while for small datasets you could consider some oversampling) but the idea is the same.
In the figure below you will find variable importances for three models created with the HR dataset. It is easy to spot that the randomForest model results in the best model and satisfaction_level is the most important variable in all three models.
There are two things that I like in this explainer.
1) Variable effects for a single model are interesting, but ability to compare effects for many modes is even more interesting. In the DALEX you can simply contrast/compare explainers across different models.
2) There is no reason to start variable importance plots in the point 0, since the initial model performance is different for different plots. It is much more informative to present both the initial model performance and drop in the performance resulting from the dropout of a variable.
If you want to learn more about DALEX package and variable importances consult following vignette or the DALEX website.
