Working on caret based models #446
Replies: 3 comments 2 replies
-
|
Can you give more information, perhaps an example of what you are looking for? Non-parametric models (like xgboot) don't really lend themselves to predictor specific effect sizes. There are variable importance measures you might want to look at: |
Beta Was this translation helpful? Give feedback.
-
|
yes variable importance is part of the solution. However conceptually if one could consider R-square of 100% to be a model that perfectly explains the dependent variable, then an R-square of 10% means 90% is unexplained effect. So hoping for a way to get model agnostic effect distribution of predictors |
Beta Was this translation helpful? Give feedback.
-
|
You can get model-wise diagnostics. But predictor-wise is much harder, and in any case would not be model agnostic. As an example, here is a simple linear model - we can compare the unique contribution of each predictor by leaving it out and seeing the change in R square. And yet... m <- lm(mpg ~ cyl + am + hp,
data = mtcars)
m_drop_cyl <- update(m, formula. = . ~ . - cyl)
m_drop_am <- update(m, formula. = . ~ . - am)
m_drop_hp <- update(m, formula. = . ~ . - hp)
R2_total <- performance::r2(m)[[1]]The difference should be the unique contribution of each model R2_delta_cyl <- R2_total - performance::r2(m_drop_cyl)[[1]]
R2_delta_am <- R2_total - performance::r2(m_drop_am)[[1]]
R2_delta_hp <- R2_total - performance::r2(m_drop_hp)[[1]]
c(R2_total = R2_total,
Sum_unique_R2 = R2_delta_cyl + R2_delta_am + R2_delta_hp)
#> R2_total.R2 Sum_unique_R2.R2
#> 0.8041352 0.1306490Created on 2022-05-04 by the reprex package (v2.0.1) In more complex models this becomes even more prominent - e.g., in tree based models or KNN, predictors interact in complex ways... |
Beta Was this translation helpful? Give feedback.
-
|
This above works only for linear models that is already a non-issue mathematically. Curious to see effect sizes in let's say xgboost or SVM. |
Beta Was this translation helpful? Give feedback.
-
|
My point was that it doesn't work even in linear models. |
Beta Was this translation helpful? Give feedback.
-
repeat of question
I want to use model agnostic caret objects to extract effect size. Wondering how to go about that.
Beta Was this translation helpful? Give feedback.
All reactions