Download to read offline
Uploaded byKeisuke Hosaka
高速・省メモリにlibsvm形式で ダンプする方法を研究してみた
The document discusses using xgboost for machine learning and summarizes steps to prepare data for xgboost models. It recommends binding feature data together and writing it out in the libsvm format for efficient reading into an xgboost DMatrix object. It also suggests using the data.table package to write out libsvm files in parallel for improved performance on large datasets.
高速・省メモリにlibsvm形式で ダンプする方法を研究してみた
- 1.
- 3. • : • :R, Python, C++ • : • :
- 4.
- 5.
- 6.
- 7.
- 10. # feature label =readRDS("label.rds") feature_set_A = readRDS("feature_set_A.rds") feature_set_B = readRDS("feature_set_B.rds") # feature cbind mat = cbind( feature_set_A, feature_set_B ) ↑ # DMatrix dmat = xgb.DMatrix(mat, label=label)
- 11.
- 12.
- 14.
- 17.
- 18.
- 19. 1. cbind libsvm 2.DMatrix
- 21.
- 22.
- 23. # feature label =readRDS("label.rds") feature_set_A = fread("feature_set_A.csv") feature_set_B = fread("feature_set_B.csv") # feature list # 1 label matrices = list(label, feature_set_A, feature_set_B) # libsvm fwrite_libsvm(matrices, "libsvm.txt") # DMatrix dmat = xgb.DMatrix("libsvm.txt")
- 25. fwrite OpenMP 8.5GB/120sec @Xeon 2.5GHz ✕ 8
- 26.
- 27.
- 28.