Logstash ã使ã£ã¦ MySQL ãã¼ã¿ã Elasticsearch ã«ã¤ã³ããã¯ã¹ããï¼åºæ¬ç·¨ï¼
ãªã¬ã¼ã·ã§ãã«ãã¼ã¿ãã¼ã¹ã§ç®¡çãã¦ãããã¼ã¿ã Elasticsearch ã§æ¤ç´¢ã»åæãããå ´åãLogstash ã便å©ã§ãã
Logstash ã¨ã¯ï¼
Logstash ã¯ãªã¼ãã³ã½ã¼ã¹ã®ãµã¼ãã¼ãµã¤ããã¼ã¿å¦çãã¤ãã©ã¤ã³ã§ããæ§ã ãªæ°ã®ã½ã¼ã¹ãããã¼ã¿ãåãè¾¼ã¿ãå¤æããæå®ãããä»»æã®ã¹ãã¢å ã«ãã¼ã¿ãæ ¼ç´ãããã¨ãã§ãã¾ãã
å¦çã®å 容ã¯ã·ã³ãã«ã§ãInput ã¹ãã¼ã¸ã§ã½ã¼ã¹å ã®æ¥ç¶å æ å ±ã管çããFilter ã¹ãã¼ã¸ã§å¤æãããOutput ã¹ãã¼ã¸ã§æ ¼ç´å æ¥ç¶å æ å ±ãå®ç¾©ãã¾ããInput åã³ Output ãã©ã°ã¤ã³ã¯ããã©ã«ãã§æ§ã ãªã½ã¼ã¹ããµãã¼ããã¦ãã¾ãããã®ãããLogstash ã使ãã°ãããã°ã©ãã³ã°ã¬ã¹ã§ MySQL ã®ãã¼ã¿ãåãè¾¼ã¿ãå¤æããElasticsearch ã¸ã¤ã³ããã¯ã¹ãããã¨ãã§ããã®ã§ãã
äºåæºå
MySQL 㨠Elasticsearch ã®ä»ã«ã¯ãLogstash 㨠Logstash ãã MySQL ã¸æ¥ç¶ããããã® JDBC Driver (MySQL Connector/J) ãå¿ è¦ã§ãã®ã§ããããããã¦ã³ãã¼ããã¦ã¤ã³ã¹ãã¼ã«ãã¦ãã ããããªããLogstash ã®ãã¼ã¸ã§ã³ã¯ã5.4.2 ã使ç¨ãã¦ã¾ãã
- Logstash (https://www.elastic.co/downloads/logstash)
- MySQL Connector/J (https://dev.mysql.com/downloads/connector/j/)
MySQL ã®ãã¼ã¿ãåãè¾¼ãè¨å®
Logstash ã®è¨å®ãã¡ã¤ã«ã¯ãinput{}
ãããã¯ãfilter{}
ãããã¯ãoutput{}
ãããã¯ããæ§æããã¾ããMySQL ã®ãã¼ã¿ãåãè¾¼ãè¨å®ã¯ãinput{}
ãããã¯ã«è¨è¿°ãã¾ããããã§ã¯ãsample.conf ã¨ãã¦è©±ãé²ãã¾ãã
# sample.conf
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/product"
jdbc_user => "product"
jdbc_password => "1234"
statement => "SELECT * FROM product"
type => "products"
}
}
filter {
...
}
output {
...
}
- jdbc_driver_library: ãã¦ã³ãã¼ããã MySQL Connector/J ã®ãã¹ãè¨è¿°ãã¾ããç¸å¯¾ãã¹ã§è¨è¿°ããå ´åã¯ãLogstash ãå®è¡ãããã£ã¬ã¯ããªããã®ç¸å¯¾ãã¹ã§ãã
- jdbc_driver_class: ããã¯ä¾ã®ã¾ã¾è¨è¿°ããã°è¯ãã§ãã
- jdbc_connection_string: jdbc:mysql://{MYSQL_HOST}:{PORT}/{DATABASE_NAME}
- jdbc_user: MySQL ã®ã¦ã¼ã¶ã¼åãè¨è¿°ãã¾ãã
- jdbc_password: MySQL ã¦ã¼ã¶ã¼ã®ãã¹ã¯ã¼ããè¨è¿°ãã¾ãã
- statement: MySQL ãããã¼ã¿ãæ½åºãããããSELECTæãè¨è¿°ãã¾ãã
- type: filter ã output ã¹ãã¼ã¸ã§ç¹å®ã®ã¤ãã³ããèå¥ããããã®èå¥åãè¨å®ãã¦ããã¾ããinput ã¹ãã¼ã¸ã¯è¤æ°ã®
å¿ è¦æä½éã®è¨å®ã¯ä»¥ä¸ã§ããã¹ã±ã¸ã¥ã¼ã«ãè¨å®ãããããã¼ã¸ã³ã°ã®è¨å®ããããè²ã ãªè¨å®ãã§ãã¾ãã®ã§è©³ç´°ã¯ä»¥ä¸ã®ãã¼ã¸ãåèã«ãã¦ãã ããã
ãã¼ã¿ã®å å·¥å¦çãå®ç¾©ãã
ãªã¬ã¼ã·ã§ãã«ãã¼ã¿ãã¼ã¹ã§å®ç¾©ããã¦ãããã¼ã¿ã¯ããã©ã°ç³»ã®ãã£ã¼ã«ããªã©åããã¥ãããã¼ã¿è¨è¨ã«ãªã£ã¦ããå ´åãå¤ãã®ã§ãã³ã¼ããå称ãªã©ã®ç解ãããããã¼ã¿ã«å¤æããããå¿ è¦ãªå å·¥å¦ç㯠Filter ã¹ãã¼ã¸ã§å®ç¾©ãã¾ãã
input {...}
filter {
if [type] == "products" {
# ---------------------------------
# {"lang_id": "1"}
# convert to
# {"language": "Japanese"}
# ---------------------------------
translate {
field => "lang_id"
destination => "language"
dictionary => {
"1" => "English"
"2" => "Japanese"
"3" => "Chinese"
}
remove_field => "lang_id"
}
# ---------------------------------
# {"colors": "Red,Blue,Yellow"}
# Convert to
# {"colors": ["Red", "Blue", "Yellow"]}
# ---------------------------------
mutate {
split => {"colors" => ","}
}
}
}
output {...}
if [type] == âproductsâ {}
ã¯ãinput ã¹ãã¼ã¸ã§ãå®ç¾©ãã MySQL ããåå¾ãããã¼ã¿ã対象ã«å¦çããããã®æ¡ä»¶å¼ã§ãã
注æ translate ãã©ã°ã¤ã³ã使ç¨ããå ´åã¯ã¤ã³ã¹ãã¼ã«ããå¿ è¦ãããã¾ãã
Elasticsearch ã®ã¤ã³ããã¯ã¹è¨å®
MySQL ãããã¼ã¿ãåå¾ããå å·¥ãããã¼ã¿ã Elasticsearch ã«ã¤ã³ããã¯ã¹ããè¨å®ã¯ãoutput ã¹ãã¼ã¸ã§å®ç¾©ãã¾ãã
input {...}
filter {...}
output {
if [type] == "products" {
elasticsearch {
manage_template => false
hosts => ["localhost:9200"]
index => "product"
document_type => "%{type}"
document_id => "%{id}"
}
stdout {codec => rubydebug {metadata => true }}
}
}
è¨å®ããå 容ãå®è¡ãã
è¨å®ããå 容ãå®è¡ããã«ã¯ã以ä¸ã®ã³ãã³ããå®è¡ãã¾ããè¨å®ã«åé¡ããªããã°ãMySQL ã®ãã¼ã¿ãå å·¥ãã Elasticsearch ã¸ã¤ã³ããã¯ã¹ãããã¯ãã§ãã
bin/logstash -f conf/sample.conf
è¨å®å 容ã¯ç°å¢å¤æ°ããåå¾ãããã¨ãã§ãã
Logstash ã®è¨å®ãã¡ã¤ã«å
ã§ã¯ãOS ã®ç°å¢å¤æ°ãåç
§ã§ãããããç°å¢ã«ä¾åããè¨å®å
容ã¯ç°å¢å¤æ°ãåç
§ããããã«ããæ¹ãè¯ãã§ããããç°å¢å¤æ°ã¯åç
§ããã ãã§ãªããããã©ã«ããè¨å®ãã¦ãããã¨ãã§ãæµã®ã§ä¾¿å©ã§ãã port => "${TCP_PORT:3306}â
ã¾ã¨ã
ãããã§ããã§ããããï¼ Logstash ã使ãã°ããªã¬ã¼ã·ã§ãã«ãã¼ã¿ãã¼ã¹ã§ç®¡çããã¦ãããã¼ã¿ãã»ã¨ãã©ããã°ã©ãã³ã°ããå¿ è¦ãªããå å·¥ã㦠Elasticsearch ã¸ã¤ã³ããã¯ã¹ãããã¨ãã§ãã¾ãã
ã¾ããLogstash ã¯æ§ã ãªãã©ã°ã¤ã³ãæ¨æºã§ç¨æãã¦ãã¾ãã®ã§ãä»åç´¹ä»ããä¾ä»¥å¤ã«ãæ§ã ãªãã¼ã¿ã¹ãã¢éã§å©ç¨ãããã¨ãã§ãã¾ãã
ãããã Logstash 使ã£ã¦ã¿ã¦ã¯ã©ãã ãããã