ニコニコ動画を検索可能にしてみよう
•
32 likes•39,719 views
This document discusses setting up Elasticsearch to make the Nicovideo video dataset searchable and analyzable. It describes importing over 25 billion comments from the 60GB JSON dataset into an Elasticsearch cluster on AWS in under 4 hours. Key steps included installing plugins, configuring the cluster, importing the data in bulk, and optimizing mappings and settings for efficiency. The dataset can now be flexibly searched and facets like date can be applied to analyze the comments.
![shrink
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
"commands" : [ {
"move" :
{
"index" : "nico2", "shard" :33,
"from_node" : "nodeA", "to_node" : "nodeB"
}
}
]}
'](https://image.slidesharecdn.com/es-130829064700-phpapp02/85/-45-320.jpg)
ニコニコ動画を検索可能にしてみよう
- 2. whoami • @PENGUINANA_ / 兼山元太 • エンジニア at *.cookpad.com/* • 検索インフラとサービス開発
- 3. 身の回りのJSON • tweet • 140 character message
- 4. 身の回りのJSON • tweet • 140 character message • user_name • datetime • location • reply or not/contains link or not/ retweeted count/reply count ...
- 5. 身の回りのJSON • access log • ip address • requested content • status code • response time • referrer
- 6. 身の回りのJSON • event log • user_id • event name • params(hash) • datetime • user agent
- 7. 身の回りのJSON • dictionary edit request • keyword • operation type • requester • status(applied or not)
- 9. kibana@cookpad • log dashboard for internal API • explore log • capacity planning • performance check • slowquery
- 10. dashboard for each application
- 13. データセット • ニコニコ動画公式データセット • 800万動画のメタデータ • 25億コメント • JSON形式(圧縮:60G 非圧縮:300G) http://goo.gl/FYtO5T
- 14. データセット • ニコニコ動画公式データセット • 800万動画のメタデータ • 25億コメント • JSON形式(圧縮:60G 非圧縮:300G) http://goo.gl/FYtO5T
- 17. 結果 • Elasticsearch on AWSで4時間でできた • s3 -> unzip -> Elasticsearch(173k doc/s) • 550円
- 25. install • wget https://download.elasticsearch.org/ elasticsearch/elasticsearch/ elasticsearch-0.90.3.noarch.rpm • sudo rpm -i elasticsearch-0.90.3.noarch.rpm
- 26. install plugins • sudo bin/plugin • .. -install elasticsearch/elasticsearch-cloud-aws • .. -install mobz/elasticsearch-head • .. -install lukas-vlcek/bigdesk • .. -install elasticsearch/elasticsearch-analysis-kuromoji
- 27. elasticsearch-cloud-aws • cluster node discovery in AWS • add config to elasticsearch.yml cloud: aws: access_key:AKI........... secret_key: mR............. discovery: type: ec2 discovery.ec2.groups: es_test (security_group)
- 29. bigdesk
- 31. config • # Set a custom allowed content length: • http.max_content_length: 1000m • # Heap Size (defaults to 256m min, 1g max) • ES_HEAP_SIZE=3g • # ElasticSearch data directory • DATA_DIR=/media/ephemeral1/es,/media/ephemeral2/ es,/media/ephemeral3/es
- 32. make AMI • elasticsearch machine image
- 33. launch ES Instances • c1.xlarge x 20 • CPU Xeon 8core(2,300MHz) • Memory 7G • Disk 420G x4 • $0.07/hour(spot instance)
- 35. • download from s3 to nodes • use s3cmd(few minutes with GNU Parallel) • unzip(60GB->300GB) deploy data
- 36. bulk import { "index" : { "_id" : "sm14784868 1", "parent": "sm14784868" } } {"date":"2011-06-18T20:15:30+09:00","no":1,"vpos": 63,"comment":"1","command":"184"} ... { "index" : { "_id" : "sm14784868 2", "parent": "sm14784868" } } {"date":"2011-07-24T02:22:58+09:00","no":2,"vpos": 4651,"comment":"2 get","command":"184"}
- 37. bulk import • ls request_file* | parallel -j N curl -X POST -s -D - 'http:// localhost:9200/nico2/comment/_bulk' -o /dev/null --data- binary @{}
- 39. wc -l requests > 4.8billion
- 40. import... import... import... • all node can handle indexing request • curl bulk import in each node (x20) • I/O into 3 disks • takes 4hours
- 41. efficiency
- 42. efficiency "mappings": { "video": { "properties": { "video_id": { "type": "string", "index": "no" }, "title": { "type": "string", "index": "analyzed" }, "description": { "type": "string", "index": "analyzed" }, "thumbnail_url": { "type": "string", "index": "no", "store": "yes" }, "upload_time": { "type": "date", "format": "YYYY-MM-dd'T'HH:mm:ss'+09:00'" }, "movie_type": { "type": "string", "index": "not_analyzed" }, "last_res_body": { "type": "string", "index": "analyzed" }, "tags": { "properties": { "tag": { "type": "string", "index": "not_analyzed" } } } } }
- 43. efficiency "mappings": { "comment": { "_parent": { "type": "video" }, "properties": { "date": { "type": "date", "format": "YYYY-MM-dd'T'HH:mm:ss'+09:00'" }, "no": { "type": "integer" }, "vpos": { "type": "integer" }, "comment": { "type": "string" }, "command": { "type": "string" }, "video_id": { "type": "string", "index": "not_analyzed" } } }
- 44. efficiency • curl -X POST 'http://localhost:9200/nico2' -d @mapping.json
- 45. shrink curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "move" : { "index" : "nico2", "shard" :33, "from_node" : "nodeA", "to_node" : "nodeB" } } ]} '
- 46. shrink curl -XPUT localhost:9200/_cluster/settings -d '{ "persistent": { "indices.recovery.concurrent_streams": 3 }}' curl -XPUT localhost:9200/_cluster/settings -d '{ "persistent": { "indices.recovery.max_bytes_per_sec": "1000mb" }}'
- 50. Why Elasticsearch? • proven scalable search engine • super flexible config with nice default conf • Great API • growing developer, user base
- 51. not covered • mapping • query DSL • search performance • cluster operation • healthcheck / cluster statistics • etc...
- 52. questions?