Elasticsearch at Wantedly

Transcript

1. Elasticsearch at

2. SEIGO UCHIDA

3. !TQFTOPWB w ೥:BIPP+"1"/৽ଔೖࣾ w ೥݄8BOUFEMZʹδϣΠϯ w ޿ٛͷΠϯϑϥΛ୲౰ w $IFGͱ)VCPUͱ͔͕޷͖

4. !TQFTOPWB

5. None

6. γΰτͰίίϩΦυϧਓΛ;΍͢

7. None
8. None

9. 8BOUFEMZ *OD w ೥݄ઃཱ w ΞϧόΠτɺΠϯλʔϯؚΊ໊ w ໊ͷΤϯδχΞ͕׆༂ w γΰτॆ࣮ԽΞϓϦ΍πʔϧͷఏڙ

   w ࣾɺਓ͕࢖͏

10. ੵۃ࠾༻த https://www.wantedly.com/projects/5106

11. WantedlyͰESΛ ͲͷΑ͏ʹ࢖͍ͬͯΔͷ͔

12. ݕࡧର৅ w γΰτ w ձࣾ w Ϣʔβ

13. γΰτ w ձ໊ࣾ w ۈ຿஍ w ࣾһ਺

14. None

15. ืू w ืू໊ w ืूৄࡉ w ืू৬छ

16. None

17. Ϣʔβ w Ϣʔβ໊ w ࣗݾ঺հจ w ͦͷଞ͍Ζ͍Ζʢޙड़ʣ

18. None

19. ΠϯσοΫε w ௨ৗͷݕࡧ༻ΠϯσοΫε w "VUPDPNQMFUF༻ΠϯσοΫε

20. None

21. λΠϓ w γΰτλΠϓ w ձࣾλΠϓ w ϢʔβʔλΠϓ

22. ௨ৗͷݕࡧ༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿ສ w ϓϥΠϚϦαΠζɿ໿(#

23. "VUPDPNQMFUF༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿ສ w ϓϥΠϚϦαΠζɿ໿(#

24. ߏ੒ w 3BJMTPO)FSPLV w &MBTUJDTFBSDIPO&$ w 1PTUHSFTRMPO3%4

25. FMBTUJDTFBSDISVCZ w UJSF͸ϝϯς͕ऴྃ w TFBSDILJDL͸UJSFʹґଘ w ެࣜHFNΛ࢖͏ͷ͕༨ܭͳ੍ݶ͕ͳ ͯ͘ྑ͍

26. Ϋϥελ w &$NTNBMMº୆ w $IFGͰ؅ཧ w +7.ׂ౰.&.ɿ౥ࡌ.&.

27. ར༻ϓϥάΠϯ w )FBE w )2ʢ/PEF%JBHOPTUJDT΍4UBUT֬ೝʹར༻ʣ w .BSWFM TFOTFͰΫΤϦ࡞੒ʹར༻ʣ ։ൃ؀ڥ ͷΈʣ

    w ,VSPNPKJ w *ORVJTJUPSʢ"OBMZ[FSͷڍಈ֬ೝʣ

28. Ͳ͏΍ֶͬͯΜ͔ͩ

29. None

30. ެࣜϦϑΝϨϯε w ඞཁͳ͜ͱ͸ॻ͍ͯ͋Δɺؾ͕͢Δ w ΫΤϦͷྫ͕෦෼తʹॻ͔Ε͍ͯͯࠔΔ w ݕࡧॳ৺ऀʹ͸શମ૾͕͔ͭΈͮΒ͍ w Ͱ΋ಡΜͩ

31. None

32. &MBTUJDTFBSDI8PSLTIPQ w શମ૾Λ௫Έͭͭɺ࣮ࡍʹ৮ΔͷͰΠ ϝʔδ͕௫Ίͨ w IUUQTTQFBLFSEFDLDPNEBEPPOFUFMBTUJDTFBSDIXPSLTIPQ

33. None

34. &MBTUJDTFBSDIνϡʔτϦΞϧ w ༻ޠɺશମ૾͕௫Έ΍͍͢ w ίϯςΩετ͕͋Γ࣮ફత w IUUQDPEFIBUFOBCMPHDPNFOUSZ

35. ͜ͷຊ͕΋ͬͱૣ͘ग़ͯΕ͹ʜ

36. ۤ࿑ͨ͠ͱ͜Ζ

37. Ϣʔβʔݕࡧ w ϓϩϑΟʔϧ߲໨͕ଟ͍ w ֤߲໨ຖʹϓϥΠόγʔϨϕϧ ͕͋Δ w NBQQJOH΋RVFSZ΋ෳࡶʹ

38. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ྫ͑͹ɺϓϩϑΟʔϧͷʮ໊લʯʮॴଐʯʮॴ ࡏ஍ʯΛجຊ৘ใͷηοτͱͯ͠ϓϥΠόγʔ ϨϕϧʮҰൠެ։ʯͱ͍ͯͨ͠ͱ͢Δ w

    ࠷ॳ͸/FTUFEEPDVNFOUΛ࢖ͬͯදݱ͠ ͍ͯͨ

39. "properties": {! "basic_privacy": {! "type": "nested"! },! "properties": {! "name":

    {! "type": "string"! },! "working_for": {! "type": "string"! },! "location": {! "type": "string"! },! "privacy_level": {! "type": "string"! }! }! }! mapping

40. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ͔͠͠ɺ/FTUFEͩͱͰ͖ͳ͍͜ͱʢϋΠϥΠτͳͲʣ ͕ग़ͯ͘ΔͷͰϑϥοτͳߏ଄ͰɺϓϥΠόγʔϨϕ ϧΛදݱ w

    ܕ@σʔληοτ@@ϓϩύςΟͷΑ͏ʹϚοϐ ϯά w ࠷ॳͷܕ͕ඞཁͳͷ͸ɺEBUF΍JOUFHFSͳͲ TUSJOHҎ֎ʹTJNQMF@RVFSZΛ࢖͏ͱFSSPSͱͳΔͨ Ί

41. mapping query {! "query": {! "simple_query_string": {! "query": <QUERY>,! "fields":

    ["s_basic__*"],! "default_operator": "and"! }! }! },! "filter": {! "term": {! "basic_privacy": <PRIVACY_LEVEL>! }! }! }! "properties": {! "basic_privacy": {! "type": "string"! },! "s_basic__name": {! "type": "string"! },! "s_basic__working_for": {! "type": "string"! },! "s_basic__location": {! "type": "string"! }! }!

42. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ݁ՌతʹɺOFTUFEΛ࢖Θ͟ΔΛಘͳ͍৔໘ ͕ग़ͯདྷͯϑϥοτͳϓϩύςΟͱOFTUFE Λซ༻͍ͯ͠Δ͕ɺࠓճ͸ͦ͜͸লུ

43. ࣭໰λΠϜ

44. 4PMS΍$MPVETFBSDI΋ධՁ্ͨ͠Ͱ &4ʹͨ͠ͷ͔ w ʮ༑ୡͱͷܨ͕ΓʯΛՃຯͯ͠ݕࡧͰ͖Δ͔ͱ͍͏͜ͱͰɺάϥϑ %#ͷ/FPKͱҰൠతͳݕࡧͱͯ͠&4Λൺֱͯ͠ɺ&4ͷํ͕அ વύϑΥʔϚϯε͕ྑ͔ͬͨ w 4PMSʹ͍ͭͯ͸4PMSϢʔβ͔Βࠓ͔Β࢖͏ͳΒ&4ͱ͍͏࿩Λฉ͍ ͍ͯͨ w

    ར༻Ϣʔβʢ(JU)VC GPVSTRVBSF 4PVOE$MPVEͳͲʣ΋ࢀߟʹ͠ ͨ w ݁Ռతʹɺ4PMSʹͳ͍OFTUFE QBSFOUDIJMEEPDVNFOUΛ࢖ͬͯ ͍ΔͷͰ&4Ͱྑ͔ͬͨ

45. ։ൃ؀ڥͰ&4Λ࢖͏ͱ NBQQJOHͳͲόϥόϥͰਏ͘ͳΒͳ͍͔ w #PPUTUSBQεΫϦϓτͱ͍͏΋ͷΛ༻ҙͯ͠HJUDMPOFޙ ͷηοτΞοϓʢMPDBMʹ&4ΛೖΕΔͳͲʣΛࣗಈԽ͠ ͍ͯΔ w .BQQJOH΍υΩϡϝϯτͷ௥Ճ΋ηοτΞοϓεΫϦϓ τΛ༻ҙͯ͠ɺϝϯόʔ͝ͱͷ؀ڥͷࠩ෼͕ग़ͳ͍Α͏ʹ ͍ͯ͠Δ

46. ! #! # Check for Elasticsearch! #! if test !

    $(which elasticsearch)! then! echo " x You need to install ElasticSearch:"! echo " + Installing Elasticsearch..."! brew install elasticsearch! else! echo " + ElasticSearch found."! fi! ! #! # Check for Elasticsearch version! #! ES_VERSION=$(elasticsearch -v | cut -d' ' -f2 | cut -d'.' -f1)! if test ! $ES_VERSION -eq 1! then! echo " x You need to upgrade elasticsearch."! brew upgrade elasticsearch! else! echo " + Your Elasticsearch version is correct."! fi! ! if test -d /usr/local/var/lib/elasticsearch/plugins/marvel! then! echo " + Elasticsearch Plugin Marvel found."! else! echo " + Installing Elasticsearch Plugins Marvel"! plugin --install elasticsearch/marvel/latest! fi! bootstrap script Ұ෦ൈਮ $ script/bootstrap! ! Wantedly! ! γΰτͰίίϩΦυϧਓΛ;΍͢! ! + Your Elasticsearch version is correct.! + ElasticSearch Plugin HQ found.! + ElasticSearch Plugin head found.! + ElasticSearch Plugin Kuromoji found.! + ElasticSearch Plugin Inquisitor found.! + ElasticSearch Plugin Marvel found.! ! Good work. We're ready.! ! Run 'elasticsearch' to start elasticsearch.! And you can see elasticsearch webui plugins 'locahost:9200/_plugin/head/'! ࣮ߦ݁Ռ γϯϓϧͳγΣϧεΫϦϓτɻelasticsearch ΍ϓϥάΠϯ͕Πϯετʔϧ͞ΕͯΔ ͔ΛνΣοΫͯ͠͞Εͯͳ͍ͳΒΠϯετʔϧ͢Δɻ ʢ࣮ࡍͷεΫϦϓτͰ͸ elasticsearch Ҏ֎ͷηοτΞοϓ΋΍ͬͯΔʣ

47. elasticsearch-ruby Λ࢖ͬͨ setup script ྫ class ES! def self.client! @client

    ||= Elasticsearch::Client.new({! url: ES.url,! log: true! })! end! end! ! ES.client.indices.create({! index: "index_name",! body: {! settings: {! analysis: {! filter: {! ...! },! tokenizer: {! ...! },! analyzer: {! ...! }! }! }! }! })! ! ES.client.indices.put_mapping({! index: “index_name”,! type: 'user',! body: {! company: {! _id: { path: 'id' },! _timestamp: { enabled: true, path: 'updated_at', store: true },! _source: { enabled: true },! _all: { enabled: true, analyzer: 'kuromoji_analyzer' },! properties: {! name: { type: 'string', store: true, index: 'analyzed', analyzer: 'kuromoji_analyzer' },! . . .! }! }! }! })! ! ! User.find_in_batches do |users|! body = users.map do |user|! {! index: {! _index: "index_name",! _type: 'user',! _id: user.id,! data: user.elasticsearch_data,! }! }! end! ES.client.bulk body: body! end!

48. εΩʔϚมߋ࣌ʹαʔϏεʹӨڹΛ༩͑ͳ͍Α͏ɺ Ͳ͏ΫϥελΛߋ৽ͯ͠Δ͔ w #MVF(SFFO%FQMPZNFOUΈ͍ͨͳ͜ͱ ʢ࣮ࡍ͸৭ʣΛ΍͍ͬͯΔ

49. blue-es.wantedly.com blue-es-elb Nginx + ES Nginx + ES blue-es-cluster green-es.wantedly.com

    green-es-elb Nginx + ES Nginx + ES green-es-cluster چεΩʔϚ ৽εΩʔϚ

50. &4ͷ#MVF(SFFO%FQMPZNFOU w CMVFFTFMC HSFFOFTFMCͷΑ͏ʹ&-# αϒυϝΠϯΛͭ༻ҙ͓ͯ͘͠ w $IFGͰlFMBTUJDTFBSDISPMF lHSFFOzFOWJSPONFOUͱͯ͠&4αʔόΛߏ ங͢ΔͱɺΫϥελϦϯάɺ&-# HSFFOFTFMC

    ΁ͷొ࿥·ͰࣗಈͰηοτ Ξοϓ͞ΕΔΑ͏ʹͯ͠Δɻ෼͘Β͍ɻ w ࣾ಺ϝϯόʔͷΈΞΫηεՄೳͳ؀ڥʹ࠷৽ͷ3BJMTΞϓϦʢ&4ͷ .BQQJOHΛؚΉʣΛσϓϩΠޙɺ͔ͦ͜ΒHSFFOFTDMVTUFSʹ৽εΩʔϚ ͷΠϯσοΫε࡞੒ɺυΩϡϝϯτ௥ՃΛߦ͏ w Ϋϥελʹσʔλ͕ೖͬͨΒɺຊ൪؀ڥͷ3BJMT΋࠷৽ʹͨ͠ޙɺ3BJMT͔Β ར༻͢Δ&4ͷΤϯυϙΠϯτʢ؀ڥม਺Ͱઃఆ͍ͯ͠ΔʣΛ੾Γସ͑ͯσϓ ϩΠ׬ྃ

51. ࠓޙͷ՝୊

52. ࠓޙͷ՝୊ w ύϑΥʔϚϯεͷ޲্ w ϢʔβʔݕࡧͷϋΠϥΠτ w ࣙॻ࢖͏ w ݕࡧਫ਼౓

53. ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ