Upgrade to Pro — share decks privately, control downloads, hide ads and more …

クックパッドの機械学習を支える基盤のつくりかた / Machine Learning ops ...

クックパッドの機械学習を支える基盤のつくりかた / Machine Learning ops at Cookpad

AWS Summit Tokyo 2017 w/ https://speakerdeck.com/kanny

http://www.awssummit.tokyo/summit/index.html

Yuichiro Someya

June 01, 2017
Tweet

More Decks by Yuichiro Someya

Other Decks in Programming

Transcript

  1. ؙحؙػحسך 堣 唒㷕 统׾ 佄ִ׷㛇 湍ךא ֻ׶ַ ׋ 2017/06/01

    AWS Summit Tokyo ΫοΫύουגࣜձࣾ છ୩ ༔Ұ࿠ ੕ ๺ే
  2. ࣗݾ ঺հ HJUIVCDPNBZFNPT UXJUUFSDPNBZFNPT@Z XXXBZFNPTNF ‣ છ୩༔Ұ࿠<:VJDIJSP4PNFZB> ‣

    ΫοΫύουגࣜձࣾ
 ݚڀ։ൃ෦ΤϯδχΞ ೥৽ଔೖࣾ
  3. ΫοΫύουͱ"84 "NB[PO&$Πϯελϯε d ୆ఔ౓ "NB[PO4ΦϒδΣΫτ ԯҎ্ ‣ ೥ʹ%$͔Β׬શҠߦ

    ‣ BQOPSUIFBTU͓ΑͼVTFBTUΛओʹར༻ άϩʔόϧαʔϏε͸ओʹ64ͰՔಇ SFRTҎ্ ϐʔΫ࣌ಉ࣌ϦΫΤετ਺
  4. ར༻͍ͯ͠ΔαʔϏε ‣ ࠓճͷ͓࿩ʹؔ܎͢ΔαʔϏε "NB[PO&$ "NB[PO71$ "84-BNCEB
 Ҏ߱&$ 71$

    -BNCEBͱදه "NB[PO424 "NB[PO4/4 Ҏ߱424 4/4ͱදه "NB[PO4 3FETIJGU Ҏ߱4 3FETIJGUͱදه "84$POpH "NB[PO$MPVE8BUDI "84$MPVE5SBJM "84*".
 Ҏ߱$POpH $MPVE8BUDI $MPVE5SBJM *".ͱදه
  5. ΫοΫύουͷݚڀ։ൃ෦ʹ͍ͭͯ ‣ ೥݄ʹൃ଍ ݚڀ։ൃνʔϜ ೥݄d ɺݚڀ։ൃࣨ ೥݄d Λܦͯ


    ݱࡏ͸ਖ਼ࣾһ໊ ΞϧόΠτʗΠϯλʔϯ໊ͷ૊৫ʹ ‣ ػցֶशΛத৺ʹɺ৯จԽܥͷݚڀʹ͓͚Δ࣮੷΋ ΫοΫύουߐށ͝൧ ୈճਓ޻஌ೳֶձશࠃେձ ‣ ྉཧ͖Ζ͘ͷެ։ ೥݄
  6. ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ ‣ ը૾ͷΞοϓϩʔυɺ൑ఆॲཧʹ͸͕͔͔࣌ؒΔ d਺ඦNT "1*αʔόʔͱ౷߹ͯ͠ಉظॲཧ͢Δͷ͸ඇݱ࣮త ඇಉظͳྉཧʗඇྉཧͷ൑ఆॲཧ͕ඞཁ

    ‣ ඞཁͳܭࢉػ؀ڥ͕ΞϓϦέʔγϣϯαʔόʔͱେ͖͘ҟͳΔ 3VCZ˱1ZUIPO 3BJMT˱$IBJOFS $16˱(16 ൑ఆॲཧͱΞϓϦέʔγϣϯΛૄ݁߹ʹ͢Δͷ͕ϕλʔ
  7. ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ ‣ "1*αʔόʔͰ41SF4JHOFE63-Λੜ੒ ੜ੒͞Εͨ63-Λར༻ͯ͠ΫϥΠΞϯτ J04 "OEPJSE ͔Β൑ఆ༻ ʹॖখ͞ΕͨࣸਅΛ4ʹ௚઀Ξοϓϩʔυ

    ൑ఆ༻ը૾ͷ
 Ξοϓϩʔυ 1SF4JHOFE63-
 ͷൃߦ https://s3.amazonaws.com/thumbnails/123.jpg? expires=1495436210&Signature=XXXYYYZZZ
  8. ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ "QQSPYJNBUF"HF0G0MEFTU.FTTBHF &$4$MVTUFS εέʔϧΞ΢τ
 (desired_count++) ‣ 424ͷ$MPVE8BUDIϝτϦΫεʹج͍ͮͨ"MBSNͰXPSLFSΛΦʔτεέʔϧ

    ΞϓϦέʔγϣϯͷੑೳཁٻΛຬͨ͢Α͏ʹ
 ྫ͑͹ɺ৽͘͠Ξοϓϩʔυ͞Εͨࣸਅ͸࣌ؒҎ಺ʹ൑ఆ͞Εͯཉ͍͠ "QQSPYJNBUF"HF0G0MEFTU.FTTBHF͕ར༻Ͱ͖Δ
  9. &$4$MVTUFS ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ "QQSPYJNBUF"HF0G0MEFTU.FTTBHF εέʔϧΞ΢τ
 (desired_count++) ‣ 424ͷ$MPVE8BUDIϝτϦΫεʹج͍ͮͨ"MBSNͰXPSLFSΛΦʔτεέʔϧ

    ΞϓϦέʔγϣϯͷੑೳཁٻΛຬͨ͢Α͏ʹ
 ྫ͑͹ɺ৽͘͠Ξοϓϩʔυ͞Εͨࣸਅ͸࣌ؒҎ಺ʹ൑ఆ͞Εͯཉ͍͠ "QQSPYJNBUF"HF0G0MEFTU.FTTBHF͕ར༻Ͱ͖Δ
  10. &$4্Ͱͷࣸਅ൑ఆ ‣ FBHMFUNUIBLP Λར༻ ZBNMܗࣜͷϑΝΠϧͰ൑ఆॲཧXPSLFSΛఆٛɺ$-*ίϚϯυͰ্ཱͪ͛

    Φʔτεέʔϧͷઃఆ΋IBLPͰߦ͑Δ scheduler: cluster: hako-production-g2 desired_count: 1 autoscaling: min_capacity: 4 max_capacity: 10 policies: - alarms: [cookpadnet-worker-busy] cooldown: 300 adjustment_type: ChangeInCapacity scaling_adjustment: 1 app: image: cookpadnet-worker cpu: 128 memory: 1024 env: AWS_REGION: ap-northeast-1 COOKPADNET_ENV: production ... &$4$MVTUFS "QQSPYJNBUF"HF0G0MEFTU.FTTBHF IUUQTHJUIVCDPNFBHMFUNUIBLP
  11. &$4$MVTUFS H 5BTL $POUBJOFS &$4্Ͱͷࣸਅ൑ఆ ‣ $POUBJOFS͔Β(16Λૢ࡞͢Δҝʹඞཁͳઃఆ /7*%*"%SJWFSͷΠϯετʔϧ


    $MVTUFSʹΠϯετʔϧ͠ɺ5BTL͔Β͸
 WPMVNFͰݟΔ ద੾ͳ-JOVY$BQBCJMJUZͷ෇༩
 QSJWJMFHFEΦϓγϣϯΛར༻ (16 /7*%*"%SJWFS
 LFSOFMNPEVMFT /7*%*"%SJWFS
 VTFSMJCSBSJFT "QQ 8PSLFS
  12. &$4$MVTUFS H &$4্Ͱͷࣸਅ൑ఆ ‣ QSJWJMFHFEͳ5BTL ͷSPPUϢʔβʔ ͸ɺ&$4Ϋϥελ ʹରͯ͠SPPUϢʔβʔͱಉ༷ͳݖݶΛ΋ͭ

    ΫϥελʹӨڹΛ༩͑ΔΑ͏ͳૢ࡞͕Մೳ
 NMPDL LJMM VUJNF ʜ &$4ͰAEFWJDFAΦϓγϣϯ͕࢖͑Δͱʜ ‣ 5BTLͰඇSPPUͳϢʔβʔΛར༻͢Δ
 %PDLFS'JMF಺ͰA64&3XPSLFSA౳ $MVTUFS্ͷ/7*%*"%SJWFS VTFSMJCSBSJFT ͕
 ݟ͑ͳ͘ͳΔ 5BTL $POUBJOFS (16 /7*%*"%SJWFS
 LFSOFMNPEVMFT /7*%*"%SJWFS
 VTFSMJCSBSJFT "QQ 8PSLFS
  13. &$4$MVTUFS H 5BTL $POUBJOFS &$4্Ͱͷࣸਅ൑ఆ ‣ $MVTUFSͱಉ͡όʔδϣϯͷVTFSMJCSBSJFTΛ
 $POUBJOFSʹΠϯετʔϧ

    $POUBJOFS͕$MVTUFSʹόʔδϣϯґଘ͢Δ
 ผͷ$MVTUFSͰ͸ಈ͔ͳ͍͔΋͠Εͳ͍ (16 /7*%*"%SJWFS
 LFSOFMNPEVMFT /7*%*"%SJWFS
 VTFSMJCSBSJFT "QQ 8PSLFS
  14. ੕๺ే ΄͠΄͘ͱ !LBOJ@C ‣ ΫοΫύουגࣜձࣾ
 ΠϯϑϥετϥΫνϟʔ෦෦௕ ೥৽ଔೖࣾ ‣

    43& ηΩϡϦςΟΤϯδχΞ ઃܭߏஙӡ༻ *%4 8"' ੬ऑੑ਍அ *4.4 FUD ‣ "84ೝఆ4" %FW0QTΤϯδχΞ 1SPGFTTJPOBM
  15. தԝ؅ཧͷݶք ‣ ن໛֦େʹ൐͍Πϯϑϥ෦ࣗ਎͕ϘτϧωοΫʹͳΔ͜ͱ͕૿͑ͨ ৽Ϧιʔεͷ࡞੒ ઃܭͳͲͷ૬ஊ ໰͍߹ΘͤͳͲ ‣ զʑ͕ಘҙͱ͢ΔΑ͏ͳӡ༻ΛٻΊΒΕͳ͍΋ͷ͕ग़͖ͯͨ

    ݚڀ։ൃ ‣ ʮͦ΋ͦ΋"84ͷར఺Λ׆͔͍ͤͯͳ͍ͷͰ͸ʯͱ͍͏ٙ໰ ηϧϑϚωʔδͰ͖Δ͜ͱ͕ྑ͞ͷҰͭͰ͋Δ͸ͣ ‣ ຊདྷ΍Δ΂͖͜ͱʹ࣌ؒΛ࢖͑ΔΑ͏ʹ͍ͨ͠ l"84ͷݕূ΍؅ཧzͰ͸ͳ͘lϢʔβʹαʔϏεΛಧ͚Δ͜ͱz͕࢓ࣄ
  16. "84૬ஊ ϊ΢ϋ΢ ‣ "84ʹؔ͢Δϊ΢ϋ΢͸جຊతʹΠϯϑϥ෦͕͍࣋ͬͯͨ ૬ஊ΋΄΅Πϯϑϥ෦͕ड͚Δ ‣ αϙʔτϓϥϯ͸σϕϩούʔ ೥ͷҠߦҎདྷ

    έʔεͷԠ౴ʹ΋͕͔͔࣌ؒΔͨΊجຊతʹࣗ෼ͨͪͰௐࠪ "84΁ͷ໰͍߹Θͤ୅ߦͷΑ͏ʹͳ͍ͬͯͨ͜ͱ΋͋ͬͨ ‣ ຊདྷࣗ෼͕ͨͪղ͔ͳͯ͘΋ྑ͍໰୊ʹ͕͔͔͍࣌ؒͬͯͨ
  17. ͦͷଞ ‣ 5".ΞαΠϯʹΑΓίϯςΩετڞ༗͕͠΍͘͢ έʔεͷऔΓ·ͱΊ΍ྉۚϨϙʔτͷ࡞੒΋͓ئ͍͍ͯ͠Δ ‣ -JNJU*ODSFBTFͷॲཧ͕ര଎ʹ ίϯγΣϧδϡ͕ΞαΠϯ͞ΕͯରԠͯ͘͠ΕΔ

    ࠷୹ ෼ ‣ "84؅ཧӡ༻Ͱ஍ຯʹ͔͔͍ͬͯͨίετ͕͔ͳΓԼ͕ͬͨ "84΁ͷґଘ౓͕ߴ͍૊৫ʹͱͬͯ͸ಛʹ༗༻ͩͱࢥ͏
  18. (16Πϯελϯε H Q ͷར༻ ‣ ϓϩϏδϣχϯάπʔϧ *UBNBF౳ Ͱͷࣗಈ؀ڥߏஙΛݕ౼

    /7*%*"%SJWFS $6%" DV%// ͳͲ͸Πϯετʔϧʹ ͕͔͔࣌ؒΔ ‣ 1BDLFS Λར༻ͯ͠όʔδϣϯຖʹϓϩϏδϣχϯάࡁΈ".*Λ
 ࡞੒ʗ؅ཧ Πϯελϯε্ཱ͕͕͙ͪͬͯ͢(16؀ڥ͕ར༻Ͱ͖Δ VTFBTUͰ͸%FFQ-FBSOJOH".*΋ࢼ༻ʗݕ౼͍ͯ͠Δ IUUQXXXOWJEJBDPN%PXOMPBEJOEFYBTQY MBOHFOVTIUUQXXXOWJEJBDPKQPCKFDUDVEBQBSBMMFMDPNQVUJOHQMBUGPSNKQIUNM IUUQTEFWFMPQFSOWJEJBDPNDVEOOIUUQTXXXQBDLFSJP
  19. ඇಉظͰૄ݁߹ͳ
 ࣸਅ൑ఆϑϩʔ 424 4 ྉཧ͖Ζ͘ͷެ։ &$4(16Ϋϥελʔ )BLPʹΑΔσϓϩΠͱ
 Φʔτεέʔϧ

    &$4 (16ܭࢉػ؀ڥ 1BDLFSΛར༻ͨ͠
 (16".*ͷ࡞੒ͱӡ༻ ".* Πϯελϯεͷ
 མͱ͠๨Ε؂ࢹ Πϯελϯε؅ཧ$IBU#PU -BNCEB $MPVE8BUDI
  20. PR