CoreOSで運用するために考えないといけないこと / CoreOS in pixiv

Transcript

1. CoreOSͰӡ༻͢ΔͨΊʹ
 ߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ Harukasan / MICHII Shunsuke CoreOS Meetup Tokyo #1

   2015-04-09

2. Harukasan / MICHII Shunsuke • 2012೥ʹ৽ଔͱͯ͠ΠϯϑϥνʔϜʹ഑ଐ • ίϯςϯπ഑৴Λ͸͡Ίͱͯ͠αʔϏεશମͷج൫Λ୲౰ • 16Gbps͘Β͍ͷը૾഑৴

   • αʔϏεϑϩϯτ(nginx) • Fluentd / Kibana / BigQuery • ٕज़ܥࡶࢽͷࣥචͳͲ΋

3. beer@harukasan http://harukasan.jp/beers • ौ୩ͰϏʔϧΛҿΉΠϕϯτΛ΍͍ͬͯ·͢ • ࠓ݄ͷΠϕϯτ͸໌೔༻ҙ͠·͢ʢͨͿΜ23೔ʁʣ

4. ͸͡Ίʹ • pixivʹ͓͚ΔCoreOS • ͳʹ͕͍ͨ͠ͷ͔ • CoreOSΛ࢖͏ͱԿΛߟ͑ͳ͍ͱ͍͚ͳ͍ͷ͔ • ͜͜1ϲ݄͘Β͍΄ͱΜͲ৮ͬͯͳ͍……

5. CoreOS in pixiv

6. CoreOS in pixiv • Relase 554͔Β࢖͍࢝Ίͨ • Ұ෦ͷϓϩμΫγϣϯ؀ڥͰ࢖༻ • IDCFΫϥ΢υ্ʹߏங

   • ࠷খߏ੒ • ·͍ͩΖ͍Ζͱࢼ͍ͯ͠Δͱ͜Ζ

7. • pixivͷອը࡞඼͔ΒΦεεϝ࡞඼Λ ӾཡɺݕࡧͰ͖ΔAndroid޲͚ΞϓϦ • 2015೥2݄4೔ϦϦʔε • Build with Play framework

   / Scala ΦεεϝϚϯΨΛ΄΅ຖ೔ߋ৽ʂ
 pixivϚϯΨ

8. App App App App LB RPC RPC RPC RPC LB

   pixiv DBs manga-app DB CoreOS CoreOS CoreOS CoreOS ΦϯϓϨϛε IDCF Cloud Request flow Manage Manage L3 Private connect

9. App App App App LB RPC RPC RPC RPC LB

   Manage pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS fleetctl Jenkins pull pploy Deployment flow ΦϯϓϨϛε IDCF Cloud L3 Private connect

10. App App App App LB RPC RPC RPC RPC LB

    pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS Monitoring flow ΦϯϓϨϛε IDCF Cloud L3 Private connect td-agent td-agent td-agent td-agent dd-agent dd-agent dd-agent dd-agent Fluentd Logs Munin Nagios

11. ਓͷԹ͔ΈΛײ͡ΔσϓϩΠ • cloud-configͰઃఆ͕ྲྀ͠ࠐ·ΕΔ • σϓϩΠ৬ਓʹΑΔϩʔϦϯάσϓϩΠ • fleetctlͰϦελʔτ͢Δ͜ͱͰσϓϩΠ͢Δ • ࣗಈԽ͍ͨ͠ export

     FLEETCTL_ENDPOINT=http://app-­‐1:4001/   NUM=1
 fleetctl  stop  app-­‐a@${NUM}.service
 fleetctl  start  app-­‐a@${NUM}.service  

12. ͳͥCoreOSͳͷ͔

13. ͳͥCoreOSΛ࠾༻ͨ͠ͷ͔ • Play framework / ScalaͷΞϓϦέʔγϣϯͷґଘ؅ ཧͱ͔؅ཧ͢Δͷ͕໘౗ͩͬͨͷͰίϯςφʹด͡ ࠐΊ͔ͨͬͨ • ίϯςφ͔͠ಈ͔͞ͳ͍ϗετΛӡ༻͢Δͷ͕໘౗

    • ίϯςφΛಈ͔͚ͩͩͬͨ͢ΒCoreOSͰྑ͍ͷͰ͸

14. ͳʹ͕͍ͨ͠ͷ͔ • ΠϯϑϥνʔϜͱ͸ࣗΒͷ࢓ࣄΛͳ͘͠ଓ͚Δ͜ͱ ͕৬຿ͱͯ͠ཁٻ͞ΕΔࣗݾໃ६ͨ͠ଘࡏͰ͋Δ • ͨͩ͠ؾΛൈ͘ͱ࢓ࣄ͕૿͑ଓ͚ͯࢮ͵ • ΊΜͲ͍͘͜͞ͱ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍

15. ͳΔ΂͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • όʔδϣϯͷࠩҟ • ੬ऑੑ • ґଘؔ܎ • Ϣʔβੜ੒σʔλ •

    etc.

16. ͳΔ΂͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • Immutable • ίϯςφҎ֎ͷঢ়ଶΛม͑ͳ͍ • /usrҎԼ͸ͦ΋ͦ΋ॻ͖׵͑Ͱ͖ͳ͍ • Disposable •

    ίϯςφΛ͍ͭͰ΋ࣺͯΔ͜ͱ͕Ͱ͖Δ • ͳΜ͔͓͔͘͠ͳͬͨΒࣺͯΕ͹ྑ͍

17. CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/fleet/docker • αʔϏε؅ཧ͸͢΂ͯsystemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ •

    fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛ؅ཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰे෼ͩͬͨΒCoreOS͸બ୒ࢶʹͳΔ

18. CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/fleet/docker • αʔϏε؅ཧ͸͢΂ͯsystemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ •

    fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛ؅ཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰे෼ͩͬͨΒCoreOS͸બ୒ࢶʹͳΔ • ͜ΕҎ্ඞཁͳΒKubernetes͕ඞཁʹͳΔ

19. CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • ΂ͭʹSSHͰ͖ͳ͍༁Ͱ͸ͳ͍ • ύοέʔδ؅ཧγεςϜ͸͍Βͳ͍ • ͍͍ͩͨͷ͜ͱ͸systemd͕΍Δ • ։ൃ࣌ʹDockerίϯςφΛಈ͔͢Πϯελϯε
 ͱͯ͠ศར

20. Container Deployment • όΠφϦʹؔ࿈͢Δ͢΂ͯͷґଘؔ܎Λίϯςφͷத ʹด͡ࠐΊΔ • ίϯςφͷ֎ͷঢ়ଶ͸ؾʹ͢Δඞཁ͕ͳ͍ /etc/ *****.conf …

    /usr/local/bin/ ***** … /usr/lib/ **** … /var/ **** … Container

21. Container Deployment • ґଘؔ܎Λ։ൃऀ͕؅ཧͰ͖Δ • Ծ૝ΠϯελϯεΈ͍ͨʹͨ͘͞ΜͷϊʔυΛ1ͭͷ ෺ཧϚγϯʹಉډͤ͞Δͷͱ͸ͪΐͬͱҧ͏ • ͋͘·Ͱ΋ґଘؔ܎Λύοέʔδϯάͨ͠1ϓϩηε ͱࢥͬͨΒΘ͔Γ΍͍͢

22. Web Application in a Container • ΞϓϦέʔγϣϯϓϩηε͸ίϯςφʹ޲͍͍ͯΔ • 1ϓϩηεʢجຊతʹ͸ʣ •

    ґଘ͢ΔϥΠϒϥϦ͕ଟ͍ • ϝϞϦҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ • LBͰϦΫΤετΛεΠονͰ͖Δ

23. CoreOSΛ࢖͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ

24. CoreOSΛ࢖͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ • CoreOSͱ͍͏͔ɺͲ͏΍ͬͯ֎෦ʹґଘ͠ͳ͍ঢ়ଶ Λͭ͘Δ͔ • ·ͩ͏·͍ํ๏͕ݟ͔ͭͬͯͳ͍͜ͱ͕ଟ͍ • γεςϜΦʔέετϨʔγϣϯ •

    σϓϩΠϝϯτ • ϞχλϦϯά • ো֐ରԠ • ࣗಈΞοϓάϨʔυ

25. etcdʹ͓͚Δ࠷খߏ੒ • ΫϥελΛ࡞Δͱ͖͸࠷௿Ͱ΋4୆ʹ͓ͯ͘͠ • 3୆ͩͱ1୆མ͚ͪͨͩͰεϓϦοτϒϨΠϯ

26. ΋͏1୆ඞཁ

27. γεςϜΦʔέετϨʔγϣϯ • fleetͰͰ͖Δͷ͸جຊతʹʮ͜ͷsystemdαʔϏεΛ Ϋϥελ಺Ͱ͍ͭ͘ಈ͔͔͢ʯͱ͍͏͜ͱ͚ͩ • αʔϏεϝϯόͱ͔Ϛελϊʔυͱ͔Λ؅ཧ͢Δػೳ ͸ͳ͍ • ΋ͪΖΜΦʔτεέʔϦϯάͱ͔ͳ͍ •

    etcdΛ࢖ͬͯࣗ෼Ͱ͕Μ͹Δʁ • KubernetesͰ΍Δʁ • LB͘Β͍΋͏ͪΐͬͱ؆୯ʹ΍Γ͍ͨ

28. ίϯςφʹର͢Δ
 ϩʔυόϥϯγϯά • ίϯςφͷstart/stop࣌ʹLBͷAPIΛୟ͘ • systemdͷαʔϏεϑΝΠϧʹίϚϯυΛ௥Ճ͢Δ
 https://github.com/coreos/elb-presence • LBͷAPIΛୟ͚ͩ͘ͳͷͰൺֱత؆୯ •

    ϋʔτϏʔτνΣοΫͰ֎ΕΔ͚ͩͰ΋ྑ͍

29. ίϯςφʹର͢Δ
 ϩʔυόϥϯγϯά • etcd͸Ωʔ͕มߋ͞Εͨͱ͖ʹϑοΫͰ͖Δ • ϑοΫͯ͠LBͷઃఆΛม͑Ε͹ྑͦ͞͏ • vulcand
 https://github.com/mailgun/vulcand •

    confd + nginx
 https://github.com/kelseyhightower/confd

30. σϓϩΠϝϯτ • fleetͩͱϩʔϦϯάϦελʔτͰ͖ͳ͍ • ϩʔϦϯάϦελʔτ͢ΔΑ͏ͳ࢓૊ΈΛͭ͘Δʹ ͸ࣗ෼ͰεΫϦϓτΛॻ͔͘͠ͳ͍ • ଞͷίϯςφ͕ಈ͍͍ͯΔϊʔυͰಈ͔ͳ͍Α͏ʹ ࢦఆ

31. [Unit]   Description=app   After=docker.service   Requires=docker.service   [Service]  

    Restart=always   TimeoutStartSec=0   ExecStartPre=-­‐/usr/bin/docker  kill  app-­‐%i   ExecStartPre=-­‐/usr/bin/docker  rm  app-­‐%i   ExecStartPre=/usr/bin/docker  pull  …   ExecStart=/usr/bin/docker  run  —name  app-­‐%i  …   ExecStop=/usr/bin/docker  stop  app-­‐%i   [X-­‐Fleet]   Conflicts=app@*.service   [email protected]:

32. ϞχλϦϯά • ίϯςφ͝ͱʹϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ • ίϯςφ໊Λ͚ͭͣʹMackerelͰ؂ࢹͯͨ͠Β
 σϓϩΠͷ౓ʹ؂ࢹର৅͕૿͑ͯ΍͹͔ͬͨ • DataDogͰͲ͏ʹ͔ͳͬͯΔ

33. ϞχλϦϯά • ͳʹΛϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ͷ͔ • αʔϏεϨϕϧ • ϊʔυϨϕϧ • ίϯςφϨϕϧ •

    ͳΜͰ΋ݟͨ͘ͳΔ͚Ͳຊ౰ʹશ෦ݟͳ͍ͱ
 ͍͚ͳ͍ͷ͔ߟ͑Δඞཁ͕͋Δ • 1ϊʔυ=1ίϯςφͰಈ͔͢ͷͰ͋Ε͹ࠓ·ͰͲ͓Γ ͷϞχλϦϯάͱͦΜͳʹมΘΒͳ͍

34. ϩάసૹ • ֤ϊʔυʹtd-agentͷίϯςφΛཱ͍ͯͯΔ • cloud-configʹॻ͍ͯىಈͯ͠΋ྑͦ͞͏

35. ো֐ରԠ • ࠓͷͱ͜Ζϗετʹґଘ͢Δো֐ʹૺ۰ͯ͠ͳ͍ • ϋʔτϏʔτ͕ಧ͍ͯΔঢ়گͩͱLB͔ΒखಈͰ
 ֎ͨ͠Γ͠ͳ͍ͱ͍͚ͳ͍͔΋ʁ

36. ࣗಈΞοϓάϨʔυ • ϊʔυ͕མͪΔ͚ͩͱࢥ͑͹ϊʔυো֐ͱมΘΒͳ͍ • ϩοΫΛ͔͚ͳ͕Β΍ͬͯ͘ΕΔͷͰϩʔϦϯάΞο ϓάϨʔυ͞ΕΔʢฒྻ਺΋ࢦఆͰ͖Δʣ • fleetͷόʔδϣϯ্͕͕ͬͨͱ͖ʹ؅ཧϗετଆͷ fleetctl͕ߋ৽͞Εͯͳࣦͯ͘ഊͨ͜͠ͱ͕͋ͬͨ

37. ࣗಈΞοϓάϨʔυ • ৽͍͠΍ͭΛ࢖͍͍͚ͨͲAlpha͸όάͬͯΔͷͰ
 Ͳ͏͠Α͏ • ࣗ෼ͰϏϧυͨ͠Βҙຯͳ͍͠

38. ωοτϫʔΫ • ಛʹωοτϫʔΫΛ੾ΔϞνϕʔγϣϯ͸ͳ͍ • ϙʔτϑΥϫʔσΟϯάͰͦΕ΄Ͳ໰୊ͳ͍ • ϗετωοτϫʔΫͰ΋ྑ͍

39. ηΩϡϦςΟ • CoreOS͕໘౗Έͯ͘ΕΔͷ͸ϗετOS͚ͩ • ίϯςφͷ੬ऑੑ͸ؔ܎ͳ͍ • Dockerͩͱίϯςφʹόʔδϣϯ͕ͳ͍ͷͰ͍ͬͣ͜ ͭ֬ೝ͠ͳ͍ͱ͍͚ͳ͍

40. ·ͱΊ • ݁ہͳΜ΍͔Μ΍ඞཁʹͳΔ • ͦΕKubernetesͰ(ུ • ͦΕGCPͰ(ུ • ίϯςφ͡Όͳͯ͘AMIͰ΋… •

    Ͳ͜·Ͱ΍Δඞཁ͕͋Δ͔ߟ͑Δඞཁ͕͋Δ

41. ·ͱΊ • ίϯςφΛಈ͔͚ͩ͢ͷϗετͰ͋Ε͹CoreOS͸ ͪΐ͏Ͳ͍͍େ͖͞ • े෼ʹγϯϓϧ • γϯϓϧ͚ͩʹߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ͸͋Δ • ຊ౰ʹKubernetesΛࣗ෼ͰϚωδϝϯτ͢Δͷʁ