LINEにおけるOpenStackの運用とUpstreamへの取り組み / our openstack operation and upstream policy

Transcript

1. LINE OpenStack Upstream LINE corporation Verda Platform development

   team Hirose Takahito

2. About Me 2011.08 ~ : livedoor → NHN Japan →

   LINE 2017.02 ~ : Infra platform dept. • Develop the private cloud in-house component(MySQL, approval) 2018.04 ~ : develop/operate the OpenStack Component (Verda platform development team) • In charge of Keystone and Designate

3. Agenda • Introduction of our private cloud • custom requirement

   in Designate • approach to upstream

4. Agenda • Introduction of our private cloud • custom requirement

   in Designate • approach to upstream

5. Introduction of our private cloud • Called‘ Verda ‘ internally

   • Launched at April 2017 • Using ‘Mitaka’ release. ◦ partially “Newton” release

6. Introduction of our private cloud services Identity VM/PM LB DNS

   KaaS (Rancher) FasS Object Storage Block Storage MySQL Redis Elastic search Kafka Managed Service IaaS Verda UI Platform Network Service Storage UI

7. Verda UI

8. Scale of our private cloud clusters Hypervisors VMs Records production

   region 1 700 13242 35000 production region 2 80 538 - production region 3 80 562 - Dev 600 20378 39000 • We have 4 clusters. • There are working single tenant. • Production has 3 regions. • Hypervisors ◦ Prod: about 860 ◦ Dev: about 600 • VMs ◦ Prod: 14342 ◦ Dev: 20378 • In August, we will start to provide new cluster. NEW!

9. Active VMs transition for dev and production Dev: 219% UP⬆

   Prod: 387% UP⏫ 2018/06 3701 VMs 2018/06 9237 VMs 2019/06 14342 VMs 2019/06 20378 VMs

10. Agenda • introduction of our private cloud • custom requirement

    in Designate • approach to upstream

11. Custom requirements in Designate • Disaster Recovery(DR) for DNS ◦

    customize the architecture • How we manage the extra API filter (API Gateway) ◦ paste deploy • Our upgrade strategy ◦ customize source code management

12. DR for DNS

13. Why we need the DR • Even in the case

    of we get the disaster and we can’t operate on region 1, we have to provide LINE service. • There are many development branch ◦ Tokyo, Fukuoka, Kyoto ◦ Korea, China, Taiwan, Thai, Vietnam, Indonesia

14. Pain point to achieve DR • We only had deployed

    the designate service to region 1 ◦ Install the designate • Designate itself doesn't provide methods for Disaster Recovery functionality. ◦ Think how to build a DR • How do we monitor the DR ◦ Use other region or

15. Design of before starting DR project from UI or API

    Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ Shovel A shovel can move messages between brokers in different administrative domains.

16. Design of before starting DR project from UI or API

    Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ Shovel

17. Deployed the designate to Region 2 from UI or API

    Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL

18. Region 1 designate gets disaster. from UI or API Region

    1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL Region 2 MySQL doesn’t have a Region 1 data.

19. Introduce pools configuration • What’s the pools.yml ◦ Designate supports

    the multiple ‘pools’ of DNS service ◦ There is `aslo_notify` and `Target` column in pools.yml that decides how to send the update data on it. ▪ aslo_notify: Optional list of additional IP/Port's for which designate- mdns will send DNS NOTIFY packets to ▪ target: List out the designate-mdns servers from which PowerDNS servers should request zone transfers (AXFRs) from ◦ It can set various backend(PowerDNS, BIND, etc.) from newton.

20. Failed Plan : Set two backend in pools.yml from UI

    or API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL MySQL doesn’t have the same zone serial number

21. Failed Plan: Set the shovel each other from UI or

    API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL

22. Actual Design: Set the cross region replication from UI or

    API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL

23. Actual Design: Set the cross region replication from UI or

    API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL configure the MySQL connection endpoint of region 1

24. Actual Design: Set the multi target on each Designate from

    UI or API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL

25. we need to change the configuration after losing the region

    1 from UI or API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL

26. we need to change the configuration after losing the region

    1 from UI or API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL Update the pools.yml (delete the region 1 information) And Update the ‘Master‘ Column

27. How to monitor the the DNS service from UI or

    API Region 1 API Gateway Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL Rabbit MQ nova compute Rabbit MQ API Gateway Designate PowerDNS MySQL Region 3 Monitor Server

28. How we manage the Extra API Filter

29. Why we need the extra API filter • Don’t register

    unmanaged IP ◦ we need a whitelist • Don’t register unmanaged Domain

30. DNS architecture from UI or API Region 1 API Gateway

    Designate BIND PowerDNS BIND BIND BIND from UI or API Region 2 nova compute MySQL RabbitMQ nova compute RabbitMQ API Gateway Designate PowerDNS MySQL

31. Pain point of this flow • This flow is difficult

    for us to troubleshooting • It takes a maintenance cost ◦ Deploy ◦ New requirement

32. How do we change • Component endpoint has been managed

    by PasteDeploy. ◦ It is easy to use and scalable. ◦ We can inject business logic. We changed the API Gateway(Flask) to WSGI Application and configured to the paste.ini.

33. Registering flow • Paste Deploy can inject a business logic

    before main logic. • Easy to manage • Easy to troubleshooting

34. Example our use case • Request filter ◦ Designate: IP,

    domain ◦ Keystone: user • replace a response format (keystone) ◦ Our Verda provides SSO authentication that uses keystone and apache Mellon. ◦ If keystone can’t find user, return 401 error with JSON format.

35. Our upgrade strategy

36. Why we decide to upgrade designate • Want to use

    a new search API ◦ We have costumed the search API, but it was a little slow ◦ A new recordset api "/v2/recordsets" is exposed with GET method allowed only. The api can be used for retrieving recordsets across all the zones under a tenant. Filtering on certain fields is supported as well. https://docs.openstack.org/releasenotes/designate/newton.html

37. What’s the pain point of upgrade Upstream Designate designate yum

    repo Change 1 Change 2 Change 3 fork Build server Change A Change B Change C The reason is that we were adding the code directly.

38. Difficulty of upgrade Upstream Designate designate yum repo Change 1

    Change 2 Change 3 fork Build server Change A Change B Change C I can not keep up with the change. I don’t know how does it affect

39. Difficulty of upgrade Upstream Designate designate yum repo Change 1

    Change 2 Change 3 fork Build server Change A Change B Change C Summarize all changes

40. How to upgrade a component Upstream Designate line-designate yum repo

    Build server patch A patch C patch B make a new repository for our changes Change A Change B Change C

41. Upgrade a component Directory is separate the cherry-pick and customs

    cherry-pick: from upstream There is two type patch in cherry-pick directory 1. .patch is NOT an original upstream patch. 2. .upstream: this is an original upstream patch. customs: requirements and internal issue

42. How to upgrade and rollback UPGRADE 1. build new package

    2. update a package version on Ansible 3. deploy ROLLBACK 1. stop process of component 2. delete installed package 3. fix a package version on Ansible 4. re-install package Designate

43. How long has the response time improved?

44. Conclusion and future plan in Designate • Conclusion ◦ We

    have a DR architecture for DNS ◦ We have been using a paste-deploy effectively ◦ We have a plan of upgrading a Designate • Future plan ◦ Build Control Plane over k8s ▪ There is an issue on current upgrading way • the upgrade process and rollback process are pain in the neck. ▪ troubleshooting ◦ upgrade to latest release

45. Agenda • introduction of our private cloud • custom requirement

    for our cloud • approach to upstream

46. Our OSS system policy • Don’t believe just document •

    Grasp code and internal State of process running • discussion inside team is based on code

47. Contribute to 1. DB Deadlock issue ◦ https://review.opendev.org/#/c/647711/ 2. service_status

    checker issue ◦ https://review.opendev.org/#/c/657382/ 3. Improve log message for better understanding ◦ https://review.opendev.org/#/c/669154/

48. Contribute to 1. DB Deadlock issue ◦ https://review.opendev.org/#/c/647711/ 2. service_status

    checker issue ◦ https://review.opendev.org/#/c/657382/ 3. Improve log message for better understanding ◦ https://review.opendev.org/#/c/669154/

49. What happened on designate • I got a report from

    the user side. • The record status is ERROR. But it can respond.

50. What happened on designate 2,3 and 7,8 are different order

51. Check log transaction1 transaction2 2. UPDATE zones SET version=(zones.version +

    1), updated_at='2019-02-21 12:30:47.183292', serial=1550752338, status='PENDING', action='UPDATE' WHERE zones.id = '5e5ffc278c5745baae5287c160f54dce' AND zones.deleted = '0' 3. UPDATE records SET version=(records.version + 1), updated_at='2019-02-21 12:30:47.508057', data='ns.example.jp. domain.example.com. 1550752338 3552 600 86400 3600', hash='39795ee18c6e3c9ad1c0190c6a3d8d4f', status='PENDING', action='UPDATE', serial=1550752338 WHERE records.id = '7a655eeda4d446cdaa81caf19ab55fcc' 7. UPDATE records SET version=(records.version + 1), updated_at='2019-02-21 12:30:11.282188', status='ACTIVE', action='NONE' WHERE records.id = '7a655eeda4d446cdaa81caf19ab55fcc' 8. UPDATE zones SET version=(zones.version + 1), updated_at='2019-02-21 12:30:47.178028', status='ACTIVE', action='NONE' WHERE zones.id = '5e5ffc278c5745baae5287c160f54dce' AND zones.deleted = '0'

52. DB Deadlock

53. Summarize for internal • When we got the something error.

    We summarize a report for internal.(share how to resolve)

54. Check bug report I sometimes find the same situation bug

    report. So I check the report before submitting a patch. https://bugs.launchpad.net/designate/+bug/1785459

55. backport to supported(not EOL) openstack releases We never forgot to

    backport to supported OpenStack release. Because this is our motivation for upgrading.

56. Session from LINE 7/22 7/23

57. Session from LINE • [Related Event] Rancher Day • :

    7/24 11:30~ • : • https://eventregist.com/e/rdt2019 •

58. Open Infrastructure Summit Shanghai • Open Infrastructure Summit • https://www.openstack.org/summit/shanghai-2019/

    • November 4-6, 2019 • • Submitted 2 CFP. • How we used RabbitMQ in wrong way at a scale • Multi-Site, Shared Zone, Extra API Filter...How we brought Designate up in Our Production Private Cloud

59. Thank you