Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

Transcript

1. 1 Grafana LokiͰߏங͢Δ େن໛ϩάϞχλϦϯάج൫ CNDT2021 LINEגࣜձࣾ Hiroki Sakamoto / @taisho6339

2. 2 ࣗݾ঺հ - Title: Senior Software Engineer@LINE Corp - Role:

   Private Cloud։ൃ૊৫ͷSRE - Mission Private CloudΛԣஅͨ͠৴པվળ - Interest: Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ - Twitter: @taisho6339

3. like Prometheus but for logs • ϩάͷอଘͱݕࡧػೳ • ϩάϕʔεͷΞϥʔςΟϯάػೳ •

   ϩάϕʔεͷϝτϦΫε࡞੒ػೳ • ϚϧνςφϯτͷDefault Support 3 Lokiͱ͸Կ͔ʁ

4. 4 Lokiͱ͸Կ͔ʁ

5. 5 Lokiͱ͸Կ͔ʁ

6. 6 ҆͘େ༰ྔͷϩάΛอଘՄೳ

7. 7 Private Cloud “Verda” in LINE is based on OpenStack.

   since 2016~ FaaS PaaS IaaS NAT LB Bare metal

8. 8 Private Cloud “Verda” in LINE Virtual Machines 74000+ 30000+

   4000+ Physical Machines Hypervisors

9. 9 20 TB / day application logs

10. 10 Loki is suitable for us!

11. 11 Lokiͷ೉͠͞ Loki͸ϚΠΫϩαʔϏε • ֤ίϯϙʔωϯτɺ֤Ωϟογϡͷ࢓૊Έͱ໾ׂ͕ෆ໌ྎ • ϩάσʔλ͸Ͳ͜ͰͲΜͳܗࣜͰͲͷ͘Β͍อ࣋͞ΕΔ͔ෆ໌ྎ • ετϨʔδো֐࣌͸Ͳ͏͍͏ڍಈʹͳΔͷ͔ෆ໌ྎ •

    ຊ൪Ͱӡ༻͢ΔͳΒԿΛߟྀ͠ͳ͍ͱ͍͚ͳ͍ͷ͔͕ෆ໌ྎ

12. 12 ຊηογϣϯͷΰʔϧ ࠃ಺Ͱ࠷΋ৄࡉʹղઆ͢Δ͜ͱΛ໨ࢦ͠·͢ • ମܥతʹLokiͷίϯϙʔωϯτͷ໾ׂͱ࢓૊ΈΛ஌Δ • τϥϒϧ࣌ʹ΋ݪҼͷಛఆ͕ਝ଎ʹͰ͖ΔΑ͏ʹͳΔ • ࣗྗͰΩϟύγςΟ؅ཧɺύϑΥʔϚϯενϡʔχϯάͰ͖ΔΑ ͏ʹͳΔ

13. 13 ຊηογϣϯ૝ఆߏ੒ Loki Version: v2.3.0 Cache: Memcached Chunk Storage: AWS

    S3 Index Storage: AWS S3 + BoltDB Shipper

14. 14 1) ϩάͷॻ͖ࠐΈϓϩηεΛ஌Δ 2) ϩάͷಡΈࠐΈϓϩηεΛ஌Δ 3) ো֐࣌ͷڍಈΛ஌Δ ຊηογϣϯͷRoadmap

15. 15 1) ϩάͷॻ͖ࠐΈϓϩηε

16. 16 Overview

17. 17 ॻ͖ࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Loki Clients Storage Distributor

    Ingesters Clients (Promtail, Fluentd)

18. 18 ॻ͖ࠐΈϓϩηε Overview Amazon S3 Chunk Cache Index Write Cache

    Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ

19. 19 Amazon S3 Chunk Cache Index Write Cache Loki Clients

    Storage Distributor Ingesters Clients (Promtail, Fluentd) • ϦΫΤετͷόϦσʔγϣϯΛߦ͏ • ద੾ͳIngester΁ϧʔςΟϯά͢Δ ॻ͖ࠐΈϓϩηε Overview

20. 20 Amazon S3 Chunk Cache Index Write Cache Loki Clients

    Storage Distributor Ingesters Clients (Promtail, Fluentd) ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ Ϩʔδʹอଘ ॻ͖ࠐΈϓϩηε Overview

21. 21 Amazon S3 Loki Clients Storage Distributor Ingesters Clients (Promtail,

    Fluentd) • S3ʹϩάΛӬଓԽ • MemcachedʹϩάͷΩϟογϡΛอଘ Chunk Cache ॻ͖ࠐΈϓϩηε Overview

22. 22 Client͔ΒDistributor΁ͷૹ৴

23. 23 Client -> Distributor Clients (Promtail, Fluentd) Distributor HTTP Headers

    X-Scope-OrgID : <Tenant ID> TenantIDΛRequest Headerʹهࡌ

24. 24 Client -> Distributor Clients (Promtail, Fluentd) Distributor {service=“keystone”, hostname=“host1”}

    00:00:02 keystone log body {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body Loki΁ૹΔϩάσʔλߏ଄

25. 25 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body

    Stream Log Body TS

26. 26 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body

    ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ ҰͭҰͭΛɺʮStreamʯͱݺͿ Stream Log Body TS

27. 27 DistributorͰͷόϦσʔγϣϯ • ϥϕϧͷܗࣜ͸ਖ਼͍͔͠ʁ • Rate limitΛ௒͑ͳ͍͔ʁ Clients (Promtail, Fluentd)

    Distributor

28. 28 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor Distributorಉ࢜ͰΫϥελϦϯά • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ •

    ingestion_rate_strategy͕global ͳΒΫϥελશମͰingestion rateΛ੍ޚ͢Δ

29. 29 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor DistributorશମͰόϦσʔγϣϯ ͢ΔͨΊͷΫϥελϦϯά

30. 30 Distributor͔ΒIngester΁ͷૹ৴

31. 31 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ݕࡧͷͨΊʹ(ޙड़)

    ෳ਺ͷIngesterʹ ৑௕Խͯ͠ϩάΛૹΔ

32. 32 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor Ingesterಉ࢜ͰΫϥελϦϯά

    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • Consistent HashͷRingʹͳͬͯ ͍Δ

33. 33 Distributor -> Ingesters Ingester Ingesters Ingester Ingester FNV1-32bitͰHash஋Λੜ੒ tenantID

    + {service=“keystone”, hostname=“host1”} a6965cd7

34. 34 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor a6965cd7

    Consistent Hashʹج͍ͮͯɺ ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester Λreplication factor෼ཁٻ

35. 35 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ؼ͖ͬͯͨෳ਺ͷIngesterʹɺ

    ಉ͡ϩάΛಉ࣌ૹ৴

36. 36 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor OK

    OK Fail

37. 37 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ա൒਺͕OKͳΒ੒ޭ

    OK OK Fail

38. 38 IngesterͷRequest Handling

39. 39 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 {service=“keystone”, hostname=“host1”} Ingester

40. 40 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks StreamͰGrouping͞Εɺ

    Chunkͱ͍͏ܗࣜͰAppend Ingester

41. 41 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester WALʹه࿥

42. 42 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester OK

43. 43 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ Out of order entry error

44. 44 ChunkόοϑΝϦϯά

45. 45 IngesterͷChunkόοϑΝϦϯά • Ұఆ਺ͷϩάΛStream͝ͱʹ ʮChunkʯʹ·ͱΊΔ • Chunk͸ϝϞϦ্ʹอଘ͞ΕΔ • HeadɺBlocksͱ͍͏഑ྻΛอ࣋ Head

    Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk

46. 46 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk {service=“keystone”, hostname=“host1”}

47. 47 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Append ड͚औͬͨϩά͸Ұ୴Head΁ Append͞ΕΔ

48. 48 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢

49. 49 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Log Log Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢

50. 50 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Blockʹ௥Ճ ҰఆαΠζཷ·ͬͨΒ ઃఆͨ͠ܗࣜͰѹॖ͠ɺ blocksʹ௥Ճ͢Δ

51. 51 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Block͕Ұఆ਺ཷ·ͬͨΒ Read Only ModeʹͳΓɺFlush Queue΁ (Default 10 blocks, target chunk sizeͰࢦఆ)

52. 52 Ingester͔ΒStorage΁Flush

53. 53 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Chunk Cache Disk సஔIndex

54. 54 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε ৚݅Λຬͨ͢ChunkΛݕ஌ • ࢦఆαΠζʹ౸ୡ • ࠷ޙͷߋ৽͔Βchunk idle periodܦա • max_chunk_ageܦա

55. 55 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Disk సஔΠϯσοΫε Flush Queue΁Enqueue

56. 56 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Disk Enqueue͞Εͨ΋ͷΛFlush 1. ChunkΛS3΁อଘ 2. Chunk Cache΁อଘ(Write Through) 3. సஔIndexΛϩʔΧϧBoltDBʹอଘ Chunk Cache సஔIndex

57. 57 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Goroutine Goroutine Amazon S3 Disk Ingester΁ͷϦΫΤετ͸ ෳ੡͞Ε͍ͯΔͷͰɺ طʹCacheʹೖ͍ͬͯΔChunk͸ Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk Chunk Cache సஔIndex

58. 58 Cacheͷෛՙ෼ࢄ Chunk Cache Ingesters Chunk Cache Chunk Cache Chunk1

    Key Chunk2 Key Chunk3 Key ઃఆʹΑͬͯConsistent HashʹΑΓɺ ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ

59. 59 Write Ahead Log

60. 60 ChunkͷFlush(࠶ܝ) Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε

61. 61 IngesterͷChunkͷ࣋ͪํ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ Chunk͕شൃ͢Δ

62. 62 WALͷҙٛ ӬଓԽͯ͠Memoryͷشൃʹඋ͑Δ

63. 63 LokiͷWALͷಛ௃ • ϩάड৴࣌ʹɺMemoryͱWAL྆ํʹॻ͖ࠐΉ • WALॻ͖ࠐΈ͕ࣦഊͯ͠΋ॲཧΛࣦഊͤ͞ͳ͍ • Ingesterͷϓϩηεىಈ࣌ʹWAL͔Βͷ෮چॲཧ͕ೖΔ • Ұఆظؒ͝ͱʹෆཁͳWAL͸ύʔδ͞ΕΔ

64. 64 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Log Entry ϩάΛૹ৴

65. 65 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Append SegmentϑΝΠϧʹ rawσʔλͷ··௥ه͞ΕΔ

66. 66 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Create! 1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ ผͷSegmentϑΝΠϧΛ࡞੒ Segment2

67. 67 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Goroutine Disk Segment1 Segment2 Ұఆظؒ͝ͱʹ ෆཁͳSegmentϑΝΠϧͷ ύʔδॲཧ͕ೖΔ

68. 68 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Create! SegmentΛҰͭਐΊΔ

69. 69 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 IngesterͷະFlush ChunkΛ Checkpointͱݺ͹ΕΔ εφοϓγϣοτͱͯ͠อଘ

70. 70 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 MemoryChunkߏ଄ͷ··อଘ͢Δ ͷͰblock͸ѹॖܗࣜɺ headঢ়ଶͷϩά͸ඇѹॖͱͳΔ

71. 71 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment3 Checkpoint1 ࠷৽ͷCheckpointͱSegmentΛ࢒ ͯ͢͠΂ͯ࡟আ͢Δ

72. 72 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment3 Checkpoint1 ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍ ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ

73. 73 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ

74. 74 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ

75. 75 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Stream1 Stream2 Stream1 Stream2 ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍

76. 76 Ingester্ͷσʔλͱEncodeܗࣜ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷

77. 77 2) ϩάͷಡΈࠐΈϓϩηε

78. 78 Overview

79. 79 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Queriers Query Result Cache Ingesters

80. 80 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ϦΫΤετΛड෇͚Δ

81. 81 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ ෼ׂͯ͠ΩϡʔΠϯά͢Δ

82. 82 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery Λड͚औΓϋϯυϦϯά͢Δ Queriers

83. 83 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ͢΂ͯͷIngesterʹରͯ͠ɺ MemoryChunk͔ΒQueryʹMatch ͢Δ΋ͷΛཁٻ Queriers Ingesters

84. 84 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ ͜ͷͱ͖CacheʹಁաతʹΞΫηε Queriers

85. 85 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy) Queriers BoltDB

86. 86 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend సஔIndex͔Βର৅ͷChunkΛׂΓग़͢ Queriers

87. 87 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ChunkΛऔಘ͢Δ Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ Cacheʹอଘ͢Δ Queriers

88. 88 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ Queriers

89. 89 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ݁ՌΛQuery Result Cache΁อଘ͢Δ Queriers Query Result Cache

90. 90 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Ϩεϙϯεฦ٫ Queriers Query Result Cache

91. 91 సஔIndex͔Βର৅Chunkͷબఆ

92. 92 సஔIndexͷ໨త S3͔ΒChunkΛ࠷খ࿑ྗͰऔಘ͢Δ͜ͱ

93. 93 సஔIndex͔Βର৅ChunkͷߜΓࠐΈ 1.LabelͷKeyͱValueͷ૊Έ߹Θ͔ͤΒStreamͷID (Series ID)Λऔಘ͢Δ 2.Series IDͱ࣌ؒൣғ͔ΒChunkͷKeyΛऔಘ͢Δ 3.ChunkͷKey͔ΒS3্ɺMemcached্ͷύεΛׂ Γग़͠ɺ௚઀ChunkΛDownload

94. 94 SeriesID {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body key, valueͷ૊Έ߹Θͤͷsha256

    9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/ ϩά SeriesID

95. 95 Chunk Key {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body Tenant1/a6965cd7:Chunk։࢝࣌ؒ:Chunkऴྃ࣌ؒ

    key, valueͷ૊Έ߹ΘͤͷHash஋ a6965cd7 ϩά FingerPrint Chunk Key

96. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value సஔIndexͷߏ଄Πϝʔδ Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index

97. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ సஔIndexͷߏ଄Πϝʔδ

98. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ

99. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value (Label Value) Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler ςʔϒϧΠϝʔδ సஔIndexͷߏ଄Πϝʔδ

100. ͜ͷIndex͸ LabelͷKeyͱValueͷύλʔϯ෼͚ͩ࡞ΒΕΔ ΧʔσΟφϦςΟͷߴ͍ϥϕϧ͸ ͜ͷIndex͕େྔʹ࡞ΒΕΔ͜ͱʹͳΔ సஔIndexͷߏ଄Πϝʔδ

101. SeriesID͔ΒChunk KeyΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Hash Value Range Value Value TenantID +

     SeriesID Chunkͷ։࢝࣌ؒ + Chunk Key nil

102. 102 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦

103. 103 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦

     IndexΛ࢖͏ͷ͸͜ͷ෦෼ͷධՁ

104. 104 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ෼ׂ

105. 105 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ͦΕͧΕͰϚονϯά৚݅ ͔ΒసஔIndexΛऔಘ సஔIndex

     సஔIndex

106. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

     + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ώοτ͢Δͷ͸͜ͷϨίʔυ

107. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

     + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ର৅SeriesID

108. 108 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} Index͔ΒSeriesIDΛநग़ ྆ํʹڞ௨͢Δ΋ͷ͚ͩ࠾༻ SeriesIDs

     సஔIndex సஔIndex

109. 109 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} SeriesIDͱ࣌ؒൣғ͔Β Chunk KeyΛநग़

     SeriesIDs సஔIndex Chunk Keys సஔIndex సஔIndex

110. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID

     + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠

111. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID

     + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠ ର৅ChunkͷRecord

112. 112 Ϩεϙϯεͷੜ੒ͱฦ٫

113. 113 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks ॳظঢ়ଶͰ͸࣮ମΛ࣋ͨͣɺ ಡΈࠐΈ໋ྩ͕͞ΕͨλΠϛϯάͰετϨʔδ(Ωϟογϡ) ʹChunkΛऔΓʹߦ͘Lazy ChunkΛੜ੒

114. 114 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Ingester͔ΒͷChunkΛ͋Θͤͯɺ IteratorΛੜ੒

115. 115 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ

     IteratorΛಡΈࠐΜͰ͍͘ ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ |= “level=ERROR”

116. 116 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response LazyChunkΛಡΈࠐΉ৔߹͸ɺ

     Ωϟογϡ΁ɺͳ͚Ε͹Storage΁ ChunkΛ໰͍߹ΘͤΔ Amazon S3 Chunk Cache

117. 117 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Storage͔ΒಡΈࠐΜͩ৔߹͸ɺ

     औಘޙʹCacheʹอଘ͢Δ Amazon S3 Chunk Cache

118. 118 Query Sharding

119. 119 Queryͷ෼ׂઓུ 1. ࣌ؒ͝ͱʹ෼ׂ͢Δ • 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4 ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ 2. సஔIndexΛSharding͢Δ •

     ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ ૠೖͯ͠QuerierʹΘͨ͢

120. Hash Value Range Value Value TenantID + LabelName Shard Number

     + Hash(Label Value) + SeriesID Label Value సஔIndex΁ͷShard൪߸ຒΊࠐΈ Shard Number = SeriesID % shard count

121. 121 సஔIndex΁ͷShard൪߸ຒΊࠐΈ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 Shard

     Number Stream

122. 122 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1

     Querier for shard 12 Querier for shard 16 QueryΛShardͰ෼ׂ

123. 123 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”}

     |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys औΕΔChunk͕shardͰ෼ׂ͞ΕΔ

124. 124 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”}

     |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ

125. 125 BoltDB ShipperʹΑΔIndex؅ཧ

126. 126 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index

     1 Index 2 Index 1 Index 2 Amazon S3

127. 127 BoltDB Shipper - Ingester side Ingester Shipper Disk Querier

     Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex ΛS3΁Ξοϓϩʔυ͢Δ Ξοϓϩʔυޙʹ࡟আ͢Δ

128. 128 BoltDB Shipper - Querier side Ingester Shipper Disk Querier

     Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓ μ΢ϯϩʔυ • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β CacheTTLܦաͨ͠IndexΛ࡟আ

129. 129 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index

     1 Index 2 Index 1 Index 2 Amazon S3 Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ

130. 130 ֤ίϯϙʔωϯτͷ໾ׂ·ͱΊ

131. 131 Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗

     Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful Memory: Chunks(raw + ѹॖ) Disk: WAL(raw + ѹॖ) సஔIndex(ѹॖ) ༗ Query Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ Querier ΫΤϦͷ࣮ߦ Stateful Disk: సஔIndexͷCache(ѹॖ) ແ Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ) Index Read Cache IndexͷRead༻Ωϟογϡ Stateful Memory: సஔIndex(Snappy) ༗(ΫϥΠΞϯταΠυ) Index Write Cache ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ (BoltDB ShipperͰ͸ෆཁ) Stateful Memory: Chunk Key(raw) ༗(ΫϥΠΞϯταΠυ) Query Result Cache ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory: Query Result(raw) ༗(ΫϥΠΞϯταΠυ)

132. 132 3) ো֐࣌ͷڍಈΛ஌Δ

133. 133 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ

134. 134 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ S3ো֐࣌ʹඋ͑Δ

135. 135 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

     Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk Segment1 Checkpoint1 ͕ࣦ͜͜ഊ సஔIndex

136. 136 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

     Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 Flush͸ඇಉظͳͷͰ ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍

137. 137 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

     Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ MemoryͱDiskʹͨ·Γଓ͚Δ

138. 138 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

     Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ ઌʹMemory͕͋;ΕɺOOM

139. 139 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

     Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ

140. 140 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

     Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ

141. 141 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

     Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 WAL͸࣮࣭MemoryͷSnapshot ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍

142. 142 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

     Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ϘτϧωοΫ͸IngesterͷMemory

143. 143 Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk

     Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷ Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)

144. 144 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ1. ଱͍͑ͨ࣌ؒ෼ͷMemoryΛੵΉ • ࣌ؒ͋ͨΓͷϩάྔ / ѹॖൺ * ࣌ؒ

     * replication factor ※ѹॖൺ͸ฐ؀ڥͰ͸gzipѹॖͰ10~18ഒͷѹॖൺ

145. 145 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ 1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ) 1000 / 24 = 41.6 GB /

     hour • Replication Factor = 1 • Ingester * 11୆ • Memory: 4GiB • Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ) • Chunk Cache * 14୆ • Memory: 3GiB

146. 146 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Replication Factor͸1Ͱ͍͍ͷ͔ʁ • Flush͞ΕΔ·ͰʹIngesterϓϩηε͕μ΢ϯ͢Δͱͦͷؒ ͚ͩͦʹ͋ͬͨϩά͸ܽଛ͢Δ ͋Δఔ౓ׂΓ੾ΓΛ͢Δ • WAL͕͋ΔͷͰ࠶ىಈޙʹ͙͢ʹ෮چͰ͖Δ

     • ࡉ͔͍ܽଛΑΓো֐࣌ʹՔಇܧଓͰ͖ΔՄೳੑΛߴΊΔํ ޲ʹৼΔ

147. 147 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ2. ো֐࣌͸Ұ࣌తʹWALΛແޮʹͯ͠ىಈ͢Δ • WALϩʔυ͕૸Βͳ͍ͷͰɺMemory͔Β͋;ΕΔྔཷ·͍ͬͯ ͯ΋ϓϩηεىಈͰ͖ΔΑ͏ʹͳΔ • ࠶༗ޮʹ͢Δͱ͖ʹϩά͕شൃ͠ͳ͍Α͏ɺreplication factorɺ

     update strategyʹ഑ྀ͢Δ

148. 148 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ

149. 149 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Amazon

     S3 Chunk Cache

150. 150 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Storageো֐࣌ʹ΋ϩάΛݕࡧ͢ΔͨΊͷ৚݅ • Ingester͕࠷௿Ұͭ͸݈ࡏͰ͋Δ͜ͱ • ݕࡧ݁ՌΛCache͔IngesterͷσʔλͰΧόʔͰ͖Δ͜ͱ • Cacheʹͳ͍࣌ؒൣғΛΫΤϦʹࢦఆ͠ͳ͍͜ͱ

151. 151 ·ͱΊ

152. 152 ·ͱΊ LokiͷίΞͰ͋Δॻ͖ࠐΈͱಡΈࠐΈϓϩηεΛৄղ • ಈ࡞ݪཧ͕Θ͔ͬͨ͜ͱͰɺτϥϒϧγϡʔςΟϯά΍νϡ ʔχϯά͕Մೳʹ • Ͳ͜ͰͲͷΤϯίʔσΟϯάͰσʔλΛ͔࣋ͭΛ೺Ѳ͢Δ͜ ͱͰΩϟύγςΟϓϥϯχϯά͕Մೳʹ •

     ো֐࣌ͷڍಈΛ೺Ѳ͢Δ͜ͱͰద੾ͳ४උ͕ݕ౼Մೳʹ

153. 153 ·ͱΊ ఻͑ΒΕͳ͔ͬͨ͜ͱ • ϩά͔ΒͷϝτϦΫεੜ੒΍ΞϥʔςΟϯά • ϩάͷϦςϯγϣϯ؅ཧʹ͍ͭͯ • LokiࣗମͷϞχλϦϯάʹ͍ͭͯ •

     ֤ίϯϙʔωϯτͷΩϟύγςΟ؅ཧʹ͍ͭͯ • Out of order entry໰୊΁ͷରࡦʹ͍ͭͯ

154. 154 ผ్ຊΛॻ͖·͢

155. 155 Twitter: @taisho6339 ࣭͝໰౳͸

156. 156 THANK YOU