[B! kafka] a_bickyã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

a_bicky id:a_bicky

kafkaã«é–¢ã™ã‚‹a_bickyã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (9)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Kafka Brokerã®compactionå‹•ä½œã®è©³ç´°ã¨ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°æ–¹æ³•ã«ã¤ã„ã¦ - Repro Tech Blog
Reproã§ãƒãƒ¼ãƒ•ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒˆã¨ã—ã¦åƒã„ã¦ã„ã‚‹id:joker1007ã§ã™ã€‚ ä»Šå›žã€Kafka Brokerã®compactionå‹•ä½œã«ã¤ã„ã¦èª¿æŸ»ã—ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°ã™ã‚‹ã“ã¨ã§ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹æ”¹å–„ã®æˆæžœãŒå¾—ã‚‰ã‚ŒãŸãŸã‚ã€ãã®ãƒŽã‚¦ãƒã‚¦ã‚’ãƒ–ãƒã‚°ã«ã¾ã¨ã‚ã¦ãŠãã¾ã—ãŸã€‚ ã‹ãªã‚Šãƒžãƒ‹ã‚¢ãƒƒã‚¯ãªå†…å®¹ãªã®ã§ã€éœ€è¦ã¯å¤šããªã„ã¨æ€ã„ã¾ã™ãŒã€ç§ãŒèª¿æŸ»ã—ãŸé™ã‚Šã§ã¯æ—¥æœ¬èªžã§åŒã˜æ§˜ãªå†…å®¹ã®ãƒ–ãƒã‚°è¨˜äº‹ã¯ã»ã¨ã‚“ã©å˜åœ¨ã—ãªã‹ã£ãŸãŸã‚ã€Kafkaã‚’è‡ªå‰ã§é‹ç”¨ã—ã¦ã„ã‚‹äººã«ã¨ã£ã¦ã¯å½¹ã«ç«‹ã¤å†…å®¹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã€‚ compactionã¨ã¯ (å‚è€ƒ: https://kafka.apache.org/documentation/#compaction) Kafkaã®åŸºæœ¬çš„ãªãƒ‡ãƒ¼ã‚¿å‰Šé™¤ãƒãƒªã‚·ãƒ¼ã¯ä¸€å®šæ™‚é–“ãŒçµŒéŽã—ãŸã‚‰éŽåŽ»ã®ãƒ‡ãƒ¼ã‚¿ã‚’ãã®ã¾ã¾å‰Šé™¤ã™ã‚‹deleteã¨ã„ã†ãƒãƒªã‚·ãƒ¼ã‚’ä½¿ã†ã€‚ ã“ã‚Œã¯ã€log.retention.hoursã¨ã„ã†è¨å®šã§ã‚³ãƒ³ãƒˆãƒãƒ¼
a_bicky 2022/06/27
kafka
ãƒªãƒ³ã‚¯
https://www.ospn.jp/osc2017.enterprise/pdf/OSC2017.enterprise_Hitachi_Kafka.pdf
a_bicky 2022/04/28
kafka
ãƒªãƒ³ã‚¯
Kafkaã®Log Compaction - çƒæ»‘ç¨½
Kafkaã«ã¯Log Compactionã¨ã„ã†æ©Ÿèƒ½ãŒã‚ã‚Šã¾ã™ã€‚ãƒãƒ¼ã‚¸ãƒ§ãƒ³0.8.1ã‹ã‚‰å°Žå…¥ã•ã‚Œã¾ã—ãŸã€‚ ã“ã®Log Compactionã€ã‹ãªã‚Šã¨ã£ã¤ãã¥ã‚‰ã„æ©Ÿèƒ½ãªã‚“ã˜ã‚ƒãªã„ã‹ã¨æ€ã£ã¦ã„ã¾ã™ã€‚Kafkaãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã®ã“ã®ã‚»ã‚¯ã‚·ãƒ§ãƒ³https://kafka.apache.org/documentation/#compactionã€ã¤ã„ã¤ã„èªã¿é£›ã°ã—ã¦ã—ã¾ã†ã‚“ã˜ã‚ƒãªã„ã§ã—ã‚‡ã†ã‹ã€‚ çŸ¥ã‚‰ãªãã¦ã‚‚æ™®é€šã®ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ãªã‚‰ä½¿ã‚ãªãã†ã§ã™ã—ã€‚ ã—ã‹ã—ã“ã„ã¤ã€ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã«ã‚ˆã£ã¦ã¯ä½¿ã£ã¦ã¿ã‚‹ã¨ãªã‹ãªã‹ä¾¿åˆ©ãªã‚·ãƒãƒ¢ãƒŽã§ã™ã€‚ ç´°ã‹ã„æŒ™å‹•ã¨ã‹ã‚‚å…¬å¼ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã‚’ç¢ºèªã—ã¤ã¤è‡ªåˆ†ã®ç†è§£ã‚’æ›¸ã„ã¦ã„ã“ã†ã‹ã¨æ€ã„ã¾ã™ã€‚*1 Log Compactionã¨ã¯ãªã«ã‹ çŠ¶æ…‹ã‚’æŒã¤ãƒ‡ãƒ¼ã‚¿ å®Ÿéš›ã®ã‚·ã‚¹ãƒ†ãƒ ã§è€ƒãˆã‚‹ compactãƒ¢ãƒ¼ãƒ‰ compactionã¯ã„ã¤ã©ã†ã‚„ã£ã¦èµ·ã“ã‚‹ï¼Ÿã©ã®ãƒ‡ãƒ¼ã‚¿ãŒå¯¾è±¡ã«ãªã‚‹ï¼Ÿ Log Compact
a_bicky 2021/10/16
kafka
ãƒªãƒ³ã‚¯
File descriptor limits
a_bicky 2020/06/15
kafka
ãƒªãƒ³ã‚¯
Configuring Kafka for Performance and Resource Management | 2.1.x | Cloudera Documentation
a_bicky 2020/06/15
kafka
ãƒªãƒ³ã‚¯
Apache Kafka Supports 200K Partitions Per Cluster | Confluent
In Kafka, a topic can have multiple partitions to which records are distributed. Partitions are the unit of parallelism. In general, more partitions leads to higher throughput. However, there are some factors that one should consider when having more partitions in a Kafka cluster. I am happy to report that the recent Apache Kafka 1.1.0 release has significantly increased the number of partitions t
a_bicky 2020/06/15
kafka
ãƒªãƒ³ã‚¯
How to Choose the Number of Topics/Partitions in a Kafka Cluster? | Confluent
[Webinar] Donâ€™t Get Left Behind: Unlock the Secrets of Shifting Left | Register Now
a_bicky 2020/06/15
kafka
ãƒªãƒ³ã‚¯
Apache Kafka Rebalance Protocol, or the magic behind your streams applications
Since Apache Kafka 2.3.0, the internal Rebalance Protocol, which is especially used by Kafka Connect and consumers, has undergone several major changes. The Rebalance Protocol is not something simple and can sometimes look like magic. In this post, I propose to go back to the foundation of this protocol, which is at the heart of the Apache Kafka consumption mechanism. Then, we will discuss its lim
a_bicky 2020/03/26
kafka
ãƒªãƒ³ã‚¯
Apache Kafkaã«å…¥é–€ã—ãŸ
Apache kafka æœ€è¿‘ä»•äº‹ã§Apache Kafkaã®å°Žå…¥ã‚’é€²ã‚ã¦ã„ã‚‹ï¼ŽKafkaã¨ã¯ä½•ã‹? ã©ã“ã§ä½¿ã‚ã‚Œã¦ã„ã‚‹ã®ã‹? ã©ã®ã‚ˆã†ãªç†ç”±ã§ä½œã‚‰ã‚ŒãŸã®ã‹? ã©ã®ã‚ˆã†ã«å‹•ä½œã™ã‚‹ã®ã‹ï¼ˆç‰¹ã«ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã®èªã¿å‡ºã—ã«ã¤ã„ã¦ï¼‰? ã‚’ç°¡å˜ã«ã¾ã¨ã‚ã¦ãŠãï¼ˆãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ãƒ³ã‚°ã¯ã¾ã ã¾ã å‹‰å¼·ä¸ãªã®ã§ãŠã‹ã—ãªã¨ã“ã‚ãŒã‚ã‚Œã°ãƒ„ãƒƒã‚³ãƒŸã‚’ã„ãŸã ã‘ã‚Œã°å¹¸ã„ã§ã™ï¼‰ï¼Ž ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã¯ 0.8.2 ã‚’å¯¾è±¡ã«æ›¸ã„ã¦ã„ã‚‹ï¼Ž Apache Kafkaã¨ã¯? 2011å¹´ã«LinkedInã‹ã‚‰å…¬é–‹ã•ã‚ŒãŸã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã®åˆ†æ•£ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ãƒ³ã‚°ã‚·ã‚¹ãƒ†ãƒ ã§ã‚ã‚‹ï¼ŽKafkaã¯ã‚¦ã‚§ãƒ–ã‚µãƒ¼ãƒ“ã‚¹ãªã©ã‹ã‚‰ç™ºã›ã‚‰ã‚Œã‚‹å¤§å®¹é‡ã®ãƒ‡ãƒ¼ã‚¿ï¼ˆe.g., ãƒã‚°ã‚„ã‚¤ãƒ™ãƒ³ãƒˆï¼‰ã‚’é«˜ã‚¹ãƒ«ãƒ¼ãƒ—ãƒƒãƒˆ/ä½Žãƒ¬ã‚¤ãƒ†ãƒ³ã‚·ã«åŽé›†/é…ä¿¡ã™ã‚‹ã“ã¨ã‚’ç›®çš„ã«é–‹ç™ºã•ã‚Œã¦ã„ã‚‹ï¼Žå…¬å¼ã®ãƒˆãƒƒãƒ—ãƒšãƒ¼ã‚¸ã«æŽ²è¼‰ã•ã‚Œã¦ã„ã‚‹ã‚»ãƒ¼ãƒ«ã‚¹ãƒã‚¤ãƒ³ãƒˆã¯ä»¥ä¸‹ã®4ã¤ï¼Ž Fast ã¨ã«ã‹ãå¤§é‡ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’æ‰±ã†ã“ã¨ãŒã§ãã‚‹ Scal
a_bicky 2018/01/16
kafka
ãƒªãƒ³ã‚¯
1