Kafka Streams ã¯ãƒ¬ã‚³ãƒ¼ãƒ‰ã‚’ã©ã®ã‚ˆã†ã«å‡¦ç†ã—ã¦ã„ã‚‹ã®ã‹ - Repro Tech Blog
Platform Team/Repro Core Unit ã®æ‘上ã§ã™ã€‚ Repro ã§ã¯ Kafka を基盤ã¨ã—ãŸã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã®ã‚¢ãƒ—リケーションを構築ã™ã‚‹éš›ã«ã€Kafka Streams ã‚’ç©æ¥µçš„ã«æ´»ç”¨ã—ã¦ã„ã¾ã™ã€‚ Kafka Streams ã¯ã€ãƒ•ã‚©ãƒ¼ãƒ«ãƒˆãƒˆãƒ¬ãƒ©ãƒ³ãƒˆãªã‚¹ãƒ†ãƒ¼ãƒˆãƒ•ãƒ«å‡¦ç†ã‚’ç°¡æ½”ã«å®Ÿè£…ã§ãã€ãƒ‡ãƒ¼ã‚¿ãƒ‘イプラインを Topology ã¨ã„ã†è¡¨ç¾ã§æŠ½è±¡åŒ–ã™ã‚‹ã“ã¨ã§ã€è¤‡é›‘ãªå‡¦ç†ã§ã‚‚管ç†ã—ã‚„ã™ã„å½¢ã§çµ„ã¿ç«‹ã¦ã¦ã„ãã“ã¨ãŒå¯èƒ½ã§ã™ã€‚ ã¾ãŸã€Apache Kafka 以外ã®å¤–部ä¾å˜ãŒãªã„ã“ã¨ã‚„ Streams DSL ã«ã‚ˆã‚‹ã‚·ãƒ³ãƒ—ルãªè¨˜è¿°ã§ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã‚’実装ã§ãã‚‹ã“ã¨ãªã©ã‚‚ã€ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã®ã‚¢ãƒ—リケーションをスムーズã«æ§‹ç¯‰ã™ã‚‹ä¸Šã§åŠ©ã‹ã£ã¦ã„ã¾ã™ã€‚ 一方ã§ã€ ãªã«ã‹ã—らã®å•é¡ŒãŒç™ºç”Ÿã—ãŸã¨ãã®ãƒˆãƒ©ãƒ–ルシューティングや影響範囲調査ã®éš›ã«ã¯ã€Kafka Streams ã®å†…部処ç†ã‚’把æ¡ã—ã¦ã„ãªã„
Kafka Streams ã® State Store ㌠tombstone ã«ã‚ˆã£ã¦é…ããªã£ã¦ã—ã¾ã£ãŸã®ã§ RocksDB ã®ãƒ‘ラメータを調整ã—ã¦ã¿ãŸè©± - Repro Tech Blog
Platform Team ã® Repro Core ã¨ã„ㆠUnit ã«æ‰€å±žã—ã¦ã„ã‚‹æ‘上ã¨ç”³ã—ã¾ã™ã€‚ Repro Core ã®å½¹å‰²ã® 1 ã¤ã¨ã—ã¦ã€å…±é€šåŸºç›¤ã¨ãªã‚‹ Kafka Streams アプリケーションã®é‹ç”¨ãŒã‚ã‚Šã¾ã™ã€‚ ã“ã®å…±é€šåŸºç›¤ã¯ Repro ã®å¤§é‡ãƒˆãƒ©ãƒ•ã‚£ãƒƒã‚¯ã‚’æŒã„ã¦ã„る基盤ã«ãªã‚‹ãŸã‚ã€æ—¥ã€…ã®é‹ç”¨ã®ä¸ã§æ§˜ã€…ãªèª²é¡Œã«ç›´é¢ã—ã¾ã™ã€‚ 今回ã¯ãã®ã‚ˆã†ãªèª²é¡Œã®ä¸ã‹ã‚‰ã€tombstone ã«ã‚ˆã£ã¦ State Store ã®ãƒ‘フォーマンスãŒä½Žä¸‹ã—ã€ãã®è§£æ±ºç–ã¨ã—㦠RocksDB ã®ãƒ‘ラメータを調整ã—ãŸè©±ã‚’ã—ã¾ã™ã€‚ å‰åŠéƒ¨åˆ†ã§ã¯ tombstone ã«ã‚ˆã£ã¦ State Store ã®ãƒ‘フォーマンスãŒä½Žä¸‹ã—ãŸä»¶ã‚’説明ã—ã¾ã™ã€‚後åŠã¯ RocksDB ã® compaction ã®æŒ™å‹•ç¢ºèªã¨ãã®ãƒ‘ラメータ調整ã«ã¤ã„ã¦èª¬æ˜Žã—ã¾ã™ã€‚ ã¡ãªã¿ã«ã€ç§ãŒæ‰€å±žã—ã¦ã„ã‚‹ Repro Core ã«ã¤ã„ã¦ã¯ã€
JBM: General JVM blocking monitor powered by eBPF and async-profiler
LINEã®Kafkaãƒãƒ¼ãƒ ã§ã¯ã€1兆件/day以上ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’å–り扱ã†ä¸–界最大è¦æ¨¡ã®Kafkaクラスタをé‹ç”¨ã—ã¦ã„ã¾ã™ã€‚ 我々ãŒã“ã‚Œã¾ã§ã«ç›´é¢ã—ãŸãƒ‘フォーマンスã«é–¢é€£ã—ãŸå•é¡Œã®å†…ã€æœ€ã‚‚典型的ãªã®ã¯JVMã®ã‚るスレッドãŒä½•ã‹ã—らã®ç†ç”±ã«ã‚ˆã‚Šãƒ–ãƒãƒƒã‚¯ã—ã¦ã—ã¾ã†ã€ã¨ã„ã†ã“ã¨ã«èµ·å› ã—ã¦ã„ã¾ã™ã€‚ 時ã«ã¯ã€ã‚…
Achieving high availability with stateful Kafka Streams applications
Kafka Streams is a java library used for analyzing and processing data stored in Apache Kafka. As with any other stream processing framework, it’s capable of doing stateful and/or stateless processing on real-time data. It’s built on top of native Kafka consumer/producer protocols and is subject to the same advantages and disadvantages of the Kafka client libraries. In this post I’ll try to descri
Kafka Streamを使ã£ãŸã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã®æ¦‚è¦ã¨é‹ç”¨æ™‚ã®è€ƒæ…®ç‚¹ - joker1007’s diary
最近ã€ä»•äº‹ã§åˆ†æ•£ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã«æ‰‹ã‚’出ã—ã¦ã„ã¦ã€ãã®åŸºç›¤ã¨ã—ã¦Apache Kafkaã¨Kafka Streamsを使ã†ã“ã¨ã«ã—ãŸã®ã§ã€å‹•ä½œæ¦‚è¦ã¨ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã®ã‚¤ãƒ¡ãƒ¼ã‚¸ã«ã¤ã„ã¦ã¾ã¨ã‚ã¦ãŠã。 kafkaãã®ã‚‚ã®ã«ã¤ã„ã¦ã¯ä»Šæ›´èª¬æ˜Žã®å¿…è¦ã¯ç„¡ã„ã ã‚ã†ã¨æ€ã†ã€‚ Kafka Streamsã¯kafkaを基盤ã«ã—ã¦åˆ†æ•£ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã‚’ç°¡å˜ã«æ›¸ããŸã‚ã®DSLライブラリ。 https://kafka.apache.org/documentation/streams/ 延々æµã‚Œã¦ãるデータを変æ›ã—ã¦åˆ¥ã®topicã«æµã—ãŸã‚Šã€æ™‚é–“ã®ã‚¦ã‚¤ãƒ³ãƒ‰ã‚¦ã‚’区切ã£ã¦ã‚«ã‚¦ãƒ³ãƒˆã—ãŸçµæžœã‚’æµã—ãŸã‚Šã€ã¿ãŸã„ãªã®ãŒã‚µã‚¯ã£ã¨æ›¸ã‘る。 Apache Flinkãªã‚“ã‹ã¨ä¼¼ãŸæ§˜ãªã“ã¨ãŒã§ãる。 Kafka StreamsãŒè‰¯ã„ã®ã¯ä»¥ä¸‹ã®ç‚¹ã€‚ ãŸã ã®Consumer/Producerã®ãƒ©ãƒƒãƒ‘ーãªã®ã§fat-jarファイル一ã¤ã§ç°¡å˜ã«å‹•ã‹ã›
LINEã®å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ãƒ‘イプラインを支ãˆã‚‹ã€Apache Kafkaプラットフォームã®é‹ç”¨ã®è£å´ - ãƒã‚°ãƒŸãƒ¼Tech
2018å¹´11月21æ—¥ã€LINEæ ªå¼ä¼šç¤¾ãŒä¸»å‚¬ã™ã‚‹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢å‘ã‘技術カンファレンス「LINE DEVELOPER DAY 2018ã€ãŒé–‹å‚¬ã•ã‚Œã¾ã—ãŸã€‚4度目ã®é–‹å‚¬ã¨ãªã‚‹ä»Šå›žã®ãƒ†ãƒ¼ãƒžã¯ã€ŒNext LINEã€ã€‚メッセージアプリã ã‘ã§ãªãã€ã•ã¾ã–ã¾ãªã‚µãƒ¼ãƒ“スã®é–‹ç™ºã‚„æ–°ãŸãªæŠ€è¡“é ˜åŸŸã¸ã®æŠ•è³‡ã‚’è¡Œã£ã¦ã„ã‚‹LINEãŒç›®æŒ‡ã™ãƒ“ジョンã«ã¤ã„ã¦ã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãŸã¡ã®æŠ€è¡“的知見や挑戦を通ã—ã¦ç´¹ä»‹ã—ã¾ã™ã€‚セッション「Multi-Tenancy Kafka cluster for LINE services with 250 billion daily messagesã€ã«ç™»å£‡ã—ãŸã®ã¯ã€LINEæ ªå¼ä¼šç¤¾ Z Part ãƒãƒ¼ãƒ ã®æ²³æ‘勇人æ°ã€‚LINEã§é‹ç”¨ã—ã¦ã„る大è¦æ¨¡Apache Kafkaプラットフォームã®èˆžå°è£ã¨ã€å®Ÿéš›ã«çµŒé¨“ã—ãŸèª²é¡Œã¨è§£æ±ºç–ã«ã¤ã„ã¦ç´¹ä»‹ã—ã¾ã—ãŸã€‚講演資料ã¯ã“ã¡ã‚‰ 大è¦æ¨¡Kafkaプラットフォームã®è£å´
Apache Kafka Best Practices
The document outlines best practices for configuring and optimizing Apache Kafka clusters, including hardware requirements, OS tuning, disk storage management, and monitoring metrics. It emphasizes the importance of replication, partition management, and producer/consumer performance tuning for ensuring reliability and efficiency in streaming data applications. Additionally, it provides guidance o
分散型メッセージングミドルウェアã®è©³ç´°æ¯”較 | POSTD
メッセージã‚ュー ã«ã¤ã„ã¦æ›¸ã„ã¦ã„る連載ã®ç¶šãã¨ã—ã¦ã€ä»Šé€±æœ«ã¯åˆ†æ•£åž‹ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ãƒ³ã‚°ã‚’実行ã™ã‚‹ãŸã‚ã®æ§˜ã€…ãªãƒ©ã‚¤ãƒ–ラリを詳細ã«åˆ†æžã—ã¦ã„ããŸã„ã¨æ€ã„ã¾ã™ã€‚今回ã®åˆ†æžã§ã¯ã€APIã®ç‰¹æ€§ã€ãƒ‡ãƒ—ãƒã‚¤ãƒ¡ãƒ³ãƒˆã‚„メンテナンスã®å®¹æ˜“ã•ã€ãã—ã¦ãƒ‘フォーマンスã®è³ªã‚’å«ã‚ã¦2ã€3種類ã®ç•°ãªã‚‹å´é¢ã«ç€ç›®ã—ã¾ã™ã€‚メッセージã‚ューã¯2ã¤ã®ã‚°ãƒ«ãƒ¼ãƒ—ã«åˆ†é¡žã§ãã¾ã™ã€‚ブãƒãƒ¼ã‚«ãƒ¬ã‚¹ï¼ˆbrokerless）ã¨ãƒ–ãƒãƒ¼ã‚«ãƒ¼ãƒ‰ï¼ˆbrokered）ã§ã™ã€‚ブãƒãƒ¼ã‚«ãƒ¼ãƒ‰ãªã‚ューã¯ã‚¨ãƒ³ãƒ‰ãƒã‚¤ãƒ³ãƒˆé–“ã«ä½•ã‹ã—らã®ã‚µãƒ¼ãƒã‚’挟んã§ã„ã¾ã™ãŒã€ãƒ–ãƒãƒ¼ã‚«ãƒ¬ã‚¹ãªãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚ューã¯ã€ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸é€ä¿¡ã®éš›ã§ã‚‚é–“ã«ä½•ã‚‚挾ã¾ãªã„P2Pã§ã™ã€‚ 今回分æžã™ã‚‹ã®ã¯ä»¥ä¸‹ã®ã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚ ブãƒãƒ¼ã‚«ãƒ¬ã‚¹ nanomsg ZeroMQ ブãƒãƒ¼ã‚«ãƒ¼ãƒ‰ ActiveMQ gnatsd Kafka Kestrel NATS NSQ RabbitMQ Redis å–り掛ã‹ã‚Šã¨ã—ã¦ã€ã»ã¼é–“é•
ストリーム処ç†ã‚’支ãˆã‚‹ã‚ューイングシステムã®é¸ã³æ–¹
This document discusses messaging queues and platforms. It begins with an introduction to messaging queues and their core components. It then provides a table comparing 8 popular open source messaging platforms: Apache Kafka, ActiveMQ, RabbitMQ, NATS, NSQ, Redis, ZeroMQ, and Nanomsg. The document discusses using Apache Kafka for streaming and integration with Google Pub/Sub, Dataflow, and BigQuery
Cloudera Standard ã®ã”案内 ~ 無償版 大幅機能強化ã®ãŠçŸ¥ã‚‰ã› | Cloudera Japan
Cloudera ã¯ã€ã‚¯ãƒ©ã‚¦ãƒ‰ã€ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼ã€ã‚¨ãƒƒã‚¸ã‚’å•ã‚ãšã€ã‚ã‚‰ã‚†ã‚‹å ´æ‰€ã®ãƒ‡ãƒ¼ã‚¿ã« AI ã®åŠ›ã‚’ã‚‚ãŸã‚‰ã™å”¯ä¸€ã®ãƒã‚¤ãƒ–リッドデータ AI プラットフォームä¼æ¥ã§ã™ã€‚ ã‚ã‚‰ã‚†ã‚‹å ´é¢ã§ AI ã®æ´»ç”¨ã‚’åŠ é€Ÿ プライベート AI を利用ã™ã‚‹ã“ã¨ã§ã€é–‹ç™ºã‹ã‚‰æŽ¨è«–ã«è‡³ã‚‹ã¾ã§ã€ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ライズ AI ãŠã‚ˆã³ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆåž‹ AI ã®ã‚¤ãƒŽãƒ™ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ã‚ã‚‰ã‚†ã‚‹å ´é¢ã§åŠ 速ã§ãã¾ã™ã€‚
Rakuten Platform as a Service and Apache Kafka | Rakuten Tech Blog
Rakuten has been running an internal Platform-as-a-Service (PaaS) for over 4 years. Rakuten application teams use our PaaS not only for testing but also for running production scale services. Because of the power of PaaS, we’ve been enabling them great productivity. For example, they can release their application and scale them out horizontally when needed using a single command. We use Cloud Foun
Apache Kafkaã«å…¥é–€ã—ãŸ
Apache kafka 最近仕事ã§Apache Kafkaã®å°Žå…¥ã‚’進ã‚ã¦ã„る．Kafkaã¨ã¯ä½•ã‹? ã©ã“ã§ä½¿ã‚ã‚Œã¦ã„ã‚‹ã®ã‹? ã©ã®ã‚ˆã†ãªç†ç”±ã§ä½œã‚‰ã‚ŒãŸã®ã‹? ã©ã®ã‚ˆã†ã«å‹•ä½œã™ã‚‹ã®ã‹ï¼ˆç‰¹ã«ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã®èªã¿å‡ºã—ã«ã¤ã„ã¦ï¼‰? ã‚’ç°¡å˜ã«ã¾ã¨ã‚ã¦ãŠã（メッセージングã¯ã¾ã ã¾ã 勉強ä¸ãªã®ã§ãŠã‹ã—ãªã¨ã“ã‚ãŒã‚ã‚Œã°ãƒ„ッコミをã„ãŸã ã‘ã‚Œã°å¹¸ã„ã§ã™ï¼‰ï¼Ž ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã¯ 0.8.2 を対象ã«æ›¸ã„ã¦ã„る． Apache Kafkaã¨ã¯? 2011å¹´ã«LinkedInã‹ã‚‰å…¬é–‹ã•ã‚ŒãŸã‚ªãƒ¼ãƒ—ンソースã®åˆ†æ•£ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ãƒ³ã‚°ã‚·ã‚¹ãƒ†ãƒ ã§ã‚る．Kafkaã¯ã‚¦ã‚§ãƒ–サービスãªã©ã‹ã‚‰ç™ºã›ã‚‰ã‚Œã‚‹å¤§å®¹é‡ã®ãƒ‡ãƒ¼ã‚¿ï¼ˆe.g., ãƒã‚°ã‚„イベント）を高スループット/低レイテンシã«åŽé›†/é…ä¿¡ã™ã‚‹ã“ã¨ã‚’目的ã«é–‹ç™ºã•ã‚Œã¦ã„る．公å¼ã®ãƒˆãƒƒãƒ—ページã«æŽ²è¼‰ã•ã‚Œã¦ã„るセールスãƒã‚¤ãƒ³ãƒˆã¯ä»¥ä¸‹ã®4ã¤ï¼Ž Fast ã¨ã«ã‹ã大é‡ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’扱ã†ã“ã¨ãŒã§ãã‚‹ Scal
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Kafka-Streams - Tips on How to Decrease Re-Balancing Impact for Real-Time Event Processing On Highly Loaded Topics
Defense Against the Dark Arts of Rebalancing in Kafka Streams
Quarkus SmallRye KafkaStream OpenTelemetry
Building Pinterest Real-Time Ads Platform Using Kafka Streams
https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Beyond-the-DSL%E2%80%94Unlocking-the-Power-of-Kafka-Streams-with-the-Processor-API-Antony-Stubbs-Confluent-Inc..pdf
https://www.ospn.jp/osc2017.enterprise/pdf/OSC2017.enterprise_Hitachi_Kafka.pdf
{{#tags}}- {{label}}
{{/tags}}