Introducing the Confluent Operator: Apache Kafka® on Kubernetes
Introducing the Confluent Operator: Apache Kafka on Kubernetes Made Simple At Confluent, our mission is to put a Streaming Platform at the heart of every digital company in the world. This means, making it easy to deploy and use Apache Kafka and Confluent Platform—the de-facto Streaming Platform—across a variety of infrastructure environments. In the last few years, the rise of Kubernetes as the c
Kafka vs Pulsar @KafkaMeetup_20180316
2. Copyright (C) 2018 Yahoo Japan Corporation. All Rights Reserved. ç„¡æ–引用・転載ç¦æ¢ 2 自己紹介 æ —åŽŸ 望 経æ´ï¼š â–ª 2012/04 ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«æ–°å’入社 â–ª 2012/10 ユーザーã®å±žæ€§æƒ…å ±ã«é–¢é€£ã™ã‚‹ç¤¾å†…å‘ã‘プラットフォームã®é–‹ç™º â–ª 2015/07 ヤフオクã®BEシステムå†æ§‹ç¯‰ â–ª 2016/10 「Pulsarã€ã‚’使ã£ãŸç¤¾å†…å‘ã‘メッセージングプラットフォームã®é–‹ç™º â–ª 2017/06~ 「Pulsarã€ã®ã‚³ãƒŸãƒƒã‚¿ãƒ¼ 趣味: â–ª ã·ã‚ˆã·ã‚ˆãƒ†ãƒˆãƒªã‚¹ â–ª ボードゲームã„ã‚ã„ã‚ 3. Copyright (C) 2018 Yahoo Japan Corporation. All Rights Reserved. ç„¡æ–引用・転載ç¦æ¢ 3 Apache Pulsar Yahoo! Inc.（ç¾ï¼šOath）ã§é–‹ç™ºã•ã‚ŒãŸ
Multitenancy: Kafka clusters for everyone at LINE
Yuto Kawamura from LINE Corporation presented on their use of Apache Kafka clusters to provide multitenancy for different internal teams. They face challenges in ensuring isolation between client workloads and preventing abusive clients. Their solutions include request quotas to limit client resource usage, slow logs to identify slow requests, and changes to the broker code to pre-warm caches and
Outshift | Kafka on Kubernetes - using etcd
Get emerging insights on innovative technology straight to your inbox. At Banzai Cloud we are building a cloud agnostic, open source next generation CloudFoundry/Heroku-like PaaS, Pipeline, while running several big data workloads natively on Kubernetes. Apache Kafka is one of the cloud native workloads we support out-of-the-box, alongside Apache Spark and Apache Zeppelin. If you’re interested in
Squeezing the firehose: getting the most from Kafka compression
Squeezing the firehose: getting the most from Kafka compression2018-03-05 We at Cloudflare are long time Kafka users, first mentions of it date back to beginning of 2014 when the most recent version was 0.8.0. We use Kafka as a log to power analytics (both HTTP and DNS), DDOS mitigation, logging and metrics. Firehose CC BY 2.0 image by RSLab While the idea of unifying abstraction of the log remain
Should You Put Several Event Types in the Same Kafka Topic? | Confluent
Should You Put Several Event Types in the Same Kafka Topic? If you adopt a streaming platform such as Apache Kafka, one of the most important questions to answer is: what topics are you going to use? In particular, if you have a bunch of different events that you want to publish to Kafka as messages, do you put them in the same topic, or do you split them across different topics? The most importan
Blog Posts | October 1, 2024 - February 15, 2024 | @lightbend
Webinar: Akka 24.05 release highlights Tyler Jewell, CEO, Jonas Boner, founder and CTO, and Michael Nash, CISO, delve into the value and power of these new features and enhancements. Q&A to follow. Lightbend aims to democratize distributed systems for developers Tyler Jewell, CEO of Lightbend, talks about some of the key challenges developers are up against with distributed systems and how Lightbe
Apache Kafkaを使ã£ãŸã‚¢ãƒ—リè¨è¨ˆã§åçœã—ã¦ã„る件をæ£ç›´ãƒ™ãƒ¼ã‚¹ã§è©±ã™
Apache Kafka: Producer, Broker and Consumer2017å¹´ã¯ç”Ÿã¾ã‚Œã¦å§‹ã‚ã¦Apache Kafkaã‚’æœ¬æ ¼çš„ã«æ¥å‹™åˆ©ç”¨ï¼ˆPoCã§ã¯ãªã本番é‹ç”¨ï¼‰ã—ãŸå¹´ã§ã—ãŸã€‚Apache Kafkaçš„ãªãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ãƒ³ã‚°ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ãã®ã‚‚ã®ã®ã¯ã€ç¤¾å†…çš„ãªäº‹æƒ…ã§ã‚ˆã使ã£ã¦ã„ãŸã®ã§ãã®ä½¿ã„å‹æ‰‹ã«å¯¾ã—ã¦å›°æƒ‘ã¯ã»ã¨ã‚“ã©ç„¡ã‹ã£ãŸã§ã™ã—ã€ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã¨ã—ã¦éžå¸¸ã«å®‰å®šã—ã¦ã„ã‚‹ãŸã‚ã€Kafkaクラスタãã®ã‚‚ã®ã§ã®ä¸å…·åˆã‚‰ã—ã„ä¸å…·åˆãŒç™ºç”Ÿã—ãŸã“ã¨ã¯ä¸€åº¦ã‚‚ã‚ã‚Šã¾ã›ã‚“ã§ã—ãŸã€‚ ã—ã‹ã—ã€Kafkaã®ãƒˆãƒ”ックè¨è¨ˆãªã©ã«é–¢ã—ã¦ã®ãƒ™ã‚¹ãƒˆãƒ—ラクティスã¯äº‹ä¾‹ãƒ™ãƒ¼ã‚¹ã§ã‚‚ã‚ã¾ã‚Šè¦‹ã‹ã‘ãŸã“ã¨ãŒãªãã€ãƒãƒ¼ãƒ メンãƒãƒ¼ã¨æ‚©ã‚€ã“ã¨ã‚‚多ã‹ã£ãŸã§ã™ã€‚ã“ã®ã‚¹ãƒˆãƒ¼ãƒªãƒ¼ã§ã¯ã€ä¸»ã«Kafkaを利用ã—ãŸã‚¢ãƒ—リè¨è¨ˆã§è€ƒãˆãŸã“ã¨ã‚„失敗ã—ãŸã“ã¨ã‚’振り返りã¤ã¤å…±æœ‰ã—ã¾ã™ã€‚ãªãŠã€ãƒ‘ーティション数やå„種ãƒãƒƒãƒ•ã‚¡ã‚µã‚¤ã‚ºãªã©ã®ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚°è¦ç´ ã¯ä»Šå›žå–
Improving Kafka at-least-once performance at Uber
At Uber, we are seeing an increasing demand for Kafka at-least-once delivery (asks=all). So far, we are running a dedicated at-least-once Kafka cluster with special settings. With a very low workload, the dedicated at-least-once cluster has been working well for more than a year. When trying to allow at-least-once producing on the regular Kafka clusters, the producing performance was the main conc
Kafka at half the price with JBOD setup
The talk introduces JBOD setup for Apache Kafka and shows how LinkedIn can save more than 30% storage cost in Kafka by adopting JBOD setup. The talk is given during the LinkedIn Streaming meetup in May, 2017.Read less
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications 1) Apache Kafka is a distributed system with many moving parts to monitor, including brokers, topics, partitions, and the applications that use Kafka. It is critical to monitor Kafka performance to ensure high availability and catch problems early. 2) Key metrics to monitor include partition replication, broker resource usa
Apache Spark Streaming + Kafka 0.10 with Joan Viladrosariera
Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, wi
Apache Kafka Goes 1.0
It has been seven years since we first set out to create the distributed streaming platform we know now as Apache Kafka®. Born initially as a highly scalable messaging system, Apache Kafka has evolved over the years into a full-fledged distributed streaming platform for publishing and subscribing, storing, and processing streaming data at scale and in real-time. Since we first open-sourced Apache
Kafkaコンシューマã®ãƒªãƒãƒ©ãƒ³ã‚¹ã¯ã„ã¤è¡Œã‚れる㋠- n-agetsumaã®æ—¥è¨˜
Kafkaã®ã‚³ãƒ³ã‚·ãƒ¥ãƒ¼ãƒžã‚°ãƒ«ãƒ¼ãƒ—ã«ã€ã‚³ãƒ³ã‚·ãƒ¥ãƒ¼ãƒžã‚’è¿½åŠ ã™ã‚‹ã¨ã€å„コンシューマãŒã©ã®ãƒ‘ーティションをèªã¿å–ã‚‹ã‹å†å‰²ã‚Šå½“ã¦ã™ã‚‹ãƒªãƒãƒ©ãƒ³ã‚¹ãŒè¡Œã‚れる。 Kafka0.11.0.1を対象ã«ã€ä»¥ä¸‹ã®è¦³ç‚¹ã§Kafkaã®å®Ÿè£…を調ã¹ã¦ã¿ãŸçµæžœã‚’ã¾ã¨ã‚る。 リãƒãƒ©ãƒ³ã‚¹ã¯ã„ã¤è¡Œã‚れるã®ã‹ ãªãœæ–°è¦ã‚³ãƒ³ã‚·ãƒ¥ãƒ¼ãƒžè¿½åŠ ã—ã¦ã‚‚ã€ã™ãã«ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸é…ä¿¡ãŒå§‹ã¾ã‚‰ãªã„ã®ã‹ リãƒãƒ©ãƒ³ã‚¹ãŒèµ·ã“ã£ãŸæ™‚ã«é‡è¤‡ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸å‡¦ç†ã¯ç™ºç”Ÿã—ãªã„㋠リãƒãƒ©ãƒ³ã‚¹ã¯ã„ã¤è¡Œã‚れるã®ã‹ KafkaConsumer.poll()メソッドã®å†…部ã§è¡Œã‚れる。 Kafkaã®ã‚³ãƒ³ã‚·ãƒ¥ãƒ¼ãƒžã®å®Ÿè£…ã¯ã ã„ãŸã„以下ã®ã‚ˆã†ã«ãªã‚‹ã€‚コード全体ã¯gistå‚照。 Kafka Comsumer Sample · GitHub KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties()); co
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Instaclustr Kafka Meetup Sydney Presentation
Monitoring Apache Kafka
End-to-End Exactly-Once Processing in Apache Flink with Apache Kafka
KIP-101 - Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation - Apache Kafka - Apache Software Foundation
Kafka 2018 - Securing Kafka the Right Way
https://jobs.zalando.com/tech/blog/rock-solid-kafka/index.html
{{#tags}}- {{label}}
{{/tags}}