CNDT2023 プレイベント 登壇資料
Kafka Streamsã§ä½œã‚‹10万rpsを支ãˆã‚‹ã‚¤ãƒ™ãƒ³ãƒˆé§†å‹•ãƒžã‚¤ã‚¯ãƒã‚µãƒ¼ãƒ“ス
CNDT2023 プレイベント 登壇資料
What Every Software Engineer Should Know about Apache Kafka: Events, Streams, Tables, Storage, Processing, And More
To help fellow engineers wrap their head around Apache Kafka and event streaming, I wrote a 4-part series on the Confluent blog on Kafka’s core fundamentals. In the series, we explore Kafka’s storage and processing layers and how they interrelate, featuring Kafka Streams and ksqlDB. In the first part, I begin with an overview of events, streams, tables, and the stream-table duality to set the stag
Apache Kafkaã§ç™ºç”Ÿã—ãŸåŽŸå› ä¸æ˜Žã®ãƒ‘フォーマンス低下ã¨ã€ãれを解決ã™ã‚‹ãŸã‚ã«ã‚„ã£ãŸã“ã¨
2019å¹´7月17æ—¥ã€kafka.apache.jpãŒä¸»å‚¬ã™ã‚‹ã‚¤ãƒ™ãƒ³ãƒˆã€ŒApache Kafka Meetup Japan #7ã€ãŒLINEæ ªå¼ä¼šç¤¾ã«ã¦é–‹å‚¬ã•ã‚Œã¾ã—ãŸã€‚分散ストリーミングプラットフォーム「Apache Kafkaã€ã«é–¢ã™ã‚‹ãƒŠãƒ¬ãƒƒã‚¸ã‚„æœ€æ–°æƒ…å ±ã‚’å…±æœ‰ã™ã‚‹æœ¬ã‚¤ãƒ™ãƒ³ãƒˆã€‚今回ã¯4人ã®ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãŒã€è‡ªèº«ã‚„自社ã«ãŠã‘る知見を語りã¾ã—ãŸã€‚プレゼンテーション「Kafka Broker performance degradation by mysterious JVM pauseã€ã«ç™»å£‡ã—ãŸã®ã¯ã€LINEæ ªå¼ä¼šç¤¾ã®æ²³æ‘勇人æ°ã€‚ã‚ã‚‹æ—¥Kafkaã«èµ·ã“ã£ãŸçªç„¶ã®ãƒ‘フォーマンス低下ã¨ãã®åŽŸå› ã«ã¤ã„ã¦ã€è§£æ±ºã¾ã§ã®è»Œè·¡ã‚’語りã¾ã—ãŸã€‚講演資料ã¯ã“ã¡ã‚‰ Apache Kafkaã®ãƒ‘フォーマンス低下ã¨ãã®åŽŸå› æ²³æ‘勇人æ°ï¼šã‚ˆã‚ã—ããŠé¡˜ã„ã—ã¾ã™ã€‚最åˆã«è‡ªå·±ç´¹ä»‹ã‚’ã—ã¾ã™ã€‚æ²³æ‘勇人ã¨ã„ã„ã¾ã™ã€‚ LINEã§å…¨ç¤¾å‘ã‘ã®
Kafka gets SQL with KSQL
Apache Kafka is a key component in data pipeline architectures when it comes to ingesting data. Confluent, the commercial entity behind Kafka, wants to leverage this position to become a platform for the enterprise and today is announcing a milestone on the road to ubiquity: SQL. Getty Images/iStockphoto Streaming is hot. The demand for real-time data processing is rising, and streaming vendors ar
内部データパイプラインã¸ã®Kafka Streamsã®é©ç”¨ - LINE ENGINEERING
! This post is also available in the following languages. 英語, 韓国語 By Kawamura Yuto | 2016.08.19 2021.01.08LINEã§ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã¨ã—ã¦åƒã„ã¦ã„ã¾ã™ã€‚Apache Kafkaã€Apache Hbaseãªã©ã®OSS技術をベースã«LINEã®ä¸æ ¸çš„ãªãƒ‡ãƒ¼ã‚¿ãƒ‘イプラインã¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚’開発・é‹å–¶ã—ã¦ã„ã¾ã™ã€‚ Kafka Streamsã®ã”紹介 ã“ã‚“ã«ã¡ã¯ã€‚LINEã§ã‚µãƒ¼ãƒé–‹ç™ºã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã¨ã—ã¦åƒã„ã¦ã„ã‚‹Yuto Kawamuraã§ã™ã€‚主ã«HBaseã€Kafkaã¨ã„ã£ãŸLINEã®ä¸æ ¸çš„ãªã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚’開発・é‹å–¶ã—ã¦ã„ã¾ã™ã€‚ 昨年下期ã‹ã‚‰ã¯ã€IMF(Internal Message Flowã¾ãŸã¯Fund)ã¨å‘¼ã°ã‚Œã‚‹æ–°è¦ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‚‚担当ã—ã¦ã„ã¾ã™ã€‚ã“ã®IMFプãƒã‚¸ã‚§ã‚¯ãƒˆã®ç›®çš„ã¯å¤§ãã2ã¤ã‚ã‚Šã¾ã™ã€‚ 内部シ
Handle Large Messages In Apache Kafka
Like many other messaging systems, Kafka has put limit on the maximum message size. User will fail to produce a message if it is too large. This limit makes a lot of sense and people usually send to Kafka a reference link which refers to a large message stored somewhere else. However, in some scenarios, it would be good to be able to send messages through Kafka without external storage. At LinkedI
Spring Cloud Streamを試ã™(kafkaã§1ã¤ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’複数ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã«æŒ¯ã‚Šåˆ†ã‘ã‚‹) - Qiita
ã¯ã˜ã‚ã« å‰å›žSpring Cloud Streamã§kafkaã«æŽ¥ç¶šã™ã‚‹ã‚µãƒ³ãƒ—ルを試ã—ãŸãŒã€ä»Šå›žã¯kafkaã§ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’å—ã‘å–ã£ã¦ã€å—ã‘å–ã£ãŸãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’複数ã®outputã«æŒ¯ã‚Šåˆ†ã‘ã‚‹multi outputを試ã—ã¦ã¿ã‚‹ã€‚ 環境 å‰å›žã¨åŒã˜ æ‰‹é † å‰å›žä½œã£ãŸ1秒ãŠãã«æ™‚間をメッセージã§é€ã‚‹ã‚µãƒ³ãƒ—ルを元ã«ã€æ™‚é–“ã®ç§’ãŒå¶æ•°ã‹å¥‡æ•°ã‹ã‚’判æ–ã—ã¦ã€ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’å¶æ•°ç”¨ã®topicã¨å¥‡æ•°ç”¨ã®topicã«æŒ¯ã‚Šåˆ†ã‘る処ç†ã‚’作æˆã™ã‚‹ã€‚ kafka接続用プãƒã‚¸ã‚§ã‚¯ãƒˆã‚’作æˆã™ã‚‹ å‰å›ž ※以下ã®ã‚µãƒ³ãƒ—ルも上記ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã‚’æµç”¨ã—ã¾ã™ã€‚ kafkaã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸å—ã‘å–り用ã®è¨å®šã‚’行ㆠapplication.ymlã«å‡ºåŠ›å…ˆã®topicã‚’è¿½åŠ å¶æ•°ç”¨ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã¨å¥‡æ•°ç”¨ã®ãƒ¡ãƒƒã‚»ãƒ¼ã‚¸ã‚’書ã込むtopicを指定 spring: cloud: stream: bindings: output: spring.cloud.
Kafkaを使ã£ãŸ マイクãƒã‚µãƒ¼ãƒ“ス基盤 part2 ＋é‹ç”¨ã—ã¦èµ·ããŸãƒˆãƒ©ãƒ–ル集
@matsu_chara 2016/5/31 Apache Kafka Meetup Japan #1 at Yahoo! JAPANRead less
Apache Kafka, Samza, and the Unix Philosophy of Distributed Data | Confluent
Apache Kafka, Samza, and the Unix Philosophy of Distributed Data One of the things I realised while doing research for my book is that contemporary software engineering still has a lot to learn from the 1970s. As we’re in such a fast-moving field, we often have a tendency of dismissing older ideas as irrelevant – and consequently, we end up having to learn the same lessons over and over again, the
Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)
Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines) I wrote a blog post about how LinkedIn uses Apache Kafka as a central publish-subscribe log for integrating data between applications, stream processing, and Hadoop data ingestion. To actually make this work, though, this "universal log" has to be a cheap abstraction. If you want to use a system as a central data hub
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training (120 slides) covering: 1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka 2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers 3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning 4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serializatio
LINE DEVELOPER DAY 2016 開催ã®ãŠçŸ¥ã‚‰ã› « LINE Engineers' Blog
LINEæ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10月1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ブãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚ LINEヤフー Tech Blog saegusa2017-04-16Yoshihiro was a network engineer at LINE, responsible for all levels of LINE's infrastructure. Since being named Infra Platform Department manager, he is finding ways to apply LINE's technology and business goals to the platform. ã“ã‚“ã«ã¡ã¯ã€‚LINEã§ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚„データセンターを担当ã—ã¦ã„る三æžã§ã™ã€‚2017å¹´1月ã«JANOG39ã§ç™»å£‡ã™ã‚‹æ©Ÿä¼šã‚’é ‚ãã¾ã—ãŸã®ã§ã€ä»Šå›ž
Apache Kafka 概è¦ï¼ˆDesign）和訳ã¾ã¨ã‚ - 夢ã¨ã‚¬ãƒ©ã‚¯ã‚¿ã®é›†ç©å ´
ã“ã‚“ã«ã¡ã¯ã€‚よã†ã‚„ã１ページ全ã¦ã¾ã¨ã‚終ã‚ã£ãŸã®ã§ã€ 最後ã«ã¾ã¨ã‚投稿ã¨ã—ã¦æŠ•ç¨¿ã—ã¦ãŠãã¾ã™ã€‚ å°šã€ãƒšãƒ¼ã‚¸ã¯ä¸‹è¨˜ã§ã™ã€‚ http://kafka.apache.org/07/design.html ï¼ï¼ï¼ï¼ï¼ï¼ï¼ï¼ï¼ï¼ 1.何故Kafkaã¯ä½œã‚‰ã‚ŒãŸã®ã‹ï¼Ÿ 元々ã¯LinkedInã®Activity Streamã¨Data Processingをパイプラインå¼ã«ç¹‹ããŸã‚ã«é–‹ç™ºã•ã‚ŒãŸãƒ—ãƒãƒ€ã‚¯ãƒˆã€‚ 最近ã¯Tumblrã€DataSiftã¨ã„ã£ãŸä¼æ¥ã§ã‚‚使用ã•ã‚Œã¦ã„る。 →　SNSã‚„ã€è¤‡æ•°ã®ã‚µãƒ¼ãƒ“スã®æƒ…å ±ã‚’çµ±åˆã™ã‚‹ã‚ˆã†ãªã‚·ã‚¹ãƒ†ãƒ ã§ä½¿ã‚ã‚Œã¦ã„るよã†ã§ã™ã€‚ ã“ã“ã§ã„ã†Activity Streamã¨ã¯ Webページã§é–²è¦§ã€æ¤œç´¢ã€ãƒªãƒ³ã‚¯è¨å®šãªã©ã‚’è¡Œã†æ´»å‹•å…¨èˆ¬ã‚’指ã™ã€‚ ã“れらã®ãƒ‡ãƒ¼ã‚¿ã¯é€šå¸¸ã®ã‚·ã‚¹ãƒ†ãƒ ãªã‚‰ã°ã€ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã¨ã—ã¦å‡ºåŠ›ã—ã€å¾Œã§åˆ¥é€”解æžã«ç”¨ã„られる。 ã‚‚ã†ä¸€ã¤è¨€è‘‰ã‚’定義ã™ã‚‹ã€‚ Operational
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Heroku Proxying becomes Free Software | Heroku
{{#tags}}- {{label}}
{{/tags}}