Streaming Dataframes
This work is supported by Anaconda Inc and the Data Driven Discovery Initiative from the Moore Foundation This post is about experimental software. This is not ready for public use. All code examples and API in this post are subject to change without warning. Summary This post describes a prototype project to handle continuous data sources of tabular data using Pandas and Streamz. Introduction Som
Google Cloud Dataflow ã£ã¦ãªã‚“ã ？ - Qiita
æ ªå¼ä¼šç¤¾WACULã€CTOã®åŒ…ã§ã™ã€‚ 会社ã®ã¿ã‚“ãªã§ãƒ‘ンを作ã£ãŸã‚Šã‚«ãƒ¬ãƒ¼ã‚’作ã£ãŸã‚Šæ˜ 画をã¿ãŸã‚Šã¨æ¥½ã—ã„日々をéŽã”ã—ã¦ã¾ã™ã€‚ ã“ã“最近ã€Google Cloud Dataflow ã‚’ã„ã˜ã£ã¦ã¿ã¦ã„ã‚‹ã®ã§ã€å…¥é–€ã—ãŸãƒ¡ãƒ¢ã¨ã—ã¦ã€æ•´ç†ã—ã¦ã¿ã¾ã—ãŸã€‚ ã“ã‚Œã‹ã‚‰è§¦ã£ã¦ã¿ã‚ˆã†ã¨ã™ã‚‹äººãŒæœ€åˆã«èªã‚“ã§ã€ã–ã£ãり概è¦ã‚’ç†è§£ã™ã‚‹åŠ©ã‘ã«ãªã‚Œã°ã¨æ€ã„ã¾ã™ã€‚ ストリーミングやãƒãƒƒãƒå‡¦ç†ã«æ·±ã„専門知è˜ãŒã‚ã‚‹ã‚ã‘ã§ã¯ãªã„ã®ã§ã€é–“é•ã£ãŸã“ã¨ã‚’言ã£ã¦ã„ãŸã‚‰æ•™ãˆã¦ã„ãŸã ã‘ã‚‹ã¨ã†ã‚Œã—ã„ã§ã™ã€‚ ã“ã®è¨˜äº‹ã‚’èªã‚“ã§ã‚ã‹ã‚‹ã“㨠Google Cloud Dataflow ã®ã ã„ãŸã„ã®æ¦‚è¦ã¨ç›®æŒ‡ã—ã¦ã„る世界感 ã ã„ãŸã„ã®ã‚³ãƒ¼ãƒ‰ã®é›°å›²æ°— 概è¦ã‚’知ã£ãŸä¸Šã§æ¬¡ã«èªã‚€ã¹ãå‚考資料 ã“ã®è¨˜äº‹ã‚’èªã‚“ã§ã‚‚ã‚ã‹ã‚‰ãªã„ã“㨠Dataflow 上ã§ã‚³ãƒ¼ãƒ‰ã‚’書ã„ã¦ã„ãã¨ãã®å…·ä½“çš„ãªçŸ¥è¦‹ é‹ç”¨ä¸Šã®ã®çŸ¥è¦‹ ã¾ãšã¯å‚考資料 ã“ã‚Œã¾ã§ã«èªã‚“ã Google Cloud
Jasper - A flexible and powerful issue reader for GitHub
You will be able to read, track and discover many active issues very flexibly by using Jasper in GitHub. It is enabled by a powerful function called "Stream" of Jasper. example repo:nodejs/node is:issue label:bug You can use the GitHub Search query to create various criteria, such as issues I created, pull requests of nodejs/node, bug labeled issues, etc. to view issues and receive update notifica
Noze.io - Evented I/O streams for Swift
Noze.io is an attempt to carry over the Node.js ideas into pure Swift. It uses libdispatch for event-driven, non-blocking I/O. Noze.io is built around type-safe back-pressure aware pull-streams (using Swift generics) operating on batches of items. Instead of working with just bytes, deal with batches of Unicode lines or database records or HTML responses or - you get the idea. Be efficient: Stream
ストリーム処ç†ã¨ã¯ä½•ã‹ï¼Ÿï¼‹2016å¹´ã®å‡ºæ¥äº‹ - Qiita
ãã®å¯¾å‡¦ã§å…¨éƒ¨ã«å¯¾å¿œã™ã‚‹ã®ã¯ç„¡ç†ãªã‚“ã˜ã‚ƒãªã„ã®ï¼Ÿ Watermarkã€Triggerã€Accumulationã®æ©Ÿæ§‹ãŒå°Žå…¥ã•ã‚Œã‚Œã°ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã¯å…¨ã¦å¯¾å¿œå¯èƒ½ã‹ã¨ã„ã†ã¨ã€ ãã‚“ãªã“ã¨ã¯ã‚ã‚Šã¾ã›ã‚“。 何故ãªã‚‰ã€ä¸‹è¨˜ã®ã‚ˆã†ãªå•é¡ŒãŒç™ºç”Ÿã—ã¦ãã‚‹ã‹ã‚‰ã§ã™ã€‚ Watermarkを実時刻ã‹ã‚‰ã©ã‚Œãらã„é…らã›ã¦è¨å®šã™ã‚Œã°ã„ã„ã®ã‹ï¼Ÿ é…れを大ããã™ã‚Œã°æ£ç¢ºæ€§ã¯å¢—ã—ã¾ã™ãŒã€é…延時間ã¯å¤§ãããªã‚Šã¾ã™ã€‚ Accumulationã®ãŸã‚ã«ã‚¦ã‚£ãƒ³ãƒ‰ã‚¦ã®é›†è¨ˆçµæžœã‚’ã©ã‚Œã ã‘ä¿æŒã™ã‚Œã°ã„ã„ã®ã‹ï¼Ÿ ä¿æŒã™ã‚‹æ™‚é–“ãŒé•·ã„ã»ã©ã€ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã‚’è¡Œã†ã‚·ã‚¹ãƒ†ãƒ ã®ãƒªã‚½ãƒ¼ã‚¹ãŒå¿…è¦ã¨ãªã‚Šã¾ã™ã€‚ データ処ç†ã‚·ã‚¹ãƒ†ãƒ （ãƒãƒƒãƒã€ã‚¹ãƒˆãƒªãƒ¼ãƒ å«ã‚€ï¼‰ã«ã¯ä¸‹è¨˜ã®ï¼“è¦ç´ ã®ãƒˆãƒ¬ãƒ¼ãƒ‰ã‚ªãƒ•ãŒã‚ã‚‹ã¨ã•ã‚Œã¦ã„ã¾ã™ã€‚ 完全性(Completeness) 低é…延(Low Latency) 低コスト(Low Cost) ã“ã®ï¼“è¦ç´ ã‚’å…¨ã¦ã«æº€ãŸã™ã“ã¨ã¯å‡ºæ¥ãšã€å…¨ã¦ã®ãƒ‡ãƒ¼ã‚¿
IoT時代ã«ãŠã‘るストリームデータ処ç†ã¨æ€¥æˆé•·ã® Apache Flink
ãˆã€ã¾ã£ã¦ã€‚ãã®ä¸¦åˆ—分散処ç†ã€Kafkaã®ã—ãã¿ã§ã‚‚ã§ãã‚‹ã®ï¼Ÿ Apache Kafkaã®æ©Ÿèƒ½ã‚’利用ã—ãŸå¤§è¦æ¨¡ã‚¹ãƒˆãƒªãƒ¼ãƒ データã®ä¸¦åˆ—分散処ç†
akka-stream を始ã‚ã‚‹ã¨ãã«è¦šãˆã¦ãŠããŸã„ã“ã¨
akka-stream 特有ã®å˜èªžã‚„概念を分ã‹ã‚Šã‚„ã™ã説明ã—ã¾ã™ã€‚
KAYAC Engineers' Blog
ã“ã‚“ã«ã¡ã¯ã€AI推進室ã®æ‘井ã§ã™ã€‚ 先日ã€åŒã˜ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆãƒ¡ãƒ³ãƒãƒ¼ã®å¤§è„‡ãŒã€AI副æ‘長「ãã£ã·ã¡ã‚ƒã‚“ã€ã®ã‚¢ãƒ¼ã‚テクãƒãƒ£ã¨æŠ€è¡“é¸å®šã«ã¤ã„ã¦æ›¸ãã¾ã—ãŸã€‚ techblog.kayac.com ãã£ã·ã¡ã‚ƒã‚“ã¯ã€ GitHub ã® README ã«ã‚‚書ã„ãŸé€šã‚Šã€AIã‚’å˜ãªã‚‹é“å…·ã¨ã—ã¦ç½®ãã®ã§ã¯ãªãã€ãã®åœ°åŸŸã®æ–‡è„ˆã‚’æŒã£ãŸå˜åœ¨ã¨ã—ã¦å—ã‘入れã¦ã‚‚らãˆã‚‹ã®ã‹ã€‚ãã‚“ãªä»®èª¬ã‚’実験ã—ã¦ã„るプãƒã‚¸ã‚§ã‚¯ãƒˆã§ã™ã€‚ コンセプトã¨ã—ã¦ã¯ã€Œå˜åœ¨ã™ã‚‹ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã€â”€â”€ ãŸã 機能ã™ã‚‹ã ã‘ã®é“å…·ã§ã¯ãªãã€æ‘ã®ä¸€å“¡ã®ã‚ˆã†ã«ãã“ã«ã„ã‚‹ã¨æ„Ÿã˜ã‚‰ã‚Œã‚‹å˜åœ¨ã«ãªã‚Œã‚‹ã‹ã‚’試ã—ã¦ã„ã¾ã™ã€‚ github.com ã¾ãŸã€ã€ŒAI副æ‘é•·ã€ã¨ã„ã†åå‰ã«ã¯ã€ã‚„ãŒã¦æ‘ã®ãƒ ードやã€ä»Šã“ã®æ‘ã§èµ·ãã¦ã„ã‚‹ã“ã¨ã‚’æ•™ãˆã¦ãれるã€æ‘é•·ã®ãƒ‘ートナーã®ã‚ˆã†ãªå˜åœ¨ã«ãªã£ã¦ã„ã‘ãŸã‚‰ã€ã¨ã„ã†æ€ã„ã‚‚è¾¼ã‚ã¦ã„ã¾ã™ã€‚ ãã£ã·ã¡ã‚ƒã‚“ãŒç›®æŒ‡ã—ã¦ã„ã‚‹ã®ã¯ã€ã²ã¨è¨€ã§ã„ãˆã°ã€Œã‚ã£ã¡ã‚ƒåœ°å…ƒã®AI
Questioning the Lambda Architecture
The brightest minds in data and AI come together at the O'Reilly Strata Data & AI Conference to develop new skills, share best practices, and discover new tools and technologies. Join us. Nathan Marz wrote a popular blog post describing an idea he called the Lambda Architecture (“How to beat the CAP theorem“). The Lambda Architecture is an approach to building stream processing applications on top
Streaming 102: The world beyond batch
Editor's note: This is the second post in a two-part series about the evolution of data processing, with a focus on streaming systems, unbounded data sets, and the future of big data. See part one. Also, check out "Streaming Systems," by Tyler Akidau, Slava Chernyak, and Reuven Lax. Check out the resource centers on O'Reilly online learning for more ways to expand your skillset. IntroductionWelcom
Streaming 101: The world beyond batch
Editor's note: This is the first post in a two-part series about the evolution of data processing, with a focus on streaming systems, unbounded data sets, and the future of big data. See part two. Also, check out "Streaming Systems," by Tyler Akidau, Slava Chernyak, and Reuven Lax. Check out the resource centers on O'Reilly online learning for more ways to expand your skillset. Streaming data proc
Kafka Streams - Qiita
Deleted articles cannot be recovered. Draft of this article would be also deleted. Are you sure you want to delete this article? Kafkaã§ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã‚’書ã‘るよã†ã«ãªã£ãŸ 2016/06 時点ã®æœ€æ–°ãƒªãƒªãƒ¼ã‚¹ v0.10 ã‹ã‚‰ã€Kafkaã«ã‚¹ãƒˆãƒªãƒ¼ãƒ 処ç†ã®ã‚¢ãƒ—リケーションを書ããŸã‚ã®ãƒ©ã‚¤ãƒ–ラリãŒå…¥ã£ãŸã€‚Kafka本体ã«åŒæ¢±ã•ã‚Œã¦ã„ã‚‹ã®ã§è¿½åŠ ã§ä½•ã‹ã‚’インストールã™ã‚‹å¿…è¦ã¯ãªã„。ã“ã®ãƒ©ã‚¤ãƒ–ラリを使ã†ã¨ã€ 「Kafkaã®ãƒˆãƒ”ックAã«ãƒ‡ãƒ¼ã‚¿ãŒå…¥ã£ã¦ããŸã‚‰ã€å³åº§ã«ã«ãれを処ç†ã—ã¦åˆ¥ã®ãƒˆãƒ”ックBã«æ ¼ç´ã™ã‚‹ã€ ã¨ã„ã†ã‚¢ãƒ—リケーションを簡å˜ã«ä½œã‚‹ã“ã¨ãŒã§ãる。ãªãŠã€Kafkaã«åŒæ¢±ã•ã‚Œã¦ã„ã‚‹ã‹ã‚‰ã¨ã„ã£ã¦ã€Kafka本体ã€ã™ãªã‚ã¡ãƒ–ãƒãƒ¼ã‚«ãƒ¼å´ã«ä½•ã‹ç‰¹æ®Šãªä»•æŽ›ã‘ãŒå°Žå…¥ã•ã‚ŒãŸã‚ã‘ã§
TCP serverã‚’SSL/TLS化ã™ã‚‹ã®ã« nginx ã® stream_ssl_module/stream_proxy_module ãŒä¾¿åˆ© - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
最近 Fluentd ã®é€šä¿¡ãƒ—ãƒãƒˆã‚³ãƒ«ã¾ã‚りをアップデートã™ã‚‹ãŸã‚ã«ã‚ã‚Œã“ã‚Œã„ã˜ã£ã¦ã„ã‚‹*1ã‚“ã ã‘ã©ã€ã“ã‚Œã¯ãŠãŠã‚€ã fluent-plugin-secure-forward ãŒã‚µãƒãƒ¼ãƒˆã—ã¦ã„ãŸå†…容を Fluentd 組込ã¿ã® forward plugin ã§ã‚‚サãƒãƒ¼ãƒˆã—ã¾ã™ã‚ˆã€ã¨ã„ã†ã‚‚ã®ã«ãªã‚‹ã€‚ ã‚“ã§å•é¡Œãªã®ãŒ secure-forward 㯠SSL/TLS ã§ã®æŽ¥ç¶šã®ã¿ã—ã‹ã‚µãƒãƒ¼ãƒˆã—ã¦ãªã‹ã£ãŸã‚“ã ã‘ã© forward ã§ã¯ç”Ÿã® TCP ã§é€šä¿¡ã™ã‚‹*2ã®ã§ã€æœ¬å½“ã« secure-forward 㨠forward ãã‚Œãžã‚Œã®å®Ÿè£…é–“ã§äº’æ›æ€§ãŒä¿ãŸã‚Œã¦ã„ã‚‹ã®ã‹ã€ç›´æŽ¥çš„ã«ã¯ç¢ºèªã™ã‚‹æ‰‹æ®µãŒãªã„ã€ã¨ã„ã†ã“ã¨ã«ãªã£ã¦ã—ã¾ã†ã€‚ TCP server ã® SSL/TLS 化 一方世ã®ä¸ã«ã¯ SSL/TLS ターミãƒãƒ¼ã‚¿ã¨ã„ã†æ©Ÿèƒ½ãŒã‚ã£ã¦ã€ãŸã¨ãˆã°ãƒãƒ¼ãƒ‰ãƒãƒ©ãƒ³ã‚µãªã‚“ã‹ãŒã“ã®æ©Ÿèƒ½ã‚’æŒã£ã¦ã„る。何をやるã‹ã¨
#golang stream caching - djherbis
djherbis/fscache Objective Our Web server needs to perform an “expensive†process to create downloads. We want to cache these files so they don’t have to be generated on every request. If two or more concurrent clients ask for the same file, don’t generate it twice. A naive cache design might look like this: Well the above code is pretty bad. We’re blocking all requests while the cache is being wr
RxJS ã®ä»£ã‚ã‚Šã« xstream を使ㆠ- Qiita
xstream を開発ã™ã‚‹å‹•æ©Ÿ xstream ã®é–‹ç™ºè€…ã§ã‚ã‚‹ Staltz ã•ã‚“㯠RxJS ã®ã‚³ãƒ³ãƒˆãƒªãƒ“ューターã§ã‚‚ã‚ã‚Šã€RxJS を使ã£ãŸãƒ•ãƒ¬ãƒ¼ãƒ ワークã§ã‚ã‚‹ Cycle.js ã®ä½œè€…ã§ã™ã€‚Staltz ã•ã‚“㯠WHY WE BUILT XSTREAM ã®è¨˜äº‹ã®ãªã‹ã§ xstream を開発ã™ã‚‹å‹•æ©Ÿã‚’è¿°ã¹ã¦ã„ã¾ã™ã€‚ ストリームã¨ã¯ä½•ã‹ã€ãƒªã‚¢ã‚¯ãƒ†ã‚£ãƒ–プãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã¨ã¯ä½•ã‹ã«ã¤ã„ã¦ã¯ã€Œã‚ãªãŸãŒæ±‚ã‚ã¦ã„ãŸãƒªã‚¢ã‚¯ãƒ†ã‚£ãƒ–プãƒã‚°ãƒ©ãƒŸãƒ³ã‚°å…¥é–€ã€ã®è¨˜äº‹ãŒå‚考ã«ãªã‚Šã¾ã™ã€‚「Everything is streamã€(ã™ã¹ã¦ãŒã‚¹ãƒˆãƒªãƒ¼ãƒ ) ãŒãƒžãƒ³ãƒˆãƒ©ã§ã™ã€‚ xstream ã®ä½ç½®ã¥ã‘㯠RxJS を補ã†ã‚‚ã®ã§ã‚ã‚Šã€ä¸€èˆ¬çš„ãªãƒªã‚¢ã‚¯ãƒ†ã‚£ãƒ–プãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã®ç”¨é€”ã§ã¯ RxJS ã®ã»ã†ãŒã™ãã‚Œã¦ã„ã‚‹ã¨ã®ã“ã¨ã§ã™ã€‚ xstream ã®ã‚ªãƒšãƒ¬ãƒ¼ã‚¿ãƒ¼ã®æ•°ã¯ RxJS ã¨æ¯”ã¹ã¦å°‘ãªãã€åˆå¿ƒè€…ã«ã‚ã‹ã‚Šã‚„ã™ã„ç›´æ„Ÿçš„ãªåå‰ã«ã—
SensorBee: An open source stream processing engine for IoT
Stateful Extracting useful information from unstructured data often requires machine learning. SensorBee works well with online machine learning algorithms by storing machine learning models as user-defined states. It supports various machine learning toolkits, including Chainer and some components of Jubatus. In addition, SensorBee’s query language enables stateful temporal data handling. Express
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Kafka Gets Streaming SQL Engine, KSQL
https://www.confluent.io/wp-content/uploads/2016/08/Making_Sense_of_Stream_Processing_Confluent_1.pdf
Kafka Streams for Confluent Platform | Confluent Documentation
Confluent Documentation | Confluent Documentation
{{#tags}}- {{label}}
{{/tags}}