[B! Spark] yukiyan_wã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yukiyan_w id:yukiyan_w

Sparkã«é–¢ã™ã‚‹yukiyan_wã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (15)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Deequ ã§å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿å“è³ªã‚’ãƒ†ã‚¹ãƒˆ | Amazon Web Services
Amazon Web Services ãƒ–ãƒã‚° Deequ ã§å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿å“è³ªã‚’ãƒ†ã‚¹ãƒˆ ä¸€èˆ¬çš„ã«ã€ã‚³ãƒ¼ãƒ‰ç”¨ã®ãƒ¦ãƒ‹ãƒƒãƒˆãƒ†ã‚¹ãƒˆã‚’æ›¸ãã¨æ€ã„ã¾ã™ãŒã€ãŠä½¿ã„ã®ãƒ‡ãƒ¼ã‚¿ã‚‚ãƒ†ã‚¹ãƒˆã—ã¦ã„ã‚‹ã®ã§ã—ã‚‡ã†ã‹? ä¸æ£ç¢ºã¾ãŸã¯ä¸æ£ãªãƒ‡ãƒ¼ã‚¿ã¯ã€æœ¬ç•ªã‚·ã‚¹ãƒ†ãƒ ã«å¤§ããªå½±éŸ¿ã‚’ä¸Žãˆã‚‹å¯èƒ½æ€§ãŒã‚ã‚Šã¾ã™ã€‚ãƒ‡ãƒ¼ã‚¿å“è³ªå•é¡Œã®ä¾‹ã¯æ¬¡ã®ã¨ãŠã‚Šã§ã™ã€‚ å€¤ãŒãªã„å ´åˆã¯ã€æœ¬ç•ªã‚·ã‚¹ãƒ†ãƒ ã§ null ä»¥å¤–ã®å€¤ã‚’å¿…è¦ã¨ã™ã‚‹ã‚¨ãƒ©ãƒ¼ (NullPointerException) ãŒç™ºç”Ÿã™ã‚‹å¯èƒ½æ€§ãŒã‚ã‚Šã¾ã™ã€‚ ãƒ‡ãƒ¼ã‚¿åˆ†å¸ƒãŒå¤‰åŒ–ã™ã‚‹ã¨ã€æ©Ÿæ¢°å¦ç¿’ãƒ¢ãƒ‡ãƒ«ã§äºˆæœŸã—ãªã„å‡ºåŠ›ã«ã¤ãªãŒã‚‹ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ ãƒ‡ãƒ¼ã‚¿ã®é›†è¨ˆã‚’èª¤ã‚‹ã¨ã€ãƒ“ã‚¸ãƒã‚¹ã§ã®åˆ¤æ–ã‚’ä¸‹ã™éš›ã«èª¤ã£ãŸæ„æ€æ±ºå®šã«ã¤ãªãŒã‚‹å¯èƒ½æ€§ãŒã‚ã‚Šã¾ã™ã€‚ ã“ã®ãƒ–ãƒã‚°è¨˜äº‹ã§ã¯ã€Amazon ã§é–‹ç™ºã—ã€ä½¿ç”¨ã•ã‚Œã¦ã„ã‚‹ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ãƒ„ãƒ¼ãƒ«ã§ã‚ã‚‹ Deequ ã‚’ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚Deequ ã§ã¯ã€ãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã®ãƒ‡ãƒ¼ã‚¿å“è³ªãƒ¡ãƒˆãƒªã‚¯ã‚¹
yukiyan_w 2020/10/17
Spark
ãƒªãƒ³ã‚¯
How Scala works at Mercari
Production-ready stream data pipeline in Merpay, Inc
yukiyan_w 2020/10/17
scala

Spark
ãƒªãƒ³ã‚¯
Databricksã§Spark Structured Streamingã‚’ã‚„ã£ã¦ã¿ã‚‹ - Qiita
ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°å‡¦ç†ã¨ã¯ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’ãƒãƒ¼ãƒ‰ã—ãªãŒã‚‰é †ç•ªã«å‡¦ç†ã—ã¦ã„ãå‡¦ç†æ–¹æ³•ã§ã™ã€‚ éžåŒæœŸã«é †ç•ªã«å‡¦ç†ã—ã¦ã„ãã ã‘ãªã®ã§ã€è«–ç†çš„ã«ã¯ã§ããã†ã ã‘ã©å¯ç”¨æ€§ã‚„ã‚¹ã‚±ãƒ¼ãƒ«æ€§ã‚’è€ƒãˆã‚‹ã¨ãªã‹ãªã‹é›£ã—ã„é ˜åŸŸã§ã™ã€‚ ãã‚“ãªã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°å‡¦ç†ã‚’ã€Databricksã¨ã„ã†Sparkã®ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ä¸Šã§ã€Spark Structured Streamingã‚’ä½¿ã£ã¦å®Ÿç¾ã™ã‚‹æ–¹æ³•ã‚’ã¾ã¨ã‚ã¦ã„ãã¾ã™ã€‚ Databricksã¯ã€Apache Sparkã‚’ä½œã£ãŸäººãŒå‰µæ¥ã—ãŸä¼šç¤¾ã§ã€AWSã‚„Azureã¨ã„ã£ãŸã‚¯ãƒ©ã‚¦ãƒ‰ä¸Šã«ã€Sparkç’°å¢ƒã‚’è‡ªå‹•çš„ã«æ§‹ç¯‰ã—ã¦ãã‚Œã‚‹ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚ æ˜¨å¹´ç´„430å„„å††èª¿é”ã—ã€ãƒãƒªãƒ¥ã‚¨ãƒ¼ã‚·ãƒ§ãƒ³ãŒï¼–7ï¼ï¼å„„å††ã¨ã„ã†ã¿ãŸã“ã¨ã‚‚ãªã„é‡‘é¡ã«ãªã£ã¦ã„ã‚‹ã‚‰ã—ã„ã§ã™ã€‚ å‚è€ƒè¨˜äº‹ ä¼šç¤¾åã‚’ã‚ã¾ã‚Šèžã„ãŸã“ã¨ãŒãªã„æ–¹ã‚‚å¤šã„ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ãŒã€Spark&AI Summitã‚’ä¸»å‚¬ã™ã‚‹ãªã©ã€ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ç•Œéšˆã§ã¯
yukiyan_w 2020/02/18
databricks

Spark
ãƒªãƒ³ã‚¯
Data Engineering & Data Analysis #8
Data Engineering & Data Analysis #8ã§ã®Talkè³‡æ–™
yukiyan_w 2019/09/13
ãƒ‡ãƒ¼ã‚¿åŸºç›¤

Spark
ãƒªãƒ³ã‚¯
Apache Sparkã‚³ãƒŸãƒƒã‚¿ãƒ¼ãŒæ•™ãˆã‚‹ã€Spark SQLã®è©³ã—ã„ä»•çµ„ã¿ã¨ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° Part1
2019å¹´3æœˆ19æ—¥ã€Data Engineering MeetupãŒä¸»å‚¬ã™ã‚‹ã‚¤ãƒ™ãƒ³ãƒˆã€ŒData Engineering Meetup #1ã€ãŒé–‹å‚¬ã•ã‚Œã¾ã—ãŸã€‚ãƒ‡ãƒ¼ã‚¿ã®åŽé›†ã‚„ç®¡ç†ã€å‡¦ç†ã€å¯è¦–åŒ–ãªã©ã€ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ã«é–¢ã™ã‚‹æŠ€è¡“ã®æƒ…å ±ã‚’å…±æœ‰ã™ã‚‹æœ¬ã‚¤ãƒ™ãƒ³ãƒˆã€‚ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ã®æœ€å‰ç·šã§æ´»èºã™ã‚‹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãŸã¡ãŒé›†ã„ã€è‡ªèº«ã®çŸ¥è¦‹ã‚’å…±æœ‰ã—ã¾ã™ã€‚ãƒ—ãƒ¬ã‚¼ãƒ³ãƒ†ãƒ¼ã‚·ãƒ§ãƒ³ã€ŒDeep Dive into Spark SQL with Advanced Performance Tuningã€ã«ç™»å£‡ã—ãŸã®ã¯ã€Databricks Inc.ã®ä¸Šæ–°å“ä¹Ÿæ°ã€‚è¬›æ¼”è³‡æ–™ã¯ã“ã¡ã‚‰ Spark SQLã®ä»•çµ„ã¿ã¨ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãƒãƒ¥ãƒ¼ãƒ‹ãƒ³ã‚° ä¸Šæ–°å“ä¹Ÿæ°ï¼šãã‚Œã§ã¯ç™ºè¡¨ã‚’å§‹ã‚ã¾ã™ã€‚ã€ŽDeep Dive into Spark SQL with Advanced Performance Tuningã€ã¨ã„ã†ã“ã¨ã§ã€Spark SQ
yukiyan_w 2019/07/03
Spark
ãƒªãƒ³ã‚¯
Databricksç¤¾ãŒOSSåŒ–ã—ãŸSparkç”¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤ã€ŒDelta Lakeã€ã«ã¤ã„ã¦ - ãŸã‘ãžã†ç€•æ»ãƒ–ãƒã‚°
å…ˆæ—¥é–‹å‚¬ã•ã‚ŒãŸSpark + AI Summit 2019ã«ã‚ã‚ã›ã¦Databricksç¤¾ã‹ã‚‰Sparkç”¨ã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤ã€ŒDelta Lakeã€ã®OSSåŒ–ãŒç™ºè¡¨ã•ã‚Œã¾ã—ãŸã€‚ databricks.com GitHubãƒªãƒã‚¸ãƒˆãƒªã¯ã“ã¡ã‚‰ã€‚ github.com Delta Lakeã¯Sparkã®ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã¨ã—ã¦å®Ÿè£…ã•ã‚Œã¦ãŠã‚Šã€åˆ†æ•£ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ä¸Šã§ä»¥ä¸‹ã®ã‚ˆã†ãªæ©Ÿèƒ½ã‚’æä¾›ã—ã¾ã™ã€‚ ACIDãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³ï¼ˆãƒ†ãƒ¼ãƒ–ãƒ«å˜ä½ï¼‰ ã‚¿ã‚¤ãƒ ãƒˆãƒ©ãƒ™ãƒ«ï¼ˆä»»æ„ã®æ™‚ç‚¹ã®éŽåŽ»ãƒ‡ãƒ¼ã‚¿ã‚’å‚ç…§å¯èƒ½ï¼‰ ã‚¹ã‚ãƒ¼ãƒžãƒãƒªãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ï¼ˆã‚¹ã‚ãƒ¼ãƒžå¤‰æ›´ã‚‚å¯èƒ½ï¼‰ å®Ÿéš›ã«å‹•ã‹ã—ã¦ã¿ã‚‹ Delta Lakeã®å‹•ä½œã«ã¯Spark 2.4.2ä»¥é™ãŒå¿…è¦ã§ã™ã€‚ãƒãƒ¼ã‚«ãƒ«ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã§ã‚‚å‹•ä½œã™ã‚‹ã®ã§spark-shellã§å‹•ãã‚’ç¢ºèªã—ã¦ã¿ã¾ã—ãŸã€‚ $ bin/spark-shell --packages io.delta:delta-core_2.
yukiyan_w 2019/05/15
Spark
ãƒªãƒ³ã‚¯
Uberã®ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ ãŒ100ãƒšã‚¿ãƒã‚¤ãƒˆè¶…ã®è¦æ¨¡ã«è‡³ã‚‹ã¾ã§
Spring Bootã«ã‚ˆã‚‹APIãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰æ§‹ç¯‰å®Ÿè·µã‚¬ã‚¤ãƒ‰ ç¬¬2ç‰ˆ ä½•åƒäººã‚‚ã®é–‹ç™ºè€…ãŒã€InfoQã®ãƒŸãƒ‹ãƒ–ãƒƒã‚¯ã€ŒPractical Guide to Building an API Back End with Spring Bootã€ã‹ã‚‰ã€Spring Bootã‚’ä½¿ã£ãŸREST APIæ§‹ç¯‰ã®åŸºç¤Žã‚’å¦ã‚“ã ã€‚ã“ã®æœ¬ã§ã¯ã€å‡ºç‰ˆæ™‚ã«æ–°ã—ããƒªãƒªãƒ¼ã‚¹ã•ã‚ŒãŸãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚ã‚‹ Spring Boot 2 ã‚’ä½¿ç”¨ã—ã¦ã„ã‚‹ã€‚ã—ã‹ã—ã€Spring Boot3ãŒæœ€è¿‘ãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã€é‡è¦ãªå¤‰...
yukiyan_w 2019/05/10
streaming

Spark

Hadoop

kafka
ãƒªãƒ³ã‚¯
Home | Delta Lake
Build Lakehouses with Delta Lake Delta Lake is an open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, and Python. Get Started
yukiyan_w 2019/04/27
ä¾¿åˆ©ãã†

Spark

DataLake
ãƒªãƒ³ã‚¯
hscj2019_ishizaki_public
2. About Me â€“ Kazuaki Ishizaki â–ª IBM Research â€“ Tokyoã§ç ”ç©¶å“¡ã‚’ã—ã¦ã„ã¾ã™ https://ibm.biz/ishizaki â€“ ã‚³ãƒ³ãƒ‘ã‚¤ãƒ©æœ€é©åŒ–ã€è¨€èªžå‡¦ç†ç³»å®Ÿè£…ã€ä¸¦åˆ—å‡¦ç†ã€ãŒå°‚é–€ â–ª 1996å¹´ã‚ˆã‚Šã€IBM Javaï¼ˆä»Šã¯Open J9ï¼‰ã®å®Ÿè£…ã«ã‹ã‹ã‚ã‚‹ â€“ ç‰¹ã«ã€PowerPCç”¨ã®å®Ÿè¡Œæ™‚ã‚³ãƒ³ãƒ‘ã‚¤ãƒ©ã®å®Ÿè£…ãƒªãƒ¼ãƒ‰ â–ª 2018å¹´9æœˆã‚ˆã‚Šã€Apache Sparkã‚³ãƒŸãƒƒã‚¿ï¼ˆSQLãƒ¢ã‚¸ãƒ¥ãƒ¼ãƒ«ï¼‰ â€“ ç¾åœ¨ã€Apache Sparkã‚³ãƒŸãƒƒã‚¿ã¯æ—¥æœ¬ã«ï¼”äºº â–ª 2018å¹´ã‚ˆã‚ŠACM Distinguished Member â–ª SNS â€“ @kiszk â€“ Slideshare: https://www.slideshare.net/ishizaki 2 DataFrameã¨Datasetã®å†…éƒ¨ã‚’ã®ãžã„ã¦ã¿ã‚‹ - Kazuaki Ishizaki 3
yukiyan_w 2019/03/15
Spark
ãƒªãƒ³ã‚¯
Spark Streaming ã¨ Kryo ã‚·ãƒªã‚¢ãƒ©ã‚¤ã‚¶ãƒ¼ã®è©± - MicroAd Developers Blog
ãƒžã‚¤ã‚¯ãƒã‚¢ãƒ‰ã§ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã‚’ã—ã¦ã„ã‚‹ã€ŒGã€ã§ã™ã€‚æ—¥ã€…ã€ãƒãƒƒãƒã‚’ä½œã£ãŸã‚Š WEB ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ä½œã£ãŸã‚Šã—ã¦ã„ã¾ã™ã€‚ ä»Šå›žã¯ãƒ©ã‚¤ãƒˆã« Spark Streaming ã§ã‚«ã‚¹ã‚¿ãƒ Kryo ï¼ˆã‚¯ãƒ©ã‚¤ã‚ªï¼‰ã‚·ãƒªã‚¢ãƒ©ã‚¤ã‚¶ãƒ¼ãŒå¿…è¦ã«ãªã£ã¦ã—ã¾ã£ãŸãŠè©±ã‚’æ›¸ã„ã¦ã„ããŸã„ã¨æ€ã„ã¾ã™ã€‚ Spark Streaming ã¨ã¯ Spark Streaming ã¨ã¯ Apache Spark ã¨ã„ã†åˆ†æ•£ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ãƒ†ã‚£ãƒ³ã‚°åŸºç›¤ä¸Šã§ã‚¹ãƒˆãƒªãƒ¼ãƒ å‡¦ç†ã‚’è¡Œã†ãŸã‚ã®ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã™ã€‚ ãƒžã‚¤ã‚¯ãƒã‚¢ãƒ‰ã§ã¯ç§’é–“æ•°ä¸‡ä»¶ä»¥ä¸Šç™ºç”Ÿã™ã‚‹ãƒ¦ãƒ¼ã‚¶ãƒ¼ã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã®å‡¦ç†ã®ãŸã‚ã« Spark Streaming ã‚’åˆ©ç”¨ã—ã¦ã„ã¾ã™ã€‚ è©³ç´°ã¯ä»¥ä¸‹ã®è¨˜äº‹ã‚’ã”ç¢ºèªãã ã•ã„ã€‚ developers.microad.co.jp ãƒ–ãƒãƒ¼ãƒ‰ã‚ãƒ£ã‚¹ãƒˆå¤‰æ•°ã¨ã‚·ãƒªã‚¢ãƒ©ã‚¤ã‚ºã®ç½ Apache Spark ã§ã¯ã€ãƒ¡ã‚¤ãƒ³ãƒ—ãƒã‚°ãƒ©ãƒ ãŒå®Ÿè¡Œã•ã‚Œã‚‹ãƒ‰ãƒ©ã‚¤ãƒãƒ¼ã¨å‘¼ã°
yukiyan_w 2019/02/09
Spark

streaming

kafka
ãƒªãƒ³ã‚¯
How to Turn Python Functions into PySpark Functions (UDF)
yukiyan_w 2018/09/13
Spark
ãƒªãƒ³ã‚¯
7 Tips to Debug Apache Spark Code Faster with Databricks
Unified governance for all data, analytics and AI assets
yukiyan_w 2018/08/29
Spark

databricks
ãƒªãƒ³ã‚¯
Databricks Community Edition ã§ Spark ã«è§¦ã‚Œã‚‹ - astamuse Lab
å±±ç¸£ã§ã™ã€‚ ä»Šå›žã¯ Spark ã‚’ç„¡æ–™ã§æ‰‹è»½ã«è§¦ã‚Œã‚‹ Databricks Community Edition ã«ã¤ã„ã¦æ›¸ããŸã„ã¨æ€ã„ã¾ã™ã€‚ Databricks Community Editionã¨ã¯ Databricks ã¯ Databricks ç¤¾ãŒæä¾›ã—ã¦ã„ã‚‹Sparkã‚µãƒ¼ãƒ“ã‚¹ã§ã€Community Edition ã¯ãã®ç„¡æ–™ç‰ˆã«ã‚ãŸã‚Šã¾ã™ã€‚ ç„¡æ–™ã«ã‚‚ã‹ã‹ã‚ã‚‰ãšãƒ¡ãƒ¢ãƒª6GBã®AWSã®ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã¨Notebook ãŒä½¿ãˆã‚‹ã¨ã„ã†ãªã‹ãªã‹å¤ªã£è…¹ãªã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚ å¼Šç¤¾ã§ã¯ Clouderaç¤¾ã® CDH ã®ã‚¯ãƒ©ã‚¹ã‚¿ãŒã‚ã‚ŠSparkã‚‚ãã®ä¸Šã§å‹•ã‹ã—ã¦ã„ã¾ã™ãŒã€å€‹äººçš„ã«Spark ãã®ã‚‚ã®ã®å‹•ä½œã‚’ç¢ºèªã—ãŸã‚Šã€ç•°ãªã‚‹ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚’è§¦ã‚ŠãŸã„ã¨ãã«æ™‚ã€…ä½¿ã‚ã›ã¦ã‚‚ã‚‰ã£ã¦ã„ã¾ã™ã€‚ Sparkã®å¦ç¿’ç”¨ç’°å¢ƒã¨ã—ã¦ã¯ååˆ†ãªæ©Ÿèƒ½ãŒå‚™ã‚ã£ã¦ã„ã‚‹ã¨æ€ã„ã¾ã™ã®ã§ã€ç°¡å˜ã«ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ ãƒ¦ãƒ¼ã‚¶ç™»éŒ² Commun
yukiyan_w 2018/04/29
Spark
ãƒªãƒ³ã‚¯
[Google Cloud Platform] Cloud Dataflowã§ãƒ•ãƒ«ãƒžãƒãƒ¼ã‚¸ãƒ‰ãªãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚’è©¦ã—ã¦ã¿ãŸ | AI tech studio
2016.11.8 [Google Cloud Platform] Cloud Dataflowã§ãƒ•ãƒ«ãƒžãƒãƒ¼ã‚¸ãƒ‰ãªãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚’è©¦ã—ã¦ã¿ãŸ ã‚¢ãƒ‰ãƒ†ã‚¯ã‚¹ã‚¿ã‚¸ã‚ªã®ä½ã€…æœ¨ã§ã™ã€‚ ã‚¢ãƒ‰ãƒ†ã‚¯ã‚¹ã‚¿ã‚¸ã‚ªã§ã¯ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã«åŠ›ã‚’ã„ã‚Œã¦ãŠã‚ŠOpenStack Saharaç‰ã‚’åˆ©ç”¨ã—ã¦å…±é€šåŸºç›¤ã‚’æ§‹ç¯‰ã—ã¦ã„ã¾ã™ã€‚ ä»Šå›žã¯Google Cloud Platform ãŒæä¾›ã—ã¦ã„ã‚‹ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®ã‚µãƒ¼ãƒ“ã‚¹ã®ã²ã¨ã¤ã§ã‚ã‚‹ Cloud Dataflow ã«ã¤ã„ã¦ã¿ã¦ã„ããŸã„ã¨æ€ã„ã¾ã™ã€‚ Cloud Dataflow ã¨ã¯ Cloud Dataflow ã®èª¬æ˜Žã¯ã€Œãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã®ãƒãƒƒãƒãŠã‚ˆã³ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°å‡¦ç†ã‚’å®Ÿç¾ã™ã‚‹ãƒ•ãƒ«ãƒžãƒãƒ¼ã‚¸ãƒ‰ã®ã‚¯ãƒ©ã‚¦ãƒ‰ã‚µãƒ¼ãƒ“ã‚¹ã€‚ã€ã¨ãªã£ã¦ã„ã¾ã™ã€‚ Google Cloud Platform ã«ã¯Cloud Dataflowã¨ã¯åˆ¥ã«Spark ã¨ Hadoop ã®ãƒžãƒãƒ¼ã‚¸ãƒ‰ã‚µãƒ¼ãƒ“ã‚¹ã§ã‚ã‚‹ Cloud Datap
yukiyan_w 2017/07/02
Hadoop

Spark

gcp

dataflow

beam
ãƒªãƒ³ã‚¯
Spark Streamingã‹ã‚‰Amazon Kinesis Analyticsã¸ç§»è¡Œã™ã‚‹è©± - Gunosyãƒ‡ãƒ¼ã‚¿åˆ†æžãƒ–ãƒã‚°
ã¯ã˜ã‚ã« ã“ã‚“ã«ã¡ã¯ã€ãƒ‡ãƒ¼ã‚¿åˆ†æžéƒ¨ã®æ£®æœ¬ã§ã™ã€‚ä¸»ãªæ¥å‹™ã¯è¨˜äº‹é…ä¿¡ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã®æ”¹å–„ã¨ãƒã‚°åŸºç›¤ã®æ•´å‚™ã§ã™ã€‚ Gunosyã§ã¯ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã¸ã‚ˆã‚Šè‰¯ã„è¨˜äº‹ã‚’æä¾›ã™ã‚‹ãŸã‚ã«ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’ã‚¹ãƒˆãƒªãƒ¼ãƒ å‡¦ç†ã—ã€é›†è¨ˆçµæžœã‚’è¨˜äº‹é…ä¿¡ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã«æ´»ç”¨ã—ã¦ã„ã¾ã™ã€‚ ã‚¹ãƒˆãƒªãƒ¼ãƒ ãƒã‚°åŸºç›¤ã«ã¯Spark Streamingã‚’åˆ©ç”¨ã—ã¦ã„ã¾ã™ãŒã€ç¾åœ¨Kinesis Analyticsã¸ç§»è¡Œä¸ã§ã™ã€‚ ã“ã®è¨˜äº‹ã§ã¯Kinesis Analyticsã¸ç§»è¡Œã™ã‚‹ç†ç”±ã‚„é‹ç”¨ä¸Šã®Tipsç‰ã«ã¤ã„ã¦ãŠè©±ã—ã¾ã™ã€‚ Spark Streamingã‚’åˆ©ç”¨ã—ãŸã‚¹ãƒˆãƒªãƒ¼ãƒ ãƒã‚°åŸºç›¤æ§‹æˆ ç¾åœ¨ã®ã‚¹ãƒˆãƒªãƒ¼ãƒ ãƒã‚°åŸºç›¤ã¯Spark Streamingã§é›†è¨ˆã‚’è¡Œã„ã€çµæžœã‚’RDSã«ä¿å˜ã—ã¦ã„ã¾ã™ã€‚ ãªãœSpark Streamingã‹ã‚‰Kinesis Analyticsã¸ç§»è¡Œã™ã‚‹ã®ã‹ ã‚µãƒ¼ãƒãƒ¼ã‚³ã‚¹ãƒˆã¨é‹ç”¨ã‚³ã‚¹ãƒˆã®å‰Šæ¸›ã‚’ç›®çš„ã¨ã—ã¦ã„ã¾ã™ã€‚ ã‚µãƒ¼ãƒãƒ¼ã‚³ã‚¹ãƒˆã«ã¤ã„
yukiyan_w 2017/02/15
streaming

Kinesis

Spark
ãƒªãƒ³ã‚¯
1