[B! Spark] karumadoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

karumado id:karumado

Sparkã«é–¢ã™ã‚‹karumadoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (5)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

AWS Solutions Architect ãƒ–ãƒã‚°
Apache Sparkã¨Amazon DSSTNEã‚’ä½¿ã£ãŸã€Amazonè¦æ¨¡ã®ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ç”Ÿæˆ Amazonã®ãƒ‘ãƒ¼ã‚½ãƒŠãƒ©ã‚¤ã‚¼ãƒ¼ã‚·ãƒ§ãƒ³ã§ã¯ã€ãŠå®¢æ§˜æ¯Žã®è£½å“ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ç”Ÿæˆã™ã‚‹ãŸã‚ã«ãƒ‹ãƒ¥ãƒ¼ãƒ©ãƒ«ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚’ä½¿ã£ã¦ã„ã¾ã™ã€‚Amazonã®è£½å“ã‚«ã‚¿ãƒã‚°ã¯ã€ã‚ã‚‹ãŠå®¢æ§˜ãŒè³¼å…¥ã—ãŸè£½å“ã®æ•°ã«æ¯”è¼ƒã—ã¦éžå¸¸ã«å·¨å¤§ãªã®ã§ã€ãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã¯æ¥µç«¯ã«ç–Žã«ãªã£ã¦ã—ã¾ã„ã¾ã™ã€‚ãã—ã¦ã€ãŠå®¢æ§˜ã®æ•°ã¨è£½å“ã®æ•°ã¯ä½•å„„ã«ã‚‚ã®ã¼ã‚‹ãŸã‚ã€æˆ‘ã€…ã®ãƒ‹ãƒ¥ãƒ¼ãƒ©ãƒ«ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã®ãƒ¢ãƒ‡ãƒ«ã¯è¤‡æ•°ã®GPUã§åˆ†æ•£ã—ãªã‘ã‚Œã°ã€ç©ºé–“ã‚„æ™‚é–“ã®åˆ¶ç´„ã‚’æº€ãŸã™ã“ã¨ãŒã§ãã¾ã›ã‚“ã€‚ ãã®ãŸã‚ã€GPUä¸Šã§å‹•ä½œã™ã‚‹DSSTNE (the Deep Scala ble Sparse Tensor Neural Engine)ã‚’é–‹ç™ºã—ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã«ã—ã¾ã—ãŸã€‚æˆ‘ã€…ã¯DSSTNEã‚’ä½¿ã£ã¦ãƒ‹ãƒ¥ãƒ¼ãƒ©ãƒ«ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚’å¦ç¿’ã—ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ç”Ÿæˆã—ã¦ã„ã¦ã€ECã®ã‚¦ã‚§ãƒ–ã‚µã‚¤ãƒˆ
karumado 2016/07/11
amazon

Spark

apache

ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰
ãƒªãƒ³ã‚¯
Elasticsearchã®Apache Sparkã‚µãƒãƒ¼ãƒˆæ©Ÿèƒ½ã§éŠã¶ - CLOVERðŸ€
Elasticsearchã«ã€Apache Sparkå‘ã‘ã®ãƒ©ã‚¤ãƒ–ãƒ©ãƒªãŒã‚ã‚‹ã“ã¨ã¯çŸ¥ã£ã¦ã„ãŸã®ã§ã™ãŒã€é•·ã‚‰ãæ‰‹ã‚’ã¤ã‘ã¦ã„ãªã„ã¾ã¾ã ã£ãŸã®ã§ã€1åº¦è©¦ã—ã¦ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ Apache Spark support | Elasticsearch for Apache Hadoop [2.3] | Elastic ã“ã¡ã‚‰ã‚’ä½¿ã†ã“ã¨ã§ã€Apache SparkãŒæä¾›ã™ã‚‹APIã‚’Elasticsearchã§ä½¿ã†ã“ã¨ãŒã§ãã‚‹ã‚ˆã†ã«ãªã‚‹ã¿ãŸã„ã§ã™ãã€‚å†…éƒ¨çš„ã«ã¯ã€elasticsearch-hadoopã«ä¾å˜ã—ã¦ã„ã‚‹æ¨¡æ§˜ã€‚ æ—¥æœ¬èªžè¨˜äº‹ã‚‚ã‚ã‚‹ã‚ˆã†ã§ã™ã€‚ æ¥½ã—ã„å¯è¦–åŒ– ï¼š elasticsearchã¨Spark Streamingã®å‡ºä¼šã„ | NTTãƒ‡ãƒ¼ã‚¿å…ˆç«¯æŠ€è¡“æ ªå¼ä¼šç¤¾ ã§ã€ä½•ã‚’ã™ã‚‹ã‹ã§ã™ãŒã€ã¾ã‚â€¦Spark Streamingã¨Twitterã§ã™ã‹ãã€‚ä»Šå›žã¯ã€ä»¥ä¸‹ã®ãƒ†ãƒ¼ãƒžã§ã‚„ã£ã¦ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚
karumado 2016/06/12
elasticsearch

Spark
ãƒªãƒ³ã‚¯
Netflixã«ãŠã‘ã‚‹Presto/Sparkæ´»ç”¨äº‹ä¾‹
2. 2 Amazon EMR - 1ã‚¯ãƒªãƒƒã‚¯ã§Hadoop/Spark â€¢ åˆ†æ•£å‡¦ç†åŸºç›¤ â€“ ã‚¯ãƒ©ã‚¹ã‚¿ã‚’ç°¡å˜ã«æ§‹ç¯‰ ã—ã¦ç ´æ£„ â€¢ åˆ†æ•£å‡¦ç†ã‚¢ãƒ—ãƒª â€“ ä½¿ã„ãŸã„ã‚¢ãƒ—ãƒªã‚’é¸ã¶ ã ã‘ â€¢ Hadoop 2.7.1 â€¢ Hive 1.0.0 â€¢ Pig 0.14.0 â€¢ Mahout 0.11.0 â€¢ Oozie 4.2.0 â€¢ Spark 1.6.0 â€¢ Presto 0.130 â€¢ Zeppelin 0.5.5 â€¢ Hue 3.7.1æ›´æ–°ã®é€Ÿã„(ã»ã¼æœˆ1ãƒšãƒ¼ã‚¹) ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ 3. 3 Amazon EMR - 1ã‚¯ãƒªãƒƒã‚¯ã§Hadoop/Spark â€¢ åˆ†æ•£å‡¦ç†åŸºç›¤ â€“ ã‚¯ãƒ©ã‚¹ã‚¿ã‚’ç°¡å˜ã«æ§‹ç¯‰ ã—ã¦ç ´æ£„ â€¢ åˆ†æ•£å‡¦ç†ã‚¢ãƒ—ãƒª â€“ ä½¿ã„ãŸã„ã‚¢ãƒ—ãƒªã‚’é¸ã¶ ã ã‘ â€¢ Hadoop 2.7.1 â€¢ Hive 1.0.0 â€¢ Pig 0.14.0 â€¢ Mahout 0.11.0 â€¢ Oozie
karumado 2016/02/09
Netflix

Spark

amazon
ãƒªãƒ³ã‚¯
åˆ†æ•£å‡¦ç†ã«å…¥é–€ã—ã¦ã¿ãŸï¼ˆHadoop + Sparkï¼‰ | Casley Deep Innovationsæ ªå¼ä¼šç¤¾ æŠ€è¡“ãƒ–ãƒã‚°
ã“ã‚“ã«ã¡ã¯ã€‚SIéƒ¨ã®è…°å¡šã§ã™ã€‚ RDBã‚„ãƒ‡ãƒ¼ã‚¿ã‚¦ã‚§ã‚¢ãƒã‚¦ã‚¹ã®ä»•äº‹ã«æºã‚ã‚‹ã“ã¨ãŒå¤šã‹ã£ãŸç†è€…ã¯ã€æ•°å¹´å‰ã‹ã‚‰ãŸã³ãŸã³èžã“ãˆãŸãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚„æ©Ÿæ¢°å¦ç¿’ã®ãŸã‚ã®åˆ†æ•£å‡¦ç†ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã«èˆˆå‘³ã‚’è¦šãˆãŸã‚‚ã®ã®ã€ã¤ã„ãžã‚¢ã‚¯ã‚»ã‚¹ã—ãªã„ã¾ã¾ã“ã“ã¾ã§æ¥ã¦ã—ã¾ã„ã¾ã—ãŸã€‚ ä»Šå›žãƒ–ãƒã‚°ã‚’æ›¸ãã«ã‚ãŸã£ã¦ã€ã›ã£ã‹ããªã®ã§ã‚¤ãƒã‹ã‚‰æ‰‹ã•ãã‚Šå…¥é–€ã—ã€ã„ã¾ã•ã‚‰ä»–äººã«èžã‘ãªã„åˆ†æ•£å‡¦ç†ã®åˆæ©ã‹ã‚‰hadoopãƒ»sparkã‚’è§¦ã£ã¦ã¿ã‚‹ã¾ã§ã‚’ã¾ã¨ã‚ãŸã„ã¨æ€ã„ã¾ã™ã€‚ 1.åˆ†æ•£å‡¦ç†ã®åŸºç¤ŽçŸ¥è˜ 1-1.åˆ†æ•£å‡¦ç†ã®å‡¦ç†æ–¹å¼ï¼šMapReduce ã¾ãšåˆ†æ•£å‡¦ç†ã¨ã¯ã€ã²ã¨ã¤ã®è¨ˆç®—å‡¦ç†ã‚’ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã§æŽ¥ç¶šã—ãŸè¤‡æ•°ã®ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ã‚¿ã§åŒæ™‚ä¸¦åˆ—ã§å‡¦ç†ã™ã‚‹ã“ã¨ã§ã™ã€‚ ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ã®å¸‚å ´ãŒæ—¥ã€…å¤§ãããªã‚‹ã«å¾“ã£ã¦ã€æ•°ç™¾ãƒ†ãƒ©ï½žãƒšã‚¿ã®ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚‚çã—ã„ã‚‚ã®ã§ã¯ãªããªã£ã¦ãŠã‚Šã€æ—¥å¸¸çš„ã«ã“ã®è¦æ¨¡ã®ãƒ‡ãƒ¼ã‚¿ã‚’æ‰±ã†ã‚·ã‚¹ãƒ†ãƒ ã§ã¯ã€ç¾å®Ÿçš„ãªæ™‚é–“çš„ãƒ»è²»ç”¨çš„ã‚³ã‚¹ãƒˆã§å‡¦ç†ã™ã‚‹å·¥å¤«ãŒå¿…è¦
karumado 2015/04/24
ã‚ã¨ã§èªã‚€

Spark

Hadoop
ãƒªãƒ³ã‚¯
Sparkã«ã‚ˆã‚‹åˆ†æ•£å‡¦ç† / 2015-01-16 PyData.Tokyo#3
[CVå‹‰å¼·ä¼š@é–¢æ± ECCV2024 èªã¿ä¼š] ã‚ªãƒ³ãƒ©ã‚¤ãƒ³ãƒžãƒƒãƒ”ãƒ³ã‚° x ãƒˆãƒ©ãƒƒã‚ãƒ³ã‚° MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping (Chen+, ECCV24)
karumado 2015/01/20
Python

spark

åˆ†æ•£å‡¦ç†
ãƒªãƒ³ã‚¯
1