[B! Spark] Dryadã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

Dryad id:Dryad

Sparkã«é–¢ã™ã‚‹Dryadã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

2015å¹´ã®Hadoopã¨ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ | gihyo.jp
ã‚ã‘ã¾ã—ã¦ãŠã‚ã§ã¨ã†ã”ã–ã„ã¾ã™ã€‚æ¿±é‡Ž è³¢ä¸€æœ—ã§ã™ã€‚ 1å¹´ã¯æ—©ã„ã‚‚ã®ã§ã™ãã€‚æœ¬å½“ã«ã‚ã£ã¨ã„ã†é–“ã«éŽãŽåŽ»ã£ã¦ã—ã¾ã†ã‚‚ã®ã§ã™ã€‚ã—ã‹ã—ã€ã„ã–1å¹´ã‚’æŒ¯ã‚Šè¿”ã£ã¦ã¿ã‚‹ã¨ã€ã„ã‚ã‚“ãªã“ã¨ãŒèµ·ãã¦ã„ãŸã®ã ã¨å®Ÿæ„Ÿã—ãŸã‚Šã‚‚ã—ã¾ã™ã€‚ã“ã“ã§ã¯ã€Œãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ï¼ˆBig Dataï¼‰æ´»ç”¨ã€ãŒ2015å¹´ã«ã©ã†å¤‰åŒ–ã—ã¦ã„ããã†ã‹ã€Hadoopã®è©±é¡Œã‚’ä¸å¿ƒã«ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ 2014å¹´ã‚’æŒ¯ã‚Šè¿”ã‚‹ ã¾ãšã¯ã€æ˜¨å¹´ã‚’æŒ¯ã‚Šè¿”ã£ã¦ã¿ã¾ã—ã‚‡ã†ã€‚æ€¥æµ®ä¸Šã—ãŸãƒã‚ºãƒ¯ãƒ¼ãƒ‰ã¨ã„ã†å°è±¡ã®ã€Œãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ã€ã‚‚ã€ã™ã“ã—è½ã¡ç€ã„ã¦ãã¾ã—ãŸã€‚ã¾ã ã¾ã æµ®ã¤ã„ãŸè©±ã‚’è€³ã«ã—ã¾ã—ãŸãŒã€ä¸€æ–¹ã§ã€å…·ä½“çš„ãªäº‹ä¾‹ã‚„æŠ€è¡“ã®è©±ã‚‚å¢—ãˆã¦ããŸ1å¹´ã§ã—ãŸã€‚ ã‚·ã‚¹ãƒ†ãƒ åŸºç›¤ã®è¦³ç‚¹ã§ã¯ã€ŒHadoopã€ã®å¤‰è²ŒãŒå§‹ã¾ã‚Šã¾ã—ãŸã€‚çš†ã•ã‚“ã”å˜ã˜ã®ã¨ãŠã‚Šã€Hadoopã¯ä¸¦åˆ—åˆ†æ•£å‡¦ç†ã‚’å®Ÿç¾ã™ã‚‹ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã™ã€‚æŠ€è¡“çš„å´é¢ã§ã®å¤‰è²Œã¯2015å¹´ã«æœ¬æ ¼åŒ–ã™ã‚‹ã¨æ€ã„ã¾ã™ãŒã€ãã®ä¸‹æº–å‚™ãŒç€å®Ÿã«è¡Œ
Dryad 2015/01/02
ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°

Hadoop

Spark
ãƒªãƒ³ã‚¯
Hadoopã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ãƒªãƒ¼ãƒ‡ã‚£ãƒ³ã‚° ç¬¬17å›žã«å‚åŠ ã—ã¦ãã¾ã—ãŸ | DevelopersIO
Hadoopã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ãƒªãƒ¼ãƒ‡ã‚£ãƒ³ã‚° ç¬¬17å›žã«å‚åŠ ã—ã¦ãã¾ã—ãŸã€‚ä»Šå›žã®ãƒ†ãƒ¼ãƒžã¯7æœˆã«Apacheã®Top-Level Projectå…¥ã‚Šã—ãŸã°ã‹ã‚Šã®Apache Tezã«ã¤ã„ã¦ã§ã—ãŸã€‚ãªãŠã€å…¨ä½“çš„ã«Apache Sparkã¨æ¯”è¼ƒã™ã‚‹å½¢ã§ã®èª¬æ˜ŽãŒå¤šã‹ã£ãŸã®ã§ã€Sparkã«ã¤ã„ã¦ã”å˜ã˜ãªã„æ–¹ã¯å‰å›žã®Hadoopã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ãƒªãƒ¼ãƒ‡ã‚£ãƒ³ã‚° ç¬¬16å›žã«å‚åŠ ã—ã¦ãã¾ã—ãŸã‚’ã”å‚ç…§ä¸‹ã•ã„ã€‚ NTTãƒ‡ãƒ¼ã‚¿æ¿±é‡Žã•ã‚“ã®å†’é ã®æŒ¨æ‹¶ ä»Šæ—¥ã¯åˆ¥ã®ã‚¤ãƒ™ãƒ³ãƒˆã‚‚å¤šã„ãŸã‚ã„ã¤ã‚‚ã®åŠåˆ†ãã‚‰ã„ã®å‚åŠ è€…ã ãŒã€ãã®åˆ†Deepã«ã‚„ã‚Œã‚Œã° ã„ã¤ã‚‚ã®ä¼šå ´ã ã¨é€”ä¸ã‹ã‚‰ãƒ”ã‚¶ã¨ãŠé…’ã ãŒã€ä»Šå›žã®ä¼šå ´ã¯é£²é£Ÿç¦æ¢ãªã®ã§æœ€å¾Œã¾ã§ã‚·ãƒ©ãƒ•ã§ Tezã«é–¢ã™ã‚‹å‹‰å¼·ä¼šã¯åˆå›žãªã®ã«ã„ããªã‚Šã‚¿ã‚¤ãƒˆãƒ«ãŒInternalsã¨ã‹ã«ãªã£ã¦ã¾ã™ãï½— Tez Internals ï¼ˆ@oza_x86 ã•ã‚“ï¼‰ @oza_x86 ã•ã‚“ã‹ã‚‰ã¯Tez Internalsã¨ã„ã†ã“ã¨ã§ã€S
Dryad 2014/09/12
ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°

Hadoop

Spark
ãƒªãƒ³ã‚¯
ã€ŒHadoopã¯ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã®â€œOSã‚«ãƒ¼ãƒãƒ«â€ã€ã€Hadoop Conference Japané–‹å‚¬
ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚½ãƒ•ãƒˆã‚¦ã‚¨ã‚¢ï¼ˆOSSï¼‰ã®åˆ†æ•£ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚½ãƒ•ãƒˆã€ŒHadoopã€ã®ãƒ¦ãƒ¼ã‚¶ãƒ¼ä¼šã€ŒHadoop Conference Japan 2014ã€ãŒ2014å¹´7æœˆ8æ—¥ã€æ±äº¬ãƒ»æ±ç•™ã§é–‹å‚¬ã•ã‚ŒãŸã€‚åŸºèª¿è¬›æ¼”ã«ã¯ã€Hadoopã®ã‚ªãƒªã‚¸ãƒŠãƒ«é–‹ç™ºè€…ã§ã‚ã‚‹ãƒ€ã‚°ãƒ»ã‚«ãƒƒãƒ†ã‚£ãƒ³ã‚°æ°ï¼ˆå†™çœŸ1ï¼‰ãªã©ãŒç™»å£‡ã€‚ã‚«ãƒƒãƒ†ã‚£ãƒ³ã‚°æ°ã¯ã€Œãƒãƒƒãƒå‡¦ç†ç”¨ã®ã€ŽMapReduceã€ä»¥å¤–ã®å‡¦ç†æ–¹å¼ã«å¯¾å¿œã—ãŸç¾åœ¨ã®Hadoopã¯ã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’å‡¦ç†ã™ã‚‹åˆ†æ•£OSã®ã‚«ãƒ¼ãƒãƒ«ã¨ã‚‚è¨€ã†ã¹ãå˜åœ¨ã«ãªã£ãŸã€ã¨èªžã£ãŸã€‚ åŸºèª¿è¬›æ¼”ã«ã¯ã‚«ãƒƒãƒ†ã‚£ãƒ³ã‚°æ°ã®ã»ã‹ã€æ—¥æœ¬Hadoopãƒ¦ãƒ¼ã‚¶ãƒ¼ä¼šã®ä¸–è©±å½¹ã§ã‚ã‚‹ãƒªã‚¯ãƒ«ãƒ¼ãƒˆãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚ºã®ç±³è°·ä¿®æ°ã‚„NTTãƒ‡ãƒ¼ã‚¿ã®æ¿±é‡Žè³¢ä¸€æœ—æ°ã€ã€ŒSparkã€ã®é–‹ç™ºå…ƒã§ã‚ã‚‹ç±³ãƒ‡ãƒ¼ã‚¿ãƒ–ãƒªãƒƒã‚¯ã‚¹ï¼ˆDatabricksï¼‰ã®ãƒ‘ãƒˆãƒªãƒƒã‚¯ãƒ»ã‚¦ã‚§ãƒ³ãƒ‡ãƒ«æ°ã€ç±³ãƒˆãƒ¬ã‚¸ãƒ£ãƒ¼ãƒ‡ãƒ¼ã‚¿ã®å¤ªç”°ä¸€æ¨¹æ°ãŒç™»å£‡ã—ãŸã€‚ NTTãƒ‡ãƒ¼ã‚¿ã®æ¿±é‡Žæ°ï¼ˆå†™çœŸ2ï¼‰ã¯åŸºèª¿è¬›æ¼”ã®å†’é ã€2009å¹´
Dryad 2014/07/12
ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°

åˆ†æ•£å‡¦ç†

Hadoop

Spark
ãƒªãƒ³ã‚¯
ã€Œ100å€é«˜é€Ÿã€ã®ä¸¦åˆ—å‡¦ç†ã‚¨ãƒ³ã‚¸ãƒ³ã‚‚å®Ÿç¾ã€SQLã‚„æ©Ÿæ¢°å¦ç¿’ã¸ã¨ç”¨é€”åºƒã’ã‚‹Hadoop
ã€ŒHadoopãŒä½¿ãˆã‚‹ã®ã¯ãƒãƒƒãƒå‡¦ç†ã ã‘ã€‚ãã‚“ãªå°è±¡ã¯ã‚‚ã¯ã‚„éŽåŽ»ã®ã‚‚ã®ã ã€ã€‚ã€ŒHadoopã€ã®ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ï¼ˆæ¤œè¨¼æ¸ˆã¿ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ï¼‰ã®ãƒ™ãƒ³ãƒ€ãƒ¼ã§ã‚ã‚‹ç±³ãƒ›ãƒ¼ãƒˆãƒ³ãƒ¯ãƒ¼ã‚¯ã‚¹ã®ãƒãƒ–ãƒ»ãƒ™ã‚¢ãƒ‡ãƒ³CEOï¼ˆæœ€é«˜çµŒå–¶è²¬ä»»è€…ï¼‰ã¯ã€2014å¹´6æœˆã«é–‹å‚¬ã—ãŸã€ŒHadoop Summit 2014ã€ã§ã“ã†å¼·èª¿ã—ãŸã€‚ Hadoopã®é–‹ç™ºå…ƒã§ã‚ã‚‹Apacheã‚½ãƒ•ãƒˆã‚¦ã‚¨ã‚¢è²¡å›£ï¼ˆASFï¼‰ãŒã€Hadoopã®æ¨™æº–çš„ãªå‡¦ç†æ–¹å¼ã€ŒMapReduceã€ä»¥å¤–ã®æ–¹å¼ã«å¯¾å¿œã—ãŸã€ŒHadoop 2ã€ã‚’ãƒªãƒªãƒ¼ã‚¹ã—ãŸã®ã¯2013å¹´10æœˆã€‚ãã‚Œã‹ã‚‰åŠå¹´ãŒçµŒéŽã—ã€Hadoopã§åˆ©ç”¨ã§ãã‚‹å‡¦ç†æ–¹å¼ã¯æ€¥é€Ÿã«å¢—åŠ ã—ã¦ã„ã‚‹ï¼ˆå›³ï¼‰ã€‚
Dryad 2014/06/24
ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°

Hadoop

Spark
ãƒªãƒ³ã‚¯
Hadoopè¶…ãˆã‚‹æ©Ÿæ¢°å¦ç¿’å‘ãã®ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã€Spark 1.0ãŒæ£å¼å…¬é–‹
ç±³Apache Software Foundationã¯ã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚’åˆ†æ•£ã‚¯ãƒ©ã‚¹ã‚¿ãƒ¼ä¸Šã§é«˜é€Ÿã«å®Ÿè¡Œã§ãã‚‹å‡¦ç†åŸºç›¤ã€ŒSpark 1.0ã€ã‚’2014å¹´5æœˆ30æ—¥ï¼ˆç±³å›½æ™‚é–“ï¼‰ã«å…¬é–‹ã—ãŸã€‚ HDFSã‚’ä»‹ã—ã¦ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸çµŒç”±ã®ã‚„ã‚Šå–ã‚ŠãŒå¤šããªã‚‹Hadoopã¨æ¯”ã¹ã¦ã€ã‚¤ãƒ³ãƒ¡ãƒ¢ãƒªãƒ¼å‡¦ç†ã‚’ä¸»ä½“ã¨ã™ã‚‹Sparkã§ã¯ã€ã‚ˆã‚Šé«˜é€Ÿã§ä½Žé…å»¶ã®åˆ†æžãŒå¯èƒ½ã¨ãªã‚‹ã€‚æ¬¡ä¸–ä»£ã®ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿å‡¦ç†åŸºç›¤ã¨ã—ã¦æœŸå¾…ãŒé›†ã¾ã£ã¦ã„ã‚‹ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã§ã‚ã‚‹ã€‚ Sparkã§ã¯Hadoopã¨åŒã˜ãã€å‡¦ç†å¯¾è±¡ã¨ãªã‚‹ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’HDFSã‹ã‚‰èªã¿å–ã‚‹ã“ã¨ãŒã§ãã‚‹ãŒã€ä»¥å¾Œã®å‡¦ç†ã¯åŸºæœ¬çš„ã«ã‚¤ãƒ³ãƒ¡ãƒ¢ãƒªãƒ¼ã§è¡Œã†ã€‚ã“ã®ãŸã‚ã€æ©Ÿæ¢°å¦ç¿’ã‚„ã‚°ãƒ©ãƒ•è¨ˆç®—ã®ã‚ˆã†ã«ç¹°ã‚Šè¿”ã—åž‹ã®è¨ˆç®—ãŒå¤šã„å‡¦ç†ã‚’ã€Hadoopã‚ˆã‚Šã‚‚é«˜é€Ÿã«å®Ÿè¡Œã§ãã‚‹ï¼ˆé–¢é€£è¨˜äº‹ï¼šNECãŒãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã®æ©Ÿæ¢°å¦ç¿’ã‚’é«˜é€ŸåŒ–ã™ã‚‹æŠ€è¡“ã‚’é–‹ç™ºã€ã‚¤ãƒ³ãƒ¡ãƒ¢ãƒªãƒ¼å‡¦ç†ã‚„MPIã‚’å°Žå…¥ï¼‰ã€‚ Sparkã¯ã€ã‚‚ã¨ã‚‚ã¨ç±³Universi
Dryad 2014/06/02
ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°

æ©Ÿæ¢°å¦ç¿’

Spark

Scala
ãƒªãƒ³ã‚¯
Spark shark
ä»Šæ³¨ç›®ã®Spark SQLã€çŸ¥ã£ã¦ãŠããŸã„ãã®æ€§èƒ½ã¨ã¯ 20151209 OSC Enterprise
Dryad 2013/10/28
ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°

æ©Ÿæ¢°å¦ç¿’

Spark

Scala
ãƒªãƒ³ã‚¯
1