[B! hadoop] karumadoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

karumado id:karumado

hadoopã«é–¢ã™ã‚‹karumadoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Twitterã€æ•°ä¸‡å°ã®Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã¨ã‚³ãƒ¼ãƒ«ãƒ‰ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚’Google Cloud Platformã¸ç§»è¡Œã™ã‚‹ã¨ç™ºè¡¨ ï¼ Publickey
Twitterã€æ•°ä¸‡å°ã®Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã¨ã‚³ãƒ¼ãƒ«ãƒ‰ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚’Google Cloud Platformã¸ç§»è¡Œã™ã‚‹ã¨ç™ºè¡¨ Twitterã¯ã€è‡ªç¤¾ã§é‹å–¶ã—ã¦ã„ã‚‹å¤§è¦æ¨¡ãªHadoopã‚¯ãƒ©ã‚¹ã‚¿ã‚’Google Cloud Platformã¸ç§»è¡Œã™ã‚‹ã¨ç™ºè¡¨ã—ã¾ã—ãŸã€‚ A new collaboration with Google Cloud - Twitter Twitterã¯åŸºæœ¬çš„ã«è‡ªç¤¾ã§ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼ã‚’ä¿æœ‰ã—é‹ç”¨ã—ã¦ãã¾ã—ãŸã€‚2017å¹´1æœˆã«Twitterã®ãƒ–ãƒã‚°ã«æŠ•ç¨¿ã•ã‚ŒãŸè¨˜äº‹ã€ŒThe Infrastructure Behind Twitter: Scaleã€ã«ã‚ˆã‚‹ã¨ã€Twitterã®ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼ã¯5ã¤ã®å¤§é™¸ã«ã‚ã‚Šã€åˆã‚ã›ã¦æ•°åä¸‡å°ã®ã‚µãƒ¼ãƒãŒé‹ç”¨ã•ã‚Œã¦ã„ã¾ã™ã€‚ åŒç¤¾ãŒé‹ç”¨ã—ã¦ã„ã‚‹Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã¯ä¸–ç•Œæœ€å¤§ç´šã®è¦æ¨¡ã¨ã—ã¦ã€ä»¥ä¸‹ã®ã‚ˆã†ã«èª¬æ˜Žã•ã‚Œã¦ã„ã¾ã™ã€‚ Hadoop: We have
karumado 2018/05/16
ãƒ‡ãƒ¼ã‚¿é‡ãŒæ¡é•ã„ã ãªã€‚

Twitter

hadoop

gcp
ãƒªãƒ³ã‚¯
ã€ŒHadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã€ã®æ„å‘³ã‚’æ£ã—ãç†è§£ã™ã‚‹ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã€ã¨ã„ã†è¨€èª¬ã‚’ãŸã¾ã«è¦‹ã‹ã‘ã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ ã‚‚ã¡ã‚ã‚“çµ‚ã‚ã£ã¦ãªã©ã„ã¾ã›ã‚“ã€‚ã—ã‹ã—ã€Hadoopã¨ãã®å–ã‚Šå·»ãç’°å¢ƒãŒå¤‰åŒ–ã—ãŸã®ã¯äº‹å®Ÿã§ã™ã€‚ æœ¬è¨˜äº‹ã§ã¯ã€ã“ã®å¤‰åŒ–ãŒä½•ãªã®ã‹ã‚’æ˜Žã‚‰ã‹ã«ã—ã€ãã®ä¸Šã§ã€ãªãœHadoopã®æ™‚ä»£ã¯çµ‚ã‚ã£ãŸã¨ã„ã†ä¸»å¼µãŒå®Ÿæ…‹ã‚’æ£ã—ãè¡¨ã—ã¦ã„ãªã„ã®ã‹ã‚’èª¬æ˜Žã—ã¦ã„ãã¾ã™ã€‚ DISCLAIMER ç§ã¯Hadoopã‚’ä¸å¿ƒã¨ã—ãŸãƒ‡ãƒ¼ã‚¿åŸºç›¤ã‚’å–ã‚Šæ‰±ã†ãƒ™ãƒ³ãƒ€ãƒ¼ã€Clouderaã®ç¤¾å“¡ã§ã™ã€‚ ä¸ç«‹çš„ã«æ›¸ãã‚ˆã†åŠªã‚ã¾ã™ãŒã€æ‰€å±žçµ„ç¹”ã«ã‚ˆã£ã¦ç™ºç”Ÿã™ã‚‹ãƒã‚¤ã‚¢ã‚¹ã®å®Œå…¨ãªæŽ’é™¤ã‚’ä¿è¨¼ã™ã‚‹ã“ã¨ã¯ã§ãã¾ã›ã‚“ã€‚ ä»¥ä¸Šã‚’ã”äº†æ‰¿ã®ä¸Šã€èªã¿é€²ã‚ã¦ãã ã•ã„ã€‚ è¦ç´„ ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã¯ã€Hadoopã®ç™»å ´ã«ã‚ˆã‚Šéžå¸¸ã«å®‰ä¾¡ã¨ãªã‚Šã€ä»Šã¾ã§ã§ã¯ä¸å¯èƒ½ã ã£ãŸå¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’å–ã‚Šæ‰±ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸã€‚ Hadoopã¯ã€NoSQLãƒ–ãƒ¼ãƒ ã®ä¸ã€å‡¦ç†ã‚¨ãƒ³ã‚¸ãƒ³ã§ã‚ã‚‹MapReduceã¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã‚ã‚‹HDFSãŒ
karumado 2017/07/10
ã‚ã¨ã§èªã‚€

hadoop
ãƒªãƒ³ã‚¯
HDFS Erasure Codingã®ç´¹ä»‹ã¨Yahoo! JAPANã«ãŠã‘ã‚‹é‹ç”¨äº‹ä¾‹
ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10æœˆ1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ãƒ–ãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚LINEãƒ¤ãƒ•ãƒ¼ Tech Blog ã¯ã˜ã‚ã« ã“ã‚“ã«ã¡ã¯ã€‚Yahoo! JAPANã§Hadoopã«æºã‚ã£ã¦ã„ã‚‹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®æµ…æ²¼ã§ã™ã€‚Hadoopã¯å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’åˆ†æ•£å‡¦ç†ã™ã‚‹ãŸã‚ã®ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚½ãƒ•ãƒˆã‚¦ã‚¨ã‚¢ã§ã™ã€‚ã“ã®å¤ã«ãƒªãƒªãƒ¼ã‚¹äºˆå®šã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³3.0ç³»ã«ã¯ã€HDFSã®æ–°æ©Ÿèƒ½ã§ã‚ã‚‹Erasure CodingãŒå°Žå…¥ã•ã‚Œã¾ã™ã€‚Yahoo! JAPANã§ã¯Hadoopã‚³ãƒŸãƒ¥ãƒ‹ãƒ†ã‚£ãƒ¼ã§Erasure Codingã®å®Ÿè£…ã«å‚åŠ ã—ã¦ãã¾ã—ãŸã€‚æœ¬è¨˜äº‹ã§ã¯Erasure Codingã®ä»•çµ„ã¿ã‚’è©³ã—ãè§£èª¬ã—ã€å¼Šç¤¾ã§ã®é‹ç”¨äº‹ä¾‹ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ æ—¢å˜ã®HDFSã®èª²é¡Œ Hadoopã®ä¸æ ¸ã‚’ãªã™åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã®HDFSï¼ˆHadoop Distributed File Systemï¼‰ã¯ã€ãƒžã‚¹
karumado 2017/03/07
ã‚ã¨ã§èªã‚€

hadoop
ãƒªãƒ³ã‚¯
Running Solr on HDFS | Apache Solr Reference Guide 6.6
Solr has support for writing and reading its index and transaction log files to the HDFS distributed filesystem. This does not use Hadoop MapReduce to process Solr data, rather it only uses the HDFS filesystem for index and transaction log file storage. To use Hadoop MapReduce to process Solr data, see the MapReduceIndexerTool in the Solr contrib area. To use HDFS rather than a local filesystem, y
karumado 2016/12/22
solr

hadoop
ãƒªãƒ³ã‚¯
åˆ†æ•£å‡¦ç†ã«å…¥é–€ã—ã¦ã¿ãŸï¼ˆHadoop + Sparkï¼‰ | Casley Deep Innovationsæ ªå¼ä¼šç¤¾ æŠ€è¡“ãƒ–ãƒã‚°
ã“ã‚“ã«ã¡ã¯ã€‚SIéƒ¨ã®è…°å¡šã§ã™ã€‚ RDBã‚„ãƒ‡ãƒ¼ã‚¿ã‚¦ã‚§ã‚¢ãƒã‚¦ã‚¹ã®ä»•äº‹ã«æºã‚ã‚‹ã“ã¨ãŒå¤šã‹ã£ãŸç†è€…ã¯ã€æ•°å¹´å‰ã‹ã‚‰ãŸã³ãŸã³èžã“ãˆãŸãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚„æ©Ÿæ¢°å¦ç¿’ã®ãŸã‚ã®åˆ†æ•£å‡¦ç†ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã«èˆˆå‘³ã‚’è¦šãˆãŸã‚‚ã®ã®ã€ã¤ã„ãžã‚¢ã‚¯ã‚»ã‚¹ã—ãªã„ã¾ã¾ã“ã“ã¾ã§æ¥ã¦ã—ã¾ã„ã¾ã—ãŸã€‚ ä»Šå›žãƒ–ãƒã‚°ã‚’æ›¸ãã«ã‚ãŸã£ã¦ã€ã›ã£ã‹ããªã®ã§ã‚¤ãƒã‹ã‚‰æ‰‹ã•ãã‚Šå…¥é–€ã—ã€ã„ã¾ã•ã‚‰ä»–äººã«èžã‘ãªã„åˆ†æ•£å‡¦ç†ã®åˆæ©ã‹ã‚‰hadoopãƒ»sparkã‚’è§¦ã£ã¦ã¿ã‚‹ã¾ã§ã‚’ã¾ã¨ã‚ãŸã„ã¨æ€ã„ã¾ã™ã€‚ 1.åˆ†æ•£å‡¦ç†ã®åŸºç¤ŽçŸ¥è˜ 1-1.åˆ†æ•£å‡¦ç†ã®å‡¦ç†æ–¹å¼ï¼šMapReduce ã¾ãšåˆ†æ•£å‡¦ç†ã¨ã¯ã€ã²ã¨ã¤ã®è¨ˆç®—å‡¦ç†ã‚’ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã§æŽ¥ç¶šã—ãŸè¤‡æ•°ã®ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ã‚¿ã§åŒæ™‚ä¸¦åˆ—ã§å‡¦ç†ã™ã‚‹ã“ã¨ã§ã™ã€‚ ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æ´»ç”¨ã®å¸‚å ´ãŒæ—¥ã€…å¤§ãããªã‚‹ã«å¾“ã£ã¦ã€æ•°ç™¾ãƒ†ãƒ©ï½žãƒšã‚¿ã®ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚‚çã—ã„ã‚‚ã®ã§ã¯ãªããªã£ã¦ãŠã‚Šã€æ—¥å¸¸çš„ã«ã“ã®è¦æ¨¡ã®ãƒ‡ãƒ¼ã‚¿ã‚’æ‰±ã†ã‚·ã‚¹ãƒ†ãƒ ã§ã¯ã€ç¾å®Ÿçš„ãªæ™‚é–“çš„ãƒ»è²»ç”¨çš„ã‚³ã‚¹ãƒˆã§å‡¦ç†ã™ã‚‹å·¥å¤«ãŒå¿…è¦
karumado 2015/04/24
ã‚ã¨ã§èªã‚€

Spark

Hadoop
ãƒªãƒ³ã‚¯
ç¬¬1å›žã€€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¨é›†åˆçŸ¥ | gihyo.jp
ã¯ã˜ã‚ã« ä»Šå›žã‹ã‚‰9å›žã«æ¸¡ã‚Šã€Hadoopã‚’ä½¿ã£ãŸãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿè£…ã«ã¤ã„ã¦ç´¹ä»‹ã•ã›ã¦ã„ãŸã ãã“ã¨ã«ãªã‚Šã¾ã—ãŸã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã‚’æ§‹ç¯‰ã—ãŸæ–¹ã¯å°‘ãªã„ã¨æ€ã„ã¾ã™ãŒã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã®ã‚µãƒ¼ãƒ“ã‚¹ã«è§¦ã‚Œã¦ã„ã‚‹æ–¹ã¯å¤šã„ã¨æ€ã„ã¾ã™ã€‚ä»Šå›žã®é€£è¼‰ã§ã€èªè€…ã®çš†æ§˜ã«ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®å¯èƒ½æ€§ã¨ãã®å®Ÿè£…ã®é¢ç™½ã•ã‚’ãŠä¼ãˆã§ãã‚Œã°ã¨æ€ã„ã¾ã™ã€‚ã‚ˆã‚ã—ããŠé¡˜ã„ç”³ã—ä¸Šã’ã¾ã™ã€‚ é€£è¼‰ã®äºˆå®šã¯æ¬¡ã®é€šã‚Šã§ã™ã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¨é›†åˆçŸ¥ï¼ˆä»Šå›žï¼‰ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®å®Ÿè£…ã¨èª²é¡Œ å”èª¿ãƒ•ã‚£ãƒ«ã‚¿ãƒªãƒ³ã‚°ï¼ˆå‰ãƒ»å¾Œç·¨ï¼‰ ã‚³ãƒ³ãƒ†ãƒ³ãƒ„ãƒ™ãƒ¼ã‚¹ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ï¼ˆå‰ãƒ»å¾Œç·¨ï¼‰ ä»Šå›žã®è¨˜äº‹ã®ãƒã‚¤ãƒ³ãƒˆã¯ä»¥ä¸‹ã®é€šã‚Šã§ã™ã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã®ç›®çš„ã¯æ°—ä»˜ãã¨é©šãã‚’ä¸Žãˆã‚‹ã“ã¨ ç†æƒ³ã®ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã¯ã‚½ãƒ ãƒªã‚¨ã®ãŠè–¦ã‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã«å¿…è¦ãªã®ã¯å—œå¥½ã¨å°‚é–€æ€§ ã§ã¯ã€æ—©é€Ÿã¯ã˜ã‚ã¾ã—ã‚‡ã†ã€‚ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¨ã¯ï¼Ÿ ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰ã‚·ã‚¹ãƒ†ãƒ ã¯æƒ…å ±ãƒ•ã‚£ãƒ«ã‚¿ãƒªãƒ³ã‚°ã®ä¸€ç¨®ã§ã€å¤§é‡ã®
karumado 2014/11/14
ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‰

hadoop
ãƒªãƒ³ã‚¯
1