[B! hadoop] somemoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

somemo id:somemo

hadoopã«é–¢ã™ã‚‹somemoã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (96)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hive Client Webã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ shib ã‚’ã¤ãã£ãŸ - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
(2013/04/02è¿½è¨˜ see: http://d.hatena.ne.jp/tagomoris/20130402/1364898063 ) ã¾ã å®Œæˆåº¦ãŒã„ã¾ã„ã¡ã ã‹ã‚‰ãªãƒ¼ã¨æ€ã£ã¦ã‚¨ãƒ³ãƒˆãƒªæ›¸ã„ã¦ãªã‹ã£ãŸã‚“ã ã‘ã©LTã§ã—ã‚ƒã¹ã£ã¡ã‚ƒã£ãŸã®ã§ã€ã¡ã‚ƒã‚“ã¨æ›¸ã„ã¦ãŠãã€‚ Hiveã«ã‚¯ã‚¨ãƒªã‚’ç™ºè¡Œã—ã¦çµæžœã‚’ç¢ºèªã™ã‚‹ãŸã‚ã®Webã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚’ç¤¾å†…ç”¨é€”ã§ä½œã£ã¦ã‚‹ã‚“ã ã‘ã©ã€æ™®é€šã«ä»–ã§ã‚‚ä½¿ãˆã‚‹ã¨æ€ã†ã®ã§å…¬é–‹ã—ã¦ã‚ã‚Šã¾ã™ã€‚ tagomoris/shib Â· GitHub ã‚·ãƒ– ã¨èªã¿ã¾ã™ã€‚ ã‚»ãƒƒãƒˆã‚¢ãƒƒãƒ—æ–¹æ³•ã¯ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã‚’å‚ç…§ã®ã“ã¨ã€‚èµ·å‹•ã—ã¦ãƒ–ãƒ©ã‚¦ã‚¶ã§ã‚¢ã‚¯ã‚»ã‚¹ã™ã‚‹ã¨ã“ã‚“ãªç”»é¢ãŒå‡ºã¦ãã‚‹ã€‚ ä½¿ã„ã‹ãŸã¯è¦‹ã‚Œã°ã‚ã‹ã‚‹ã€ã¨æ€ã†ã€‚ãŸã¶ã‚“ã€‚ã‚¯ã‚¨ãƒªã¯å‚ç…§å°‚ç”¨(SELECTã®ã¿)ã€‚ __KEY__ ã¨ã‹ __KEY1__ ã¨ã‹ãŒãƒ—ãƒ¬ãƒ¼ã‚¹ãƒ›ãƒ«ãƒ€ã§ã™ã‚ˆã£ã¦ãã‚‰ã„ã‹ãªã€‚ã‚¨ãƒ‡ã‚£ã‚¿å†…ã§ãƒ—ãƒ¬ãƒ¼ã‚¹ãƒ›ãƒ«ãƒ€ã‚’æ›¸ãã¨ãƒ—ãƒ¬ãƒ¼ã‚¹ãƒ›ãƒ«ãƒ€ã‚’ç½®æ›ã™ã‚‹å€¤
somemo 2015/07/28
hive

hadoop
ãƒªãƒ³ã‚¯
Apache Hiveã®ä»Šã¨ã“ã‚Œã‹ã‚‰
1. Â© Hortonworks Inc. 2011 â€“ 2015. All Rights Reserved Apache Â Hiveã®ä»Šã¨ã“ã‚Œã‹ã‚‰ Yifeng Â Jiang Solutions Â Engineer, Â Hortonworks, Â inc. June Â 11, Â 2015 2. Â© Hortonworks Inc. 2011 â€“ 2015. All Rights Reserved è‡ªå·±ç´¹ä»‹ è’‹â€ƒã€€é€¸å³° Â (Yifeng Â Jiang) â€¢â€¯ Solutions Â Engineer Â @ Â Hortonworks Â Japan â€¢â€¯ HBase Â book Â author â€¢â€¯ â½‡æ—¥æœ¬ã«æ¥ã¦ï¼‘ï¼å¹´ï¦ŽçµŒã¡ã¾ã—ãŸâ€¦ â€¢â€¯ è¶£å‘³ã¯â¼å±±ç™»ã‚Š â€¢â€¯ Twitter: Â @uprush 4. Â© Hortonworks Inc. 2011 â€“ 2015. All Rights Rese
somemo 2015/06/20
hive

hadoop
ãƒªãƒ³ã‚¯
Amazon.co.jp: HBaseå¾¹åº•å…¥é–€: Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã«ã‚ˆã‚‹é«˜é€Ÿãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã®å®Ÿç¾: ã‚µã‚¤ãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ: æœ¬
somemo 2015/01/15
hbase

hadoop
ãƒªãƒ³ã‚¯
LanguageManual LateralView - Apache Hive - Apache Software Foundation
lateralView: LATERAL VIEW udtf(expression) tableAlias AS columnAlias (',' columnAlias)* fromClause: FROM baseTable (lateralView)* DescriptionLateral view is used in conjunction with user-defined table generating functions such as explode(). As mentioned in Built-in Table-Generating Functions, a UDTF generates zero or more output rows for each input row. A lateral view first applies the UDTF to eac
somemo 2014/10/28
hive

hadoop
ãƒªãƒ³ã‚¯
Hcj2014 myui
1. ç‹¬ï¼‰ç”£æ¥æŠ€è¡“ç·åˆç ”ç©¶æ‰€ æƒ…å ±æŠ€è¡“ç ”ç©¶éƒ¨é–€ æ²¹äº• èª m.yui@aist.go.jp, @myui Hivemall: Apache Hiveã‚’ç”¨ã„ãŸã‚¹ã‚±ãƒ¼ãƒ© ãƒ–ãƒ«ãªæ©Ÿæ¢°å¦ç¿’åŸºç›¤ Hadoop Conference Japan 2014 English-version of this slide available on bit.ly/hivemall-slide 1 / 42 2. ç™ºè¡¨ã®æ§‹æˆ â€¢ What is Hivemall â€¢ Why Hivemall â€¢ What Hivemall can do â€¢ How to use Hivemall â€¢ How Hivemall works â€¢ ã‚¤ãƒ†ãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ã¸ã®å¯¾å‡¦æ–¹æ³•ï¼ˆSparkã¨æ¯”è¼ƒï¼‰ â€¢ æ€§èƒ½è©•ä¾¡ â€¢ ã¾ã¨ã‚ Hadoop Conference Japan 2014 åŸºæœ¬çš„ã«Hadoop Summit 2014, San J
somemo 2014/10/28
hadoop

hive
ãƒªãƒ³ã‚¯
å¹³æˆï¼’ï¼‘å¹´åº¦ç”£å¦é€£æºã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦å®Ÿè·µäº‹æ¥ï¼ˆé«˜ä¿¡é ¼ã‚¯ãƒ©ã‚¦ãƒ‰å®Ÿç¾ç”¨ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆåˆ†æ•£åˆ¶å¾¡å‡¦ç†æŠ€è¡“ç‰ã«ä¿‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼ã®é«˜ä¿¡é ¼åŒ–ã«å‘ã‘ãŸå®Ÿè¨¼äº‹æ¥ï¼‰ï¼‰äº‹æ¥æˆæžœå ±å‘Šæ›¸
somemo 2014/10/21
hadoop

distributed system
ãƒªãƒ³ã‚¯
HDFS HA ã‚»ãƒŸãƒŠãƒ¼ #hadoop
2013/05/30ã«é–‹å‚¬ã—ãŸã€HDFS HA(High Availability: é«˜å¯ç”¨æ€§)ã‚»ãƒŸãƒŠãƒ¼ã®è³‡æ–™ã§ã™ã€‚åŒã˜ãã”ç™»å£‡é ‚ã„ãŸã€æ ªå¼ä¼šç¤¾ã‚µã‚¤ãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆã®ä¸ŠåŽŸèª æ§˜ã®è³‡æ–™ã¯â†“ã§ã™ã€‚ http://www.slideshare.net/makotouehara39/cl-st-20130530nnhaRead less
somemo 2014/10/21
hadoop

distributed system

ZooKeeper
ãƒªãƒ³ã‚¯
Treasure Dataã®ã‚µãƒ¼ãƒ“ã‚¹ã¯ã‚¯ãƒ©ã‚¦ãƒ‰ä¸Šã§ã©ã†æ§‹ç¯‰ã•ã‚Œã¦ã„ã‚‹ã®ã‹ï¼ˆå¾Œç·¨ï¼‰ï½žJuly Tech Festa 2013
Treasure Dataã®ã‚µãƒ¼ãƒ“ã‚¹ã¯ã‚¯ãƒ©ã‚¦ãƒ‰ä¸Šã§ã©ã†æ§‹ç¯‰ã•ã‚Œã¦ã„ã‚‹ã®ã‹ï¼ˆå¾Œç·¨ï¼‰ï½žJuly Tech Festa 2013 Treasure Dataã¨ã„ãˆã°ã€æ—¥æœ¬äººãŒã‚·ãƒªã‚³ãƒ³ãƒãƒ¬ãƒ¼ã§å‰µæ¥ã—ãŸãƒ™ãƒ³ãƒãƒ£ãƒ¼ã¨ã—ã¦çŸ¥ã‚‰ã‚Œã¦ã„ã‚‹ä¼æ¥ã€‚ãã®ã‚·ãƒ‹ã‚¢ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ä¸å·çœŸå®æ°ãŒã€7æœˆ14æ—¥ã«è¡Œã‚ã‚ŒãŸJuly Tech Festa 2013ã®åŸºèª¿è¬›æ¼”ã§ã€åŒç¤¾ãŒã‚¯ãƒ©ã‚¦ãƒ‰ä¸Šã§æ§‹ç¯‰ã—ãŸã‚µãƒ¼ãƒ“ã‚¹ã«ã¤ã„ã¦ãã®ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ã‚’ä¸å¿ƒã«è§£èª¬ã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ ã“ã®è¨˜äº‹ã¯ã€ŒTreasure Dataã®ã‚µãƒ¼ãƒ“ã‚¹ã¯ã‚¯ãƒ©ã‚¦ãƒ‰ä¸Šã§ã©ã†æ§‹ç¯‰ã•ã‚Œã¦ã„ã‚‹ã®ã‹ï¼ˆå‰ç·¨ï¼‰ï½žJapan Tech Festa 2013ã€ã®ç¶šãã§ã™ã€‚ ãƒ‡ãƒ¼ã‚¿ã‚’è§£æžã™ã‚‹ã€ŒPlazmaã€ã®ä»•çµ„ã¿ ãƒ‡ãƒ¼ã‚¿ã‚’è§£æžã™ã‚‹ã¨ã“ã‚ã§ã¯ã€ŒPlazmaã€ã¨å‘¼ã¶ã€Hadoopã®ã‚¨ã‚³ã‚·ã‚¹ãƒ†ãƒ ã¨ã‚«ãƒ©ãƒ ã‚¹ãƒˆã‚¢ãªã©ã‚’çµ„ã¿åˆã‚ã›ãŸã‚‚ã®ã‚’ç”¨ã„ã¦ã„ã¾ã™ã€‚
somemo 2014/09/28
hadoop

fluentd
ãƒªãƒ³ã‚¯
Cloudera Manager + EC2 ã§ Mahout ã‚’5åˆ†ï¼ˆãã‚‰ã„ï¼‰ã§å‹•ã‹ã™ - Qiita
ã‚´ãƒ¼ãƒ« Hadoopã‚„Mahoutã¯ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã‹ã‚‰å‹•ãçŠ¶æ…‹ã¾ã§æŒã£ã¦ã„ãã®ã«ã€æ…£ã‚Œãªã„ã†ã¡ã¯çµæ§‹æ™‚é–“ã‹ã‹ã£ã¦ã—ã¾ã„ã¾ã™ã€‚ ã—ã‹ã—ã€ã“ã‚Œã‚’ã•ãã£ã¨çµ‚ã‚ã‚‰ã›ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ ãã†ã€Cloudera Managerãªã‚‰ãã€‚ ã•ã‚‰ã«ã“ã‚Œã«EC2ã‚’çµ„ã¿åˆã‚ã›ã‚‹ã¨ã€ã•ã‚‰ã«ã•ãã£ã¨çµ‚ã‚ã‚‰ã›ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ ãã‚“ãªè¨³ã§ã€ä»Šå›žã¯ClouderaManager + EC2ã®çµ„ã¿åˆã‚ã›ã‚’åˆ©ç”¨ã—ã¦ã€ã¾ã£ã•ã‚‰ãªçŠ¶æ…‹ã‹ã‚‰ Hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« ã‚¯ãƒ©ã‚¹ã‚¿æ§‹ç¯‰ Mahoutã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« ãƒ‡ãƒ¢å®Ÿè¡Œ ã¾ã§ã‚’5åˆ†ï¼ˆãã‚‰ã„ï¼‰ã§çµ‚ã‚ã‚‰ã‚‹æ‰‹é †ã‚’ç¤ºã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ â€»æ³¨ - ã‚µãƒ¼ãƒãŒEC2ã§ãªãã¦ã‚‚ã€å¤§ä½“ä¼¼ãŸæ„Ÿã˜ã§ã„ã‘ã‚‹ã¨æ€ã„ã¾ã™ - ã»ã‚“ã¨ã«5åˆ†ï¼ˆãã‚‰ã„ï¼‰ã§çµ‚ã‚ã‚‹ã‹ã©ã†ã‹ã¯ã€å›žç·šé€Ÿåº¦ã‚„ã‚µãƒ¼ãƒã‚¹ãƒšãƒƒã‚¯ã«ã‚ˆã‚Šã¾ã™ã€‚ä¿¡ã˜ã¡ã‚ƒã„ã‘ã¾ã›ã‚“ã€‚ å‰æã¨ã—ã¦ã„ã‚‹çŸ¥è˜ EC2ã«é–¢ã™ã‚‹åŸºæœ¬çš„ãªçŸ¥è˜ Hadoopã«é–¢ã™ã‚‹åŸºæœ¬çš„ãªçŸ¥è˜ ss
somemo 2014/09/25
hadoop

aws
ãƒªãƒ³ã‚¯
Python + Hive on AWS EMR ã§è²§è€…ã®ãƒã‚°ã‚µãƒžãƒª
1. Akira Chiku is an engineer who works on an engineering team. Their requirements include collecting between 10-20GB of data per day from various sources like Hadoop and Hive. 2. Data is collected from sources like Fluentd and parsed using Query String and stored in Hive. It is then processed and visualized. 3. Data can be stored in S3, processed using services like AWS EMR, and visualized in das
somemo 2014/09/14
Python

Hadoop

aws
ãƒªãƒ³ã‚¯
Hadoopãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ãƒ¥ãƒ¼ã‚¿ MapRã¯ã©ã†ã‚„ã£ã¦HDFSã®å¼±ç‚¹ã‚’å…‹å¾©ã—ãŸã®ã‹ï¼Ÿ
åŠ¹çŽ‡çš„ã«ä¸¦åˆ—åˆ†æ•£å‡¦ç†ãŒè¡Œãˆã‚‹Hadoopã®ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ã¯ã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿æ™‚ä»£ã«ã‹ãªã‚Šæœ‰åŠ¹ãªä»•çµ„ã¿ã ã‚ã†ã€‚ã€Œã¨ã¯ã„ãˆã€Hadoopã‚’ã„ã–ä¼æ¥ãŒåˆ©ç”¨ã—ã‚ˆã†ã¨ã™ã‚‹ã¨ã€ã•ã¾ã–ã¾ãªå•é¡ŒãŒç™ºç”Ÿã—ã¾ã™ã€‚ç™ºç”Ÿã™ã‚‹å•é¡Œã®å¤šãã¯ã€Hadoopã®åˆ†æ•£å‡¦ç†ã‚’å®Ÿç¾ã™ã‚‹ãŸã‚ã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã€ŽHDFSã€ã«ã‚ã‚Šã¾ã™ã€ã¨è¿°ã¹ã‚‹ã®ã¯ã€ãƒžãƒƒãƒ—ã‚¢ãƒ¼ãƒ«ãƒ»ãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚ºã®ã‚»ãƒ¼ãƒ«ã‚¹ãƒ‡ã‚£ãƒ¬ã‚¯ã‚¿ãƒ¼ å¹³æž—è‰¯æ˜æ°ã ã€‚ ã“ã®HDFSã®èª²é¡Œã‚’è§£æ±ºã™ã‚‹ãŸã‚ã«ã€MapRã§ã¯ç‹¬è‡ªã®MapRãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã‚’ç”Ÿã¿å‡ºã—ãŸã€‚ãã®MapRãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã®Lockless Storage Serviceã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ã‚’è€ƒãˆå‡ºã—ãŸã®ãŒã€åŒç¤¾ã®å‰µæ¥ãƒ¡ãƒ³ãƒãƒ¼ã§ã‚ã‚ŠChief Techno logy Officerã®M.C. Srivasæ°ã€‚å½¼ã¯ã€NetAppã®å‰èº«ã§ã‚ã‚‹Spinnaker Networksã§Chief Architectã‚’å‹¤ã‚ãŸäººç‰©ã§ã‚ã‚Šã€å°‚é–€ã¯ãƒ•ã‚¡
somemo 2014/09/04
hadoop
ãƒªãƒ³ã‚¯
Hadoop Tutorial Series, Issue #3: Counters In Action
somemo 2014/09/02
hadoop
ãƒªãƒ³ã‚¯
ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ãƒ«ãªã‚·ã‚¹ãƒ†ãƒ ã®ãŸã‚ã®HBaseã‚¹ã‚ãƒ¼ãƒžè¨è¨ˆ #hcj13w
Cloudera HBase ãƒˆãƒ¬ãƒ¼ãƒ‹ãƒ³ã‚°: http://tiny.cloudera.com/jptraininghbase Hadoop Conference Japan 2013 Winter ã§ç™ºè¡¨ã—ãŸã€HBaseã®ã‚¹ã‚ãƒ¼ãƒžè¨è¨ˆã«é–¢ã™ã‚‹è³‡æ–™ã§ã™ã€‚ Cloudera ã® HBase ã‚µãƒãƒ¼ãƒˆã€Cloudera Enterprise RTD http://tiny.cloudera.com/jpcertdRead less
somemo 2014/08/28
hbase

hadoop
ãƒªãƒ³ã‚¯
[hadoop][AWS] hadoopã‚’å§‹ã‚ã¦ã¿ã‚‹(7) ï½žEC2Ã—MapRÃ—Eclipseã§é–‹ç™ºç’°å¢ƒã‚’ä½œã£ã¦ã¿ã‚‹ï½ž: HelloWorldã¯çµ‚ã‚ã‚‰ãªã„
EC2ä¸Šã«MapRã‚µãƒ¼ãƒãŒç”¨æ„ã§ããŸã®ã§ã€æ¬¡ã¯ã‚¯ãƒ©ã‚¤ã‚¢ãƒ³ãƒˆWindowsä¸Šã«é–‹ç™ºç’°å¢ƒã‚’æ§‹ç¯‰ã™ã‚‹ã€‚ ã¾ãšã¯ã€çŽ‹é“ã§MapReduceã‚’Javaã§é–‹ç™ºã—ã¦ã¿ã‚‹ã“ã¨ã«ã—ãŸã€‚ Javaã¨ã„ãˆã°Eclipseï¼ï¼ï¼ ã£ã¦ã“ã¨ã§ã€EC2Ã—MapRÃ—Eclipseã§é–‹ç™ºç’°å¢ƒã‚’ä½œã£ã¦ã¿ã‚‹ã“ã¨ã«ã—ãŸã€‚ è©¦è¡ŒéŒ¯èª¤ã‚’ã—ãŸçµæžœã®æ‰‹é †ã‚’ãƒ¡ãƒ¢ã—ã¦ãŠãã€‚ â‘ Javaã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ« é–‹ç™ºæ‹…å½“ã®Windowsç’°å¢ƒã«JDKã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã™ã‚‹ã€‚ MapRãŒä¸ã§æŒã¤hadoopãŒ0.20.2ãªã®ã§ã€ãã®hadoopãŒå‹•ãJavaãªã‚‰ä½•ã§ã‚‚è‰¯ã„ã¨æ€ã†ãŒã€MapRã®å‹•ä½œè¦ä»¶ãŒJava SE 1.6.0_24ä»¥é™ãªã®ã§ã€å¿µã®ãŸã‚ã€ãã®è¦ä»¶ã¯æº€ãŸã—ã¦ãŠã„ãŸæ–¹ãŒè‰¯ã„ã ã‚ã†ã€‚ â‘¡Eclipseã®ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ï¼†è§£å‡ ä¾å˜é–¢ä¿‚ãŒé¢å€’ãªã®ã§ã€ç§ã¯ã„ã¤ã‚‚æ—¥æœ¬èªžåŒ–ã•ã‚ŒãŸAll in One Eclipseã‚’ä½¿ã£ã¦ã„ã‚‹ã€‚ ã“ã“ã§ã‚‚ã€ä»¥ä¸‹ã‹ã‚‰ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰
somemo 2014/08/25
eclipse

hadoop
ãƒªãƒ³ã‚¯
HBase Hadoopãƒ¡ãƒ¢(Hishidama's HBase Hadoop Memo)
S-JIS[2010-07-11/2012-04-28] å¤‰æ›´å±¥æ´ HBaseã®Hadoopé–¢é€£ã‚¯ãƒ©ã‚¹ HBaseã«Hadoopã§ã‚¢ã‚¯ã‚»ã‚¹ã™ã‚‹ç‚ºã®ã‚¯ãƒ©ã‚¹ã€‚ WordCountã‚µãƒ³ãƒ—ãƒ« HBaseã®ãƒ†ãƒ¼ãƒ–ãƒ«ã®å…¨é …ç›®ã‚’èªã¿è¾¼ã¿ã€å˜èªžæ•°ã‚’æ•°ãˆã‚‹ã‚µãƒ³ãƒ—ãƒ«ã€‚ å‡ºåŠ›å…ˆã¯ãƒ•ã‚¡ã‚¤ãƒ«ã€‚ï¼ˆé€šå¸¸ã®Hadoopã¨åŒã˜ï¼‰ ï¼ˆâ†’HBaseã®ãƒ†ãƒ¼ãƒ–ãƒ«ã«å‡ºåŠ›ã™ã‚‹WordCountï¼‰ package jp.hishidama.sample.hbase; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; impor
somemo 2014/08/25
hadoop

hbase
ãƒªãƒ³ã‚¯
HBaseã®ãƒãƒ«ã‚¯ãƒãƒ¼ãƒ‰ã¨ã„ã†ã‹HFileç”Ÿæˆæ–¹æ³• - wyukawa's diary
HBaseã«ãƒ‡ãƒ¼ã‚¿ã‚’ãƒãƒ¼ãƒ‰ã™ã‚‹å ´åˆã¯ãƒãƒ«ã‚¯ãƒãƒ¼ãƒ‰ã‚’ä½¿ã†ã®ãŒæ€§èƒ½çš„ã«ã¯è‰¯ã„ã¨æ€ã„ã¾ã™ã€‚ HTable#putã ã¨WALã‚’ã‚ªãƒ•ã«ã—ã¦ã‚‚é…ã„ã‚“ã§ã™ã‚ˆãã€‚ ãƒãƒ«ã‚¯ãƒãƒ¼ãƒ‰ã«é–¢ã—ã¦ã¯é¦¬æœ¬ã®12.2.3ã‚’èªã‚€ã¨è‰¯ã„ã¨æ€ã„ã¾ã™ã€‚ ãƒãƒ«ã‚¯ãƒãƒ¼ãƒ‰ã‚’ä½¿ã†ãŸã‚ã«ã¯HBaseç”¨ã®ãƒ‡ãƒ¼ã‚¿ãƒ•ã‚¡ã‚¤ãƒ«ã€HFileã‚’ç”Ÿæˆã™ã‚‹å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ MapReduceã‚¸ãƒ§ãƒ–ã§HFileã‚’ç”Ÿæˆã™ã‚‹ãŸã‚ã«ã¯å‡ºåŠ›ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã¨ã—ã¦HFileOutputFormatã‚’ä½¿ã„ã¾ã™ã€‚ MapReduceã‚¸ãƒ§ãƒ–ã‚’åŠ¹çŽ‡çš„ã«å‹•ä½œã•ã›ã‚‹ãŸã‚ã«ã¯å‡ºåŠ›ã—ãŸHFileã‚’1ã¤ã®ãƒªãƒ¼ã‚¸ãƒ§ãƒ³ã«åŽã¾ã‚‹ã‚ˆã†ã«ã™ã‚‹ã‚ã‘ã§ã™ãŒã€ãã®è¾ºã®å‡¦ç†ã¯HFileOutputFormat#configureIncrementalLoadã‚’å‘¼ã¹ã°ã„ã„ã§ã™ã€‚å‘¼ã¹ã°TotalOrderPartitionerã‚’partitionerã¨ã—ã¦ã‚»ãƒƒãƒˆã—ã¦ã‚ˆã‚ã—ãã‚„ã£ã¦ãã‚Œã¾ã™ã€‚ MapReduceã§
somemo 2014/08/25
hbase

hadoop
ãƒªãƒ³ã‚¯
MapReduceã§HBaseã¸æ›¸ãè¾¼ã‚€ã¨ãã®æ³¨æ„ç‚¹
ã›ã¡ã‚ãƒ¼ @sechiro HBaseã®TableOutputFormatã®å±é™ºæ€§ã£ã¦ã€å‰ã«è¦‹ã¦ãŸautoFlushãŒFalseã«ãªã£ã¦ã‚‹ã‹ã‚‰ã£ã¦ã“ã¨ã‹ãªï¼Ÿ Hadoop Hacksã‚’è¦‹ã‚Œã°ã‚ˆã„ã‹ã€‚å˜ç´”ãªè¿½è¨˜ã§ä½¿ã†ãªã‚‰å•é¡Œãªã„ã‘ã©ã€ãƒ‡ãƒ¼ã‚¿ã®æ›´æ–°ã¿ãŸã„ãªå‡¦ç†ã ã¨TaskãŒè½ã¡ãŸæ™‚ã«ã¯æˆ»ã›ãªã„ã‚ˆã£ã¦ã“ã¨ã ã¨äºˆæƒ³ 2012-04-29 01:33:44
somemo 2014/08/25
hbase

hadoop
ãƒªãƒ³ã‚¯
HBaseã¸ãƒ‡ãƒ¼ã‚¿æŠ•å…¥(importtsvã®ä½¿ã„æ–¹) - ãã¾ãã¾ãƒ¼Z
create 'sample', 'data' ('a'..'z').each {|i| put 'sample', i, 'data:alpha', i} scan 'samples' ROW COLUMN+CELL a column=data:alpha, timestamp=1333387516755, value=a b column=data:alpha, timestamp=1333387516772, value=b ... æ‰‹ã”ã‚ãªfixtureä½œæˆã«ã¯ã‚ˆã„ãŒã€ã“ã‚Œã§1å„„ä»¶ã‚„ã‚‹æ°—ã¯ã—ãªã„ b) importtsv ã®ä½¿ã„æ–¹ æ—¢å˜ã®ã‚¤ãƒ³ãƒãƒ¼ãƒˆãƒ„ãƒ¼ãƒ«ã‚’åˆ©ç”¨ã™ã‚‹æ–¹æ³•ã€‚ å…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã¯tsv,csvã«é™å®šã•ã‚Œã‚‹ãŒã€hbase.jarã«å«ã¾ã‚Œã‚‹ImportTsvãŒç›®çš„ã«åˆè‡´ã™ã‚‹ã€‚ å®Ÿè¡Œã«ã¯hbase.jarãƒ•ã‚¡ã‚¤ãƒ«ã®ãƒ•ãƒ«ãƒ‘ã‚¹ãŒå¿…è¦ãªã®ã§ã€ã¾ãšã¯jarã‚’locateãªã©ã§æŽ¢ã™ã€‚ % lo
somemo 2014/08/25
hbase

hadoop
ãƒªãƒ³ã‚¯
é›»åæ›¸ç±ã€Žå®Ÿè·µ æ©Ÿæ¢°å¦ç¿’ã€ã®ç„¡æ–™ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ãŒå¯èƒ½ã«ï¼
æ©Ÿæ¢°å¦ç¿’ã®åŸºç¤Žã«ã¤ã„ã¦è§£èª¬ã—ãŸé›»åæ›¸ç±ã€Žå®Ÿè·µ æ©Ÿæ¢°å¦ç¿’ â€• ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã«ãŠã‘ã‚‹ã‚¤ãƒŽãƒ™ãƒ¼ã‚·ãƒ§ãƒ³ã€ãŒHadoopæƒ…å ±ã‚µã‚¤ãƒˆã€ŒHadoop Timesã€ã‹ã‚‰ç„¡æ–™ã§ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã§ãã¾ã™ã€‚ è‘—è€…ã¯ã€ŽMahoutã‚¤ãƒ³ãƒ»ã‚¢ã‚¯ã‚·ãƒ§ãƒ³ã€ã®è‘—ä½œã§ã‚‚ãŠé¦´æŸ“ã¿ã®Ted Dunningã¨Ellen Friedmanã€‚Apache Mahoutãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã§ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆãƒžãƒã‚¸ãƒ¡ãƒ³ãƒˆå§”å“¡ã‚„ã‚³ãƒŸãƒƒã‚¿ã¨ã—ã¦æ´»èºã—ãªãŒã‚‰ã€MapRç¤¾ã§ãƒãƒ¼ãƒ•ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒˆã‚„ã‚³ãƒ³ã‚µãƒ«ã‚¿ãƒ³ãƒˆã‚’å‹™ã‚ã¦ã„ã‚‹ä¸¡æ°ãŒã€æ©Ÿæ¢°å¦ç¿’ã®åˆå¦è€…ã®ãŸã‚ã«æ›¸ãä¸‹ã‚ã—ãŸã®ä¸€å†Šã§ã™ã€‚50ãƒšãƒ¼ã‚¸ã»ã©ã®æ‰‹è»½ãªãƒœãƒªãƒ¥ãƒ¼ãƒ ãªãŒã‚‰ã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã‚’æ´—ç·´ã•ã›ã‚‹ãŸã‚ã®é‡è¦ãªã‚¨ãƒƒã‚»ãƒ³ã‚¹ãŒè©°ã¾ã£ã¦ã„ã¾ã™ã€‚ æ©Ÿæ¢°å¦ç¿’ã¨ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã«ã¤ã„ã¦å¦ç¿’ã—ãŸã„ã‘ã‚Œã©ã€ã©ã“ã‹ã‚‰å§‹ã‚ã‚Œã°ã‚ˆã„ã‹è¿·ã£ã¦ã„ã‚‹ã¨ã„ã†æ–¹ã¯ã€æ‰‹å§‹ã‚ã«æœ¬æ›¸ã‚’èªã‚€ã¨ã“ã‚ã‹ã‚‰å§‹ã‚ã¦ã¿ã¦ã¯ã„ã‹ãŒã§ã—ã‚‡ã†ã‹ã€‚ H
somemo 2014/08/22
hadoop

Mahout

Machine

learning
ãƒªãƒ³ã‚¯
Hadoop World 2011: Replacing RDB/DW with Hadoop and Hive for Telco Big Data - Jason Han, NexR
2. NexR Introduction Big data analytics firm Working on Hadoop and big data for 5 years Provided a NexR Hadoop solution to all major Korea telcos (KT, SKT, LG U+) Leading a Korean Hadoop community and holding Hadoop conferences Products NexR Data Analytics Platform (NDAP) iCube Cloud: cloud computing platform (like OpenStack) Massive em ail archiving solution (presented in Hadoop World 2009) Next R
somemo 2014/08/12
hadoop

hive
ãƒªãƒ³ã‚¯
1 2 3 4 5 æ¬¡ã®ãƒšãƒ¼ã‚¸