[B! EMR] toritori0318ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

toritori0318 id:toritori0318

EMRã«é–¢ã™ã‚‹toritori0318ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (4)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Apache Hive ã«ã‚ˆã‚‹é–‹ç™º (Frozen) - Ayutaya.com
ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã€‚è¾žæ›¸é †ã€‚partition ã¯ã•ã‚‰ã«ã‚µãƒ–ãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã€‚ã“ã®ã¸ã‚“ã€ã©ã“ã‹ã«ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆã¯ã‚ã‚Šã¾ã›ã‚“ã‹ï¼Ÿ Hadoop Hive - Hadoop Hive- Data Manipulation Statements EMR ã¯ã€æœ€åˆã®å…¥åŠ›ã¨æœ€å¾Œã®å‡ºåŠ›ã«ã ã‘ã¯ã€å¤–éƒ¨ã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ï¼ˆæ™®é€šã¯ S3ï¼‰ã‚’åˆ©ç”¨ã›ã–ã‚‹ã‚’å¾—ãªã„ãŸã‚ã€HDFS ã®ãƒãƒ¼ã‚«ãƒªãƒ†ã‚£ãŒç”Ÿãã¾ã›ã‚“ã€‚ãªã‚‹ã¹ã Hive ã®ä¸é–“ãƒ‡ãƒ¼ã‚¿ã‚’ä½¿ã‚ãªã„ã‚ˆã†ã«ã—ãŸã„ã®ã§ã€ä»¥ä¸‹ã®ã‚ˆã†ã«ã™ã‚Œã°è‰¯ã„ã®ã ã¨æ€ã‚ã‚Œã¾ã™ã€‚ ã‚µãƒ³ãƒ—ãƒ«ã¨ã—ã¦ã€ä¸‹è¨˜ã® 3 ã¤ã®å…¥åŠ›ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ç”¨æ„ã—ã¾ã™ã€‚ã“ã“ã§ã¯ã€ã‚«ãƒ©ãƒ ã®åŒºåˆ‡ã‚Šã«ã¯ã‚¿ãƒ–ã‚’ä½¿ã£ã¦ã„ã¾ã™ã€‚ input/employees/foo.tsv10 NAKA Kiichiro 100 20 LUCAS Geroge 200 30 KANAME Madoka 100input/employees/bar.tsv40 J
toritori0318 2013/03/05
EMR

hive

aws
ãƒªãƒ³ã‚¯
AWSã®Amazon Elastic MapReduce(EMR)ã§Hiveã‚’ä½¿ã†æ‰‹é † | takemikami's note
è‡ªå‰ã§Hadoopã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã›ãšã¨ã‚‚ã€ AmazonWebService(AWS)ã§æ‰‹è»½ã«Hadoopã‚’åˆ©ç”¨ã™ã‚‹äº‹ãŒã§ãã‚‹ã¨ã„ã†ã“ã¨ã§ã€ Elastic MapReduce(EMR)ç’°å¢ƒã§Hiveã‚’åˆ©ç”¨ã—ã¦ã¿ã¾ã—ãŸã€‚ ã“ã“ã§ã¯ã€ä»¥ä¸‹ã®ã‚ˆã†ãªæ‰‹é †ã§Hiveã‚’åˆ©ç”¨ã™ã‚‹æ‰‹é †ã‚’ç¤ºã—ã¾ã™ã€‚ S3ã«Hiveç”¨ã®bucketã‚’ä½œæˆAmazon Elastic MapReduce Ruby Clientã®ã‚»ãƒƒãƒˆã‚¢ãƒƒãƒ—S3ã«ãƒ‡ãƒ¼ã‚¿ã‚’é…ç½®Hiveã‚¤ãƒ³ã‚¿ãƒ©ã‚¯ãƒ†ã‚£ãƒ–ãƒ¢ãƒ¼ãƒ‰ã§Elastic MapReduceã‚’èµ·å‹•ä½œæˆã—ãŸHiveã‚¹ã‚¯ãƒªãƒ—ãƒˆã‚’ãƒãƒƒãƒå‡¦ç†ã§å®Ÿè¡ŒS3ã«Hiveç”¨ã®bucketã‚’ä½œæˆHiveç”¨ã®ãƒ‡ãƒ¼ã‚¿ã‚’ä¿å˜ã™ã‚‹ãŸã‚ã€S3ã«bucketã‚’ä½œã‚Šã¾ã™ã€‚ ã€ŒAWS Management Consoleã€ã§S3ã‚’é¸ã³ã€ ã€ŒBucketsã€ã®ã€ŒCreate Bucketã€ã‚’é¸æŠžã—ã¦ã€ ã€ŒBucket Nameã€ã‚’
toritori0318 2013/03/05
EMR

aws

hive
ãƒªãƒ³ã‚¯
EMRã£ã¦ãªã‚“ã˜ã‚ƒï¼Ÿï¼ˆãƒã‚°ã€ã‚†ã‚Šã‹ã”ã‹ã‚‰å¢“å ´ã¾ã§ï¼‰
AWS Advent Calendar 2012 ã«å‚åŠ ã•ã›ã¦ã‚‚ã‚‰ã„ã¾ã—ãŸã€‚21æ—¥æ‹…å½“ã§ã™ã€‚ AWSã§WEBã‚µã‚¤ãƒˆã‚’ãƒ›ã‚¹ãƒˆã™ã‚‹ã¨ãã®ãƒã‚°ã®ãƒ©ã‚¤ãƒ•ã‚µã‚¤ã‚¯ãƒ«ã«ã¤ã„ã¦ã€ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ WEBã‚µãƒ¼ãƒãƒ¼ã®ä¸€èˆ¬çš„ãªãƒã‚°ã®æ‰±ã„ã¯ä»¥ä¸‹ã®ã‚ˆã†ãªæ„Ÿã˜ã ã¨æ€ã„ã¾ã™ã€‚ å„ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’1ç®‡æ‰€ã«é›†ã‚ã‚‹ è¤‡æ•°ã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ï¼‘ãƒ•ã‚¡ã‚¤ãƒ«ã«ã¾ã¨ã‚ã¦ã‚½ãƒ¼ãƒˆã™ã‚‹ é›†è¨ˆã‚’ã™ã‚‹ å¤ã„ãƒã‚°ã®ãƒãƒƒã‚¯ã‚¢ãƒƒãƒ—ã‚’ã¨ã‚Šã€å‰Šé™¤ã™ã‚‹ ã“ã‚Œã‚’AWSã§è¡Œãªã£ã¦ã¿ã‚‹ã¨ä¾‹ãˆã°ä»¥ä¸‹ã®ã‚ˆã†ã«ãªã‚Šã¾ã™ã€‚ fluentdã‚’ä½¿ã£ã¦å„ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã®ãƒã‚°ã‚’ï¼‘ã¤ã®ãƒã‚±ãƒƒãƒˆAã«é€ã‚‹ EMRã§ï¼‘ã¤ã«ã¾ã¨ã‚ã¦ã‚½ãƒ¼ãƒˆã€‚åˆ¥ã®ãƒã‚±ãƒƒãƒˆBã«ä¿å˜ EMRã§é›†è¨ˆã‚‚ã—ã¦ã€åˆ¥ã®ãƒã‚±ãƒƒãƒˆBã«ä¿å˜ ãƒã‚±ãƒƒãƒˆAã®æœŸé™ãŒéŽãŽãŸã‚‚ã®ã‚’Glacierã«é€ã‚‹ å›³ã«ã™ã‚‹ã¨ä»¥ä¸‹ã®ã‚ˆã†ãªã‚¤ãƒ¡ãƒ¼ã‚¸ã§ã™ã€‚ ï¼‘ï¼Žå„ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’1ç®‡æ‰€ã«é›†ã‚ã‚‹ ã“ã‚Œã¯fluentdã§tailâ†’s3ã§è¡Œ
toritori0318 2012/12/21
aws

s3

fluentd

EMR
ãƒªãƒ³ã‚¯
2012/03/03 JAWS Summit 2012 PROGRAM Day2 ä¸Šç´šè€…å‘ã‘ãƒ–ãƒ¼ãƒˆã‚ãƒ£ãƒ³ãƒ— #jawsug #ãƒ–ãƒ¼ãƒˆã‚ãƒ£ãƒ³ãƒ—
PROGRAM DAY2 | JAWS Summit 2012 http://jaws-ug.jp/summit2012/day2.html AWSã®ä¸ã§ã‚‚ã€ç‰¹å¾´çš„ãª3ã¤ã®ã‚µãƒ¼ãƒ“ã‚¹ã«ã¤ã„ã¦ã€å®Ÿéš›ã®åˆ©ç”¨è€…ãŒTIPSã‚’å«ã‚ã”ç´¹ä»‹ã™ã‚‹ä¸Šç´šè€…å‘ã‘ã‚»ãƒƒã‚·ãƒ§ãƒ³ã€‚AWSã®ã‚¨ã‚ã‚¹ãƒ‘ãƒ¼ãƒˆã‚’ç›®æŒ‡ã™æ–¹ã«ã¯å¿…è¦‹ã®ã‚»ãƒƒã‚·ãƒ§ãƒ³ã§ã™ã€‚ ç¶šãã‚’èªã‚€
toritori0318 2012/03/04
aws

RDS

EMR

DynamoDB
ãƒªãƒ³ã‚¯
1