[B! redshift] yanbeã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yanbe id:yanbe

redshiftã«é–¢ã™ã‚‹yanbeã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

100å€ã§è€ƒãˆã‚‹ - Preferred Networks Research & Development
ç§ãŒæœ€è¿‘å¼·ãå°è±¡ã«æ®‹ã£ãŸè¨€è‘‰ãŒ10å€ã§ç‰©äº‹ã‚’è€ƒãˆã‚‹ã§ã™[wired]ã€‚ ã“ã‚ŒãŒç§ã®è¨˜æ†¶ã®ä¸ã§æ‹¡å¤§è§£é‡ˆã•ã‚Œã€ä»Šã¯100å€ã§ç‰©äº‹ã‚’è€ƒãˆã‚‹ã‚ˆã†ã«ãªã£ã¦ã„ã¾ã™ã€‚ ã€Œ100å€ã€ã¨ã„ã†ã®ã¯ä¸€è¦‹ã™ã‚‹ã¨ä¸å¯èƒ½ãªã“ã¨ã®ä¾‹ãˆã®ã‚ˆã†ã«æ€ãˆã¾ã™ãŒã€æ±ºã—ã¦ãã‚“ãªã“ã¨ã¯ã‚ã‚Šã¾ã›ã‚“ã€‚ ã©ã®ç¨‹åº¦ç¾å®Ÿçš„ã‹ä¾‹ã‚’ã‚ã’ã¦è€ƒãˆã¦ã¿ã¾ã—ã‚‡ã†ã€‚ DWHï¼ˆDBã¨è€ƒãˆã¦ã‚‚è‰¯ã„ã§ã™ï¼‰ã¨ã„ã†åˆ†é‡Žã‚’è€ƒãˆã¦ã¿ã¾ã™*1ã€‚ *1 ã“ã®åˆ†é‡Žã¯å°‚é–€å®¶ã§ã¯ãªã„ã®ã§ã‚ãã¾ã§å¤–ã‹ã‚‰è¦‹ã¦ã„ã‚‹ç´ äººã®æ„è¦‹ã§ã™ã€‚ 2014å¹´10æœˆç¾åœ¨ Google BigQueryã¯1GBã®ä¿å˜ã«æœˆã‚ãŸã‚Š ç´„3å††ã€ã‚¯ã‚¨ãƒªæ™‚1TBã‚¹ã‚ãƒ£ãƒ³ã‚ãŸã‚Š500å††ã¨ã„ã†ä¾¡æ ¼è¨å®šã§ã™ã€‚åŸºæœ¬çš„ãªDBã®æ“ä½œã¯å…¨éƒ¨ã§ãã¦ã€ãã®ä¸Šã§ãƒ¦ãƒ¼ã‚¶ãƒ¼ãŒè‡ªç”±ã«é–¢æ•°ã‚’å®šç¾©ã§ãã¦ã€ç”»é¢ã¨ã¤ãªãŒã£ã¦çµæžœãŒæ•°åç§’ã§è¿”ã£ã¦ãã¦ã§ã™ã€‚ã“ã‚Œã¯ã“ã®åˆ†é‡Žã‚’çŸ¥ã‚‹äººã«ã¨ã£ã¦ã¯è¡æ’ƒçš„ãªä¾¡æ ¼ã§ã™ã€‚ 1æ˜”å‰ã€DWHã®ä¸–ç•Œã§ã¯è£½å“ãŒæ•°åƒä¸‡ã‹ã‚‰æ•°å„„å††
yanbe 2014/10/11
redshift
ãƒªãƒ³ã‚¯
MPP on Hadoop, Redshift, BigQuery - Go ahead!
Twitterã§ã€Œæ—©ãä»Šæµè¡Œã®MPPã®å¤§ã¾ã‹ãªä½¿ã„æ–¹ã®é•ã„æ›¸ã‘ã‚ˆï¼ã€ã¨ã„ã†ãƒ—ãƒ¬ãƒƒã‚·ãƒ£ãƒ¼ãŒåŠç«¯ãªã„ã®ã§ã¦ãã¨ã†ã«æ›¸ãã¾ã™ï¼Žã“ã®è¨˜äº‹ã¯ä¿ºã®çµŒé¨“ã¨å‹‰å¼·ä¼šãªã©ã§ãƒ¦ãƒ¼ã‚¶ã‹ã‚‰èžã„ãŸè©±ã‚’ã‚‚ã¨ã«æ›¸ã„ã¦ã„ã‚‹ã®ã§ï¼Œã™ã¹ã¦ãŒä¿ºã®çµŒé¨“ã§ã¯ã‚ã‚Šã¾ã›ã‚“(ç‰¹ã«BigQuery)ï¼Žå„ç¤¾ã®SAã®äººã¨ã‹ã«èžã‘ã°ï¼Œã‚‚ã£ã¨è‰¯ã„ã‚¢ãƒ—ãƒãƒ¼ãƒã¨ã‹è©³ç´°ã‚’æ•™ãˆã¦ãã‚Œã‚‹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ï¼Ž ã‚ªãƒ³ãƒ—ãƒ¬ãƒŸã‚¹ã®å•†ç”¨MPPã¯ä½¿ã£ãŸã“ã¨ãªã„ã®ã§ãƒŽãƒ¼ã‚³ãƒ¡ãƒ³ãƒˆã§ã™ï¼Ž MPP on Hadoopã§PrestoãŒãƒ¡ã‚¤ãƒ³ãªã®ã¯ä»Šä¸€ç•ªä½¿ã£ã¦ã„ã‚‹ã‹ã‚‰ã§ï¼ŒImpalaãªã©ä»–ã®MPP on Hadoopçš„ãªã‚‚ã®ã‚‚ä¼¼ãŸã‚ˆã†ãªæ„Ÿã˜ã‹ãªã¨æ€ã£ã¦ã„ã¾ã™ï¼Ž ã‚‚ã¡ã‚ã‚“å®Ÿè£…ã®é•ã„ãªã©ãŒã‚ã‚‹ã®ã§ï¼Œãã®è¾ºã¯é©å®œè‡ªåˆ†ã§è£œé–“ã—ã¦ãã ã•ã„ï¼Ž å‰æ ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚’é–‹ç™ºã—ã¦ã„ã¦ï¼Œãã®ãŸã‚ã®è§£æžåŸºç›¤ã‚’ä¸€ã‹ã‚‰ä½œã‚‹ï¼Ž ç°¡å˜ãªã¾ã¨ã‚ ãƒ‡ãƒ¼ã‚¿ã‚’è²¯ã‚ã‚‹æ‰€ãŒä½œã‚Œã‚‹ã®ã§ã‚ã‚Œã°ï¼Œãã“ã«ç›´æŽ¥ã‚¯ã‚¨ãƒªã‚’æŠ•ã’ã‚‰ã‚Œã‚‹Pre
yanbe 2014/09/09
å‚è€ƒã«ãªã£ãŸ

bigquery

redshift

hadoop
ãƒªãƒ³ã‚¯
Amazon Kinesis/Redshiftç·¨ï½žã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’kinesisã§åŠ å·¥ã—ã¦Tableauã§è¡¨ç¤ºã—ã¦ã¿ã‚ˆã†â‘ [å…¨4å›ž]ï½ž
$ vim putrecords.py #!/usr/bin/python # -*- coding: utf-8 -*- import boto.kinesis,datetime,time import threading, Queue, subprocess, sys tailq = Queue.Queue(maxsize=10) def tail_forever(fn): p = subprocess.Popen(["tail", "-f", fn], stdout=subprocess.PIPE) while 1: line = p.stdout.readline() tailq.put(line) if not line: break def main(): fn = sys.argv[1] threading.Thread(target=tail_forever, args=(
yanbe 2014/08/25
redshift

kinesis

aws
ãƒªãƒ³ã‚¯
Engineering
Engineering Building the platform where people around the world come to search, save and shop Our Engineering team is at the core of bringing our platform to life for Pinners worldwide. Working collaboratively and cross-functionally with teams across the company, our engineers tackle growth-driving challenges to build an inspired and inclusive platform for all. Natasha Magliui Engineering Manager,
yanbe 2014/08/25
Pinterestã§ãƒã‚°é›†è¨ˆã‚’Hiveã‹ã‚‰Redshiftã«åˆ‡ã‚Šæ›¿ãˆãŸã‚‰ç´„100å€é€Ÿããªã£ãŸã‚‰ã—ã„ã€‚ via http://stackoverflow.com/questions/24220590/can-i-use-amazon-kinesis-to-connect-to-amazon-redshift-for-data-load-in-every-co

pinterest

aws

hive

redshift
ãƒªãƒ³ã‚¯
Amazon Kinesisã¯Redshiftã«åŒ¹æ•µã™ã‚‹ç ´å£ŠåŠ› â”€Hapyrusè—¤å·æ°ã«èžãAWSãƒ“ã‚¸ãƒã‚¹ã®ãƒˆãƒ¬ãƒ³ãƒ‰ | gihyo.jp
ä»Šå›žã®re:Inventã«å‚åŠ ã—ã¦ã€ã“ã®å‹¢ã„ã¯æ¢ã‚ã‚‰ã‚Œãªã„ã¨ã¤ãã¥ãæ„Ÿã˜ã¾ã™ã€‚ã‚ªãƒ³ãƒ—ãƒ¬ãƒŸã‚¹ã¨AWSã®ã™ã‚Šåˆã‚ã›ã‚‚ã‹ãªã‚Šé€²ã‚“ã§ãŠã‚Šã€ã‚¹ã‚¿ãƒ¼ãƒˆã‚¢ãƒƒãƒ—ã‹ã‚‰ã¯AWSã‚’ãƒ™ãƒ¼ã‚¹ã«ã—ãŸæ–°ãŸãªã‚¯ãƒ©ã‚¦ãƒ‰ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ãŒæ¬¡ã€…ã¨å‡ºã¦ã„ã¾ã™ã€‚ã»ã‹ã®ã‚¯ãƒ©ã‚¦ãƒ‰ãƒ™ãƒ³ãƒ€ã¨ã®å·®ã¯ç¸®ã¾ã‚‹ã©ã“ã‚ã‹ã€æ‹¡ãŒã‚‹ä¸€æ–¹ã§ã—ã‚‡ã†ãã€‚ æ—¢å˜ã®RDBMSã‹ã‚‰Redshiftã«JOIN â”€â â”€å‰å›žãŠè©±ã‚’ä¼ºã£ãŸã¨ãã¯RedshiftãŒå‡ºã¦ã¾ã‚‚ãªã„ã“ã‚ã§ã—ãŸãŒã€ãã®å¾Œã€Hapyrusã®ãƒ“ã‚¸ãƒã‚¹ã‚‚å«ã‚ã€Redsiftã®å¸‚å ´ã«ãŠã‘ã‚‹èªçŸ¥åº¦ã¯ã©ã‚Œãã‚‰ã„é«˜ã¾ã£ã¦ããŸæ„Ÿã˜ã§ã™ã‹ã€‚ è—¤å·ï¼šã‚‚ã®ã™ã”ãå¿œç”¨ç¯„å›²ãŒæ‹¡ãŒã£ã¦ãã¦ã„ã‚‹ã¨å®Ÿæ„Ÿã—ã¾ã™ã€‚ç¾åœ¨AWSãŒå…¬é–‹ã—ã¦ã„ã‚‹äº‹ä¾‹ã‹ã‚‰ç´¹ä»‹ã™ã‚‹ã¨ã€è‹±BBCãŒã‚°ãƒãƒ¼ãƒãƒ«ã§å±•é–‹ã™ã‚‹ã‚³ãƒ³ãƒ†ãƒ³ãƒ„ã®åˆ†æžã‚’RedshiftçµŒç”±ã§ã‚ªãƒ³ãƒ‡ãƒžãƒ³ãƒ‰ã§è¡Œã†ãªã©ã€å¤§è¦æ¨¡ç’°å¢ƒã§ã®åˆ©ç”¨ãŒå¢—ãˆã¦ãã¦ã„ã¾ã™ã€‚ç±³å›½ã§ã¯é‡‘èžã‚„ã‚¢ãƒ‰ãƒ†ã‚¯ã®äº‹ä¾‹ãŒå¤šã„ã§ã™ãã€‚ã¨ãã«ãƒˆ
yanbe 2014/08/25
redshift
ãƒªãƒ³ã‚¯
1