KLab-データ分æžã‚°ãƒ«ãƒ¼ãƒ—ã®blog : pythonã§Elastic MapReduce
2013å¹´02月06æ—¥ カテゴリpythonHadoop pythonã§Elastic MapReduce ã¾ã¨ã‚Elastic MapReduceã¯ã€amazonã®AWS上ã§Hadoopを使ãˆã‚‹ã‚µãƒ¼ãƒ“スã§ã™ã€‚一時的ã«ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã‚’ãŸãã•ã‚“ç«‹ã¡ä¸Šã’ã‚‹ã“ã¨ã§ã€é‡ã‚ã®ãƒãƒƒãƒå‡¦ç†ã‚’分散処ç†ã§ãã¾ã™ã€‚料金ã¯EC2インスタンス分 +αã§ä½¿ç”¨ã§ãã¾ã™ã€‚http://aws.amazon.com/jp/elasticmapreduce/#pricing æ¥½ï¼ ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã„ã£ã±ã„ãŸã¡ã‚ã’ã¾ã™ã€è¨ˆç®—ã—ã¾ã™ã€çµæžœã¾ã¨ã‚ã¾ã™ã¨ã„ã†ä¸€é€£ã®å‡¦ç†ã‚’気軽ã«æ›¸ã‘ã¦ã€ã‚³ãƒžãƒ³ãƒ‰ä¸€ç™ºã§å®Ÿè¡Œã§ãã¾ã™ã€‚特ã«é›†è¨ˆå‡¦ç†ã€ãƒãƒƒãƒå‡¦ç†ã«ã¯ä¾¿åˆ©ã§ã™ã€‚ãŸã デãƒãƒƒã‚°ã¯æ‰‹é–“ãŒã‹ã‹ã‚Šã¾ã™ã€‚ MapReduceã«ã¤ã„ã¦MapReduceã¯ã€å¤§é‡ã®ãƒ‡ãƒ¼ã‚¿ã‚’複数ã®ãƒžã‚·ãƒ³ã§åˆ†æ•£ã—ã¦æ‰±ã†ãŸã‚ã®æŠ€è¡“ã§ã™(デザインパターン的ãª)。基本的ãªè€ƒãˆæ–¹ã¯ã€å‡¦ç†ã‚’以下ã®
PyCon 2014ã§Python+EMRã«ã¤ã„ã¦ç™ºè¡¨ã—ã¦ããŸ
今週末ã€PyCon JPã«åˆå‚åŠ ã—ã€åˆæŠ€è¡“系発表ã—ã¦ããŸã€‚ YouTubeã§ã®ç™ºè¡¨ã‚‚ã¿ãŸã‚“ã§ã™ãŒã€å™›ã¿ã¾ãã£ã¦ã¦ãƒ¤ãƒãƒ¡ãªè¡¨æƒ…ã—ã¦ã‚‹ã‚¿ã‚¤ãƒŸãƒ³ã‚°ãŒã‚ã‚‹ã‚“ã§ã™ãŒã€ã‚ã‚Œã¯ç™ºè¡¨ã—ã¦ã‚‹æ™‚間を計測ã—ã¦ã„ã‚‹iPhoneã®é›»æ± ãŒåˆ‡ã‚ŒãŸæ™‚ã¨ã€å¾Œã‚ã«ã€Œã‚ã¨5分ã€ã‚’ãŠçŸ¥ã‚‰ã›ã™ã‚‹ç´™ãŒä¸ŠãŒã£ãŸæ™‚ã§ã™ã€‚スーパー焦ã£ãŸã€‚ ã¾ãŸã€å…¨èˆ¬çš„ã«ãƒ†ãƒ³ã‚·ãƒ§ãƒ³é«˜ã‚ã ã£ãŸã®ã¯å˜ã«å…ƒæ°—ã«ãªã‚‹é£²ã¿ç‰©ã‚’ã‚メã¦ãŸã‹ã‚‰ã§ã™ã€‚ è¬è¾ž PyConã®é‹å–¶ã‚’ã—ã¦ãã ã•ã£ãŸã¿ãªã•ã‚“ã€æœ¬å½“ã«ã‚ã‚ŠãŒã¨ã†ã”ã–ã„ã¾ã—ãŸã€‚ã‚ã®ã‚ˆã†ãªå¤§è¦æ¨¡ãªã‚¤ãƒ™ãƒ³ãƒˆã‚’ã‚ã‚Œã ã‘スムーズã«é‹å–¶ã•ã‚Œã‚‹ã‚¹ã‚ルã€æœ¬å½“ã«å‡„ã„ã§ã™ã€‚ tagomorisã•ã‚“ã®Data Analysis Flowã®å›³ã‚’ベースã«å‰åŠã¯èª¬æ˜Žè¡Œã„ã¾ã—ãŸã€‚æらãã‚ã®å›³ãŒãªã‹ã£ãŸã‚‰ç™ºè¡¨å‰æ—¥ã«ã‚ã®ã‚ˆã†ãªè³‡æ–™ã‚’作るã“ã¨ã¯ã§ããªã‹ã£ãŸã¨æ€ã„ã¾ã™ã€‚ã¼ã‚“ã‚„ã‚Šã—ã¦ã„る概念を図ã«è½ã¨ã—ã“ã‚“ã§å…¬é–‹ã—ã¦é ‚ãã€æœ¬å½“ã«ã‚ã‚ŠãŒã¨ã†ã”ã–ã„ã¾ã™ã€‚(ã¡ã‚‡ã£
Python + Hive on AWS EMR ã§è²§è€…ã®ãƒã‚°ã‚µãƒžãƒª
1. Akira Chiku is an engineer who works on an engineering team. Their requirements include collecting between 10-20GB of data per day from various sources like Hadoop and Hive. 2. Data is collected from sources like Fluentd and parsed using Query String and stored in Hive. It is then processed and visualized. 3. Data can be stored in S3, processed using services like AWS EMR, and visualized in das
Private Presentation
Private content!This content has been marked as private by the uploader.
EMR上ã§Mahoutを使ã£ã¦ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ | DevelopersIO
7月ã«AWS Big Data Blogã¨ã„ã†ãƒ–ãƒã‚°ãŒå§‹ã¾ã£ãŸã®ã§ã™ãŒã€æœ€åˆã®è¨˜äº‹ãŒBuilding a Recommender with Apache Mahout on Amazon Elastic MapReduce (EMR)ã¨ã„ã†ã‚¿ã‚¤ãƒˆãƒ«ã§EMR上ã§Mahoutを使ã£ã¦ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã‚’è¡Œã£ã¦ã¿ã‚‹ã¨ã„ã†ã‚‚ã®ã§ã—ãŸã€‚EMR上ã§Mahoutã¨ã„ã†ã¨æ—¢ã«Amazon Elastic MapReduce入門 〜 Apache Mahoutã§ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ï¼ã¨ã„ã†ã‚¨ãƒ³ãƒˆãƒªãƒ¼ãŒã‚ã‚Šã¾ã™ãŒã€ã“ã¡ã‚‰ã¯Amazon EMR CLIを使ã£ã¦ã„ã‚‹ã“ã¨ã‚‚ã‚ã‚Šã€ãƒ–ãƒã‚°ã«ã—ã¦ã¿ã¾ã—ãŸã€‚ Building a Recommender with Apache Mahout on Amazon Elastic MapReduce (EMR)ã«ã¤ã„㦠ã¾ãšæ©Ÿæ¢°å¦ç¿’ã®æ¦‚è¦ã«ã¤ã„ã¦èª¬æ˜Žã—ãŸä¸Šã§Mahoutを使ã£ã¦
Amazon EMRã§Mahoutã®ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã®ãƒ¯ã‚¯ãƒ¯ã‚¯æ„Ÿã‚’体験ã™ã‚‹ï¼š30分ã§ã§ãる分散レコメンデーション - tetsuya_odakaã®æ—¥è¨˜
追記：2013/9/17 ã“ã®ãƒã‚°ã®ç¶šç·¨ã®æŠ•ç¨¿ãŒå®Œäº†ã—ã¾ã—ãŸã®ã§ã€è¨˜äº‹ã®æœ«ã«ãƒªãƒ³ã‚¯ã‚’è¿½åŠ ã—ã¾ã—ãŸã€‚ã“ã‚Œã§ã€ã“ã®ãƒã‚°ã®æ–¹æ³•ã‚’応用ã—ãŸåˆ†æ•£ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã‚¨ãƒ³ã‚¸ãƒ³ã®æ§‹ç¯‰ã‚‚ã°ã£ã¡ã‚Šï¼ˆã®ã¯ãšï¼‰ã§ã™ã€‚ å…ˆã®ãƒã‚°ã§ã¯ã€Parallel ALS(Alternating Least Squares)を用ã„ãŸãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã®ç†è«–é¢ã®ãƒ•ã‚©ãƒãƒ¼ã‚¢ãƒƒãƒ—ã¨ã€Apache Mahoutã§ã®å®Ÿè£…ã‚’å°‘ã—詳ã—ãã¿ãŸã€‚ ã“ã®ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã¯æ¥µã‚ã¦ã‚·ãƒ³ãƒ—ルã§ã€ã‹ã¤ã€Apache Mahoutã§ã¯Hadoop上ã§ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ルã«å®Ÿè£…ã•ã‚Œã¦ã„る。 Apache Mahoutã¯ç™ºå±•é€”上ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆãªã®ã§ã€ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ルã«å®Ÿè£…ã•ã‚Œã¦ã„るアルゴリズムã¨ãã†ã§ãªã„アルゴリズムãŒã‚ã£ã¦ã€ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã«ã¤ã„ã¦ã„ãˆã°ã€0.8ã§ã¯ã€ã‚¢ã‚¤ãƒ†ãƒ ベースã®ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ã€Slope Oneレコメンデーションã€Parallel ALSを使ã£ãŸãƒ¬
1
ランã‚ング
ランã‚ング
ãŠçŸ¥ã‚‰ã›
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
EMR — boto v2.49.0
Starting an EMR job with Boto – An infinite monkey – Nico Brailovsky's blog
Config file format and location — mrjob v0.5.10 documentation
Modifying the Number of Mappers or Reducers on a Running EMR Cluster - Stochastic Nonsense
609?_encoding=UTF8&jiveRedirect=1
{{#tags}}- {{label}}
{{/tags}}