Kafkaã«ã‚ˆã‚‹ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ 処ç†
This document discusses Kafka, a distributed messaging system originally created by LinkedIn and now an Apache project. It provides an overview of how Kafka works and how various companies use it, including for log processing (LinkedIn), analytics (Facebook, Twitter, Google), and integrating with other technologies like Hadoop, Zookeeper, HBase and Storm. It also covers Kafka's scalability, perfor
S4
S4 - Presentation Transcript S4: the open-source distributed stream computing platformNachi Ueno 自己紹介 日本ã®ã©ã“ã‹ã®ç ”究所ã¨å‘¼ã°ã‚Œã¦ã„る組織ã§æ—¥ã€…åƒã„ã¦ã„ã¾ã™ 最近ã¯ã€Hadoop,OpenStack,スケールã™ã‚‹RDBMSã¨ã‹ã«èˆˆå‘³ãŒã‚ã‚Šã¾ã™ SimCloudã¨ã„ã†ã‚¯ãƒ©ã‚¦ãƒ‰ç®¡ç†ãƒ„ールを作ã£ã¦ã¾ã™ã€€->　OSS公開予定　 http://wiki.openstack.org/simcloud Twitter @nati OpenStackeré”ã¨@築地 S4 (Simple Scalable Streaming System) 　http://s4.io/ S4 is a general-purpose, distributed, scalable, partially fault-tole
リアルタイム分散処ç†ã®å¸¸è˜ã‚’Apache S4ã§èº«ã«ã¤ã‘ã‚‹
リアルタイム分散処ç†ã®å¸¸è˜ã‚’Apache S4ã§èº«ã«ã¤ã‘る：ビッグデータ処ç†ã®å¸¸è˜ã‚’Javaã§èº«ã«ã¤ã‘る（6）（1/2 ページ） Hadoopã‚’ã¯ã˜ã‚ã€Java言語を使ã£ã¦æ§‹ç¯‰ã•ã‚Œã‚‹ã“ã¨ãŒå¤šã„「ビッグデータã€å‡¦ç†ã®ãŸã‚ã®ãƒ•ãƒ¬ãƒ¼ãƒ ワークï¼ãƒ©ã‚¤ãƒ–ラリを紹介ã—ãªãŒã‚‰ã€å¤§é‡ãƒ‡ãƒ¼ã‚¿ã‚’活用ã™ã‚‹ãŸã‚ã®æŠ€è¡“ã®å¸¸è˜ã‚’身ã«ä»˜ã‘ã¦ã„ã連載 Hadoopã®å¼±ç‚¹ã€Œãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ 分散処ç†ã€ã¨ã¯ 「ビッグデータã€å‡¦ç†ã®ãŸã‚ã«Hadoopを用ã„ã‚‹ã¨ã€ã€Œè¤‡æ•°ã®ãƒžã‚·ãƒ³ã«å¤§é‡ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚’分散ã—ã¦é£›èºçš„ã«æ€§èƒ½ã‚’å‘上ã™ã‚‹ã€ã“ã¨ãŒå®¹æ˜“ã«ã§ãã¾ã™ã€‚ ã¨ã“ã‚ãŒHadoopã®å¼±ç‚¹ã¨ã—ã¦ã€ãƒ“ッグデータをã„ã£ãŸã‚“è“„ç©ã—ã€ãƒãƒƒãƒã§ä¸€æ‹¬å‡¦ç†ã™ã‚‹å½¢æ…‹ã§å‡¦ç†ã™ã‚‹ã®ã§ã€å‡¦ç†ãƒ‡ãƒ¼ã‚¿ãŒç™ºç”Ÿã—ã¦ã‹ã‚‰ã€ãã‚Œã«å¯¾ã™ã‚‹å‡¦ç†çµæžœãŒå¾—られるã¾ã§ã€å¿…ãšã‚¿ã‚¤ãƒ ラグãŒç™ºç”Ÿã—ã¾ã™ã€‚ã“ã®ãŸã‚ã€ã‚¯ãƒ¬ã‚¸ãƒƒãƒˆã‚«ãƒ¼ãƒ‰ã®ä¸æ£ã‚¢ã‚¯ã‚»ã‚¹æ¤œçŸ¥ã€ã‚»ãƒ³ã‚µãƒ‡ãƒ¼ã‚¿ãªã©ã§ã®ç•°å¸¸å€¤æ¤œå‡ºã®ã‚ˆã†ãªãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãª
GPUã¸ã®MapReduceã®é©ç”¨ã«é–¢ã™ã‚‹èª¿æŸ» - nishio-dens's diary
æ¦‚è¦ è¿‘å¹´ã€GPUã®æ€§èƒ½ã¯é£›èºçš„ã«å‘上ã—ã¦ãŠã‚Šã€ã‚°ãƒ©ãƒ•ã‚£ãƒƒã‚¯å°‚用ã®å‡¦ç†è£…ç½®ã¨ã—ã¦ã§ã¯ãªãã€æ•°å€¤è¨ˆç®—ç‰ã®æ±Žç”¨å‘ã‘ã®å‡¦ç†ã«åˆ©ç”¨ã™ã‚‹ã€GPGPUã«é–¢ã™ã‚‹ç ”究ãŒç››ã‚“ã«è¡Œã‚ã‚Œã¦ã„る。GPUã¯å†…部ã«å¤šãã®ã‚³ã‚¢ã‚’å‚™ãˆã¦ãŠã‚Šã€NVIDIA社ã®GeForce GTX 580ã§ã¯ã€512個もã®ã‚³ã‚¢ã‚’ä¿æŒã—ã¦ã„る。ã“れらã®ã‚³ã‚¢ã™ã¹ã¦ã‚’効率よã利用ã™ã‚‹ã“ã¨ã§ã€GPUã®æŒã£ã¦ã„る高ã„パフォーマンスを引ã出ã™ã“ã¨ãŒå‡ºæ¥ã‚‹ãŒã€GPUã®é«˜ã„並列性を利用ã™ã‚‹ãŸã‚ã«ã¯GPU特有ã®å‡¦ç†ã‚’実装ã™ã‚‹å¿…è¦ãŒã‚ã‚Šã€GPUプãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã«ãªã˜ã¿ã®ç„¡ã„利用者ã«ã¨ã£ã¦ã¯å‡¦ç†ã®è¨˜è¿°ãŒå›°é›£ã§ã‚る。 ãã“ã§ã€GPUを大é‡ã®ã‚³ã‚¢ã‚’æŒã£ãŸãƒ—ãƒã‚»ãƒƒã‚µã ã¨è€ƒãˆã€ã“れらã«å¯¾ã—ã¦MapReduceã‚’é©ç”¨ã™ã‚‹ã“ã¨ã‚’検討ã™ã‚‹ã€‚親ã—ã¿ã‚„ã™ã„MapReduceインターフェースã«ã‚ã¦ã¯ã‚ã¦å‡¦ç†ã‚’記述ã™ã‚‹ã ã‘ã§ã€GPUã®é«˜ã„並列性を生ã‹ã—ãŸãƒ—ãƒã‚°ãƒ©ãƒ ãŒå®Ÿè£…å¯èƒ½ã¨ãªã‚‹ã€‚åŠ
Apache Hadoop MapReduce (3) — MapReduce を多段ã§å‹•ã‹ã™ : MYH
MapReduce ã§å°‘ã—複雑ãªã“ã¨ã‚’ã—よã†ã¨æ€ã†ã¨ã€MapReduce を複数回é©ç”¨ã—ãªã‘ã‚Œã°ãªã‚‰ãªã„ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ã“ã‚Œã«ã¯ã„ãã¤ã‹ã®æ–¹æ³•ãŒã‚ã‚Šã¾ã™ã€‚ ã‚‚ã—ã€Map+ Reduce Map* (１回以上㮠Map, Reduce, ï¼å›žä»¥ä¸Šã® Map) ã¨ã„ã†å½¢ã§è¨˜è¿°ã™ã‚‹ã“ã¨ãŒå‡ºæ¥ã‚‹ãªã‚‰ã°ã€ChainMapper, ChainReducer ã¨ã„ã†ã‚¯ãƒ©ã‚¹ãŒç”¨æ„ã•ã‚Œã¦ãŠã‚Šã€ã“れを Mapper, Reducer ã¨ã—ã¦åˆ©ç”¨ã™ã‚‹ã®ãŒç°¡å˜ã§ã™ã€‚ ã‚‚ã—ã€ã‚·ãƒ¼ã‚±ãƒ³ã‚·ãƒ£ãƒ«ã« MapReduce ã‚’é©ç”¨ã™ã‚‹ã®ã§ã‚ã‚Œã°ã€Job ã®çµ‚了を待ã£ã¦ã‹ã‚‰æ¬¡ã® Job を投入ã™ã‚Œã°æ¸ˆã¿ã¾ã™ã€‚ ChainMapper / ChainReducer ã“ã®ã‚¯ãƒ©ã‚¹ã¯æ—¢å˜ã® Mapper ã‚’ç°¡å˜ã«ã¤ãªã’られるã“ã¨ã‚’目的ã¨ã—ã¦ã„ã¾ã™ã€‚ 仮㫠ChainMapper, ChainReducer ã¨ã„ã†ã‚¯ãƒ©ã‚¹ãŒãŒãªã‹ã£ãŸã¨ã—ã¾
Pig, Hiveãªã©8種類ã®MapReduce言語ã«ã¤ã„ã¦ã®æ¯”較 - nokunoã®æ—¥è¨˜
以下ã®è¨˜äº‹ã§ã¯Pig, Hiveãªã©8種類ã®è¨€èªžã§MapReduceã®ä¸€ç•ªå˜ç´”ãªä¾‹ã€WordCountを記述ã—ãŸå ´åˆã®ã‚¹ã‚¯ãƒªãƒ—トã®æ¯”較を行ã£ã¦ã„ã¾ã™ã€‚ã¨ã¦ã‚‚興味深ã‹ã£ãŸã®ã§ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚Pigs, Bees, and Elephants: A Comparison of Eight MapReduce Languages « Dataspora ãªãŠã€ä»Šå›žã¯RSSè³¼èªã—ã¦ã„ã‚‹DataSporaã®ãƒ–ãƒã‚°ã‹ã‚‰ç™ºè¦‹ã—ã¾ã—ãŸãŒã€ã‚¨ãƒ³ãƒˆãƒªã‚’寄稿ã—ãŸã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®å€‹äººãƒ–ãƒã‚°ã‚‚大変興味深ã„ã®ã§ã‚ªã‚¹ã‚¹ãƒ¡ã§ã™ã€‚ ã¯ã˜ã‚ã«MapReduceを美ã—ã効率的ã«æ›¸ããŸã‚ã«ã€ç§ã¯æ§˜ã€…ãªè¨€èªžã‚’比較ã—ã¾ã—ãŸã€‚æžœãŸã—ã¦ãã®å‹è€…ã¯ï¼ï¼Ÿç§ã®å€‹äººãƒ–ãƒã‚°ã§ã¯çµ±è¨ˆã‚„グラフã®ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã‚’MapReduceã§è¨˜è¿°ã™ã‚‹æ–¹æ³•ã‚’紹介ã—ã€æ“¬ä¼¼ã‚³ãƒ¼ãƒ‰ã«ã‚ˆã‚‹å®Ÿè£…を示ã—ã¾ã—ãŸã€‚擬似コードã«ã¯2ã¤ã®å•é¡Œç‚¹ãŒã‚ã‚Šã¾ã™ï¼šèª°ã‚‚ãŒãã®å‘½ä»¤ã‚’ç†è§£ã§ãã‚‹ã¨
Ruby/Amazon S3を使ㆠ- 俺ã®åŸºåœ°
ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã¨è£½é€ 年月日 2009å¹´09月08æ—¥ Ruby 1.8.6 aws-s3 0.6.2 WindowsXP SP3 ▲ ▼ 環境構築 Amazonã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚µãƒ¼ãƒ“スをS3ã‹ã‚‰ä½¿ã†ãƒ©ã‚¤ãƒ–ラリ　aws-s3　ãŒã‚ã‚‹ã®ã§ã“ã„ã¤ã‚’インストールã™ã‚‹ gem install aws-s3 ã“ã„ã¤ã«ã¯3ã¤ã®ä¾å˜æ€§ãŒã‚ã‚Šgemãªã‚‰è‡ªå‹•çš„ã«ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã•ã‚Œã‚‹ã‚“ã ãŒã€ä¸€å¿œä¹—ã£ã‘ã¦ãŠã xml-simple http://rubyforge.org/projects/xml-simple/ builder http://rubyforge.org/projects/builder/ mime-types http://rubyforge.org/projects/mime-types/ ▲ ▼ 接続è¨å®š ã¾ãšæŽ¥ç¶šè¨å®šã‚’è¡Œã†ã€‚ã“ã„ã¤ã¯Amazonã®ã‚µãƒ¼ãƒ“スã§æ¯Žåº¦ä½¿ã†ã€ç”Ÿæˆidã¨keyã®
Amazon Elastic MapReduce – madoverload
Amazon Web Servicesã«ã¯ã€Amazon Elastic MapReduce(EMR)ã¨ã„ã†ã‚µãƒ¼ãƒ“スもã‚ã‚Šã€Hadoopジョブを簡å˜ã«èµ°ã‚‰ã›ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚Hadoopトレーニングã§ã¯ã€CDHを使ã£ã¦æ¼”ç¿’ã‚’è¡Œãªã„ã¾ã—ãŸãŒã€ã‚·ãƒ³ã‚°ãƒ«ãƒŽãƒ¼ãƒ‰ã ã£ãŸãŸã‚分散処ç†ã®é†é†å‘³ãŒãªã‹ãªã‹å‘³ã‚ãˆã¾ã›ã‚“。 家ã«è¤‡æ•°ãƒŽãƒ¼ãƒ‰ã®ãƒžã‚·ãƒ³ãŒãªãã¦ã‚‚ç°¡å˜ã«è©¦ã™ã“ã¨ãŒã§ãã‚‹EMRを試ã—ã¦ã¿ã¾ã—ãŸã€‚ 準備 EMRã®åˆ©ç”¨å‰ã«ã„ãã¤ã‹ã®æº–備をã—ã¾ã™ã€‚詳細ã¯ã€Getting Started Guide ã‚’å‚ç…§ã—ã¦ãã ã•ã„。 EMRã®åˆ©ç”¨é–‹å§‹ EMRを利用ã™ã‚‹å‰ã«ã€EMRã®åˆ©ç”¨ç”³è«‹ã‚’è¡Œã†å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ã“ã‚Œã¯ã€ãƒžãƒãƒ¼ã‚¸ãƒ¡ãƒ³ãƒˆã‚³ãƒ³ã‚½ãƒ¼ãƒ«ã§Elastic MapReduceタブをクリックã—ãŸéš›ã«ã€åˆ©ç”¨ç”³è«‹æ¸ˆã¿ã§ãªã‘ã‚Œã°ç”³è«‹ãƒšãƒ¼ã‚¸ã¸ã®ãƒªãƒ³ã‚¯ãŒå‡ºã‚‹ã®ã§ã€ãã“ã‹ã‚‰ç”³è«‹ã™ã‚Œã°OKã§ã™ã€‚ EMRコマンドラインクライアントã®ãƒ€ã‚¦ãƒ³
Amazon Elastic MapReduceã«ä»Šã•ã‚‰å…¥é–€ã—ã¦ã¿ãŸ - nokunoã®æ—¥è¨˜
ã¨ã„ã†ã‚ã‘ã§å†¬ä¼‘ã¿æœ€å¾Œã®è‡ªç”±ç ”究ã¨ã—ã¦ã€Amazon Elastic MapReduce（EMR）を使ã£ã¦ã¿ã¾ã—ãŸã€‚今ãªã‚‰å…¬å¼ãƒšãƒ¼ã‚¸ã‚‚ã»ã¼æ—¥æœ¬èªžåŒ–ã—ã¦ã„ã¦æ¥½ãƒãƒ³ã§ã™ã€‚Amazon Web Services (日本語) Amazon Elastic MapReduceã¨ã¯Amazon EMRã¯ã€Amazonã®ã‚¤ãƒ³ãƒ•ãƒ©ä¸Šã§å‹•ä½œã™ã‚‹ä»®æƒ³ã‚µãƒ¼ãƒãƒ¼ã‚’使ã£ãŸHadoopクラスタを時間å˜ä¾¡ã§è²¸ã—出ã™ã‚µãƒ¼ãƒ“スã§ã™ã€‚少々ã‚ã‹ã‚Šã«ãã„ã®ã§ã€Amazon Web Service(AWS)ã®é–¢é€£ã™ã‚‹è£½å“群ã«ã¤ã„ã¦æ•´ç†ã—ã¦ãŠãã¾ã™ã€‚EC2 (Elastic Compute Cloud) EC2ã¯ã€ä»®æƒ³ãƒžã‚·ãƒ³ã‚’時間å˜ä¾¡ã§è²¸ã—出ã™ã‚µãƒ¼ãƒ“スã§ã™ã€‚ EMRを使ã‚ãšã«ã€EC2ã«è‡ªå‰ã§Hadoopをインストールã—ã¦ä½¿ã†ã‚„り方もã‚ã‚Šã¾ã™ï¼ˆEMRãŒå‡ºæ¥ã‚‹ä»¥å‰ã¯ãã‚Œã—ã‹ãªã‹ã£ãŸï¼‰ã€‚ EMRを使ã†å ´åˆã§ã‚‚ã€ãƒãƒƒã‚¯ã‚°ãƒ©ã‚¦ãƒ³ãƒ‰ã§ã¯è‡ªå‹•çš„ã«
Hadoop Streaming
Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run map/reduce jobs with any executable or script as the mapper and/or the reducer. For example: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \ -input myInputDirs \ -output myOutputDir \ -mapper /bin/cat \ -reducer /bin/wc In the above example, both the mapper and the reduce
blog.katsuma.tv
å‰å›žã€JavaScriptã§Map Reduceã®ã‚³ãƒ¼ãƒ‰ãŒæ›¸ã‘ã‚‹Hadoop Streamingã«ã¤ã„ã¦ç´¹ä»‹ã—ã¾ã—ãŸã€‚ 標準入出力ã•ãˆã‚µãƒãƒ¼ãƒˆã•ã‚Œã¦ã‚ã‚Œã°ã€ä»»æ„ã®ã‚³ãƒ¼ãƒ‰ã§Map Reduuceã®å‡¦ç†ãŒæ›¸ã‘ã‚‹ã€ã¨ã„ã†ã‚‚ã®ã§ã—ãŸãŒã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã¯ãã‚‚ãã‚‚é¢å€’ãã•ãŒã‚Šã€‚コードも書ãã®ã‚‚é¢å€’ã§ã™ã€‚ ã¨ã€ã„ã†ã‚ã‘ã§ã€ä»Šå›žã¯ã‚‚ã†ã‚³ãƒ¼ãƒ‰ã™ã‚‰æ›¸ã‹ãšã«SQLライクã§Map Reduceã§ãã‚‹Hiveã¨ã„ã†ãƒ—ãƒãƒ€ã‚¯ãƒˆã«ã¤ã„ã¦ã€ã¾ã¨ã‚ãŸã„ã¨æ€ã„ã¾ã™ã€‚ Hive Hiveã¨ã¯ã€ç°¡å˜ã«è¨€ã†ã¨Hadoop上ã§å‹•ä½œã™ã‚‹RDBã®ã‚ˆã†ãªã‚‚ã®ã§ã™ã€‚ HDFSãªã©ã®åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ 上ã«å˜åœ¨ã™ã‚‹ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦ã€HiveQLã¨ã„ã†SQLライクãªè¨€èªžã§æ“作ã§ãã¾ã™ã€‚ ã§ã€é¢ç™½ã„ã®ãŒHiveQLã®æ“作ã¯åŸºæœ¬çš„ã«Map Reduceã®ãƒ©ãƒƒãƒ‘ーã«ãªã£ã¦ã„ã‚‹ã“ã¨ã€‚ è¦ã™ã‚‹ã«ã€SELECT文実行ã™ã‚‹ã¨è£ã§Map&Reduceã®ã‚¿ã‚¹ã‚¯ãŒèµ°ã‚Šå‡º
Sector/Sphere: High Performance Distributed Data Storage and Processing
High Performance Distributed File System and Parallel Data Processing Engine Sector/Sphere supports distributed data storage, distribution, and processing over large clusters of commodity computers, either within a data center or across multiple data centers. Sector is a high performance, scalable, and secure distributed file system. Sphere is a high performance parallel data processing engine tha
Hadoop Streaming - naoyaã®ã¯ã¦ãªãƒ€ã‚¤ã‚¢ãƒªãƒ¼
id:naoya:20080511:1210506301 ã®ã‚¨ãƒ³ãƒˆãƒªã®ã‚³ãƒ¡ãƒ³ãƒˆæ¬„㧠kzk ã•ã‚“ã«æ•™ãˆã¦ã„ãŸã ã„㟠Hadoop Streaming を試ã—ã¦ã„ã¾ã™ã€‚ Hadoop ã¯ã‚ªãƒ¼ãƒ—ンソース㮠MapReduce + 分散ファイルシステムã§ã™ã€‚Java ã§ä½œã‚‰ã‚Œã¦ã„ã¾ã™ã€‚Yahoo! Inc ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã‚„ã€Facebookã€Amazon.com ãªã©ã§ã‚‚利用ã•ã‚Œã¦ã„ã‚‹ã¨ã®ã“ã¨ã§ã™ã€‚詳ã—ã㯠http://codezine.jp/a/article/aid/2448.aspx (kzk ã•ã‚“ã«ã‚ˆã‚‹é€£è¼‰è¨˜äº‹)ã‚’å‚ç…§ã—ã¦ãã ã•ã„。 Hadoop Streaming 記事ã«ã‚‚ã‚ã‚Šã¾ã™é€šã‚Šã€Hadoop 拡張㮠Hadoop Streaming を使ã†ã¨æ¨™æº–入出力を介ã™ã‚‹ãƒ—ãƒã‚°ãƒ©ãƒ を記述ã™ã‚‹ã ã‘ã§ã€Hadoop ã«ã‚ˆã‚‹ MapReduce を利用ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ã¤ã¾ã‚Šã€Java 以外
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
AWS for Developers | Programming Languages, Tools, Community | AWS Developer Center
Amazon Elastic MapReduceã®ä½¿ã„方─Hadoopより手軽ã«ã¯ã˜ã‚る大è¦æ¨¡è¨ˆç®— 記事一覧 | gihyo.jp
Hadoop Streaming
IBM Developer
Big Data Platform - Amazon EMR - AWS
{{#tags}}- {{label}}
{{/tags}}