GitHub - twitter/summingbird: Streaming MapReduce with Scalding and StormYou signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert
TwitterãŒMapReduceストリーミングフレームワークSummingbirdをオープンソースã«
Spring Bootã«ã‚ˆã‚‹APIãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰æ§‹ç¯‰å®Ÿè·µã‚¬ã‚¤ãƒ‰ 第2版 何åƒäººã‚‚ã®é–‹ç™ºè€…ãŒã€InfoQã®ãƒŸãƒ‹ãƒ–ック「Practical Guide to Building an API Back End with Spring Bootã€ã‹ã‚‰ã€Spring Bootを使ã£ãŸREST API構築ã®åŸºç¤Žã‚’å¦ã‚“ã 。ã“ã®æœ¬ã§ã¯ã€å‡ºç‰ˆæ™‚ã«æ–°ã—ãリリースã•ã‚ŒãŸãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚ã‚‹ Spring Boot 2 を使用ã—ã¦ã„る。ã—ã‹ã—ã€Spring Boot3ãŒæœ€è¿‘リリースã•ã‚Œã€é‡è¦ãªå¤‰...
MapReduceã¯ä»Šå¾Œã©ã†ãªã‚‹ã®ã‹ï¼Ÿ - 急ãŒã°å›žã‚Œã€é¸ã¶ãªã‚‰è¿‘é“
2012å¹´ã®ç¾åœ¨ã€å‰²ã¨æ‚©ã‚“ã§ã„ã‚‹ã®ã§ãƒ¡ãƒ¢ã£ã¦ãŠã。 年度末ãらã„ã«å†èª¿æŸ»ã®äºˆå®šã€‚・・ãªã®ã§æš«å®šã§ã™ã‚ˆã€‚ ã¾ãšå‰æã¨ã—ã¦ã€ç¾è¡Œã®Hadoopã®å®Ÿè¡Œãƒ•ãƒ¬ãƒ¼ãƒ ワークã§ã‚ã‚‹MapReduceã¯ã€å®Ÿè¡ŒåŠ¹çŽ‡ã¯æ±ºã—ã¦è‰¯ãã¯ãªã„ã§ã™ã€‚ã“ã®è¾ºãŒå‰²ã¨è¾›ã„。 ã¨ã¯ã„ãˆã€å¤§è¦æ¨¡ä¸¦åˆ—処ç†ã‚’一般的ã«è¡Œã†ã¨ã„ã†è¦³ç‚¹ã§ã®å“質やå–り回ã—を考ãˆãŸå ´åˆã€â€çµæžœã¨ã—ã¦â€éžå¸¸ã«ãƒãƒ©ãƒ³ã‚¹ãŒã¨ã‚Œã¦ãŠã‚Šã€æ™®åŠã—ã¦ã„る。ãã®ä¸Šã§ã€ã“ã®MapReduceã§ã™ãŒã€ä»Šå¾Œã®è¦‹é€šã—ã«ã¤ã„ã¦ã¯ã€æ½®æµã¯ä»Šã®ã¨ã“ã‚二ã¤ã«å‰²ã‚Œã¦ã„るよã†è¦‹ãˆã‚‹ã€‚ã®ã§ã€ãã®è¾ºã®ãƒ¡ãƒ¢ã€‚ â– YARN 一ã¤ã®æ–¹å‘性ã¯ã€ç¾åœ¨ã®Hadoop2.0ç³»ã§å®Ÿè£…ã•ã‚Œã¦ã„ã‚‹MapReduce2.0ã€ã¨ã„ã†ã‹ã€MapReduceã¨ã¯åˆ¥ã®å®Ÿè¡ŒåŸºç›¤ã‚’利用ã™ã‚‹ã¨ã„ã†æ–¹å‘ã§ã™ã。ã™ãªã‚ã¡BSPã‚„ã€MPIを利用ã™ã‚‹ã€‚è¦ã¯ã€ä»Šã¾ã§ã®ä¸¦åˆ—処ç†ã®æˆæžœã‚’ãã®ã¾ã¾åˆ©ç”¨ã—ã¾ã—ょã†ã€ã¨ã„ã†æµã‚Œã«è¿‘ã„。 MapReduce
Hadoop summit 2012 report
第10回Hadoopソースコードリーディングã§ç™ºè¡¨ã—ãŸè³‡æ–™ã§ã™ã€‚ 2012å¹´6月ã«é–‹å‚¬ã•ã‚ŒãŸHadoop Summit ã®å‚åŠ ãƒ¬ãƒãƒ¼ãƒˆã§ã€YARNã€HBaseã€HDFS HA ãªã©ã®ã‚»ãƒƒã‚·ãƒ§ãƒ³ã‚’紹介ã—ã¦ã„ã¾ã™ã€‚Read less
MapReduceã§ãã‚‹10個ã®ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ - データサイエンティスト上ãŒã‚Šã®DXå‚謀・起æ¥å®¶
Hadoopã¨Mahoutã«ã‚ˆã‚Šã€ãƒ“ッグデータã§ã‚‚機械å¦ç¿’ã‚’è¡Œã†ã“ã¨ãŒã§ãã¾ã™ã€‚Mahoutã§å®Ÿè£…ã•ã‚Œã¦ã„る手法ã¯ã€å…¨ã¦åˆ†æ•£å‡¦ç†ã§ãるアルゴリズムã¨ã„ã†ã“ã¨ã«ãªã‚Šã¾ã™ã€‚Mahoutã§å®Ÿè£…ã•ã‚Œã¦ã„るアルゴリズムã¯ã€ã“ã“ã«åˆ—挙ã•ã‚Œã¦ã„ã¾ã™ã€‚è«–æ–‡ã¨ã—ã¦ã‚‚ã€2006å¹´ã«ã€ŒMap-Reduce for Machine Learning on Multicoreã€ã¨ã—ã¦ã„ãã¤ã‹ã®ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ãŒç´¹ä»‹ã•ã‚Œã¦ã„ã¾ã™ã€‚ ãã“ã§ä»Šå›žã¯ã€ï¼ˆä½•ç•ªç…Žã˜ã‹åˆ†ã‹ã‚Šã¾ã›ã‚“ãŒè‡ªåˆ†ã®ç†è§£ã®ãŸã‚ã«ã‚‚）ã“ã®è«–æ–‡ã§ç´¹ä»‹ã•ã‚Œã¦ã„るアルゴリズムã¨ã€ã©ã†ã‚„ã£ã¦åˆ†æ•£å‡¦ç†ã™ã‚‹ã®ã‹ã‚’ç°¡å˜ã«ãƒ¡ãƒ¢ã—ã¦ãŠããŸã„ã¨æ€ã„ã¾ã™ã€‚計算ã™ã‚‹ã¹ã統計é‡ãŒã€summation form（足ã—ç®—ã§è¡¨ç¾ã§ãる形）ã«ãªã£ã¦ã„ã‚‹ã‹ã©ã†ã‹ãŒã€é‡è¦ãªãƒã‚¤ãƒ³ãƒˆã§ã™ã€‚ãªã£ã¦ãªã„å ´åˆã¯ã€â€ã†ã¾ãâ€MapReduceã®å½¢ã«ãƒãƒ©ã™å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ ※例ã«ã‚ˆã£ã¦ã€é–“é•ã„ãŒã‚ã£ãŸå ´åˆã¯éšæ™‚
ビッグデータã®ä¾¡æ ¼ç ´å£Šï¼Ÿã€€GoogleãŒã€Œå‡¦ç†é‡100GB/月ã¾ã§ç„¡æ–™ã€ã®è§£æžã‚µãƒ¼ãƒ“スBigQueryã‚’æ供開始 - ITジャーナリスト星æšé›„ã®"æƒ…å ±è«–"ノート
æƒ…å ±ã¨æŠ€è¡“ã¯æœªæ¥ã‚’ã©ã†å¤‰ãˆã‚‹ã®ã‹â”€â”€ITã€ã‚¹ãƒžãƒ¼ãƒˆãƒ‡ãƒã‚¤ã‚¹ã€ãƒãƒœãƒƒãƒˆã€é›»å工作ã€ãƒ¡ãƒ‡ã‚£ã‚¢ã®ã‚¢ãƒ¼ã‚テクãƒãƒ£ Google勤務ã®Kazunori Satoã•ã‚“ãŒGoogle+ã«ç°¡æ½”ãªè§£èª¬ã‚’ãƒã‚¹ãƒˆã—ã¦ãã‚Œã¦ã„ã¾ã™ã€‚ ãƒã‚¹ãƒˆ1 BigQueryãŒä¸€èˆ¬å…¬é–‹ã•ã‚Œã¾ã—ãŸï¼æ•°100億件ã®å…¨æ¤œç´¢ãŒæ•°å秒ã§å®Œäº†ã™ã‚‹è¶…並列クエリサービスã§ã€MapReduceã¨ä¸¦ã³Googleã®æ ¹å¹¹ã‚’支ãˆã‚‹è™Žã®å技術ã§ã™ã€‚ Google BigQuery brings Big Data analytics to all businesses - Google Developers Blog ãƒã‚¹ãƒˆ2 BigQueryプãƒè§£èª¬ï¼šBigQueryã¯Google社内ã§ã¯ã€ŒDremelã€ã¨å‘¼ã°ã‚Œã‚‹è¶…並列クエリインフラを利用ã—ãŸä¸€èˆ¬å‘ã‘サービスã§ã™ã€‚Dremelã¯Sybase IQã‚„Oracle Exadataã¨åŒæ§˜ã®Columar DB
Cascading
Please note that all new project news and releases have moved to https://cascading.wensel.net The Cascading Ecosystem is a collection of applications, languages, and APIs for developing data-intensive applications. At the ecosystem core is Cascading, a Java API for defining complex data flows and integrating those flows with back-end systems, and a query planner for mapping and executing logical f
MapReduceã®ãƒ‘ターンã€ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã€ãã—ã¦ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ - ãã—ã ã®Hatena
Ilya Katsovæ°ã«ã‚ˆã‚‹ã€ŒMapReduce Patterns, Algorithms, and Use Casesã€ã®ç¿»è¨³ http://highlyscalable.wordpress.com/2012/02/01/mapreduce-patterns/ (下書ãã«å…¥ã‚Œã¦æŽ¨æ•²ã™ã‚‹ã¤ã‚‚ã‚ŠãŒã€ãªã‚“ã‹å…¬é–‹ã•ã‚Œã¦ã—ã¾ã£ã¦ã„ãŸã®ã§ã€ã‚ã¨ã§ã„ã‚ã„ã‚ä¿®æ£ã™ã‚‹ã¨æ€ã„ã¾ã™) February 1, 2012 ã“ã®è¨˜äº‹ã§ã¯ã€Webや科å¦è«–æ–‡ã§è¦‹ã‚‰ã‚Œã‚‹ç•°ãªã‚‹ãƒ†ã‚¯ãƒ‹ãƒƒã‚¯ã®ä½“系的ãªè¦–点を与ãˆã‚‹ãŸã‚ã«ã€æ•°ã€…ã®MapReduceパターンã¨ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ ã‚’ã¾ã¨ã‚ãŸã€‚ ã„ãã¤ã‹ã®å®Ÿç”¨çš„ãªã‚±ãƒ¼ã‚¹ã‚¹ã‚¿ãƒ‡ã‚£ã‚‚æä¾›ã—ã¦ã„る。 ã™ã¹ã¦ã®èª¬æ˜Žã¨ã‚³ãƒ¼ãƒ‰ã‚¹ãƒ‹ãƒšãƒƒãƒˆã§ã¯ã€Mapperã€Reducerã€Combinerã€Partitionaerã€ã‚½ãƒ¼ãƒ†ã‚£ãƒ³ã‚°ã«ãŠã„ã¦Hadoopã®æ¨™æº–çš„ãªMapReduceモデルを利用ã—ã¾ã™ã€‚ã“ã®ãƒ•ãƒ¬ãƒ¼
liris.org - ã“ã®ã‚¦ã‚§ãƒ–サイトã¯è²©å£²ç”¨ã§ã™ï¼Â - liris リソースãŠã‚ˆã³æƒ…å ±
This webpage was generated by the domain owner using Sedo Domain Parking. Disclaimer: Sedo maintains no relationship with third party advertisers. Reference to any specific service or trade mark is not controlled by Sedo nor does it constitute or imply its association, endorsement or recommendation.
Introduction to Parallel Programming and MapReduce - Google Code University - Google Code
Introduction to Parallel Programming and MapReduce Table of Contents Audience and Pre-Requisites This tutorial covers the basics of parallel programming and the MapReduce programming model. The pre-requisites are significant programming experience with a language such as C++ or Java, and data structures & algorithms. Serial vs. Parallel Programming In the early days of computing, programs were s
Celery - Distributed Task Queue — Celery 3.0.11 documentation
Documentation has moved Celery is now using Read the Docs to host the documentation for the development version, where the pages are automatically updated as new changes are made (YAY!) The new location for this page is: http://docs.celeryproject.org/en/master/index.html. If you wanted documentation for the latest stable version instead, please go to: http://docs.celeryproject.org/en/latest/index.
優良ä¼æ¥ã¯ãªãœHadoopã«èµ°ã‚‹ã®ã‹
ã¡ãªã¿ã«ã€ã“ã®åˆ†æžã®ãŸã‚ã«å¿…è¦ã¨ã•ã‚Œã‚‹MapReduceã®ã‚³ãƒ¼ãƒ‰ã§ã‚ã‚‹ãŒã€ãã®ã‚µã‚¤ã‚ºã¯ã‚ãšã‹20ステップã ã¨ã„ã†ã€‚Yahoo!ã®ãƒ—レゼンテーターã§ã‚ã‚‹ã€ã‚¨ãƒªãƒƒã‚¯ãƒ»ãƒãƒ«ãƒ‡ã‚·ãƒ¥ãƒã‚¤ãƒ©ãƒ¼æ°ã«ã‚ˆã‚‹ã¨ã€ãŸã¨ãˆçµŒé¨“ã®æµ…ã„エンジニアã§ã‚ã£ã¦ã‚‚ã€MapReduceã«ã‚ˆã‚‹ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã¯å¯èƒ½ã§ã‚ã‚‹ã¨ã•ã‚Œã‚‹ã€‚ ã¾ãŸã€VISAã®ã‚¸ãƒ§ãƒ¼ãƒ»ã‚«ãƒ‹ãƒ³ã‚¬ãƒ æ°ã‹ã‚‰ã‚‚ã€è²´é‡ãªãƒ‡ãƒ¼ã‚¿ãŒæä¾›ã•ã‚Œã¦ã„ãŸã®ã§ä»¥ä¸‹ã«ç´¹ä»‹ã™ã‚‹ã€‚åŒç¤¾ã§ã¯ã€1æ—¥ã«1億トランザクションãŒç™ºç”Ÿã™ã‚‹ãŸã‚ã€2å¹´é–“ã§700å„„å¼·ã®ãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³ãƒã‚°ãŒè“„ç©ã•ã‚Œã€ãã®ãƒ‡ãƒ¼ã‚¿é‡ã¯36テラãƒã‚¤ãƒˆã«è‡³ã‚‹ã¨ã„ã†ã€‚ã“ã†ã—ãŸã‚¹ã‚±ãƒ¼ãƒ«ã®ãƒ‡ãƒ¼ã‚¿ã‚’ã€å¾“æ¥ã®RDBを用ã„ã¦åˆ†æžã™ã‚‹ã«ã¯ã€ç´„1カ月ã®æ™‚é–“ãŒå¿…è¦ã¨ã•ã‚Œã¦ããŸãŒã€Hadoopを用ã„ã‚‹ã“ã¨ã§13分ã«çŸç¸®ã•ã‚ŒãŸã¨ã„ã†ã€‚ ã“ã‚Œã¾ã§ã¯ã€Yahoo!ã«ã—ã‚VISAã«ã—ã‚ã€è†¨å¤§ãªãƒ‡ãƒ¼ã‚¿ã‚’RDBã«æŠ¼ã—込むã»ã‹ã«æ–¹æ³•ã¯ãªãã€ãã®åˆ†æžã«æ•°å日をè¦ã™ã‚‹
Amazon Elastic MapReduceを使ã£ã¦ã¿ãŸ - moratorium
Amazon Elastic MapReduceを使ã£ã¦ã¿ãŸ 2009-04-03 (Fri) 3:06 Amazon EC2 連日ã®EC2ãƒã‚¿ã§ã™ã€‚本日ã€Amazonã‹ã‚‰Elastic MapReduceã¨ã„ã†ã‚µãƒ¼ãƒ“スãŒãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¾ã—ãŸã€‚大è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†æŠ€è¡“ãŒä¸€æ°—ã«æ°‘é–“ã®æ‰‹ã«ä¸‹ã‚Šã¦ãã‚‹ã€ã¾ã•ã«é©å‘½çš„ãªã‚µãƒ¼ãƒ“スã ã¨æ€ã„ã¾ã™ã€‚ Amazon Elastic MapReduce Amazon ElasticMapReduce 紹介ビデオ With Hadoop, Amazon Adds A Web-Scale Data Processing Engine To Its Cloud Computer by techcrunch.com Elastic MapReduceã¯ã€Googleã®åŸºç›¤æŠ€è¡“ã®ä¸€ã¤ã§ã‚ã‚‹MapReduceを時間å˜ä½èª²é‡‘ã§å®Ÿè¡Œã§ãるサービスã§ã™ã€‚MapReduceã«ã¤ã„ã¦ã¯ä»¥
Disco MapReduce
Disco is a lightweight, open-source framework for distributed computing based on the MapReduce paradigm. Disco is powerful and easy to use, thanks to Python. Disco distributes and replicates your data, and schedules your jobs efficiently. Disco even includes the tools you need to index billions of data points and query them in real-time. Disco was born in Nokia Research Center in 2008 to solve rea
)
Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G. Noll
What we want to do In this tutorial, I will describe the required steps for setting up a multi-node Hadoop cluster using the Hadoop Distributed File System (HDFS) on Ubuntu Linux. Hadoop is a framework written in Java for running applications on large clusters of commodity hardware and incorporates features similar to those of the Google File System and of MapReduce. HDFS is a highly fault-tolera
Hadoop Python: Writing An Hadoop MapReduce Program In Python - Michael G. Noll
In this tutorial, I will describe how to write a simple MapReduce program for Hadoop in the Python programming language. Motivation Even though the Hadoop framework is written in Java, programs for Hadoop need not to be coded in Java but can also be developed in other languages like Python or C++ (the latter since version 0.14.1). However, the documentation and the most prominent Python example o
Kansai.pm ã§ã®ç™ºè¡¨è³‡æ–™ (Hadoop Streaming 㧠MapReduce) - naoyaã®ã¯ã¦ãªãƒ€ã‚¤ã‚¢ãƒªãƒ¼
Kansai.pm ã«å‚åŠ ã—ã¾ã—ãŸã€‚ã¨ã¦ã‚‚楽ã—ã‹ã£ãŸã§ã™ã€‚自分も "Hadoop Streaming 㧠MapReduce" ã¨ã„ã†é¡Œç›®ã§ç™ºè¡¨ã—ã¾ã—ãŸã€‚å–り急ãŽã€è³‡æ–™ã‚’以下ã«å…¬é–‹ã—ã¾ã™ã€‚ http://bloghackers.net/~naoya/ppt/080530kansaipm.ppt MapReduce 㯠Google ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã§å‹•ã„ã¦ã„る分散並列ãƒãƒƒãƒå‡¦ç†ã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚GFS 㯠Google ã®åˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚Google ウェアã®ã‚¯ãƒãƒ¼ãƒ³ã¨ã—ã¦ã‚ªãƒ¼ãƒ—ンソースã§é–‹ç™ºã•ã‚Œã¦ã„ã‚‹ã®ãŒ Hadoop。Hadoop 㯠Yahoo! Inc ã‚„ Facebook, Amazon.com ãªã©ã§ã‚‚利用ã•ã‚Œã¦ã„ã‚‹ã¨ã®ã“ã¨ã€‚Hadoop 㯠Java ã§ã™ãŒã€Hadoop Streaming を使ã†ã¨ã€€Java 以外ã§ã‚‚ MapReduce ã§ãã¾ã™ã€‚ 以下ã®ã‚¨ãƒ³ãƒˆãƒªã‚‚åˆ
MapReduce - naoyaã®ã¯ã¦ãªãƒ€ã‚¤ã‚¢ãƒªãƒ¼
"MapReduce" 㯠Google ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã§åˆ©ç”¨ã•ã‚Œã¦ã„る並列計算システムã§ã™ã€‚検索エンジンã®ã‚¤ãƒ³ãƒ‡ãƒƒã‚¯ã‚¹ä½œæˆã‚’ã¯ã˜ã‚ã¨ã™ã‚‹ã€å¤§è¦æ¨¡ãªå…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹ãƒãƒƒãƒå‡¦ç†ã‚’想定ã—ã¦ä½œã‚‰ã‚ŒãŸã‚·ã‚¹ãƒ†ãƒ ã§ã™ã€‚ MapReduce ã®é¢ç™½ã„ã¨ã“ã‚ã¯ã€map() 㨠reduce() ã¨ã„ã†äºŒã¤ã®é–¢æ•°ã®çµ„ã¿åˆã‚ã›ã‚’定義ã™ã‚‹ã ã‘ã§ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã™ã‚‹æ§˜ã€…ãªè¨ˆç®—å•é¡Œã‚’解決ã™ã‚‹ã“ã¨ãŒã§ãる点ã§ã™ã€‚ MapReduce ã®è¨ˆç®—モデル map() ã«ã¯ãã®è¨ˆç®—å•é¡Œã®ãƒ‡ãƒ¼ã‚¿ã¨ã—ã¦ã® key-value ペアãŒæ¬¡ã€…ã«æ¸¡ã£ã¦ãã¾ã™ã€‚map() ã§ã¯ key-value 値ã®ãƒšã‚¢ã‚’ç•°ãªã‚‹è¤‡æ•°ã® key-value ペアã«å¤‰æ›ã—ã¾ã™ã€‚reduce() ã«ã¯ã€map() ã§ä½œã£ãŸ key-value ペアをåŒä¸€ã® key ã§æŸããŸã‚‚ã®ãŒé †ç•ªã«æ¸¡ã£ã¦ãã¾ã™ã€‚ãã® key-values ペアを任æ„ã®å½¢å¼ã«å¤‰æ›ã™ã‚‹ã“ã¨
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Treasure Data, Inc. | Finding Gems in Your Big Data
Hadoopã®MapRecudeã‚’Python(Jython)ã§æ›¸ããŸã‚ã®ãƒ•ãƒ¬ãƒ¼ãƒ ワーク「Happy
{{#tags}}- {{label}}
{{/tags}}