OASIS - Data Analysis Platform for Multi-tenant Hadoop Cluster
OASIS - Data Analysis Platform for Multi-tenant Hadoop Cluster OASIS is a web-based data analysis platform that enables employees at LINE to analyze data from their services in a multi-tenant Hadoop cluster. It provides features like query execution, result visualization, code execution in Scala/Python/R, scheduled execution, and result sharing. OASIS addresses security, stability, and feature req
Amazon Elastic MapReduce Training
Elastic MapReduce Training What is Amazon Elastic MapReduce? Amazon Elastic MapReduce (EMR) is a web service that makes it easy to efficiently process and analyze vast amounts of data in the cloud. It uses a hosted Hadoop framework and is closely integrated with Amazon EC2, Amazon S3, and other AWS services. EMR also supports Hadoop-related technologies such as Hive, Pig, and HBase. Amazon EMR tak
Build Hadoop from Source
Instructions in this write-up were tested and run successfully on Ubuntu 10.04, 10.10, 11.04, and 11.10. The instructions should run, with minor modifications, on most flavors and variants of Linux and Unix. For example, replacing apt-get with yum should get it working on Fedora and CentOS. If you are starting out with Hadoop, one of the best ways to get it working on your box is to build it from
NTTデータã€OSS分散処ç†åŸºç›¤ã€ŒHadoopã€ã®æ§‹ç¯‰ã€é‹ç”¨ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ã‚’æ供開始
å°åˆ·ã™ã‚‹ メールã§é€ã‚‹ テã‚スト HTML é›»åæ›¸ç± PDF ダウンãƒãƒ¼ãƒ‰ テã‚スト é›»åæ›¸ç± PDF クリップã—ãŸè¨˜äº‹ã‚’Myページã‹ã‚‰èªã‚€ã“ã¨ãŒã§ãã¾ã™ NTTデータã¯7月1æ—¥ã€åŒç¤¾ã®ã‚¯ãƒ©ã‚¦ãƒ‰ã‚µãƒ¼ãƒ“ス「BizXaaSã€ï¼ˆãƒ“ズエクサース）ã®ã€Œã‚¯ãƒ©ã‚¦ãƒ‰æ§‹ç¯‰ã‚µãƒ¼ãƒ“スã€ã®ãƒ©ã‚¤ãƒ³ã‚¢ãƒƒãƒ—ã¨ã—ã¦ã€ŒHadoop構築・é‹ç”¨ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ã€ã®æ供を開始ã—ãŸã€‚ オープンソースã®åˆ†æ•£å‡¦ç†åŸºç›¤ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã‚る「Hadoopã€ã¯ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’多数ã®ã‚µãƒ¼ãƒã«åˆ†æ•£ã—ã¦è“„ç©ã™ã‚‹ã¨ã¨ã‚‚ã«ã€å¤§é‡ã‹ã¤è¤‡é›‘ãªè¨ˆç®—を並列処ç†ã•ã›ã¦ã€ãƒ‡ãƒ¼ã‚¿ã‚’高速ã«å‡¦ç†ã§ãã‚‹ã‚‚ã®ã€‚従æ¥ã€åˆ†æ•£å‡¦ç†ã‚’活用ã—ãŸã‚·ã‚¹ãƒ†ãƒ を構築ã™ã‚‹ãŸã‚ã«ã¯ã€ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã®é–‹ç™ºè€…å´ã«é«˜åº¦ãªãƒŽã‚¦ãƒã‚¦ãŒå¿…è¦ã¨ã•ã‚Œã¦ã„ãŸãŒã€Hadoopã§ã¯ã€MapReduceフレームワークを利用ã™ã‚‹ã“ã¨ã§ã€å€‹ã€…ã®åˆ†æ•£åˆ¶å¾¡ã‚’æ„è˜ã™ã‚‹ã“ã¨ãªã分散処ç†ã®ã‚·ã‚¹ãƒ†ãƒ を構築ã§ãã‚‹ã€‚è¡Œå‹•æƒ…å ±ã€ã‚»ãƒ³ã‚µæƒ…å ±ã‚„ãƒ©ã‚¤
SQLライクã«Hadoop Hiveを使ã„倒ã™!
パーティションを利用ã™ã‚‹ 今回ã¯å°‘ã—å‡ã£ãŸãƒ†ãƒ¼ãƒ–ルを定義をã—ã¦ã¿ã¾ã—ょã†ã€‚ 郵便番å·ãƒ‡ãƒ¼ã‚¿ã¯æ¯Žæœˆæ›´æ–°ã•ã‚Œã‚‹ã®ã§ã€ãƒ†ãƒ¼ãƒ–ル指定時ã«ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚‚指定ã§ãるよã†ã«ã—ã¾ã™ã€‚ã“ã®ã‚ˆã†ãªå ´åˆã€Hiveã§ã¯ãƒ‘ーティションを使ã„ã¾ã™ã€‚ 以下ã«éƒµä¾¿ç•ªå·ã‚’ä¿å˜ã™ã‚‹ãƒ†ãƒ¼ãƒ–ル「zipã€ã‚’定義ã—ã¾ã™ãŒã€æ—¥ä»˜åž‹DATEã®ãƒ‘ーティションverã‚’è¨å®šã™ã‚‹ã‚ˆã†ã«ã—ã¾ã™ã€‚ hive> CREATE TABLE zip (zip STRING, pref INT, city STRING, town STRING) > PARTITIONED BY (ver DATE) > ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' > LINES TERMINATED BY '\n'; OK Time taken: 0.128 seconds
優良ä¼æ¥ã¯ãªãœHadoopã«èµ°ã‚‹ã®ã‹
ã¡ãªã¿ã«ã€ã“ã®åˆ†æžã®ãŸã‚ã«å¿…è¦ã¨ã•ã‚Œã‚‹MapReduceã®ã‚³ãƒ¼ãƒ‰ã§ã‚ã‚‹ãŒã€ãã®ã‚µã‚¤ã‚ºã¯ã‚ãšã‹20ステップã ã¨ã„ã†ã€‚Yahoo!ã®ãƒ—レゼンテーターã§ã‚ã‚‹ã€ã‚¨ãƒªãƒƒã‚¯ãƒ»ãƒãƒ«ãƒ‡ã‚·ãƒ¥ãƒã‚¤ãƒ©ãƒ¼æ°ã«ã‚ˆã‚‹ã¨ã€ãŸã¨ãˆçµŒé¨“ã®æµ…ã„エンジニアã§ã‚ã£ã¦ã‚‚ã€MapReduceã«ã‚ˆã‚‹ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ã¯å¯èƒ½ã§ã‚ã‚‹ã¨ã•ã‚Œã‚‹ã€‚ ã¾ãŸã€VISAã®ã‚¸ãƒ§ãƒ¼ãƒ»ã‚«ãƒ‹ãƒ³ã‚¬ãƒ æ°ã‹ã‚‰ã‚‚ã€è²´é‡ãªãƒ‡ãƒ¼ã‚¿ãŒæä¾›ã•ã‚Œã¦ã„ãŸã®ã§ä»¥ä¸‹ã«ç´¹ä»‹ã™ã‚‹ã€‚åŒç¤¾ã§ã¯ã€1æ—¥ã«1億トランザクションãŒç™ºç”Ÿã™ã‚‹ãŸã‚ã€2å¹´é–“ã§700å„„å¼·ã®ãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³ãƒã‚°ãŒè“„ç©ã•ã‚Œã€ãã®ãƒ‡ãƒ¼ã‚¿é‡ã¯36テラãƒã‚¤ãƒˆã«è‡³ã‚‹ã¨ã„ã†ã€‚ã“ã†ã—ãŸã‚¹ã‚±ãƒ¼ãƒ«ã®ãƒ‡ãƒ¼ã‚¿ã‚’ã€å¾“æ¥ã®RDBを用ã„ã¦åˆ†æžã™ã‚‹ã«ã¯ã€ç´„1カ月ã®æ™‚é–“ãŒå¿…è¦ã¨ã•ã‚Œã¦ããŸãŒã€Hadoopを用ã„ã‚‹ã“ã¨ã§13分ã«çŸç¸®ã•ã‚ŒãŸã¨ã„ã†ã€‚ ã“ã‚Œã¾ã§ã¯ã€Yahoo!ã«ã—ã‚VISAã«ã—ã‚ã€è†¨å¤§ãªãƒ‡ãƒ¼ã‚¿ã‚’RDBã«æŠ¼ã—込むã»ã‹ã«æ–¹æ³•ã¯ãªãã€ãã®åˆ†æžã«æ•°å日をè¦ã™ã‚‹
Cascading
Please note that all new project news and releases have moved to https://cascading.wensel.net The Cascading Ecosystem is a collection of applications, languages, and APIs for developing data-intensive applications. At the ecosystem core is Cascading, a Java API for defining complex data flows and integrating those flows with back-end systems, and a query planner for mapping and executing logical f
InfoQ: Cascading -- Hadoop MapReduceå‘ã‘データ処ç†ç³»API
Spring Bootã«ã‚ˆã‚‹APIãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰æ§‹ç¯‰å®Ÿè·µã‚¬ã‚¤ãƒ‰ 第2版 何åƒäººã‚‚ã®é–‹ç™ºè€…ãŒã€InfoQã®ãƒŸãƒ‹ãƒ–ック「Practical Guide to Building an API Back End with Spring Bootã€ã‹ã‚‰ã€Spring Bootを使ã£ãŸREST API構築ã®åŸºç¤Žã‚’å¦ã‚“ã 。ã“ã®æœ¬ã§ã¯ã€å‡ºç‰ˆæ™‚ã«æ–°ã—ãリリースã•ã‚ŒãŸãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚ã‚‹ Spring Boot 2 を使用ã—ã¦ã„る。ã—ã‹ã—ã€Spring Boot3ãŒæœ€è¿‘リリースã•ã‚Œã€é‡è¦ãªå¤‰...
MapReduceã®Java実装Apache Hadoopを使ã£ã¦ã¿ãŸ (1/3) - @IT
MapReduceã®Java実装Apache Hadoopを使ã£ã¦ã¿ãŸï¼šã„ã¾å†æ³¨ç›®ã®åˆ†æ•£å‡¦ç†æŠ€è¡“（後編）（1/3 ページ） 最近注目を浴ã³ã¦ã„る分散処ç†æŠ€è¡“「MapReduceã€ã®åˆ©ç‚¹ã‚’サンプルã‹ã‚‰ã‚¢ãƒ«ã‚´ãƒªã‚ºãƒ レベルã§ç†è§£ã—ã€æ˜”ã‹ã‚‰ã‚ã‚‹Java関連ã®åˆ†æ•£å‡¦ç†æŠ€è¡“を見直ã™ç‰¹é›†ä¼ç”»ï¼ˆç·¨é›†éƒ¨ï¼‰ Apache Hadoopプãƒã‚¸ã‚§ã‚¯ãƒˆã¨ã¯ä½•ã‹ï¼Ÿ 本特集ã§ã¯ã€ã„ã¾å†æ³¨ç›®ã®åˆ†æ•£å‡¦ç†æŠ€è¡“ã«ã¤ã„ã¦ã„ã‚ã„ã‚ã¨ç´¹ä»‹ã—ã¦ãã¾ã—ãŸã€‚å‰ç·¨ã®ã€ŒGoogleã®MapReduceアルゴリズムをJavaã§ç†è§£ã™ã‚‹ã€ã§ã¯ã€åˆ†æ•£å‡¦ç†æŠ€è¡“ã§æœ€è¿‘注目を浴ã³ã¦ã„ã‚‹ã‚‚ã®ã¨ã—ã¦ã€ã‚°ãƒ¼ã‚°ãƒ«ã®MapReduceアルゴリズムを紹介ã—ã€ä¸ç·¨ã®ã€Œã‚¤ãƒã‚¤ãƒãªåˆ†æ•£å‡¦ç†æŠ€è¡“ã¨ã‚¤ãƒžãƒ‰ã‚ã®Webサービスã€ã§ã¯ã€MapReduceã«è‡³ã‚‹ã¾ã§ã®ã•ã¾ã–ã¾ãªåˆ†æ•£å‡¦ç†æŠ€è¡“ã‚„Webサービスã«ã¤ã„ã¦ç´¹ä»‹ã—ã¾ã—ãŸã€‚ ãã®MapReduceアルゴリズムをJavaã§å®Ÿè£…ã—ãŸã‚‚
Hadoopã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã¨ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã®å®Ÿè¡Œ
å‰å›žã¯Googleã®åŸºç›¤æŠ€è¡“ã¨ãã‚Œã«å¯¾å¿œã™ã‚‹ã‚ªãƒ¼ãƒ—ンソースソフトウェアã¨ã—ã¦ã€Hadoop & hBaseを紹介ã—ã¾ã—ãŸï¼ˆå›³1 å‚照）。今回ã¯Hadoopã‚’1å°ã«ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã€ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã‚’å‹•ã‹ã—ã¾ã™ã€‚次ã«HDFSã¨MapReduceã®ã‚¢ãƒ¼ã‚テクãƒãƒ£ã‚’解説ã—ã¾ã™ã€‚最後ã«ã‚µãƒ³ãƒ—ルプãƒã‚°ãƒ©ãƒ ã®ã‚½ãƒ¼ã‚¹ã‚³ãƒ¼ãƒ‰ã‚’解説ã—ã¾ã™ã€‚ 2. Hadoopã®æ¦‚è¦ Hadoopã¯ä¸»ã«Yahoo! Inc.ã®Doug Cuttingæ°ã«ã‚ˆã£ã¦é–‹ç™ºãŒé€²ã‚られã¦ã„るオープンソースソフトウェアã§ã€GoogleFileSystemã¨MapReduceã¨ã„ã†Googleã®åŸºç›¤æŠ€è¡“ã®ã‚ªãƒ¼ãƒ—ンソース実装ã§ã™ã€‚Hadoopã¨ã„ã†åå‰ã¯é–‹ç™ºè€…ã®åä¾›ãŒæŒã£ã¦ã„る黄色ã„象ã®ã¬ã„ãã‚‹ã¿ã®åå‰ã«ç”±æ¥ã—ã¦ã„ã¾ã™ã€‚Hadoopã¯HDFS（Hadoop Distributed File System）ã€Hadoop MapReduce F
Welcome to Apacheâ„¢ Hadoopâ„¢!
Apache Hadoop The Apache® Hadoop® project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation an
複数マシンã¸Hadoopをインストールã™ã‚‹
CodeZine編集部ã§ã¯ã€ç¾å ´ã§æ´»èºã™ã‚‹ãƒ‡ãƒ™ãƒãƒƒãƒ‘ーをスターã«ã™ã‚‹ãŸã‚ã®ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã€ŒDevelopers Summitã€ã‚„ã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã®ç”Ÿãã–ã¾ã‚’ブーストã™ã‚‹ãŸã‚ã®ã‚¤ãƒ™ãƒ³ãƒˆã€ŒDevelopers Boostã€ãªã©ã€ã•ã¾ã–ã¾ãªã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã‚’ä¼ç”»ãƒ»é‹å–¶ã—ã¦ã„ã¾ã™ã€‚
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
http://infra-engineer.com/hadoop/hadoop%E8%B1%A1%E6%9C%AC%E7%AC%AC2%E7%89%88-11%E7%AB%A0-pig-%E3%81%AE%E3%81%BE%E3%81%A8%E3%82%81/
{{#tags}}- {{label}}
{{/tags}}