[B! hadoop][development] yukungã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yukung id:yukung

hadoopã¨developmentã«é–¢ã™ã‚‹yukungã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (17)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Hadoopã®SQLå¯¾å¿œåˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒCloudera Impalaã€ã€‚ClouderaãŒã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹
Hadoopã®SQLå¯¾å¿œåˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒCloudera Impalaã€ã€‚ClouderaãŒã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ Hadoopã®ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ãƒ™ãƒ³ãƒ€ã¨ã—ã¦çŸ¥ã‚‰ã‚Œã‚‹Clouderaã¯10æœˆ25æ—¥ã€SQLã«å¯¾å¿œã—ã€ãƒ‡ãƒ¼ã‚¿ã®åˆ†æžé€Ÿåº¦ã¯MapReduceã‚ˆã‚Šã‚‚ä½•å€ã‚‚é«˜é€Ÿã ã¨ã„ã†æ–°ã—ã„åˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã€ŒCloudera Impalaã€ï¼ˆè£½å“åã€ŒCloudera Enterprise RTQã€ï¼‰ã‚’ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã§å…¬é–‹ã—ã¾ã—ãŸã€‚ ã“ã‚Œã¾ã§Hadoopã§ã¯å†…éƒ¨ã§MapReduceã¨å‘¼ã°ã‚Œã‚‹å‡¦ç†ãŒç”¨ã„ã‚‰ã‚Œã¦ã„ã¾ã—ãŸãŒã€Impalaã§ã¯MapReduceã‚’ä½¿ã‚ãšã€ClouderaãŒ2å¹´ã‹ã‘ã¦é–‹ç™ºã—ãŸç‹¬è‡ªã®åˆ†æ•£ã‚¯ã‚¨ãƒªã‚¨ãƒ³ã‚¸ãƒ³ã‚’ç”¨ã„ã¦å‡¦ç†ã‚’è¡Œã„ã¾ã™ã€‚Hiveã®ä¸Šä½äº’æ›ã®SQLãŒåˆ©ç”¨ã§ãã€Hive/MapReduceã§æ•°åˆ†ã‹ã‹ã£ã¦ã„ãŸå¿œç”æ™‚é–“ã‚’æ•°ç§’ã«çŸç¸®ã™ã‚‹ã¨èª¬æ˜Žã•ã‚Œã¦ã„ã¾ã™ã€‚ ã‚°ãƒ¼ã‚°ãƒ«ã®Dremel
yukung 2012/10/26
development

hadoop

hugedata

impala

cloudera

database

sql

mapreduce
ãƒªãƒ³ã‚¯
æ—¥ã€…é€²åŒ–ã™ã‚‹Hadoopã€‚ã“ã‚Œã¾ã§ã®ãŠã•ã‚‰ã„ã¨æœ€è¿‘ã®å‹•å‘ï¼ˆå¾Œç·¨ï¼‰
Hadoopã¨ã¯ã©ã†ã„ã†ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã‚ã‚Šã€ã„ã¾ã©ã®ã‚ˆã†ãªçŠ¶æ³ã«ãªã£ã¦ã„ã‚‹ã®ã‹ã€‚NTTãƒ‡ãƒ¼ã‚¿ã®æ¿±é‡Žè³¢ä¸€æœ—æ°ãŒã€å…ˆæ—¥è¡Œã‚ã‚ŒãŸç¬¬2å›ž NHNãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã§è¡Œã‚ã‚ŒãŸã‚»ãƒƒã‚·ãƒ§ãƒ³ã€Œæ—¥ã€…é€²åŒ–ã™ã‚‹Hadoopã®ã€Žã„ã¾ã€ã€ã§åˆ†ã‹ã‚Šã‚„ã™ãè§£èª¬ã—ã¦ã„ã¾ã™ã€‚ ï¼ˆæœ¬è¨˜äº‹ã¯ã€Œæ—¥ã€…é€²åŒ–ã™ã‚‹Hadoopã€‚ã“ã‚Œã¾ã§ã®ãŠã•ã‚‰ã„ã¨æœ€è¿‘ã®å‹•å‘ï¼ˆå‰ç·¨ï¼‰ã€ã®ç¶šãã§ã™ï¼‰ Hadoopã®1.0ç³»ã¨2.0ç³» Hadoopã®é–‹ç™ºã¯ã‹ãªã‚Šæ´»ç™ºã«é€²ã‚“ã§ã„ã¾ã™ã€‚ Hadoopã§ã„ã¡ã°ã‚“åˆ†ã‹ã‚‰ãªã„ã¨ã„ã‚ã‚Œã‚‹ã®ãŒãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã€ã“ã‚Œã¯ä»¥å‰ã‹ã‚‰Hadoopã«æŽ¥ã—ã¦ã„ã‚‹ç§ãŸã¡ã§ã‚‚ã‚ˆãåˆ†ã‹ã‚‰ãªã„ãã‚‰ã„ã§ã™ã€‚ Hadoopã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã«ã¯å¤§ããäºŒç³»çµ±ã‚ã£ã¦ã€å°‘ã—å‰ã¾ã§0.20ç³»ã¨å‘¼ã°ã‚Œã¦ã„ãŸã‚‚ã®ã§ã€ã“ã‚ŒãŒç¾è¡Œã¯1.0ç³»ã¨å‘¼ã°ã‚Œã¦ã„ã¾ã™ã€‚ã‚‚ã†ã²ã¨ã¤ãŒ0.21ã€0.22ã€0.23ç³»ã¨å‘¼ã°ã‚Œã¦ã„ãŸã‚‚ã®ã€ã“ã‚ŒãŒã„ã¾2.0ç³»ã¨å‘¼ã°ã‚Œã¦ã„ã¾ã™ã€‚ å¯¾å¤–çš„ã«ã¯
yukung 2012/08/27
development

hadoop
ãƒªãƒ³ã‚¯
æ—¥ã€…é€²åŒ–ã™ã‚‹Hadoopã€‚ã“ã‚Œã¾ã§ã®ãŠã•ã‚‰ã„ã¨æœ€è¿‘ã®å‹•å‘ï¼ˆå‰ç·¨ï¼‰
å¤§è¦æ¨¡ãªåˆ†æ•£å‡¦ç†ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã¨ã—ã¦HadoopãŒç™»å ´ã—ãŸã“ã¨ã«ã‚ˆã‚Šã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã®ãƒ–ãƒ¼ãƒ ã‚„ã€å¤§è¦æ¨¡ãªã‚½ãƒ¼ã‚·ãƒ£ãƒ«ã‚²ãƒ¼ãƒ ã§ã®ãƒã‚°è§£æžã«ã‚ˆã‚‹æ”¹å–„ã€ã‚³ãƒžãƒ¼ã‚¹ã‚µã‚¤ãƒˆã§ã®æ©Ÿæ¢°å¦ç¿’ã«ã‚ˆã‚‹ãƒ¬ã‚³ãƒ¡ãƒ³ãƒ‡ãƒ¼ã‚·ãƒ§ãƒ³ãªã©ã€å¤šãã®å¤‰åŒ–ãŒå¼•ãèµ·ã“ã•ã‚Œã¦ãã¾ã—ãŸã€‚ ãã—ã¦ãã®Hadoopè‡ªä½“ã‚‚ã€æ—¥ã€…é€²åŒ–ã—ç¶šã‘ã¦ã„ã¾ã™ã€‚ Hadoopã¨ã¯ã©ã†ã„ã†ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã§ã‚ã‚Šã€ã„ã¾ã©ã®ã‚ˆã†ãªçŠ¶æ³ã«ãªã£ã¦ã„ã‚‹ã®ã‹ã€‚NTTãƒ‡ãƒ¼ã‚¿ã®æ¿±é‡Žè³¢ä¸€æœ—æ°ãŒã€å…ˆæ—¥è¡Œã‚ã‚ŒãŸç¬¬2å›ž NHNãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã§è¡Œã£ãŸã‚»ãƒƒã‚·ãƒ§ãƒ³ã€Œæ—¥ã€…é€²åŒ–ã™ã‚‹Hadoopã®ã€Žã„ã¾ã€ã€ã§åˆ†ã‹ã‚Šã‚„ã™ãè§£èª¬ã—ã¦ã„ã¾ã™ã€‚ ã“ã®è¨˜äº‹ã§ã¯ãã®ã‚»ãƒƒã‚·ãƒ§ãƒ³ã®å†…å®¹ã‚’ãƒ€ã‚¤ã‚¸ã‚§ã‚¹ãƒˆã§ç´¹ä»‹ã—ã¾ã—ã‚‡ã†ã€‚ æ—¥ã€…é€²åŒ–ã™ã‚‹Hadoopã®ã€Œã„ã¾ã€ NTTãƒ‡ãƒ¼ã‚¿ åŸºç›¤ã‚·ã‚¹ãƒ†ãƒ äº‹æ¥éƒ¨ æ¿±é‡Žè³¢ä¸€æœ—æ°ã€‚ NTTãƒ‡ãƒ¼ã‚¿ã¨ã„ã†ã¨ã“ã‚ã§ä»•äº‹ã‚’ã—ã¦ã„ã¾ã™ã€‚NTTãƒ‡ãƒ¼ã‚¿è‡ªä½“ã¯ã‚‚ã†5å¹´ãã‚‰ã„Hadoopã‚’ã‚„ã£ã¦ã¾ã—ã¦ã€ã
yukung 2012/08/27
development

hadoop
ãƒªãƒ³ã‚¯
ã€17-E-3ã€‘Hadoopï¼šé»„è‰²ã„è±¡ä½¿ã„ã¸ã®é“ ï½žã€ŒHadoopå¾¹åº•å…¥é–€ã€ã‚ˆã‚Šï½ž
1. Hadoop é»„è‰²ã„è±¡ä½¿ã„ã¸ã®é“ ï½žHadoopå¾¹åº•å…¥é–€ã‚ˆã‚Šï½ž æ ªå¼ä¼šç¤¾NTTãƒ‡ãƒ¼ã‚¿ åŸºç›¤ã‚·ã‚¹ãƒ†ãƒ äº‹æ¥æœ¬éƒ¨ ã‚·ã‚¹ãƒ†ãƒ æ–¹å¼æŠ€è¡“BU èª²é•·ä»£ç† ä¸‹åž£ å¾¹ 2. æœ¬æ—¥ã®ã‚¢ã‚¸ã‚§ãƒ³ãƒ€ Hadoopæ¦‚è¦ Hadoopé©ç”¨äº‹ä¾‹ ã€ŒHadoopå¾¹åº•å…¥é–€ã€ã®èªã¿æ–¹ é»„è‰²ã„è±¡ (= Hadoop) ä½¿ã„ã¸ã®é“ Hadoopåˆ©ç”¨æ™‚ã®ã‚ˆãã‚ã‚‹è³ªå• Copyright Â© 2010 NTT DATA CORPORATION 1 4. ãã®å‰ã«ãƒ»ãƒ»ãƒ» Googleã®åŸºç›¤æŠ€è¡“ Googleã¯ç‹¬è‡ªã®åŸºç›¤æŠ€è¡“ã‚’ç”¨ã„ã¦ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã‚’å¯¾è±¡ã¨ã—ãŸã‚µãƒ¼ãƒ“ã‚¹ã‚’å±•é–‹ Googleã¯ã€è‡ªã‚‰ ã€Œã‚¯ãƒ©ã‚¦ãƒ‰ã‚³ãƒ³ãƒ”ãƒ¥ãƒ¼ã‚¿ã‚’æŒã£ã¦ã‚µãƒ¼ãƒ“ã‚¹å±•é–‹ã—ã¦ã„ã‚‹ã€ ã¨ã„ã† Webæ¤œç´¢ ãƒã‚°è§£æž Gmail Google Maps ãƒ»ãƒ»ãƒ» ãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°è¨€èªž Sawzall åˆ†æ•£å‡¦ç†ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ ã‚ãƒ¼ãƒ»ãƒãƒªãƒ¥ãƒ¼åž‹ãƒ‡ãƒ¼ã‚¿ã‚¹ãƒˆã‚¢ è«–æ–‡ï¼š MapReduc
yukung 2011/10/20
development

hugedata

tutorial

hadoop

mapreduce
ãƒªãƒ³ã‚¯
Hortonworks Documentation
yukung 2011/10/17
development

hugedata

hadoop

flume

document
ãƒªãƒ³ã‚¯
CDH3 Installation - Cloudera Support
Ways To Install CDH3 You can install CDH3 in any of the following ways: Automated method using SCM Express; instructions here. Service and Configuration Manager Express Edition (SCM Express) automates the installation and configuration of CDH3 on an entire cluster (up to 50 nodes), requiring only that you have root SSH access to your cluster's machines. SCM Express is not supported on 32-bit Red
yukung 2011/10/17
Flume ã®ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«æ–¹æ³•ã€‚

development

hugedata

hadoop

cassandra

logging

flume
ãƒªãƒ³ã‚¯
Cloudera Blog
We are excited to announce the acquisition of Octop ai, a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. Clouderaâ€™s mission since its inception has been to empower organizations to transf orm all their data to deliver trusted, valuable, and predictive insights. With AI and [â€¦] Read blog post
yukung 2011/10/17
development

hugedata

hadoop

cassandra

flume

logging
ãƒªãƒ³ã‚¯
https://www.gemini-bigdata.com/2011/03/flume-cassandra-log-processing-system.html
yukung 2011/10/17
development

hugedata

hadoop

logging

cassandra

flume
ãƒªãƒ³ã‚¯
GitHub - cloudera/flume: WE HAVE MOVED to Apache Incubator. https://cwiki.apache.org/FLUME/ . Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexib
h1. Welcome to Flume! NOTE: We have moved to the Apache Incubator. https://cwiki.apache.org/FLUME/ Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and rec
yukung 2011/10/17
development

hugedata

hadoop

cassandra

document

flume
ãƒªãƒ³ã‚¯
Flume ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆ (v0.9.5-SNAPSHOT)
Flume ã¯ã€å¤§é‡ã®ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚’åŠ¹çŽ‡çš„ã«åŽé›†ã€é›†ç´„ã€ç§»å‹•ã™ã‚‹ã“ã¨ã‚’ç›®çš„ã«é–‹ç™ºã•ã‚ŒãŸã€é«˜ã„ä¿¡é ¼æ€§ã¨å¯ç”¨æ€§ã‚’æŒã¤åˆ†æ•£åž‹ã®ã‚µãƒ¼ãƒ“ã‚¹ã§ã™ã€‚Flume ã‚’ä½¿ãˆã°ã€ã‚¯ãƒ©ã‚¹ã‚¿å†…ã®å„ãƒžã‚·ãƒ³ã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åŽé›†ã—ã€ã“ã‚Œã‚‰ã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ Hadoop Distributed File System (HDFS) ãªã©ã®ä¸å¤®ã®æ°¸ç¶šçš„ã‚¹ãƒˆã‚¢ã«é›†ç´„ã™ã‚‹ãƒã‚®ãƒ³ã‚°ã‚·ã‚¹ãƒ†ãƒ ã‚’æ§‹ç¯‰ã§ãã¾ã™ã€‚
yukung 2011/10/17
development

document

hadoop

flume

hugedata

cassandra
ãƒªãƒ³ã‚¯
ã€ŽHadoop/Hiveã‚’ç”¨ã„ãŸãƒã‚°è§£æžåŸºç›¤ã®æ§‹ç¯‰ã€
ã“ã‚“ã«ã¡ã¯ã€‚Amebaã®ãƒã‚°è§£æžåŸºç›¤ã‚’æ‹…å½“ã—ã¦ã„ã‚‹ICHIROã§ã™ã€‚ ä»Šå›žã¯æ§‹ç¯‰ãƒ»é‹ç”¨ä¸ã®ãƒã‚°è§£æžåŸºç›¤ã€ŒPatriotã€ã«ã¤ã„ã¦æ›¸ããŸã„ã¨æ€ã„ã¾ã™ã€‚ Webã‚µãƒ¼ãƒ“ã‚¹ã‚’é‹å–¶ã—ã¦ã„ã‚‹ã¨æ—¥ã€…å¤§é‡ã®ãƒã‚°ãƒ‡ãƒ¼ã‚¿ã‚„ãƒ¦ãƒ¼ã‚¶ãƒ‡ãƒ¼ã‚¿ãŒè“„ç©ã•ã‚Œã¾ã™ã€‚ ä»Šã¾ã§ã¯PVï¼ˆãƒšãƒ¼ã‚¸ãƒ“ãƒ¥ãƒ¼ï¼‰ã‚„UUï¼ˆãƒ¦ãƒ‹ãƒ¼ã‚¯ãƒ¦ãƒ¼ã‚¶ï¼‰ãªã©ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ãªã©ã‹ã‚‰å–ã‚Œã‚‹ã€å¤§æž ã®æŒ‡æ¨™ã®ã¿ã‚’è¦‹ã‚‹ã“ã¨ãŒã»ã¨ã‚“ã©ã§ã—ãŸã€‚ ãƒšãƒ¼ã‚¸ãƒ“ãƒ¥ãƒ¼ã«åˆã‚ã›ã¦ã‚·ã‚¹ãƒ†ãƒ ã‚’å¢—å¼·ã™ã‚‹ãªã©ã€ã‚·ã‚¹ãƒ†ãƒ å´ã¨ã—ã¦ã¯ååˆ†ã¨ã‚‚è¨€ãˆã‚‹æŒ‡æ¨™ã§ã™ãŒã€åºƒå‘Šã‚„èª²é‡‘ã‚µãƒ¼ãƒ“ã‚¹ã¨ã„ã†è¦–ç‚¹ã‹ã‚‰è¦‹ã‚‹ã¨ååˆ†ã¨ã¯è¨€ãˆã¾ã›ã‚“ã€‚ ä»Šã¾ã§ã¯Amebaå†…ã®å€‹ã€…ã®ã‚µãƒ¼ãƒ“ã‚¹æ‹…å½“è€…ãŒå¿…è¦ã¨ã™ã‚‹æŒ‡æ¨™ã‚’å‡ºã™ãŸã‚ã«ã‚¢ãƒ—ãƒªã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã‚„DB ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã«éƒ½åº¦ä¾é ¼ã‚’ã™ã‚‹å½¢ã§ãƒ‡ãƒ¼ã‚¿ã‚’æŠ½å‡ºã—ã¦ã„ã¾ã—ãŸã€‚ ä»Šå¾Œã®èª²é‡‘ã‚µãƒ¼ãƒ“ã‚¹ã®ç™ºå±•ãŒè¦‹è¾¼ã¾ã‚Œã€ã‚ˆã‚Šãƒ‡ãƒ¼ã‚¿åˆ†æžã®é‡è¦æ€§ãŒé«˜ã¾ã‚‹ã¨è€ƒãˆãŸç§ã¯ã€ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã§ãªã„ã‚µãƒ¼ãƒ“ã‚¹æ‹…å½“è€…ï¼ˆä¸»ã«ãƒ—ãƒãƒ‡ãƒ¥ãƒ¼ã‚µï¼‰ãŒã‚ˆã‚Šç°¡å˜
yukung 2011/10/17
development

hadoop

hive

hugedata

logging
ãƒªãƒ³ã‚¯
Flume cassandra real time log processing (æ—¥æœ¬èªžï¼‰
2. æ¦‚è¦ 1. ãƒã‚°åŽé›†ã€ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã¸ã®æ ¼ç´ ãƒã‚°åŽé›†ã€ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã¸ã®æ ¼ç´ åŽé›† â€¢ è¤‡æ•°ã®ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ãƒ»ãƒŽãƒ¼ãƒ‰ã‹ã‚‰ã®ã€ Flumeã«ã‚ˆã‚‹ä¿¡é ¼æ€§ã‚„åŠ¹çŽ‡ã®é«˜ã„ãƒã‚°åŽé›†ã€‚ â€¢ Cassandraãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã¸ã®ç”Ÿãƒã‚°ã‚„å‡¦ç†æ¸ˆã¿ãƒã‚°ã®æ ¼ç´ã€‚ 2. ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã€ã‚ªãƒ³ãƒ‡ãƒžãƒ³ãƒ‰ã®ãƒ¬ãƒãƒ¼ãƒˆ ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ã€ â€¢ ã‚¦ã‚§ãƒ–GUIã‹ã‚‰Cassandraã¸ã®ã‚¯ã‚¨ãƒªãƒ¼ã€‚ ï¼ˆä¾‹ï¼‰1ç§’å½“ãŸã‚Šãƒˆãƒ©ãƒ³ã‚¶ã‚¯ã‚·ãƒ§ãƒ³å‡¦ç†ä»¶æ•°ï¼ˆTPS) VS æ™‚é–“ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã®CDRæ¤œç´¢ã€‚ ã«ã‚ˆã‚‹ã‚µãƒžãƒªãƒ¼ãƒ¬ãƒãƒ¼ãƒˆ 3. Map-Reduceã«ã‚ˆã‚‹ã‚µãƒžãƒªãƒ¼ãƒ¬ãƒãƒ¼ãƒˆ â€¢ ï¼ˆä¾‹ï¼‰ãƒ¦ãƒ¼ã‚¶ãƒ¼ç¾¤æ¯Žã®ç¨®é¡žåˆ¥ï¼ˆéŸ³å£°ã€ãƒ‡ãƒ¼ã‚¿ã€ãƒ¡ãƒ¼ãƒ«ç‰ï¼‰æœˆé–“åˆ©ç”¨ ã‚¢ãƒ—ãƒªã‚±ãƒ¼ ã‚¢ãƒ—ãƒªã‚±ãƒ¼ â€¦ ã‚¢ãƒ—ãƒªã‚±ãƒ¼ ã‚·ãƒ§ãƒ³ãƒ»ãƒŽãƒ¼ãƒ‰ ã‚·ãƒ§ãƒ³ãƒ»ãƒŽãƒ¼ãƒ‰ ã‚·ãƒ§ãƒ³ãƒ»ãƒŽãƒ¼ãƒ‰ ãƒã‚°ãƒ» ãƒã‚°ãƒ» OA&M ã‚¢ã‚°ãƒªã‚²ãƒ¼ã‚¿ ã‚¢ã‚°ãƒªã‚²ãƒ¼ã‚¿ ãƒ¬ãƒãƒ¼ãƒˆ (Web GUI) Cassandra Cassandra
yukung 2011/10/17
development

hadoop

cassandra

hugedata

slide

logging

flume
ãƒªãƒ³ã‚¯
Hadoopã§ã®MapReduceå®Ÿè¡Œæ–¹æ³• - wyukawa's diary
Hadoopã§ã®MapReduceå®Ÿè¡Œæ–¹æ³•ã¯ä»¥ä¸‹ã®5ã¤ã‚ã‚‹ã ã‚ã†ã€‚ Javaã§ã”ã‚Šã”ã‚Šå®Ÿè£… Asakusaã¤ã‹ã£ã¦å®Ÿè£… Hadoopã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ã‚’ä½¿ã£ã¦å®Ÿè£… Pigä½¿ã† Hiveä½¿ã† åƒ•è‡ªèº«ã¯ã‚‚ã£ã±ã‚‰Hiveã ã€‚ Asakusaã¯å…¨ç„¶ã‚ã‹ã‚‰ãªã„ã®ã§ã“ã“ã§ã¯è§¦ã‚Œãªã„ãŒã€ãã‚Œä»¥å¤–ã¯è±¡æœ¬èªã‚“ã§ä½•ã¨ãªãé›°å›²æ°—ã¯ã¤ã‹ã‚ãŸã€‚ åˆ†é¡žçš„ã«è¨€ã†ã¨Pigã¨Hiveã¯æ‰‹ç¶šãçš„ã€å®£è¨€çš„ã¨ã„ã†é•ã„ã¯ã‚ã‚‹ãŒDSLã¨ã„ã†æ„å‘³ã§ã¯åŒã˜ã‚ˆã†ãªã‚‚ã®ãªã®ã§ã“ã“ã§ã¯ã‚ã¾ã‚Šè§¦ã‚Œãªã„ã€‚ ã¾ãšJavaã§ã”ã‚Šã”ã‚Šå®Ÿè£…ã™ã‚‹å ´åˆã ãŒã“ã‚Œã¯ãªã‹ãªã‹å¤§å¤‰ã ã€‚ ã¨ãã«çµåˆãŒå¤§å¤‰ã€‚çµåˆãŒã©ã‚“ãªæ„Ÿã˜ã‹ã¯è±¡æœ¬ã®8ç« ã«æ›¸ã„ã¦ã‚ã‚‹ã€‚ ãã‚Œä»¥å¤–ã®è³‡æ–™ã¨ã—ã¦ã¯ä»¥å‰@n3104ã•ã‚“ãŒ 5æœˆ11æ—¥ DevLOVE é»„è‰²ã„è±¡ä½¿ã„ãŒã€ç„é•·ã«å‡ºä¼šã†ã¾ã§ã€‚ã€œè±¡ã€é‚‚é€…ç·¨ã€œ(æ±äº¬éƒ½)ã§èª¬æ˜Žã—ã¦ã„ã‚‹è³‡æ–™ã¨ã‚µãƒ³ãƒ—ãƒ«ã‚³ãƒ¼ãƒ‰ãŒè‰¯ã„é¡Œæã«ãªã‚‹ã ã‚ã†ã€‚ã‚»ã‚«ãƒ³ãƒ€ãƒªã‚½ãƒ¼ãƒˆã‚„map side
yukung 2011/09/26
development

hadoop

mapreduce

hive
ãƒªãƒ³ã‚¯
Hiveã®ãƒ™ã‚¹ãƒˆãƒ—ãƒ©ã‚¯ãƒ†ã‚£ã‚¹(ã‹ã‚‚ã—ã‚Œãªã„ã“ã¨)ã‚’ã‚ã‚‚ã£ã¦ãŠã - wyukawa's diary
Hiveã®ç¾å ´ã«æ¥ã¦4ã‹æœˆçµŒã¡å›žã‚Šã®ã‚„ã‚Šæ–¹ã‚’è¦³å¯Ÿã—ãŸã‚Šã€ä»–ã®äº‹ä¾‹ã‚’èª¿ã¹ãŸã‚Šã—ã¦ã„ã‚‹ã†ã¡ã«Hiveã‚’ä½¿ç”¨ã™ã‚‹éš›ã®ãƒ‘ã‚¿ãƒ¼ãƒ³ã¨ã„ã†ã‹ãƒ™ã‚¹ãƒˆãƒ—ãƒ©ã‚¯ãƒ†ã‚£ã‚¹ãŒè¦‹ãˆã¦ããŸæ°—ãŒã™ã‚‹ã®ã§æ›¸ã„ã¦ã¿ã‚‹ã‚ˆã€‚ ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã¯ãƒã‚°è§£æžã§ã™ã€‚ 1. ãƒ•ã‚¡ã‚¤ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã¨åœ§ç¸® ãƒã‚°ã¯åœ§ç¸®çŽ‡é«˜ã„ã®ã§åœ§ç¸®ã™ã¹ã—ã€‚ã§ã‚‚ã‚¹ãƒ—ãƒªãƒƒãƒˆå¯èƒ½ã§ã‚ã‚‹ãŸã‚ã«ã¯ã£ã¦ã“ã¨ã§SequenceFileã§ãƒ–ãƒãƒƒã‚¯åœ§ç¸®ã¯ç¢ºå®šã€‚ åœ§ç¸®ã‚³ãƒ¼ãƒ‡ãƒƒã‚¯ã¯åœ§ç¸®çŽ‡ã‚’è€ƒãˆã¦gzipã€‚ ã¡ãªã¿ã«åœ§ç¸®çŽ‡ã¯ bzip2>gzip>LZO ã§ã‚¹ãƒ”ãƒ¼ãƒ‰ã¯ LZO>gzip>bzip2 ã ã€‚bzipã¯ã‚¹ãƒ—ãƒªãƒƒãƒˆå¯èƒ½ã§åœ§ç¸®çŽ‡ã‚‚é«˜ã„ã‚“ã ã‘ã©Hadoop 0.20ç³»ã§ã¯ã¾ã ä½¿ãˆãªã‹ã£ãŸã¨æ€ã†ã€‚Hadoop 0.20ç³»ã§ã‚‚ä½¿ãˆã‚‹ã‚‰ã—ã„ã§ã™ãŒé…ãã¦ä½¿ã„ç‰©ã«ãªã‚‰ãªã„ã‚ˆã†ã§ã™ã€‚ æœ€è¿‘ã¯snappyãªã‚“ã¦ã®ã‚‚å‡ºã¦ã¦CDH3 Update 1ã§ä½¿ãˆã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸãã€‚ snappyã¯åœ§ç¸®çŽ‡ã¯L
yukung 2011/09/26
development

hadoop

hive

tips
ãƒªãƒ³ã‚¯
ãã‚ãã‚Hadoopã«ã¤ã„ã¦ã²ã¨ã“ã¨è¨€ã£ã¦ãŠãã‹ - nokunoã®æ—¥è¨˜
ã‚‚ã†ã“ã®æ‰‹ã®è©±é¡Œã¯å‡ºå°½ãã—ãŸæ„ŸãŒã‚ã‚Šã¾ã™ãŒã€æœ€è¿‘Hadoopã«ã¤ã„ã¦è€ƒãˆã‚‹ã“ã¨ãŒå¤šã„ã®ã§ã€ã‚¨ãƒ³ãƒˆãƒªã«ã—ã¦ã¿ã¾ã™ã€‚ãªãŠã€ã“ã“ã§ã¯ãƒ™ãƒ¼ã‚·ãƒƒã‚¯ãªMapReduce+HDFSã®ã“ã¨ã‚’Hadoopã¨å‘¼ã¶ã“ã¨ã«ã—ã¾ã™ã€‚ Hadoopã¨ã¯Hadoopã¨ã¯è¨€ã‚ãšã¨çŸ¥ã‚ŒãŸGoogleã®MapReduce/GFSã®ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã®ã‚¯ãƒãƒ¼ãƒ³ã§ã™ã€‚MapReduceã§ã¯ãƒ—ãƒã‚°ãƒ©ãƒžã¯Mapã¨Reduceã¨ã„ã†2ã¤ã®é–¢æ•°ã‚’æ›¸ãã ã‘ã§ã€ä¸¦åˆ—åˆ†æ•£å‡¦ç†ã‚’ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ã“ã‚Œã¯(1) ãƒ‡ãƒ¼ã‚¿ã‚’å®Ÿéš›ã«æŒã¤ãƒžã‚·ãƒ³ã«ãƒ—ãƒã‚°ãƒ©ãƒ ã‚’é…å¸ƒã™ã‚‹ (2) Mapã¨Reduceã‚’ã¤ãªãShuffleãƒ•ã‚§ãƒ¼ã‚ºã§ã‚ãƒ¼ã‚’ã‚°ãƒ«ãƒ¼ãƒ—åŒ–ã—ã¦ã‚½ãƒ¼ãƒˆã™ã‚‹ã€(3) éšœå®³æ™‚ã®ãƒ•ã‚§ãƒ¼ãƒ«ã‚ªãƒ¼ãƒãƒ¼ã‚„ãƒ¬ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã€ã¨ã„ã£ãŸå‡¦ç†ã‚’ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯å´ãŒå—ã‘æŒã¤ã“ã¨ã«ã‚ˆã£ã¦ã€ãƒ—ãƒã‚°ãƒ©ãƒžå´ã®è² æ‹…ã‚’æ¸›ã‚‰ã™ã‚‚ã®ã§ã™ã€‚GFSã«å¯¾å¿œã™ã‚‹HDFSã«ã¯ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚¯ãƒ©ã‚¹ã‚¿ã«åˆ†æ•£ã—ã¦ä¿å˜
yukung 2011/01/13
development

hadoop

tutorial
ãƒªãƒ³ã‚¯
NTTãƒ‡ãƒ¼ã‚¿ã®Hadoopå ±å‘Šæ›¸ãŒã™ã”ã‹ã£ãŸ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
æ¥ç•Œãƒˆãƒƒãƒ— ã®ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ãƒ©ã‚¤ã‚º Hadoop ä¼æ¥ Cloudera ã«å…¥ç¤¾ã—ã¾ã—ãŸ http://www.cloudera.co.jp/ ä»Šå¹´ã®6æœˆã«ã€ã€Œå¹³æˆï¼’ï¼‘å¹´åº¦ ç”£å¦é€£æºã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦å®Ÿè·µäº‹æ¥å ±å‘Šæ›¸ã€ã¨ã„ã†ãƒ‰ã‚ãƒ¥ãƒ¡ãƒ³ãƒˆç¾¤ãŒçµŒç”£çœã‹ã‚‰å…¬è¡¨ã•ã‚Œã¾ã—ãŸã€‚ ãã®ã†ã¡ã®ä¸€ã¤ã«ã€NTTãƒ‡ãƒ¼ã‚¿ã«å§”è¨—ã•ã‚ŒãŸHadoopã«é–¢ã™ã‚‹å®Ÿè¨¼å®Ÿé¨“ã®å ±å‘Šæ›¸ãŒã‚ã‚Šã¾ã—ãŸã®ã§ã€ä»Šæ›´ãªãŒã‚‰èªã‚“ã§ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ Hadoopç•Œéšˆã®äººã¯ã‚‚ã†ã¿ã‚“ãªã¨ã£ãã«èªã‚“ã§ã‚‹ã®ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã‘ã©ã€‚ http://www.meti.go.jp/policy/mono_info_service/joho/downloadfiles/2010software_research/clou_dist_software.pdf ã€Œé«˜ä¿¡é ¼ã‚¯ãƒ©ã‚¦ãƒ‰å®Ÿç¾ç”¨ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆåˆ†æ•£åˆ¶å¾¡å‡¦ç†æŠ€è¡“ç‰ã«ä¿‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼é«˜ä¿¡é ¼åŒ–ã«å‘ã‘ãŸå®Ÿè¨¼äº‹æ¥ï¼‰ã€ã¨ã„ã†
yukung 2010/12/26
programming

hadoop

development

cloud

document

mapreduce
ãƒªãƒ³ã‚¯
ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã¨Hadoop - ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰é–‹ç™ºè€…ãƒ–ãƒã‚°
ã¯ã˜ã‚ã¾ã—ã¦ã€‚ä»Šå¹´ã®5æœˆã«å…¥ç¤¾ã—ãŸå‹é–“@ã•ãŒã™ãƒãƒ¼ãƒ ã§ã™ã€‚ å…¥ç¤¾ã—ã¦ã‹ã‚‰ã¯ã€ãªã‹ãªã‹å¤§å¤‰ãªã“ã¨ã‚‚å¤šã„ã§ã™ãŒã€æœ€è¿‘ã¯ãŠé…’å¥½ããŒé›†ã¾ã£ã¦æœˆæ›œã‹ã‚‰é£²ã¿åˆã† ã€Œå‹é–“ä¼šã€ãªã‚‹ã‚‚ã®ã‚‚ç™ºè¶³ã—ã¦ã€ä»•äº‹é¢ã§ã‚‚ä»•äº‹ä»¥å¤–ã®é¢ã§ã‚‚å¯†åº¦ã®é«˜ã„æ¯Žæ—¥ã‚’éŽã”ã—ã¦ã„ã¾ã™ï¼ ã•ã¦ã€åƒ•ã¯ã€Œã•ãŒã™ã€ãƒãƒ¼ãƒ æ‰€å±žã¨ã„ã†ã“ã¨ã§ã€æ™®æ®µã¯ãƒ¬ã‚·ãƒ”ã‚’ã€Œã•ãŒã™ã€ãƒ¦ãƒ¼ã‚¶ã®æº€è¶³åº¦ã‚’ä¸Šã’ã‚‹ãŸã‚ã«ã€ ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®æ¤œç´¢ã¾ã‚ã‚Šã«ã¤ã„ã¦ã€ã„ã‚ã„ã‚ãªé–‹ç™ºã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ ä¸€æ–¹ã§ã€ãƒ¦ãƒ¼ã‚¶ã®ã€Œã•ãŒã™æ¬²æ±‚ã€ã«ã¤ã„ã¦æ·±ãçŸ¥ã‚‹ãŸã‚ã«ã€å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿è§£æžã‚’è¡Œã„ã€æ¬²æ±‚ã®åˆ†æžã‚’è¡Œã†æ©Ÿä¼šã‚‚å¢—ãˆã¦ãã¾ã—ãŸã€‚ ã¨ã“ã‚ãŒã€ã‚¯ãƒƒã‚¯ãƒ‘ãƒƒãƒ‰ã®ãƒã‚°ã¯è†¨å¤§ãªæ•°ãŒã‚ã‚‹ã®ã§ã€ä¸€å£ã®ãƒ‡ãƒ¼ã‚¿è§£æžã¨è¨€ã£ã¦ã‚‚é€šå¸¸ã®ãƒãƒƒãƒå‡¦ç†ã ã¨é–“ã«åˆã‚ãªã„ãŸã‚ã€ åˆ†æ•£å‡¦ç†ç’°å¢ƒã®å¿…è¦æ€§ãŒé«˜ã¾ã£ã¦ãã¾ã—ãŸã€‚ ãã“ã§ã€ã¾ãšã¯æ‰‹è»½ã«è©¦ã›ã‚‹åˆ†æ•£å‡¦ç†ã®çŽ‹é“ã¨ã„ã†ã“ã¨ã§ã€æœ€è¿‘ã§ã¯Hadoopã‚’ä½¿ã£ãŸãƒ‡ãƒ¼ã‚¿è§£æžç’°å¢ƒã‚’æ•´å‚™ã—ã¦ã„ã¾ã™ã€‚
yukung 2009/09/17
webservice

development

hadoop

cookpad
ãƒªãƒ³ã‚¯
1