[B! hadoop] nubesã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

nubes id:nubes

hadoopã«é–¢ã™ã‚‹nubesã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (30)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

ãƒ‡ãƒ¼ã‚¿ãƒ‰ãƒªãƒ–ãƒ³ä¼æ¥ã«ãŠã‘ã‚‹ã€hadoopåŸºç›¤ã¨ETL #hcj2016
Hadoop Conference Japan 2016 ã§ç™ºè¡¨ã—ãŸè³‡æ–™ã§ã™ã€‚ http://www.eventbrite.com/e/hadoop-spark-conference-japan-2016-tickets-20809016328 ã“ã®è³‡æ–™ã¯å‰åŠéƒ¨åˆ†ã§ã™ã€‚ãƒ‰ãƒ¯ãƒ³ã‚´å¿—æ‘æ§˜ã«ã‚ˆã‚‹å¾ŒåŠéƒ¨åˆ†ã¯ã“ã¡ã‚‰: http://www.slideshare.net/smrmkt/hadoopetl-niconico ---- (ã‚¢ãƒ–ã‚¹ãƒˆãƒ©ã‚¯ãƒˆ) ETL(Extract, Transf orm, Load) ã¯ã€å¤–éƒ¨ã®ã‚½ãƒ¼ã‚¹ã‹ã‚‰ãƒ‡ãƒ¼ã‚¿ã‚’æŠ½å‡ºã—ã€æ¥å‹™ä¸Šå¿…è¦ãªå½¢ã«åŠ å·¥ã—ã€ãƒ‡ãƒ¼ã‚¿åŸºç›¤ã«ãƒãƒ¼ãƒ‰ã™ã‚‹ãŸã‚ã®ä¸€é€£ã®æŠ€è¡“ã§ã™ã€‚Hadoopã®ç™»å ´ã«ã‚ˆã‚Šã€ã‚¹ã‚±ãƒ¼ãƒ©ãƒ–ãƒ«ã‹ã¤æŸ”è»Ÿãªã‚¹ã‚ãƒ¼ãƒžæ§‹é€ ã‚’æŒã¤ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ãƒ¤ãƒ¼ã§ã‚ã‚‹HDFSã«ä»£è¡¨ã•ã‚Œã‚‹ã€ETLã«ãŠã‘ã‚‹å¤šãã®ç‰©ç†ã‚¢ãƒ¼ã‚ãƒ†ã‚¯ãƒãƒ£ã«å¤‰åŒ–ãŒã‚‚ãŸã‚‰ã•ã‚Œã¾ã—ãŸã€‚ã—ã‹ã—ä¸€æ–¹ã§ã€ETL
nubes 2016/02/08
hadoop

slide
ãƒªãƒ³ã‚¯
Hadoopã¯ã©ã®ã‚ˆã†ã«å‹•ãã®ã‹ â”€ä¸¦åˆ—ãƒ»åˆ†æ•£ã‚·ã‚¹ãƒ†ãƒ æŠ€è¡“ã‹ã‚‰èªã¿è§£ãHadoopå‡¦ç†ç³»ã®è¨è¨ˆã¨å®Ÿè£… è¨˜äº‹ä¸€è¦§ | gihyo.jp
ç¬¬21å›žSparkã®è¨è¨ˆã¨å®Ÿè£…ï¼»2ï¼½ï½žSparkã«ãŠã‘ã‚‹ãƒ‡ãƒ¼ã‚¿å…±æœ‰ã®ä»•çµ„ã¿ã¨è€éšœå®³æ€§ã®å®Ÿç¾æ–¹æ³• çŒ¿ç”°æµ©è¼”ï¼Œå±±ç”°æµ©ä¹‹ 2016-06-08
nubes 2015/11/06
hadoop
ãƒªãƒ³ã‚¯
Hortonworksã®ã‚¤ãƒ™ãƒ³ãƒˆã«è¡Œã£ã¦ããŸ - wyukawa's diary
Hadoop and the Modern Data Architecture ã«è¡Œã£ã¦ãã¾ã—ãŸã€‚ ç«‹æ´¾ãªãƒ›ãƒ†ãƒ«ã§è‰¯ã„ã‚¤ãƒ™ãƒ³ãƒˆã§ã—ãŸã€‚ã‚ã‚ŠãŒã¨ã†ã”ã–ã„ã¾ã—ãŸã€‚ï¼ž Hortonworksã®ã¿ãªã•ã¾ ã‚»ãƒƒã‚·ãƒ§ãƒ³ã‚„ä¼šå ´ã«ã„ãŸäººã¨ã®ä¼šè©±ã«ã¤ã„ã¦å°‘ã—æ›¸ãã¾ã™ã€‚ ã¾ãšåƒ•ãŒä½¿ã£ã¦ã„ã‚‹Ambariã«é–¢ã—ã¦ç™ºè¡¨ãŒã‚ã‚Šã¾ã—ãŸã€‚ ãã‚Œã«ã‚ˆã‚‹ã¨ã€Hueã¿ãŸã„ãªã‚¯ã‚¨ãƒªã‚’submitã™ã‚‹æ©Ÿèƒ½ãŒå…¥ã‚‹æ¨¡æ§˜ã€‚ã©ã†ã‚‚Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã¸ã®ã‚¢ã‚¯ã‚»ã‚¹ã‚’ã™ã¹ã¦AmbariçµŒç”±ã«ã—ãŸã„ã‚ˆã†ã ã€‚ã†ãƒ¼ã‚“ã€ãã‚Œã¯ã©ã†ãªã‚“ã ã‚ã€‚ã€‚ã€‚Prestoã‚ã‚‹ã—ã€‚å€‹äººçš„ã«ã¯ãã‚Œã‚ˆã‚Šã‚‚ã£ã¨ã‚¯ãƒ©ã‚¹ã‚¿ç®¡ç†ã«æ³¨åŠ›ã—ã¦ã»ã—ã„ã¨æ€ã£ãŸã‚Šã€‚ã€‚ã€‚ä¾‹ãˆã°ã‚¨ãƒ©ãƒ¼é€šçŸ¥ã‚’ãƒ¡ãƒ¼ãƒ«ã˜ã‚ƒãªãã¦HipChatã¨ã‹Slackã«ã¨ã°ã›ã‚‹ã‚ˆã†ã«ã™ã‚‹ã¨ã‹ã€‚ Ambariã§ä½¿ã£ã¦ã„ã‚‹Nagios, Gangilaã¯deprecatedã«ãªã‚Šã€ãƒ¡ãƒˆãƒªã‚¯ã‚¹ã‚’HBaseã«ãŸã‚ã¦Phenioxã§ã‚¯ã‚¨
nubes 2015/03/12
hadoop

yarn
ãƒªãƒ³ã‚¯
Hadoop World 2011: Hadoop Troubleshooting 101 - Kate Ting - Cloudera
Attend this session and walk away armed with solutions to the most common customer probl ems. Learn proactive configuration tweaks and best practices to keep your cluster free of fetch failures, job tracker hangs, and the like.Read less
nubes 2015/01/04
hadoop

slide

TroubleShooting
ãƒªãƒ³ã‚¯
SQL on Hadoop æ¯”è¼ƒæ¤œè¨¼ ã€2014æœˆ11æ—¥ã«ãŠã‘ã‚‹æ¤œè¨¼ãƒ¬ãƒãƒ¼ãƒˆã€‘
Impala Meetup 2014/10/31 @Tokyo è¬›æ¼”è³‡æ–™ ã€æ³¨æ„äº‹é …ã€‘ æœ¬è³‡æ–™ã§ç´¹ä»‹ã—ã¦ã„ã‚‹æ¤œè¨¼çµæžœã¯2014å¹´å½“æ™‚ã®ã‚‚ã®ã§ã™ã€‚å½“è©²ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã¯æˆé•·ã‚„æ”¹å–„ãŒæ—©ãã€ç¾æ™‚ç‚¹ã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã¯å¤§ããç•°ãªã‚‹æ©Ÿèƒ½ã‚„æ€§èƒ½ã¨ãªã£ã¦ã„ã¾ã™ã€‚ SQL on Hadoopã®æœ€æ–°æƒ…å ±ã«åŸºã¥ãã‚µãƒ¼ãƒ“ã‚¹ã‚„ã‚·ã‚¹ãƒ†ãƒ ã‚¤ãƒ³ãƒ†ã‚°ãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ã«ã”èˆˆå‘³ã‚’ãŠæŒã¡ã®æ–¹ã¯ã€NTTãƒ‡ãƒ¼ã‚¿ åŸºç›¤ã‚·ã‚¹ãƒ†ãƒ äº‹æ¥æœ¬éƒ¨ OSSãƒ—ãƒãƒ•ã‚§ãƒƒã‚·ãƒ§ãƒŠãƒ«ã‚µãƒ¼ãƒ“ã‚¹ï¼ˆé›»åãƒ¡ãƒ¼ãƒ«ï¼š hadoop [AT] kits.nttdata.co.jpï¼‰ ã«ã”ç›¸è«‡ãã ã•ã„ã€‚Read less
nubes 2014/12/09
hadoop

presto

hive

slide
ãƒªãƒ³ã‚¯
2014å¹´ã®è±¡(Hadoop)å››æ–¹å±±è©±
Spark, SQL on Hadoop etc. Advent Calendar 2014 1æ—¥ç›®ã€ãŠã‚ˆã³Hadoopé–¢é€£ï¼ˆå…¨éƒ¨ä¿ºï¼‰ Advent Calendar 2014ã®1æ—¥ç›®ã§ã™ã€‚ åˆæ—¥ãªã®ã§ã€æ˜¨å¹´ãƒ–ãƒã‚°ã«ç¶šã„ã¦2014å¹´ã®Hadoopé–¢é€£ã®è©±é¡Œã‚’ã¾ã¨ã‚ã¾ã™ã€‚ â€»ã“ã®ãƒ–ãƒã‚°ã®å†…å®¹ã¯æ‰€å±žã—ã¦ã„ã‚‹çµ„ç¹”ã«ã¯é–¢ä¿‚ãªãã€å€‹äººçš„ãªé›‘æ„Ÿã§ã™ã€‚é–“é•ã„ãŒã‚ã‚Œã°ã”æŒ‡æ‘˜ä¸‹ã•ã„ã€‚ â€»ãªãŠã€ä¸ç«‹çš„ã«æ›¸ã„ã¦ã„ã‚‹ã¤ã‚‚ã‚Šã§ã™ãŒã€å¤šå°‘åã‚ŠãŒã‚ã‚‹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã€‚ãŒã€ç‰¹ã«æ‚ªæ„ã¯ã‚ã‚Šã¾ã›ã‚“ :p ä¿ºçš„Hadoopã®è©±é¡Œ2014: ãƒ™ã‚¹ãƒˆ3 Apache Sparkã®ç››ã‚Šä¸ŠãŒã‚ŠãŒåŠ é€Ÿ SQL on Hadoop ã¯æ··æ²Œ Hadoopã¯å …èª¿ã«é€²åŒ–ã€‚å¾ã€…ã«æ¶ˆãˆã¦ã„ãï¼ï¼Ÿ ï¼ˆæ¬¡ç‚¹: Hueã‚‚å¤§å¹…ã«é€²åŒ–ã—ã¦ã¾ã™ãŒã€ã“ã‚Œã¯ã„ãšã‚Œç´¹ä»‹ã—ã¾ã™ï¼‰ 1. Apache Sparkã®ç››ã‚Šä¸ŠãŒã‚ŠãŒåŠ é€Ÿ ä»Šå¹´ã«ãªã£ã¦ã¾ã™ã¾ã™ç››ã‚Šä¸ŠãŒã£ã¦ã„ã‚‹Ap
nubes 2014/12/08
hadoop
ãƒªãƒ³ã‚¯
Workflow Engines for Hadoop
Building a reliable pipeline of data ingress, batch computation, and data egress with Hadoop can be a major challenge. Most folks start out with cron to manage workflows, but soon discover that doesn't scale past a handful of jobs. There are a number of open-source workflow engines with support for Hadoop, including Azkaban (from LinkedIn), Luigi (from Spotify), and Apache Oozie. Having deployed a
nubes 2014/11/19
hadoop

slide
ãƒªãƒ³ã‚¯
Genie - Hadoop Platform as a Service at Netflix
In a prior tech blog (http://nflx.it/XoySYR), we had discussed the architecture of our petabyte-scale data warehouse in the cloud. Salient features of our architecture include the use of Amazonâ€™s Simple Storage Service (S3) as our "source of truth", leveraging the elasticity of the cloud to run multiple dynamically resizable Hadoop clusters to support various workloads, and our horizontally scalab
nubes 2014/11/19
hadoop

netflix

slide
ãƒªãƒ³ã‚¯
S3 ä¸Šã®å¤§é‡ãƒ‡ãƒ¼ã‚¿ã‚’ EMR ã™ã‚‹ã¨ãã¯ S3DistCp ã‚’ä½¿ã†ã¨æ—ã‚‹ â€“ I'm Sei.
CloudFront ã®ã‚¢ã‚¯ã‚»ã‚¹ãƒã‚°ã‚’ S3 ã«å‡ºåŠ›ã™ã‚‹ã‚ˆã†ã«è¨å®šã—ã¦ã„ã‚‹ã¨ã€æ—¥ã«æ—¥ã«å¤§é‡ã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ãŒæºœã¾ã£ã¦ã„ãã¾ã™ã€‚ ä¸ã«ã¯ 1 MB ç¨‹åº¦ã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚‚ã‚ã‚Œã°ã€æ•°ç™¾ bytes ç¨‹åº¦ã®å°ã•ãªã‚‚ã®ã‚‚ã‚ã‚Šã¾ã™ã€‚ ä½•ã«ã›ã‚ˆã€ã¨ã«ã‹ãå¤§é‡ã«ã‚ã‚Šã¾ã™ã€‚ å°ã•ã„ãƒ•ã‚¡ã‚¤ãƒ«ã¯ã¾ã¨ã‚ã‚‹CloudFront ã®ãƒã‚°ã®ã‚ˆã†ã«å°ã•ã„ãƒ•ã‚¡ã‚¤ãƒ«ãŒå¤§é‡ã«ã‚ã‚‹ã‚ˆã†ãªã‚±ãƒ¼ã‚¹ã¯ Hadoop ãŒè‹¦æ‰‹ã¨ã™ã‚‹ãƒ‘ã‚¿ãƒ¼ãƒ³ã§ã€ãã®ã¾ã¾æ‰±ã†ã¨å¤§æŠµã‚ã¡ã‚ƒã‚“ã“æ™‚é–“ãŒã‹ã‹ã‚Šã¾ã™ã€‚ ãã†ã„ã†å ´åˆã¯ã€S3DistCp ã¨ã„ã†ä¾¿åˆ©ãƒ„ãƒ¼ãƒ«ã‚’ä½¿ã£ã¦ S3 ä¸Šã®ãƒã‚°ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ã‚‚ã†ã™ã“ã—å¤§ãã„ã‚µã‚¤ã‚ºã«é€£çµã—ã¦ã€HDFS ã«ã‚³ãƒ”ãƒ¼ã™ã‚‹ã¨ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ãŒã‚ãŒã‚‹ã“ã¨ãŒã‚ã‚Šã¾ã™ã€‚ ã¡ãªã¿ã«ã€EMR ã®ãƒ™ã‚¹ãƒˆãƒ—ãƒ©ã‚¯ãƒ†ã‚£ã‚¹ã«ã‚ˆã‚‹ã¨ã€Gzip ã§åœ§ç¸®ã•ã‚Œã¦ã„ã‚‹å ´åˆã¯ 1 ~ 2GBã€LZO ã§åœ§ç¸®ã•ã‚Œã¦ã„ã‚‹å ´åˆã¯ 2 ~ 4GB ãŒæœ€é©ãªãƒ•ã‚¡ã‚¤ãƒ«ã‚µã‚¤ã‚ºã‚‰ã—ã„ã§ã™
nubes 2014/09/22
AmazonElasticMapReduce

hadoop

s3
ãƒªãƒ³ã‚¯
SQL on Hadoop in Taiwan
This document discusses SQL engines for Hadoop, including Hive, Presto, and Impala. Hive is best for batch jobs due to its stability. Presto provides interactive queries across data sources and is easier to manage than Hive with Tez. Presto's distributed architecture allows queries to run in parallel across nodes. It supports pluggable connectors to access different data stores and has language bi
nubes 2014/09/21
slide

hadoop

presto
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿åˆ†æžç’°å¢ƒã«ã¤ã„ã¦æ›¸ã„ã¦ã¿ã‚‹ - wyukawa's diary
ãƒã‚°ã‚’HDFSã«é›†ã‚ã¦Hiveã§ETLã‚„é›†è¨ˆã‚’è¡Œã„é›†è¨ˆçµæžœã‚’RDBMSã«è“„ç©ã—ã¦ãƒ¬ãƒãƒ¼ãƒ†ã‚£ãƒ³ã‚°ãƒ„ãƒ¼ãƒ«ã§å¯è¦–åŒ–ã™ã‚‹ã¨ã„ã†ã®ã¯ä¸€èˆ¬çš„ãªè©±ã ã¨æ€ã„ã¾ã™ã€‚ ãƒ‡ãƒ¼ã‚¿ã®æµã‚Œã§ã„ã†ã¨ App -> HDFS -> RDBMS -> ãƒ¬ãƒãƒ¼ãƒ†ã‚£ãƒ³ã‚°ãƒ„ãƒ¼ãƒ« ã¨ã„ã†æ„Ÿã˜ã§ã™ã€‚ ä»–ã«ã‚‚Prestoã®ã‚ˆã†ãªlow latencyãªãƒ„ãƒ¼ãƒ«ãŒåŠ ã‚ã‚‹ã“ã¨ãŒã‚ã‚‹ã¨æ€ã„ã¾ã™ã€‚ã“ã‚Œã‚‰ã®ãƒ„ãƒ¼ãƒ«ã‚’ã©ã†çµ„ã¿åˆã‚ã›ã¦ã©ã†ETLã‚’ã¾ã‚ã—ã¦ã„ãã®ãŒã„ã„ã®ã‹ã¤ã‚‰ã¤ã‚‰ã¨æœ€è¿‘è€ƒãˆã¦ãŠã‚Šã¾ã™ã€‚ åƒ•ãŒçµŒé¨“ã—ãŸã®ã¯Pythonã§ETLå‡¦ç†ã‚’æ›¸ã„ã¦ï¼ˆå†…éƒ¨çš„ã«ã¯hiveserverã«hiveã‚¯ã‚¨ãƒªã‚’æŠ•ã’ãŸã‚Šã€MySQLã«é›†è¨ˆçµæžœã‚’ä¿å˜ã—ãŸã‚Šï¼‰ã€ã‚¹ã‚±ã‚¸ãƒ¥ãƒ¼ãƒªãƒ³ã‚°ã¯cron, Azkabanã§ã€é›†è¨ˆçµæžœã¯MySQLã§ãƒ¬ãƒãƒ¼ãƒ†ã‚£ãƒ³ã‚°ãƒ„ãƒ¼ãƒ«ã¯è‡ªä½œã§ã¨ã„ã†ã‚‚ã®ã§ã™ã€‚adhocãªãƒ‡ãƒ¼ã‚¿åˆ†æžã¯shibä½¿ã„ã¾ã™ã€‚ã¾ã‚ã“ã‚Œã§ã‚‚ååˆ†é‹ç”¨å›žã£ã¦ã‚‹ã‚“ã ã‘ã©ã€ä»–ã«ã‚‚è‰¯ã„æ–¹æ³•ãŒç„¡
nubes 2014/09/14
hadoop

DataMining
ãƒªãƒ³ã‚¯
Python + Hive on AWS EMR ã§è²§è€…ã®ãƒã‚°ã‚µãƒžãƒª
1. Akira Chiku is an engineer who works on an engineering team. Their requirements include collecting between 10-20GB of data per day from various sources like Hadoop and Hive. 2. Data is collected from sources like Fluentd and parsed using Query String and stored in Hive. It is then processed and visualized. 3. Data can be stored in S3, processed using services like AWS EMR, and visualized in das
nubes 2014/09/14
hive

hadoop

presto

slide
ãƒªãƒ³ã‚¯
Blog - LINE ENGINEERING
As of October 1, 2023, LINE has been rebranded as LY Corporation. Visit the new blog of LY Corporation here: LY Corporation Tech Blog
nubes 2014/09/03
hadoop
ãƒªãƒ³ã‚¯
mixi ã®è§£æžåŸºç›¤ã¨Apache Hive ã§ã® JSON ãƒ‘ãƒ¼ã‚µã®æ´»ç”¨ã®ç´¹ä»‹ - mixi engineer blog
ã“ã‚“ã«ã¡ã¯ï¼Žæœ€è¿‘ãƒ”ã‚¯ãƒ«ã‚¹ä½œã‚Šã§ç²¾ç¥žçµ±ä¸€ã‚’ã—ã¦ã„ã‚‹ï¼ŒãŸã‚“ã½ã½ã‚°ãƒ«ãƒ¼ãƒ—è§£æžãƒãƒ¼ãƒ ã®çŸ³å·æœ‰ã§ã™ï¼Ž ã“ã®ãƒ–ãƒã‚°ã§ã¯ãŠé¦´æŸ“ã¿ã®ãŸã‚“ã½ã½ã‚°ãƒ«ãƒ¼ãƒ—ã§ã™ãŒï¼Œ"No More ã€Œåˆºèº«ã®ä¸Šã«ã‚¿ãƒ³ãƒãƒã‚’ã®ã›ã‚‹ä»•äº‹ã€ - å˜ç´”ä½œæ¥ã®ç¹°ã‚Šè¿”ã—ã§é–‹ç™ºè€…ã®æ™‚é–“ã‚’æµªè²»ã—ãªã„ãŸã‚ã«ã€‚"ã¨ã„ã†ãƒŸãƒƒã‚·ãƒ§ãƒ³ã‚’æŒã£ã¦ã„ã¾ã™ï¼Žãã®ä¸ã§è§£æžãƒãƒ¼ãƒ ã¯ï¼Œãƒ‡ãƒ¼ã‚¿è§£æžåŸºç›¤ã®æ§‹ç¯‰ï¼Œãƒ‡ãƒ¼ã‚¿ãƒžã‚¤ãƒ‹ãƒ³ã‚°ï¼Œãƒ‡ãƒ¼ã‚¿è§£æžã®ç¤¾å†…ã‚³ãƒ³ã‚µãƒ«ãƒ†ã‚£ãƒ³ã‚°ã‚’è¡Œãªã„æŠ€è¡“ã‹ã‚‰ã®æ”¹å–„ã‚’æ‹…å½“ã—ã¦ã„ã¾ã™ï¼Ž ä»Šå›žã®è¨˜äº‹ã§ã¯ï¼Œmixi ã«ãŠã‘ã‚‹è§£æžåŸºç›¤ã«ã¤ã„ã¦ç°¡å˜ã«è§¦ã‚ŒãŸã‚ã¨ï¼Œãã®åŸºç›¤ã«ãŠã‘ã‚‹ã€Œåˆºèº«ã®ä¸Šã«ã‚¿ãƒ³ãƒãƒã‚’ã®ã›ã‚‹ä»•äº‹ã€ã‚’ã©ã†æ¸›ã‚‰ã™ã‹ã®ï¼’ç‚¹ã«ã¤ã„ã¦æ›¸ãã¾ã™ï¼Ž mixi ã®è§£æžåŸºç›¤ ã¾ãšã¯è§£æžç’°å¢ƒã«ã¤ã„ã¦ï¼Œç°¡å˜ã«ãŠè©±ã—ã¾ã™ï¼Ž2012-08 ç¾åœ¨ mixi ã§ã¯ï¼Œä¸»ãªè§£æžç”¨ã®ãƒ„ãƒ¼ãƒ«ã¨ã—ã¦ã¯ï¼ŒApache Hadoop, Hive ã‚’åˆ©ç”¨ã—ã¦ã„ã¾ã™ï¼Žã¾ãŸã‚ã‚ã›ã¦ï¼Œè‡ªåˆ†ãªã©ä¸€éƒ¨ã®äººã¯ï¼Œ
nubes 2014/09/03
hive

hadoop
ãƒªãƒ³ã‚¯
ã‚ªãƒ©ã‚¤ãƒªãƒ¼ã®Hadoopé–¢é€£æœ¬ãƒªãƒªãƒ¼ã‚¹äºˆå®š (2014/8æ™‚ç‚¹)
æœ€è¿‘ã€ç±³Oâ€™Reillyã‹ã‚‰Hadoopé–¢é€£æ›¸ç±ã®å‡ºç‰ˆäºˆå®šãŒç›®ç™½æŠ¼ã—ã§ã™ã€‚ãƒªãƒªãƒ¼ã‚¹äºˆå®šãŒã‚ã‹ã‚Šã«ãã„ã®ã§ã€è¡¨ã«ã¾ã¨ã‚ã¦ã¿ã¾ã—ãŸã€‚ â€»2015/1/8è£œè¶³: æœ€æ–°æƒ…å ±ã¯åˆ¥ãƒšãƒ¼ã‚¸ã«ã¾ã¨ã‚ã¾ã—ãŸ Hadoopã‚’ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ãƒ©ã‚¤ã‚ºã§åˆ©ç”¨ã—ãŸã„æ–¹ã¯å¿…èªã§ã™ãã€‚ï¼ˆã¡ã‚‡ã£ã¨ãƒªãƒªãƒ¼ã‚¹ãƒ©ãƒƒã‚·ãƒ¥ã§ç©ã‚“èªã«ãªã‚Šãã†ãªã¨ã“ã‚ãŒæ€–ã„ã§ã™ï¼‰ ã‚‚ã¡ã‚ã‚“ã€æ—¥æœ¬èªžã«ç¿»è¨³ã•ã‚Œã‚‹ã®ã‚‚æ¥½ã—ã¿ã§ã™ ðŸ™‚ NoTitleRelease dateå’Œæ›¸åå‡ºç‰ˆæ—¥é›‘æ„Ÿ
nubes 2014/09/02
hadoop

book
ãƒªãƒ³ã‚¯
Powering Interactive Data Analysis at Pinterest by Amazon Redshift
Amazon Redshift is a fast, petabyte-scale data warehouse that makes it easy to analyze your data for a fraction of the cost of traditional data warehouses. In this webinar, you will learn how to easily migrate your data from other data warehouses into Amazon Redshift, efficiently load your data with Amazon Redshift's massively parallel processing (MPP) capabilities, and automate data loading with
nubes 2014/07/28
Pinterest

redshift

hadoop

slide
ãƒªãƒ³ã‚¯
MapReduce Patterns, Algorithms, and Use Cases
In this article IÂ digestedÂ a number of MapReduce patterns and algorithms to give a systematic view of the different techniques that can be found on the web or scientific articles. Several practical case studies are also provided. All descriptions and code snippets use the standard Hadoopâ€™s MapReduce model with Mappers, Reduces, Combiners, Partitioners, and sorting. This framework is depicted in th
nubes 2014/07/27
hadoop

algorithm
ãƒªãƒ³ã‚¯
MPP on Hadoop, Redshift, BigQuery - Go ahead!
Twitterã§ã€Œæ—©ãä»Šæµè¡Œã®MPPã®å¤§ã¾ã‹ãªä½¿ã„æ–¹ã®é•ã„æ›¸ã‘ã‚ˆï¼ã€ã¨ã„ã†ãƒ—ãƒ¬ãƒƒã‚·ãƒ£ãƒ¼ãŒåŠç«¯ãªã„ã®ã§ã¦ãã¨ã†ã«æ›¸ãã¾ã™ï¼Žã“ã®è¨˜äº‹ã¯ä¿ºã®çµŒé¨“ã¨å‹‰å¼·ä¼šãªã©ã§ãƒ¦ãƒ¼ã‚¶ã‹ã‚‰èžã„ãŸè©±ã‚’ã‚‚ã¨ã«æ›¸ã„ã¦ã„ã‚‹ã®ã§ï¼Œã™ã¹ã¦ãŒä¿ºã®çµŒé¨“ã§ã¯ã‚ã‚Šã¾ã›ã‚“(ç‰¹ã«BigQuery)ï¼Žå„ç¤¾ã®SAã®äººã¨ã‹ã«èžã‘ã°ï¼Œã‚‚ã£ã¨è‰¯ã„ã‚¢ãƒ—ãƒãƒ¼ãƒã¨ã‹è©³ç´°ã‚’æ•™ãˆã¦ãã‚Œã‚‹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ï¼Ž ã‚ªãƒ³ãƒ—ãƒ¬ãƒŸã‚¹ã®å•†ç”¨MPPã¯ä½¿ã£ãŸã“ã¨ãªã„ã®ã§ãƒŽãƒ¼ã‚³ãƒ¡ãƒ³ãƒˆã§ã™ï¼Ž MPP on Hadoopã§PrestoãŒãƒ¡ã‚¤ãƒ³ãªã®ã¯ä»Šä¸€ç•ªä½¿ã£ã¦ã„ã‚‹ã‹ã‚‰ã§ï¼ŒImpalaãªã©ä»–ã®MPP on Hadoopçš„ãªã‚‚ã®ã‚‚ä¼¼ãŸã‚ˆã†ãªæ„Ÿã˜ã‹ãªã¨æ€ã£ã¦ã„ã¾ã™ï¼Ž ã‚‚ã¡ã‚ã‚“å®Ÿè£…ã®é•ã„ãªã©ãŒã‚ã‚‹ã®ã§ï¼Œãã®è¾ºã¯é©å®œè‡ªåˆ†ã§è£œé–“ã—ã¦ãã ã•ã„ï¼Ž å‰æ ã‚¢ãƒ—ãƒªã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚’é–‹ç™ºã—ã¦ã„ã¦ï¼Œãã®ãŸã‚ã®è§£æžåŸºç›¤ã‚’ä¸€ã‹ã‚‰ä½œã‚‹ï¼Ž ç°¡å˜ãªã¾ã¨ã‚ ãƒ‡ãƒ¼ã‚¿ã‚’è²¯ã‚ã‚‹æ‰€ãŒä½œã‚Œã‚‹ã®ã§ã‚ã‚Œã°ï¼Œãã“ã«ç›´æŽ¥ã‚¯ã‚¨ãƒªã‚’æŠ•ã’ã‚‰ã‚Œã‚‹Pre
nubes 2014/07/24
çŸ¥è¦‹ã‚ã‚ŠãŒãŸã„ã€‚

bigquery

presto

Redshift

hadoop
ãƒªãƒ³ã‚¯
2013å¹´ Hadoop é‹ç”¨ãƒã‚° @ Treasure Data | Post Moratorium
2013å¹´ Hadoop é‹ç”¨ãƒã‚° @ Treasure Data Hadoop Advent Calendar 2013ã€5æ—¥ç›®ã®ã‚¨ãƒ³ãƒˆãƒªã§ã™ã€‚2æ—¥é…ã‚Œã¦ã™ã„ã¾ã›ã‚“ï¼ ç´°ã‹ã„ã®ã¯ä¸Šã’ã‚Œã°ã‚ãƒªãŒç„¡ã„ã‚“ã ã‘ã©ã€èª°ã‹ã®å½¹ã«ç«‹ã¦ã°ã¨æ€ã„ Treasure Data ã§Hadoopã‚¯ãƒ©ã‚¹ã‚¿ã‚’é‹ç”¨ã—ã¦ã¿ãŸãƒã‚° 2013å¹´åº¦ç‰ˆã‚’å…¬é–‹ã—ã¦ã¿ã¾ã™ã€‚ å¯¾è±¡ãƒãƒ¼ã‚¸ãƒ§ãƒ³2013å¹´ã«ã¯ã€ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³ã®ã‚¢ãƒƒãƒ—ã‚°ãƒ¬ãƒ¼ãƒ‰ã‚’5å›žã»ã©è¡Œã„ã¾ã—ãŸã€‚ CDH3u0CDH3u1CDH4.1.2CDH4.2.0CDH4.2.1ãƒ¡ã‚¸ãƒ£ãƒ¼ã‚¢ãƒƒãƒ—ã‚°ãƒ¬ãƒ¼ãƒ‰ã¯è¤‡æ•°ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚’åŒæ™‚ã«èµ°ã‚‰ã›ã¦å•é¡ŒãŒç„¡ã„ã“ã¨ã‚’ç¢ºèªå¾Œã€åˆ‡ã‚Šæ›¿ãˆã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ã—ã‹ã—CDH3ã£ã¦æ—¢ã«End of Maintenanceãªã‚“ã§ã™ãã€çŸ¥ã‚‰ãªã‹ã£ãŸã€‚ é‹ç”¨ä½“åˆ¶ç´„3åã€å¹´æœ«ã«ã¯ç´„8åç¨‹ã€‚ç´„100ç¤¾ã«å‘ã‘ã¦ã‚µãƒ¼ãƒ“ã‚¹é‹ç”¨ã‚’ã—ã¦ã„ã¦ã€æ•°åˆ†ã‚¸ãƒ§ãƒ–ãŒåˆºã•ã‚‹ã ã‘ã§ã‚‚ã‚µãƒãƒ¼ãƒˆãƒ
nubes 2014/07/08
hadoop
ãƒªãƒ³ã‚¯
Hadoop 2 @Twitter, Elephant Scale. Presented at
1) Twitter migrated their Hadoop infrastructure from Hadoop 1 to Hadoop 2 to address scalability challenges and improve efficiency from tens of thousands of servers and hundreds of petabytes of data. 2) The migration involved extensive testing of Hadoop 2 configurations and dependent components before rolling out to production clusters supporting various use cases such as analytics, personalizatio
nubes 2014/07/08
twitter

hadoop

slide
ãƒªãƒ³ã‚¯
1 2 æ¬¡ã®ãƒšãƒ¼ã‚¸