Apache Nutchâ„¢
Nutch is a highly extensible, highly scalable, matured, production-ready Web crawler which enables fine grained configuration and accomodates a wide variety of data acquisition tasks. Scalable Relying on Apache Hadoopâ„¢ data structures, Nutch is great for batch processing large data volumes but can also be tailored to smaller jobs. Pluggable Out of the box Nutch offer powerful plugins i.e., parsing
Facebook open sources Corona — a better way to do webscale Hadoop – Old GigaOm
Facebook (s fb) is at it again, building more software to make Hadoop a better way to do big data at web scale. Its latest creation, which the company has also open sourced, is called Corona and aims to make Hadoop more efficient, more scalable and more available by re-inventing how jobs are scheduled. As with most of its changes to Hadoop over the years — including the recently unveiled AvatarNod
データ解æžç”¨ãƒ¯ãƒ¼ã‚¯ãƒ•ãƒãƒ¼ãƒ•ãƒ¬ãƒ¼ãƒ ワーク Honey ã®ç´¹ä»‹ - mixi engineer blog
最近，もã£ã±ã‚‰ä¸ŠåŽŸã²ã‚ã¿ã•ã‚“ã®æ›²ã‚’エンドレスã«è´ã„ã¦ç™’ã—ã‚’å¾—ã¦ã„ã¾ã™ï¼Žã‚‚ã¡ã‚んピクルス作りも最高ã§ã™ï¼Žã¿ãªã•ã‚“ã¯ä½•ã§ç™’ã—ã‚’å¾—ã¦ã„ã¾ã™ã§ã—ょã†ã‹ï¼Žã“ã‚“ã«ã¡ã¯ï¼ŒæŠ€è¡“部ã®çŸ³å·æœ‰ã§ã™ï¼Ž 以å‰ï¼Œã€Œmixi ã®è§£æžåŸºç›¤ã¨Apache Hive ã§ã® JSON パーサã®æ´»ç”¨ã®ç´¹ä»‹ã€ã§ mixi ã«ãŠã‘ã‚‹ Hadoop/ Hive ã®æ´»ç”¨ã®ä»•æ–¹ã«ã¤ã„ã¦è¨˜äº‹ã‚’書ã‹ã›ã¦ã„ãŸã ãã¾ã—ãŸï¼Žä»Šå›žã®è¨˜äº‹ã§ã¯ï¼Œã¡ã‚‰ã£ã¨è§¦ã‚Œã¦ã„㟠Hive ãªã©ã§å®šæœŸå®Ÿè¡Œã™ã‚‹å¿…è¦ã®ã‚る処ç†ã‚’ワークフãƒãƒ¼ã¨ã—ã¦å®šç¾©ã™ã‚‹ãƒ•ãƒ¬ãƒ¼ãƒ ワークã«ã¤ã„ã¦æ›¸ãã¾ã™ï¼Ž æœ¬æ–‡ç« ã®æ§‹æˆ ã¾ãšæœ€åˆã«ï¼Œä»Šå›žã”紹介ã™ã‚‹ãƒ‡ãƒ¼ã‚¿è§£æžç”¨ãƒ¯ãƒ¼ã‚¯ãƒ•ãƒãƒ¼ãƒ•ãƒ¬ãƒ¼ãƒ ワーク Honey ã¨ã¯ä½•ã‹ï¼Œãªãœä½œã£ãŸã®ã‹ã‚’説明ã—ã¾ã™ï¼Žã¤ãŽã«ï¼Œã©ã®ã‚ˆã†ãªæ§‹æˆã‚„機能ãŒã‚ã‚‹ã®ã‹ã‚’ç°¡å˜ã«èª¬æ˜Žã—ã¾ã™ï¼Žãã‚Œã‹ã‚‰å…·ä½“çš„ãªãƒ‡ãƒ¼ã‚¿è§£æžå‡¦ç†ã‚’記述ã™ã‚‹æ–¹æ³•ã«ã¤ã„ã¦èª¬æ˜Žã—ã¾ã™ï¼Žãã®ä¸ã§ï¼Œå®šåž‹çš„ãªå‡¦ç†ã‚’ YAML ã¨ã—
scribedã®ã‚»ãƒƒãƒˆã‚¢ãƒƒãƒ—æ‰‹é †ã²ã¨é€šã‚Š(hdfs書ãè¾¼ã¿æœ‰åŠ¹ç‰ˆ) - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
ãƒã‚°ã®åŽé›†ã‚’scribeã§ã‚„ã‚ŠãŸã„ãœï¼ ã¨æ€ã£ãŸãŒREADMEã«æ›¸ã„ã¦ã‚る通りã«ã‚„ã‚ã†ã¨ã—ã¦ã‚‚ã†ã¾ãã„ã‹ãšä¸ƒè»¢å…«å€’ã—ã€ã—ã‹ã‚‚ãã®ä¸Šhdfsã«æ›¸ã込もã†ã¨ã—ãŸã‚‰Hadoopã¾ã‚ã‚Šã§æ›´ã«è‹¦æ‚¶ã®æ—¥ã€…ã‚’é€ã‚‹ã¨ã„ã†è‹¦é›£ã®é“ã®ã‚Šã‚’よã†ã‚„ã完走ã—ãŸã®ã§ã€ãã‚Œã«ã¤ã„ã¦æ›¸ã„ã¦ã¿ã‚‹ã€‚ æ£ç›´ã«è¨€ã£ã¦ã‹ãªã‚Šãƒ†ã‚トーã«ã‚„ã£ã¤ã‘ãŸãŒã€å‹•ããƒã‚¤ãƒŠãƒªãŒã§ãã¦ã„ã‚‹ã‹ã‚‰å¤šåˆ†å¤§ä¸ˆå¤«ã ã‚ã†ã€‚ç´°ã‹ã„å•é¡Œã«ã¤ã„ã¦ã¯ã€çŸ¥ã‚‰ã‚“。 (1/24追記 CentOS 5.5 ã§ã‚‚ビルドã§ããŸã®ã§ä½•ç®‡æ‰€ã‹ã«è¿½è¨˜ã—ã¾ã—ãŸ) ãªãŠä»Šå›žã¯ Fedora 13 ã§å®Ÿè¡Œã—ã¾ã—ãŸã€‚(01/24追記)CentOS 5 ã ã¨boostãŒå¤ã„ã®ã§ã€ãã“を自分ã§ã©ã†ã«ã‹ã™ã‚‹å¿…è¦ãŒã‚ã‚‹(ç´ ã®CentOS 5.5ã ã¨scribeã®ãƒ“ルド時ã«boostã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ 1.36 以é™ã‚’è¦æ±‚ã•ã‚Œã¦å¤±æ•—ã™ã‚‹)。Ubuntu ã‚„ Debian ã ã¨å‰²ã¨ã„ã‘ãã†ãªè©±ã¯å¤šã„ãŒã€è©¦ã—ã¦ãª
NTTデータã®Hadoopå ±å‘Šæ›¸ãŒã™ã”ã‹ã£ãŸ - 科å¦ã¨éžç§‘å¦ã®è¿·å®®
æ¥ç•Œãƒˆãƒƒãƒ— ã®ã‚¨ãƒ³ã‚¿ãƒ¼ãƒ—ライズ Hadoop ä¼æ¥ Cloudera ã«å…¥ç¤¾ã—ã¾ã—㟠http://www.cloudera.co.jp/ 今年ã®6月ã«ã€ã€Œå¹³æˆï¼’１年度 産å¦é€£æºã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢å·¥å¦å®Ÿè·µäº‹æ¥å ±å‘Šæ›¸ã€ã¨ã„ã†ãƒ‰ã‚ュメント群ãŒçµŒç”£çœã‹ã‚‰å…¬è¡¨ã•ã‚Œã¾ã—ãŸã€‚ ãã®ã†ã¡ã®ä¸€ã¤ã«ã€NTTデータã«å§”託ã•ã‚ŒãŸHadoopã«é–¢ã™ã‚‹å®Ÿè¨¼å®Ÿé¨“ã®å ±å‘Šæ›¸ãŒã‚ã‚Šã¾ã—ãŸã®ã§ã€ä»Šæ›´ãªãŒã‚‰èªã‚“ã§ã¿ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ Hadoop界隈ã®äººã¯ã‚‚ã†ã¿ã‚“ãªã¨ã£ãã«èªã‚“ã§ã‚‹ã®ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã‘ã©ã€‚ http://www.meti.go.jp/policy/mono_info_service/joho/downloadfiles/2010software_research/clou_dist_software.pdf ã€Œé«˜ä¿¡é ¼ã‚¯ãƒ©ã‚¦ãƒ‰å®Ÿç¾ç”¨ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢é–‹ç™ºï¼ˆåˆ†æ•£åˆ¶å¾¡å‡¦ç†æŠ€è¡“ç‰ã«ä¿‚ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³ã‚¿ãƒ¼é«˜ä¿¡é ¼åŒ–ã«å‘ã‘ãŸå®Ÿè¨¼äº‹æ¥ï¼‰ã€ã¨ã„ã†
Hadoopを使ã„ã“ãªã™(3)
ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10月1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ブãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚LINEヤフー Tech Blog ã“ã‚“ã«ã¡ã¯ã€R&D統括本部ã®å‰ç”°ã§ã™ã€‚ å‰å›žã¨å‰ã€…回ã®è¨˜äº‹ã§ã¯ã€Hadoopã®ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºãƒã‚¤ãƒ³ãƒˆã‚’解説ã—ã¾ã—ãŸã€‚ å˜ç´”ãªå•é¡Œã«å¯¾ã—ã¦ã¯ã€map関数ã¨reduce関数ã®çµ„ã¿åˆã‚ã›ã ã‘ã§ã‚‚ã€å‡¦ç†ã‚’è¡Œã†ã“ã¨ãŒã§ãã¾ã™ãŒã€ã‚る程度複雑ãªå•é¡Œã«å¯¾ã—ã¦ã¯ã€ä»Šã¾ã§ã«è§£èª¬ã—ãŸã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºãƒã‚¤ãƒ³ãƒˆã®æ´»ç”¨ãŒé‡è¦ã«ãªã‚‹ã¨ã„ã†è©±ã§ã—ãŸã€‚ 今回ã¯å®Ÿéš›ã®ã‚µãƒ¼ãƒ“スã®äº‹ä¾‹ã‚’紹介ã—ã€ã‚«ã‚¹ã‚¿ãƒžã‚¤ã‚ºãƒã‚¤ãƒ³ãƒˆãŒã©ã†æ´»ç”¨ã•ã‚Œã¦ã„ã‚‹ã‹ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ ABYSS ABYSSã®äº‹ä¾‹ã‚’紹介ã—ã¾ã—ょã†ã€‚ ABYSSã¨ã¯ã€æ¤œç´¢ã‚µãƒ¼ãƒ“スを簡å˜ã«æ§‹ç¯‰ã§ãる社内プラットフォームã®ã“ã¨ã§ã€è©³ã—ãã¯ä»¥ä¸‹ã®TechBlog記事ã§ç´¹ä»‹ã—ã¦ã„ã¾ã™ã€‚先日無事ã«ç¤¾å†…リリースã•ã‚Œã¾ã—ãŸã€‚ 新検索
マーケティングã®ãŸã‚ã®Hadoop利用
This document discusses Hadoop and its use on Amazon Web Services. It describes how Hadoop can be used to process large amounts of data in parallel across clusters of computers. Specifically, it outlines how to run Hadoop jobs on an Elastic Compute Cloud (EC2) cluster configured with Hadoop and store data in Amazon Simple Storage Service (S3). The document also provides examples of using Hadoop St
Scala on Hadoop: Hadoop Conference - stanaka's blog
先日ã€Hadoop Conferenceã§Scala on Hadoopã¨ã„ã†ã‚¿ã‚¤ãƒˆãƒ«ã§ç™ºè¡¨ã—ã¦ãã¾ã—ãŸã€‚スライドを以下ã«ç½®ã„ã¦ãŠãã¾ã™ã€‚ Scala on HadoopView more presentations from Shinji Tanaka. ダイジェストã¨ã—ã¦ã€Scalaã‚’Hadoopã§å‹•ã‹ã™ãŸã‚ã®æ–¹æ³•ã‚’書ã„ã¦ãŠãã¾ã™ã€‚ ã¾ãšã€Hadoop上ã§Scalaを実行ã•ã›ã‚‹ãŸã‚ã«ã¯ã€Javaã¨Scalaを接続ã™ã‚‹ãƒ©ã‚¤ãƒ–ラリãŒå¿…è¦ã¨ãªã‚Šã¾ã™ã€‚ã“ã“ã§ã¯ã€SHadoop( http://code.google.com/p/jweslley/source/browse/#svn/trunk/scala/shadoop )を使用ã—ã¾ã™ã€‚SHadoopã¯ã€åž‹å¤‰æ›ã‚’è¡Œã†ã‚·ãƒ³ãƒ—ルãªãƒ©ã‚¤ãƒ–ラリã§ã™ã€‚ よãã‚ã‚‹WordCountã®ã‚µãƒ³ãƒ—ルã€WordCount.scala (http://blog.jo
Installation notes ‎(Hadoop)‎
Hadoop WebDAV server is based the earlier work and is released under an Apache License, Version 2.0 Notice The WebDAV server is built on the top of Hadoop distribution, so make sure you have sources from hadoop-0.16.* or hadoop-0.17.* unpacked in some location. WebDAV server may be deployed on any node of Hadoop subnet, inside or outside of cluster. It uses some libraries from Hadoop's lib/ dir, s
The Hadoop Distributed File System: Architecture and Design
Hardware Failure Hardware failure is the norm rather than the exception. An HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that each component has a non-trivial probability of failure means that some component of HDFS is always non-functional. Therefore, detection of faults
1
ランã‚ング
ランã‚ング
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
ãƒãƒ¥ãƒ¼ãƒˆãƒªã‚¢ãƒ« - Azure HDInsight 㧠Apache HBase を使用ã™ã‚‹
Riak at shareaholic
http://jonhnnyweslley.net/blog/shadoop/
Scala on Hadoop
GitHub - klbostee/dumbo: Python module that allows one to easily write and run Hadoop programs.
{{#tags}}- {{label}}
{{/tags}}