Architecting a Next Generation Data Platform – Strata Singapore 2017
Architecting a Next Generation Data Platform – Strata Singapore 2017 This document discusses the high-level architecture for a data platform to support a customer 360 view using data from connected vehicles (taxis). The architecture includes data sources, streaming data ingestion using Kafka, schema validation, stream processing for transformations and routing, and storage for analytics, search an
èªè¨¼/èªå¯ãŒå®Ÿç¾ã™ã‚‹å®‰å…¨ã§é«˜é€Ÿåˆ†æžå¯èƒ½ãªåˆ†æžå‡¦ç†åŸºç›¤
分æžå‡¦ç†åŸºç›¤ã«ãŠã„ã¦ã€ãƒ‡ãƒ¼ã‚¿ã®æ°‘主化・マルãƒãƒ†ãƒŠãƒ³ãƒˆåŒ–を進ã‚ã‚‹ãŸã‚ã«ã¯ã„ãã¤ã‹ã®ãƒãƒ¼ãƒ‰ãƒ«ãŒã‚ã‚Šã¾ã™ã€‚ãã®ãƒãƒ¼ãƒ‰ãƒ«ã®ä¸€ã¤ã«ã€ãƒ‡ãƒ¼ã‚¿ãƒªã‚¹ã‚¯ã‚’低減ã™ã‚‹ãŸã‚ã®èªè¨¼ãƒ»èªå¯ã®å®Ÿç¾ãŒæŒ™ã’られã¾ã™ã€‚ 本講演ã§ã¯ã€ãƒ‰ãƒ¯ãƒ³ã‚´ãŒåˆ†æžå‡¦ç†åŸºç›¤ã®èªè¨¼ãƒ»èªå¯æ©Ÿæ§‹ã‚’一新ã—ãŸå†…容ã¨ãƒã‚¤ãƒ³ãƒˆã€ã¾ãŸãれらã®èªè¨¼ãƒ»èªå¯æ©Ÿæ§‹ã‚’ベースã¨ã—ã€åˆ†æžå‡¦ç†ã‚’æ•°å€ã‹ã‚‰æ•°åå€ã«é«˜é€ŸåŒ–ã—ãŸäº‹ä¾‹ã‚’紹介致ã—ã¾ã™ã€‚Read less
Apache Hadoopã®HDFS Erasure Codingã®ç´¹ä»‹
原文: https://blog.cloudera.com/blog/2015/09/introduction-to-hdfs-erasure-coding-in-apache-hadoop/ 訳注：2015/9/23ã«å…¬é–‹ã•ã‚ŒãŸãƒ–ãƒã‚°ã®ç¿»è¨³ã§ã™ã€‚ブãƒã‚°å…¬é–‹å½“時ã¨ã¯ç•°ãªã‚‹ç‚¹ã‚‚ã‚ã‚Šã¾ã™ã®ã§ã€æœ€æ–°æƒ…å ±ã¯HDFS-7285ã€HDFS-8031ãªã©ã‚’ã”確èªãã ã•ã„。 HDFSã®æ–°æ©Ÿèƒ½ã§ã‚るイレイジャーコーディング(Erasure Coding)ã¯ã€ãƒ¬ãƒ—リケーション（複製）ã¨æ¯”較ã—ã¦ã€åŒç‰ã®æ°¸ç¶šæ€§ã®ä¿è¨¼ã‚’ç¶æŒã—ãªãŒã‚‰ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã®ã‚ªãƒ¼ãƒãƒ¼ãƒ˜ãƒƒãƒ‰ã‚’ç´„50％削減ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ ã“ã®ãƒ–ãƒã‚°ã§ã¯ã€Erasure CodingãŒã©ã®ã‚ˆã†ã«å‹•ä½œã™ã‚‹ã‹ã‚’説明ã—ã¾ã™ã€‚ デフォルトã§HDFSã¯å„ブãƒãƒƒã‚¯ã‚’3回複製ã—ã¾ã™ã€‚ レプリケーションã¯ã€ã»ã¨ã‚“ã©ã®éšœå®³ã‚·ãƒŠãƒªã‚ªã‚’回é¿ã™ã‚‹ãŸã‚ã®ã€ã‚·ãƒ³ãƒ—ルã§å …牢ãªå†—長性ã®å½¢å¼ã‚’æ
[HADOOP-1687] Name-node memory size estimates and optimization proposal. - ASF JIRA
I've done some estimates on how much space our data structures take on the name-node per block, file and directory. Brief overview of the data structures: Directory tree (FSDirectory) is built of inodes. Each INode points either to an array of blocks if it corresponds to a file or to a TreeMap<String, INode> of children INodes if it is a directory. [Note: this estimates were made before Dhruba rep
Yahoo! JAPAN MeetUp #8 （インフラ技術カンファレンス）セッション②
1. Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. 2017å¹´1月30æ—¥ 1 ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ システム統括本部 サイトオペレーション本部 インフラ技術3部 DCNW1 深澤 é–‹ Network for the Large-scale Hadoop Cluster at Yahoo! JAPAN 2. Copyright © 2017 Yahoo Japan Corporation. All Rights Reserved. 自己紹介 2 ïµ åå‰ ïµ æ·±æ¾¤ é–‹ (ãµã‹ã–ã‚ ã‹ã„) ïµ æ‰€å±ž ïµ ãƒ‡ãƒ¼ã‚¿&サイエンスソリューション統括本部 データプラットフォーム本部 開発2部 ïµ ã‚·ã‚¹ãƒ†ãƒ çµ±æ‹¬æœ¬éƒ¨ サイトオペレーション本部 インフラ技術3部 (兼務) ïµ æ¥å‹™ ïµ å…¨ç¤¾Hadoopã®è¨è¨ˆãƒ»æ§‹ç¯‰ãƒ»é‹ç”¨ ïµ ãƒ‡ãƒ¼ã‚¿ã‚»ãƒ³
基幹æ¥å‹™ã‚‚Hadoop(EMR)ã§!!ã®ãã®å¾Œ
Impala Meetup 2014/10/31 @Tokyo 講演資料 ã€æ³¨æ„äº‹é …ã€‘ 本資料ã§ç´¹ä»‹ã—ã¦ã„る検証çµæžœã¯2014年当時ã®ã‚‚ã®ã§ã™ã€‚当該ソフトウェアã¯æˆé•·ã‚„改善ãŒæ—©ãã€ç¾æ™‚点ã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã¯å¤§ããç•°ãªã‚‹æ©Ÿèƒ½ã‚„性能ã¨ãªã£ã¦ã„ã¾ã™ã€‚ SQL on Hadoopã®æœ€æ–°æƒ…å ±ã«åŸºã¥ãサービスやシステムインテグレーションã«ã”興味をãŠæŒã¡ã®æ–¹ã¯ã€NTTデータ 基盤システム事æ¥æœ¬éƒ¨ OSSプãƒãƒ•ã‚§ãƒƒã‚·ãƒ§ãƒŠãƒ«ã‚µãƒ¼ãƒ“ス（電åメール： hadoop [AT] kits.nttdata.co.jp） ã«ã”相談ãã ã•ã„。
基幹æ¥å‹™ã‚‚Hadoopã§!! -ãƒãƒ¼ã‚½ãƒ³ã«ãŠã‘る店舗発注æ¥å‹™ã¸ã®ï¿½Hadoop + Hive導入㨠ãã®å–り組ã¿ã«ã¤ã„ã¦-
基幹æ¥å‹™ã‚‚Hadoopã§!! -ãƒãƒ¼ã‚½ãƒ³ã«ãŠã‘る店舗発注æ¥å‹™ã¸ã®ï¿½Hadoop + Hive導入㨠ãã®å–り組ã¿ã«ã¤ã„ã¦-
Cloud Native Hadoop #cwt2016
クラウド時代ã®ä»Šã€"Cloud Native" ã‚„ "Microservices" ãªã©ã®ãƒ¯ãƒ¼ãƒ‰ã‚’よã見ã‹ã‘るよã†ã«ãªã‚Šã¾ã—ãŸã€‚ã“れらã¯åŸºæœ¬çš„ã«ã€Œã‚¯ãƒ©ã‚¦ãƒ‰ä¸Šã§ã‚¢ãƒ—リケーションを開発ã™ã‚‹ãŸã‚ã®ãƒ™ã‚¹ãƒˆãƒ—ラクティスã€ã‚’æ„味ã™ã‚‹è¨€è‘‰ã§ã™ã€‚一方ã€Hadoop ãŒã‚¯ãƒ©ã‚¦ãƒ‰ã®æ–‡è„ˆã§èªžã‚‰ã‚Œã‚‹ã“ã¨ã¯ã¾ã ã¾ã å°‘ãªã„状æ³ã§ã™ã€‚ãã‚Œã¯ã‚¢ãƒ—リケーションã¨æ¯”較ã—ã¦ã€ã‚ˆã‚Š H/W ã‚„ OS ã«è¿‘ã„レイヤー㮠Hadoop をクラウド上ã§ç¨¼åƒã•ã›ã‚‹ãŸã‚ã«ã¯ä»Šã¾ã§ã¨ã¯é•ã†æ ¹æœ¬çš„ãªã‚¢ãƒ¼ã‚テクãƒãƒ£ãƒ¼ã®å¤‰æ›´ã‚’ä¼´ã†ã‚±ãƒ¼ã‚¹ãŒã‚ã‚‹ãŸã‚ã§ã™ã€‚本セッションã§ã¯ "Cloud Native" 㪠Hadoop ã¨ã¯ä½•ã‹ã€ã¾ãŸãã®ãƒ™ã‚¹ãƒˆãƒ—ラクティスをデモを交ãˆã¦ç´¹ä»‹ã—ã¾ã™ã€‚Read less
Hadoop Summit Tokyo 2016ã«è¡Œã£ã¦ã㟠- wyukawa's diary
http://hadoopsummit.org/tokyo ãƒã‚±ãƒƒãƒˆä»£ãŒç´„4万円ã§é«˜ã„ã¨å™‚ã«ãªã£ãŸHadoop Summit Tokyo 2016ã«è¡Œã£ã¦ãã¾ã—ãŸã€‚ ãŸã 海外ã§ã¯ã“ã®ãらã„ã®å€¤æ®µã¯æ™®é€šã‚‰ã—ãã€ã‚€ã—ã‚日本ãŒç•°å¸¸ã«å®‰ã™ãŽã‚‹ã¨ã„ã†ã€‚ ãã®ã—ã‚寄ã›ãŒã‚¤ãƒ™ãƒ³ãƒˆé‹å–¶è€…ã«ã„ã£ã¦ã—ã¾ã£ã¦ã„ã‚‹ã®ãŒç¾çŠ¶ãªã®ã§ã€ä¸–界基準を知る良ã„機会ã ã£ãŸã‹ã‚‚。 僕ã¯2日間ã¨ã‚‚最åˆã‹ã‚‰å‚åŠ ã—ã¾ã—ãŸã€‚ 基調講演を除ãã¨èžã„ãŸã‚»ãƒƒã‚·ãƒ§ãƒ³ã¯ä¸‹è¨˜ã®é€šã‚Šã€‚ 10/26 Real-time Analytics in Financial: Use Case, Architecture and Challenges Path to 400M Members: LinkedIn’s Data Powered Journey Hadoop 3.0 in a Nutshell Apache Eagle - Monitor Hadoo
Using Apache Parquet Data Files with CDH | 6.3.x | Cloudera Documentation
Apache Parquet is a columnar storage format available to any component in the Hadoop ecosystem, regardless of the data processing framework, data model, or programming language. The Parquet file format incorporates several features that support data warehouse-style operations: Columnar storage layout - A query can examine and perform calculations on all values for a column while reading only a sma
第6回　Hiveã§ãƒœãƒˆãƒ«ãƒãƒƒã‚¯ã¨ãªã£ã¦ããŸãƒ¡ã‚¿ãƒ‡ãƒ¼ã‚¿ã€HBaseを使ã£ã¦ãƒ¬ã‚¤ãƒ†ãƒ³ã‚·ã®æ”¹å–„ã«æŒ‘ã‚€ ―「Hadoop Summit 2016 San Joseã€ã‹ã‚‰ | gihyo.jp
IT Cutting Edge ─世界を変ãˆã‚‹ãƒ†ã‚¯ãƒŽãƒã‚¸ã®æœ€å‰ç·š 第6回Hiveã§ãƒœãƒˆãƒ«ãƒãƒƒã‚¯ã¨ãªã£ã¦ããŸãƒ¡ã‚¿ãƒ‡ãƒ¼ã‚¿ã€HBaseを使ã£ã¦ãƒ¬ã‚¤ãƒ†ãƒ³ã‚·ã®æ”¹å–„ã«æŒ‘ã‚€ ―「Hadoop Summit 2016 San Joseã€ã‹ã‚‰ 2016å¹´6/28～6/30（米国時間）ã®3日間ã«æ¸¡ã£ã¦ç±³ã‚µãƒ³ãƒŽã‚¼ã§é–‹å‚¬ã•ã‚ŒãŸã€ŒHadoop Summit 2016 San Joseã€ï¼ˆâ 主催: Hortornworksï¼Yahoo!）ã§ã¯ã€2016å¹´ã®Hadoopトレンドを紹介ã™ã‚‹æ•°å¤šãã®æŠ€è¡“ï¼äº‹ä¾‹ã‚»ãƒƒã‚·ãƒ§ãƒ³ãŒè¡Œã‚ã‚Œã¾ã—ãŸã€‚本稿ã§ã¯ãã®ã²ã¨ã¤ã€Hortonwokrsã«ã‚ˆã‚‹ã‚»ãƒƒã‚·ãƒ§ãƒ³ã€Œhive HBase Metastore - Improving Hive with a Big Data Metadata Storageã€ã‚’ã‚‚ã¨ã«ã€Hiveã®ãƒ¡ã‚¿ãƒ‡ãƒ¼ã‚¿ã‚’HBaseã§ã‚¹ãƒˆã‚¢ã™ã‚‹ã“ã¨ã§Hiveã®ä½Žãƒ¬ã‚¤ãƒ†ãƒ³ã‚·åŒ–を図る技
Using CombineInputFormat to Combat Hadoop’s Small Files Problem | Amazon Web Services
AWS Big Data Blog Using CombineInputFormat to Combat Hadoop’s Small Files Problem James Norvell is a Big Data Cloud Support Engineer for AWS Many Amazon EMR customers have architectures that track events and streams and store data in S3. This frequently leads to many small files. It’s now well known that Hadoop doesn’t deal well with small files. This issue can be amplified when migrating from Had
Tez 0.5.2ã‚’Hadoop 2.6.0ã§å‹•ã‹ã—ã¦ã¿ã‚‹ - Qiita
diff --git a/pom.xml b/pom.xml index dbf5923..28cd805 100644 --- a/pom.xml +++ b/pom.xml @@ -37,7 +37,7 @@ <properties> <maven.test.redirectTestOutputToFile>true</maven.test.redirectTestOutputToFile> <clover.license>${user.home}/clover.license</clover.license> - <hadoop.version>2.4.0</hadoop.version> + <hadoop.version>2.6.0</hadoop.version> <jetty.version>7.6.10.v20130312</jetty.version> <distMgmt
ランã‚ング
障害
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Tez on CDH 5.2.5 -- CDH-4212
Slider getting InvalidProtocolBufferException error
Compression vs. Performance · Apache Big_Data Notes: Hadoop, Spark, Flink, etc.
Cloudera Hybrid Data Community
https://mail-archives.apache.org/mod_mbox/hive-user/201605.mbox/%
Introduction to Apache Apex
Apache Hadoop 2.7.1 – Hadoop: Fair Scheduler
{{#tags}}- {{label}}
{{/tags}}