Apache Sentryã§å§‹ã‚るアクセス制御 (Hiveç·¨)
Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.Twitter / Photos Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers. Twitter / Photos ã“ã®æŠ•ç¨¿ã¯Spark, SQL on Hadoop etc. Advent Calendar 2014ã®5日目ã®è¨˜äº‹ã§ã™ã€‚ Apache Sentry(以下Sentry)を使ãˆã°ã€Hiveã‚„Impalaã€Cloudera Search(SolrCloud
[db tech showcase Tokyo 2016] B15: サイãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ アドテクスタジオã®æ¬¡ä¸–代データ分æžåŸºç›¤ç´¹ä»‹ by æ ªå¼ä¼šç¤¾ã‚µã‚¤ãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ æˆå°¾ 文秀
[db tech showcase Tokyo 2016] B15: サイãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ アドテクスタジオã®æ¬¡ä¸–代データ分æžåŸºç›¤ç´¹ä»‹ by æ ªå¼ä¼šç¤¾ã‚µã‚¤ãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ æˆå°¾ 文秀
![[db tech showcase Tokyo 2016] B15: サイãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ アドテクスタジオã®æ¬¡ä¸–代データ分æžåŸºç›¤ç´¹ä»‹ by æ ªå¼ä¼šç¤¾ã‚µã‚¤ãƒãƒ¼ã‚¨ãƒ¼ã‚¸ã‚§ãƒ³ãƒˆ æˆå°¾ 文秀](https://cdn-ak-scissors.b.st-hatena.com/image/square/6b8984e6962fe144e222d0696bde3fc99da1a0f2/height=288;version=1;width=512/https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2F6i9kohhirrcoablwy2vg-signature-de1723da869232faf642797bf4411177191c3ef937923c1c3c4b18848e6dd30c-poli-160721053413-thumbnail.jpg%3Fwidth%3D640%26height%3D640%26fit%3Dbounds)
Tuning YARN | 6.3.x | Cloudera Documentation
This overview provides an abstract description of a YARN cluster and the goals of YARN tuning. A YARN cluster is composed of host machines. Hosts provide memory and CPU resources. A vcore, or virtual core, is a usage share of a host CPU. Tuning YARN consists primarily of optimally defining containers on your worker hosts. You can think of a container as a rectangular graph consisting of memory and
Cloudera Blog
We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices, part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scalable AI inference. Background The generative AI landscape is evolving […] Read blog post
Spark shark
6. ãŠã¾ãˆã れよ？ Tsuyoshi Ozawa @oza_x86 OSS developer Apache Hadoop ã®é–‹ç™ºã‚’ã—ã¦ã¾ã™ github : oza â†ã®22ç« ã‚’æ›¸ã„ãŸã‚ˆï¼ Tuesday, October 22, 13
2013å¹´ Hadoop é‹ç”¨ãƒã‚° @ Treasure Data | Post Moratorium
2013å¹´ Hadoop é‹ç”¨ãƒã‚° @ Treasure Data Hadoop Advent Calendar 2013ã€5日目ã®ã‚¨ãƒ³ãƒˆãƒªã§ã™ã€‚2æ—¥é…ã‚Œã¦ã™ã„ã¾ã›ã‚“ï¼ ç´°ã‹ã„ã®ã¯ä¸Šã’ã‚Œã°ã‚リãŒç„¡ã„ã‚“ã ã‘ã©ã€èª°ã‹ã®å½¹ã«ç«‹ã¦ã°ã¨æ€ã„ Treasure Data ã§Hadoopクラスタをé‹ç”¨ã—ã¦ã¿ãŸãƒã‚° 2013年度版を公開ã—ã¦ã¿ã¾ã™ã€‚ 対象ãƒãƒ¼ã‚¸ãƒ§ãƒ³2013å¹´ã«ã¯ã€ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ューションã®ã‚¢ãƒƒãƒ—グレードを5回ã»ã©è¡Œã„ã¾ã—ãŸã€‚ CDH3u0CDH3u1CDH4.1.2CDH4.2.0CDH4.2.1メジャーアップグレードã¯è¤‡æ•°ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚’åŒæ™‚ã«èµ°ã‚‰ã›ã¦å•é¡ŒãŒç„¡ã„ã“ã¨ã‚’確èªå¾Œã€åˆ‡ã‚Šæ›¿ãˆã‚’è¡Œã£ã¦ã„ã¾ã™ã€‚ã—ã‹ã—CDH3ã£ã¦æ—¢ã«End of Maintenanceãªã‚“ã§ã™ãã€çŸ¥ã‚‰ãªã‹ã£ãŸã€‚ é‹ç”¨ä½“制約3åã€å¹´æœ«ã«ã¯ç´„8å程。約100社ã«å‘ã‘ã¦ã‚µãƒ¼ãƒ“スé‹ç”¨ã‚’ã—ã¦ã„ã¦ã€æ•°åˆ†ã‚¸ãƒ§ãƒ–ãŒåˆºã•ã‚‹ã ã‘ã§ã‚‚サãƒãƒ¼ãƒˆãƒ
20130626 kawasaki.rb NKT77
1. Hadoop with Ruby - 僕ãŒPythonã‚’é¸ã‚“ã ç†ç”± 実際㯠「Kawasaki.rbã®ãã£ã‹ã‘ã¨ãªã£ãŸ “Rubyã®ãƒãƒƒã‚·ãƒ¥ãŒé…ã‹ã£ãŸäº‹ä¾‹â€ã®ç´¹ä»‹ã€ - NKT77 2. <B,1> <C,1> <E,1> MapReduce with Hadoop • WordCountã®ä¾‹ mapper1 mapper2 mapperN ... <å˜èªž,é »åº¦> <A,1> <C,1> <D,1> <C,1> <D,1> <E,1> <A,1> <B,1> <C,1> <C,1> <C,1> <D,1> <D,1> <E,1> <A,1> <B,1> <C,3> <D,2> <E,1> reducer Map Shuffle Reduce 3. <B,1> <C,1> <E,1> MapReduce with Hadoop • WordCountã®ä¾‹ mapper1 mapper2
Cloudera Standard ã®ã”案内 ~ 無償版 大幅機能強化ã®ãŠçŸ¥ã‚‰ã› | Cloudera Japan
Cloudera ã¯ã€æœ€å¤§è¦æ¨¡ã®ä¼æ¥ãŒã‚ã‚‰ã‚†ã‚‹å ´æ‰€ã«å˜åœ¨ã™ã‚‹ã™ã¹ã¦ã®ãƒ‡ãƒ¼ã‚¿ã‚’ä¿¡é ¼æ€§ã®é«˜ã„有æ„義ãªæ´žå¯Ÿã«å¤‰æ›ã§ãるよã†æ”¯æ´ã—ã¾ã™ã€‚ Trusted Data Today for Tomorrow’s AI (今日ã®ä¿¡é ¼ã§ãるデータをã€æ˜Žæ—¥ã® AI ã«è¨—ã™) データ㨠AI ãŒãƒ†ãƒ¼ãƒžã®ãƒ—レミアカンファレンスã§ã€æ¥ç•Œæœ€å‰ç·šã®æ˜Žç¢ºãªãƒ“ジョンをæŒã£ãŸæ–¹ã€…やソートリーダーã‹ã‚‰è²´é‡ãªæ´žå¯Ÿã‚’å¾—ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚
How to convert non-HA NameNode to QJM HA on CDH4 | 外é“父ã®åŒ
CDH3ã‹ã‚‰å§‹ã‚ã¦CDH4.1ã¾ã§ã‚¢ãƒƒãƒ—グレードã—ã¦åˆ©ç”¨ã—続ã‘ã¦ã„ã¾ã™ãŒã€ã“ã®éŽç¨‹ã§NameNodeã®æ§‹æˆã¯å¤‰æ›´ã›ãšã«é‹ç”¨ã—ã¦ãã¾ã—ãŸã€‚ 当然ã€CDH4ã‹ã‚‰ã®å…¬å¼HA構æˆã«é–¢å¿ƒã¯ã‚ã£ãŸã®ã§ã™ãŒã€è¤‡æ•°ã®æ›´æ–°ã‚’åŒæ™‚ã«è¡Œã†ã¨å±ãªã„ã¨ã‹ã€è‹±èªžãƒžãƒ‹ãƒ¥ã‚¢ãƒ«èªã‚€ã®ã‚ã‚“ã©ãã›ãƒ¼æ„Ÿã‹ã‚‰ãƒŸãƒ‰ãƒ«ãƒ¬ãƒ³ã‚¸ã«è·é›¢ã‚’ç½®ã„ã¦ã„ãŸã¨ã“ã‚ã«ã€ã‚‚ã‚Šã™å…ˆç”ŸãŒãƒˆãƒ©ã‚¤ã—ã¦ãã‚Œã¦ã€ä¹—ã‚‹ã—ã‹ãªã„ビッグウェーブ到æ¥ã€‚å¾…ã¤ã¹ãã‚‚ã®ã¯ä»–人ã®æ¤œè¨¼ã¨ã¯ã‚ˆã言ã£ãŸã‚‚ã®ã§ã™ãªã。 タイトルã¯NameNode構æˆã®åˆ‡ã‚Šæ›¿ãˆã€ã¨ãªã£ã¦ã„ã¾ã™ãŒã€ã“ã‚Œã‹ã‚‰æ–°ã—ãQJM HAã§çµ„む人ã«ã‚‚å½¹ã«ç«‹ã¤å†…容ã¨ãªã£ã¦ã„ã¾ã™ã‚†ãˆã€ç§ãŒè¡€ååã‚’åã„ã¦ã¾ã¨ã‚ãŸæƒ…å ±ã‚’æ˜¯éžã”覧ãã ã•ã„ã¾ã›ã€‚ リンク CDH4.1ã«ãŠã‘るクォーラムベースジャーナリング Quorum-Journal Design CDH4.1オーãƒãƒ¼ãƒ“ュー Software Configuration for Qu
「オンプレミス・システムã®çµ‚ã‚ã‚Šã€ã®å§‹ã¾ã‚Šã€œAWSã§ã®ãƒŸãƒƒã‚·ãƒ§ãƒ³ã‚¯ãƒªãƒ†ã‚£ã‚«ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã®ç¨¼åƒ - 急ãŒã°å›žã‚Œã€é¸ã¶ãªã‚‰è¿‘é“
個人的ã«ã¯å‰²ã¨å¤§å¤‰ã ã£ãŸã®ã§ã€ãã®è¾ºã‚’ã¾ã¨ã‚ã¦ãŠãã¾ã™ã€‚ ニュースリリースã¯ã“ã¡ã‚‰ã€‚ http://www.nautilus-technologies.com/topics/20130409.html è¦ã™ã‚‹ã«æœ¬éƒ¨ç³»ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰åŸºå¹¹ã‚·ã‚¹ãƒ†ãƒ ã®ã€Œä¸€å¼ã€ã®ã‚¯ãƒ©ã‚¦ãƒ‰ç§»è¡Œã§ã™ã€‚完全ãªãƒŸãƒƒã‚·ãƒ§ãƒ³ã‚¯ãƒªãƒ†ã‚£ã‚«ãƒ«ã‚·ã‚¹ãƒ†ãƒ ã§ã€æ¢ã¾ã£ãŸæ®µéšŽã§æ¥å‹™ã«ç¢ºå®Ÿã«å½±éŸ¿ãŒå‡ºã¾ã™ã€‚ システムã®æ©Ÿèƒ½æ¦‚è¦ 1.売上ã®ç¢ºå®šå‡¦ç†ã¨å‚µæ¨©ç®¡ç† POSデータã®ç›´çµã§ã™ã€‚売上確定処ç†ã‚’è¡Œã„ã¾ã™ã€‚åŒæ™‚ã«å‚µæ¨©ç®¡ç†ã‚‚è¡Œã„ã€F/Bã‹ã‚‰ã®å…¥é‡‘データをãã®ã¾ã¾ã¤ãªãŽè¾¼ã¿ã€å…¥é‡‘処ç†ãƒ»å‚µæ¨©ã®æ¶ˆã—è¾¼ã¿å‡¦ç†ã‚’実行ã—ã¾ã™ã€‚マッãƒãƒ³ã‚°ã¯è‡ªå‹•å‡¦ç†ã§ãã‚‹ã‚‚ã®ã¯å‡¦ç†ã‚’è¡Œã„ã€ãƒ’ューリスティックãªã‚‚ã®ã¯ãƒ¦ãƒ¼ã‚¶ãƒ¼åˆ¤æ–ã«å¾“ã„ã¾ã™ã€‚ 2.仕入・費用ã®è¨ˆä¸Šã¨ç¢ºå®šå‡¦ç†ã€ãŠã‚ˆã³æ”¯æ‰•ã„データã®ä½œæˆ 費用・在庫ã®è¨ˆä¸Šç¢ºå®šå‡¦ç†ã§ã™ã€‚当時ã«æ”¯æ‰•ãƒ‡ãƒ¼ã‚¿ã®ç¢ºå®šå‡¦ç†ã‚’è¡Œã„ã¾ã™ã€‚EDI（BMS）ã¨ã®
The dark side of Hadoop | gihyo.jp
ツイートやブãƒã‚°ã‚³ãƒ¡ãƒ³ãƒˆã®è§£æžãƒ—ラットフォームä¼æ¥ã§ã‚る「BackTypeã€ã®æŠ€è¡“ブãƒã‚°è¨˜äº‹ã§ã™ã€‚BackTypeã§ã¯30TBã‚‚ã®ã‚½ãƒ¼ã‚·ãƒ£ãƒ«ãƒ‡ãƒ¼ã‚¿ã‚’Hadoopã§å‡¦ç†ã—ã¦ã„ã¾ã™ã€‚ã¾ãŸã€Hadoopを利用ã—ãŸãƒ—ãƒãƒ€ã‚¯ãƒˆã‚’ã„ãã¤ã‹é–‹ç™ºã—オープンソースã«ã—ã¦å…¬é–‹ã—ã¦ãŠã‚Šã€Hadoopをヘビーã«æ´»ç”¨ã—ã¦ã„ã‚‹ä¼æ¥ã®ä¸€ã¤ã¨è¨€ãˆã‚‹ã§ã—ょã†ã€‚ã—ã‹ã—ã€æ·±ã利用ã—ã¦ã„ã‚‹ãŒã‚†ãˆã«ä¸æº€ç‚¹ã‚‚多ã„よã†ã§ã€Hadoopã«é–¢ã™ã‚‹å•é¡Œç‚¹ã®ã„ãã¤ã‹ã‚’挙ã’ã¦ã„ã¾ã™ã€‚ ã¾ãšã€éžå¸¸ã«é‡è¦ãªè¨å®šã«ã¤ã„ã¦ã€ä½•ã‚‚ドã‚ュメント化ã•ã‚Œã¦ã„ãªã„ã“ã¨ã‚’指摘ã—ã¦ã„ã¾ã™ã€‚具体的ã«ã¯ã€HFDSã®DataNodeã¸ã®ä¸Šé™æŽ¥ç¶šæ•°ã‚’表ã™è¨å®šé …目「dfs.datanode.max.xcieversã€ã«ã¤ã„ã¦ã€ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆå€¤ãŒ256ã¨éžå¸¸ã«ä½Žãè¨å®šã•ã‚Œã¦ã„ã‚‹ãŸã‚ã€ã™ãã«ã‚¨ãƒ©ãƒ¼ãŒç™ºç”Ÿã—ã¦ã„ã¾ã™ã€‚ã—ã‹ã—ã€å…¬å¼ãƒ‰ã‚ュメントã«ã¯ã€ã„ã£ãŸã„何ã®è¨å®šã‚’表ã™ã®ã‹ã‚„ã€ã©ã®ã‚ˆã†ãª
å“å·æµ·è‹”ã¯ä¼Šè±†ã®ç£¯é¤…» Blog Archive » Hadoopセットアップ-ãã®3(完全分散モード)
ã„やーã‹ãªã‚Šé–“ãŒç©ºã„ã¦ã—ã¾ã„ã¾ã—ãŸâ€¦ é‡ã„è…°ã‚’ã‚ã’ã¦3å°(仮想環境ã«)セットアップã—ã¾ã—ãŸï¼ 3å°ã«hadoopをセットアップã™ã‚‹ ã²ã¨ã¾ãšã€å‰ã€…回ã®è¨˜äº‹ã®é€šã‚Šã€hadoopã‚’rpmã‹ã‚‰ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ã¾ã—ãŸã€‚ ホストåã¯ãã‚Œãžã‚Œã€hadoop1ã€hadoop2ã€hadoop3ã§ã™ã€‚(ãã®ã¾ã‚“ã¾â€¦) ã—ã°ã‚‰ã見ãªã„ã†ã¡ã«hadoop1.0.3ãŒå‡ºã¦ã¾ã—ãŸã。 今回ç§ãŒã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã—ãŸã®ã¯ hadoop-1.0.3-1.x86_64.rpmã«ãªã‚Šã¾ã™ã€‚ ã¨ã‚Šã‚ãˆãšã‚¹ã‚¿ãƒ³ãƒ‰ã‚¢ãƒãƒ³ãƒ¢ãƒ¼ãƒ‰ã§å‹•ä½œç¢ºèª インストールã—ãŸhadoopをスタンドアãƒãƒ³ãƒ¢ãƒ¼ãƒ‰ã§å‹•ãã‹ç¢ºèªã—ã¾ã™ã€‚ 状態ã¨ã—ã¦ã¯å‰ã€…回ã®è¨˜äº‹ã§è¨å®šã™ã‚‹å†…容ã¯å…¨ã¦è¨å®šã—ã¦ã‚ã‚Šã¾ã™ã€‚ ãã‚Œã§ã¯hadoop1ã«ã¦å®Ÿè¡Œã—ã¾ã™ã€‚ # hadoop jar /usr/share/hadoop/hadoop-examples-1.0.3.jar word
Hadoopクラスタã§ulimitã‚’è¨å®šã™ã‚‹ã¨ãã®æ³¨æ„点 - ãŸã”ã‚‚ã‚Šã™ãƒ¡ãƒ¢
Hadoopクラスタをé‹ç”¨ã™ã‚‹éš›ã« ulimit 㧠nofile (プãƒã‚»ã‚¹ãŒopenã§ãるファイルディスクリプタ数ã®ä¸Šé™)ã®è¨å®šã‚’変更ã—ã¦ãŠãã¹ã*1ã¨ã„ã†ã®ã¯ã‚‚ã¯ã‚„常è˜çš„ãªãŠè©±ã§ã™ãŒã€ãã“ã«ã¯å®Ÿã¯ç½ ãŒã‚る。ãŸã¶ã‚“RHELã®ãƒ‡ãƒ•ã‚©ãƒ«ãƒˆé€šã‚Šãªã‚‰ãƒãƒžã‚‰ãªã„ã‚“ã ã‚ã†ã‘ã©ã€æ‰‹ã‚’入れã¦ã„る環境ã ã¨ãƒãƒžã‚‹ã€‚ã®ã§ã€ãã®è©±ã€‚ è¦ã™ã‚‹ã«ãƒãƒžã£ãŸã€‚ã®ã‚’解決ã—ãŸã‚ˆå¤šåˆ†ï¼ ã¾ã 最終的ãªç¢ºèªã§ãã¦ãªã„ã‘ã©ï¼ å„書ç±ã§ã®è§£èª¬ ã¨ã‚Šã‚ãˆãšã€å›½å†…ã§å£²ã‚‰ã‚Œã¦ã„ã‚‹Hadoop関連書ç±ã®è¨˜è¿°ã‚’確èªã—ã¦ãŠã“ã†ã€‚ã¾ãšHadoop徹底入門。 ファイディスクリプタã®è¨å®šã¯ã€/etc/security/limits.conf ã«è¨˜è¿°ã—ã¾ã™ã€‚エディタを利用ã—ã¦ã€limits.conf ã«ä»¥ä¸‹ã®ã‚ˆã†ã«è¨˜è¿°ã—ã¾ã™ã€‚ã“ã“ã§ã¯ã€Hadoopã®å„種ノードを起動ã™ã‚‹ãƒ¦ãƒ¼ã‚¶ãƒ¼ã‚’ hadoop ã¨ã—ã¾ã™[12]。 hadoop soft nofile
Hadoop error logs: org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException
Hadoop error logs: org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException 2010-06-06 23:28:04,195 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.13.53.128:50010, storageID=DS-1219700830-127.0.0.1-50010-1266876153854, infoPort=50075, ipcPort=50020):DataXceiver org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_-76478369313042311
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Discover thousands of collaborative articles on 2500+ skills
Apache Sparkâ„¢ - Unified Engine for large-scale data analytics
GitHub - myui/hivemall: Scalable machine learning library for Apache Hive/Spark/Pig
GitHub - recruitcojp/WebHive: ブラウザよりHiveQLを実行ã§ãã‚‹WEBアプリケーション
Apache whirrã®è¨å®šç³»ã«ã¤ã„㦠: ã¸ã£ã½ã“技術者備忘録
BOOT_COMPLETEDãŒå—信出æ¥ãªã„ - Google グループ
{{#tags}}- {{label}}
{{/tags}}