[B! HDFS] yassan0627ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

yassan0627 id:yassan0627

HDFSã«é–¢ã™ã‚‹yassan0627ã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (53)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

å°ã•ãªãƒ•ã‚¡ã‚¤ãƒ«ãŒå¤§ããªå•é¡Œã‚’å¼•ãèµ·ã“ã™ï¼šHadoopã‚¯ãƒ©ã‚¹ã‚¿ãƒ¼ã§ã®ã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ•ã‚¡ã‚¤ãƒ«ã®äºˆé˜²ã¨å¯¾å‡¦ã«ã¤ã„ã¦ - Clouderaæ ªå¼ä¼šç¤¾ å…¬å¼ãƒ–ãƒã‚°
åŽŸæ–‡: https://blog.cloudera.com/blog/2019/05/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/ è‘—è€…: Shashank Naik, Bhagya Gummalla è¨³æ³¨: æœ¬ç¨¿ã¯2019/5/9ã«å…¬é–‹ã•ã‚ŒãŸãƒ–ãƒã‚°è¨˜äº‹ã®ç¿»è¨³ã§ã™ã€‚ ã€Œã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ»ãƒ•ã‚¡ã‚¤ãƒ«ã€ã¯Apache Hadoopã®ä¸–ç•Œã§ã¯ä¸€èˆ¬çš„ãªèª²é¡Œã§ã‚ã‚Šã€æ³¨æ„ã—ã¦æ‰±ã‚ãªã„ã¨ã€ã„ãã¤ã‹ã®è¤‡é›‘ãªå•é¡Œã‚’å¼•ãèµ·ã“ã™å¯èƒ½æ€§ãŒã‚ã‚Šã¾ã™ã€‚Apache Hadoopåˆ†æ•£ãƒ•ã‚¡ã‚¤ãƒ«ã‚·ã‚¹ãƒ†ãƒ ï¼ˆHDFSï¼‰ã¯ã€ãƒ†ãƒ©ãƒã‚¤ãƒˆã‹ã‚‰ãƒšã‚¿ãƒã‚¤ãƒˆã®ç¯„å›²ã«ã‚ãŸã‚‹ã€å¤§ããªãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã‚’æ ¼ç´ã—ã¦å‡¦ç†ã™ã‚‹ãŸã‚ã«é–‹ç™ºã•ã‚Œã¾ã—ãŸã€‚ã—ã‹ã—HDFSã¯ã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ»ãƒ•ã‚¡ã‚¤ãƒ«ã‚’åŠ¹çŽ‡çš„ã«æ ¼ç´ã—ãªã„ãŸã‚ã€ã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ»ãƒ•ã‚¡ã‚¤ãƒ«ã«ã‚ˆã£ã¦NameNo
yassan0627 2023/03/24
small fileå•é¡Œ

HDFS
ãƒªãƒ³ã‚¯
ç¶šãƒ»Apache Ozone ã‚’ã‚„ã£ã¦ã„ãŸä¸€å¹´ - Preferred Networks Research & Development
PFN ã§ã¯ HDFS ã‹ã‚‰ Apache Ozone ã¸ã®ç§»è¡Œã‚’é€²ã‚ã¦ã„ã¾ã™ã€‚Ozone ã‚¯ãƒ©ã‚¹ã‚¿ã¯é †èª¿ã«ç¤¾å†…ã®ãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã§æŽ¡ç”¨ãŒé€²ã‚“ã§ãƒ‡ãƒ¼ã‚¿ãŒå¢—åŠ ã—ã¦ãŠã‚Šã€ã“ã‚Œã¾ã§ã® 2 å¹´é–“ã¯æ•°åº¦ã«ã‚ãŸã£ã¦ã‚¯ãƒ©ã‚¹ã‚¿ã‚’æ‹¡å¼µã—ã¾ã—ãŸã€‚ã¨ã“ã‚ãŒã€æ®µéšŽçš„ã«ã‚¯ãƒ©ã‚¹ã‚¿ã‚’æ‹¡å¼µã—ãŸã“ã¨ã§ãƒ‡ãƒ¼ã‚¿ãƒŽãƒ¼ãƒ‰é–“ã§ã®ãƒ‡ã‚£ã‚¹ã‚¯ä½¿ç”¨é‡ã®ä¸å‡è¡¡ãŒå•é¡Œã«ãªã£ã¦ãã¾ã—ãŸã€‚ä¾‹ãˆã°ã€å…¨ã¦ã®ãƒ‡ã‚£ã‚¹ã‚¯ãŒ 8 å‰²åŸ‹ã¾ã£ã¦ã„ã‚‹ Ozone ã‚¯ãƒ©ã‚¹ã‚¿ã«ã¾ã£ãŸãç©ºã®ãƒ‡ãƒ¼ã‚¿ãƒŽãƒ¼ãƒ‰ã‚’è¿½åŠ ã™ã‚‹ã¨ã€ãƒ‡ãƒ¼ã‚¿ä½¿ç”¨é‡ã«å¤§ããªä¸å‡è¡¡ãŒç”Ÿã˜ã¾ã™ã€‚æ–°è¦è¿½åŠ ã—ãŸç©ºã®ã‚µãƒ¼ãƒã«ãƒ‡ãƒ¼ã‚¿ã‚’ç§»å‹•ã™ã‚‹ã“ã¨ã‚’ãƒªãƒãƒ©ãƒ³ã‚¹ã¨ã„ã„ã¾ã™ãŒã€é‹ç”¨ã—ã‚„ã™ã„ã‚ˆã†ã«ãƒªãƒãƒ©ãƒ³ã‚¹ã‚’å®Ÿè£…ã™ã‚‹æ–¹æ³•ã¯è‡ªæ˜Žã§ã¯ã‚ã‚Šã¾ã›ã‚“ã€‚HDFS ã§ã¯ãƒŽãƒ¼ãƒ‰å˜ä½ã§ãƒ‡ã‚£ã‚¹ã‚¯ä½¿ç”¨é‡ã‚’è¦å®šã®ç¯„å›²å†…ã«å‡ã™ã‚ˆã†ã«ç§»å‹•ã™ã‚‹ Balancer ã‚„ãƒ‡ãƒ¼ã‚¿ãƒŽãƒ¼ãƒ‰å†…ã®ãƒ‡ã‚£ã‚¹ã‚¯ã‚’åŒæ§˜ã«ç§»å‹•ã™ã‚‹ Disk Balancer ãŒã‚ã‚Šã¾ã™ã€‚ä¸€æ–¹ã§ Ozon
yassan0627 2022/11/15
HDFS

hadoop

Apache Ozone

ãƒ‡ãƒ¼ã‚¿
ãƒªãƒ³ã‚¯
Ozone Write Pipeline V2 with Ratis Streaming - Cloudera Blog
yassan0627 2022/11/12
Ozoneã¨HDFSã®ãƒ•ã‚¡ã‚¤ãƒ«æ›¸ãè¾¼ã¿ã®ä»•çµ„ã¿

hadoop

HDFS

Apache Ozone

ãƒ‡ãƒ¼ã‚¿
ãƒªãƒ³ã‚¯
Apache Ozoneã‚’ã‚„ã£ã¦ã„ãŸä¸€å¹´ - Preferred Networks Research & Development
Why Apache Ozone? ã“ã‚Œã¾ã§PFNã§ã¯å¢—ãˆç¶šã‘ã‚‹ãƒ‡ãƒ¼ã‚¿ã‚„ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã«å¯¾å¿œã™ã‚‹ãŸã‚ã«ã€ã‚¹ã‚±ãƒ¼ãƒ«ã‚¢ã‚¦ãƒˆå¯èƒ½ãªã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚·ã‚¹ãƒ†ãƒ ã‚’ãšã£ã¨æ¨¡ç´¢ã—ç¶šã‘ã¦ãã¾ã—ãŸã€‚ã‚·ãƒŸãƒ¥ãƒ¬ãƒ¼ã‚·ãƒ§ãƒ³ã‚’åŸºè»¸ã¨ã—ãŸæˆ¦ç•¥ã‚’æŽ¡ç”¨ã—ãŸ[1]ã“ã¨ã«ã‚ˆã‚Šãƒ‡ãƒ¼ã‚¿é‡ã¯ã•ã‚‰ã«å¢—åŠ ã—ã€ãƒ‡ãƒ¼ã‚¿ä¿ç®¡ã‚·ã‚¹ãƒ†ãƒ ã®é‡è¦æ€§ã¯é«˜ã¾ã£ã¦ã„ã¾ã™ã€‚ Preferred Networks ã«ãŠã‘ã‚‹Hadoop â€“ Preferred Networks Research ã§è§£èª¬ã—ãŸåŸºæœ¬çš„ãªè¦ä»¶ã¯ä»Šã§ã‚‚å¤‰ã‚ã£ã¦ã„ã¾ã›ã‚“ãŒ[2]ã€ç¾åœ¨ãƒ¡ã‚¤ãƒ³ã®ã‚·ã‚¹ãƒ†ãƒ ã¨ã—ã¦é‹ç”¨ã—ã¦ã„ã‚‹ Hadoop (HDFS) ã«ã¯ã„ãã¤ã‹ã®ã‚·ã‚¹ãƒ†ãƒ é‹ç”¨ä¸Šã®èª²é¡ŒãŒã‚ã‚Šã¾ã™ã€‚ãŸã¨ãˆã°ã€ä¸€ç•ªå¤§ããªHadoopã‚¯ãƒ©ã‚¹ã‚¿ã¯ç¾æ™‚ç‚¹ã§ç‰©ç†çš„ã«10PBè¿‘ãã®ãƒ‡ã‚£ã‚¹ã‚¯å®¹é‡ã‚’æŒã£ã¦ã„ã¾ã™ãŒã€Ubuntu 16.04ã§å‹•ä½œã—ã¦ã„ã¾ã™ã€‚OSã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚¢ãƒƒãƒ—ã‚’ä¼´ã†ã‚¯ãƒ©ã‚¹ã‚¿ã®In-placeãªã‚¢ãƒƒãƒ—ã‚°ãƒ¬ãƒ¼
yassan0627 2021/12/10
hadoop

HDFS

é‹ç”¨

Apache Ozone

äº‹ä¾‹

ãƒ‡ãƒ¼ã‚¿
ãƒªãƒ³ã‚¯
ã‚ˆã‚Šã‚»ã‚ãƒ¥ã‚¢ãªHadoopã®ä½œã‚Šæ–¹ ã€œ ApacheCon Asia 2021ç™»å£‡å ±å‘Š
HDFSã®ãƒ–ãƒãƒƒã‚¯ãƒ‡ãƒ¼ã‚¿è»¢é€ã®æš—å·åŒ– HDFSã®å®Ÿãƒ‡ãƒ¼ã‚¿ã«ç›¸å½“ã™ã‚‹ãƒ–ãƒãƒƒã‚¯ãƒ‡ãƒ¼ã‚¿ã®è»¢é€ã«å¯¾ã™ã‚‹æš—å·åŒ–ã¯RPCæš—å·åŒ–ã¨ã¯ã•ã‚‰ã«åˆ¥ã®ãƒ‘ãƒ©ãƒ¡ãƒ¼ã‚¿ã§ç®¡ç†ã•ã‚Œã¦ã„ã¾ã™ã€‚ hdfs-siteã® dfs.encrypt.data.transfer ã‚’ true ã«ã€ dfs.encrypt.data.transfer.cipher.suites ã‚’ AES/CTR/NoPadding ã«ãã‚Œãžã‚ŒæŒ‡å®šã™ã‚‹ã“ã¨ã§æœ‰åŠ¹åŒ–ã—ã¾ã™ã€‚ ã“ã®è¨å®šã‚’ç„¡åœæ¢ã§åæ˜ ã™ã‚‹ã«ã‚ãŸã£ã¦ã¯ã€è»¢é€ã®æš—å·åŒ–è¨å®šã®åæ˜ æ¸ˆ/æœªåæ˜ ã‚’åˆ¤åˆ¥ã™ã‚‹ dfs.trustedchannel.resolver.class ã«ã‚ˆã‚‹ãƒŽãƒ¼ãƒ‰ã®å‹•çš„ãªç®¡ç†ã€å ´åˆã«ã‚ˆã£ã¦ã¯æ‹¡å¼µã«ã‚ˆã‚‹ç‹¬è‡ªå®Ÿè£…ãŒå¿…è¦ã¨ãªã‚Šã¾ã™ã€‚ HDFSã‚¯ãƒ©ã‚¤ã‚¢ãƒ³ãƒˆãŠã‚ˆã³NameNodeãƒ»DataNodeé–“ã§ dfs.encrypt.data.transfer ã®çœŸå½ãŒä¸€è‡´ã—ã¦ã„ãªã„å ´åˆã¯ãƒ–ãƒãƒƒã‚¯ãƒ‡
yassan0627 2021/12/07
hadoop

HDFS

ã‚»ã‚ãƒ¥ãƒªãƒ†ã‚£
ãƒªãƒ³ã‚¯
Containerizing Apache Hadoop Infrastructure at Uber
Youâ€™re seeing information for Japan . To see local features and services for another location, select a different city. Show more Introduction As Uberâ€™s business grew, we scaled our Apache Hadoop (referred to as â€˜Hadoopâ€™ in this article) deployment to 21000+ hosts in 5 years, to support the various analytical and machine learning use cases. We built a team with varied expertise to address the chal
yassan0627 2021/07/23
hadoop

YARN

docker

HDFS

apache
ãƒªãƒ³ã‚¯
ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªã‚¤ãƒ™ãƒ³ãƒˆã«Flafkaã‚’ä½¿ã£ã¦Kafkaã¨ãƒ‡ãƒ¼ã‚¿ã®ã‚„ã‚Šå–ã‚Šã‚’è¡Œã† - Qiita
Apache Flumeã‚„Apache Kafkaã¯ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ ãªã‚¤ãƒ™ãƒ³ãƒˆå‡¦ç†ã®ãƒãƒƒã‚¯ã‚¨ãƒ³ãƒ‰ã¨ã—ã¦åºƒãåˆ©ç”¨ã•ã‚Œã¦ã„ã¾ã™ã€‚ã“ã‚Œã‚‰ï¼’ã¤ã®ã‚·ã‚¹ãƒ†ãƒ ã¯ä¼¼ã¦ã„ã‚‹éƒ¨åˆ†ã‚‚ã‚ã‚Šã¾ã™ãŒã€ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã«ã‚ˆã‚Šã©ã¡ã‚‰ã‹ä¸€æ–¹ã€ã‚ã‚‹ã„ã¯é‡ã‚’çµ„ã¿åˆã‚ã›ã¦ä½¿ã†å ´åˆã‚‚ã‚ã‚Šã¾ã™ã€‚ Flumeã¨Kafkaã®é•ã„ã¯æ¬¡ã®ãƒ–ãƒã‚°ã‚‚å‚è€ƒã«ãªã‚Šã¾ã™ã€‚ https://www.linkedin.com/pulse/flume-kafka-real-time-event-processing-lan-jiang Apache Kafka Apache Kafkaã¯pub-subã€å‡ºç‰ˆ-è³¼èªåž‹ã®ã‚·ã‚¹ãƒ†ãƒ ã§ã€å¤šæ•°ã®ã‚·ã‚¹ãƒ†ãƒ ã¨ã®é€£æºã«åºƒãåˆ©ç”¨ã•ã‚Œã¦ã„ã¾ã™ã€‚ [ç”»åƒã¯https://kafka.apache.org/ã‚ˆã‚Šå¼•ç”¨] ã—ã‹ã—ã€Kafkaã‚’ä½¿ã†å ´åˆã€ä¸€èˆ¬çš„ã«ãƒ—ãƒãƒ‡ãƒ¥ãƒ¼ã‚µã‚„ã‚³ãƒ³ã‚·ãƒ¥ãƒ¼ãƒžã®ãŸã‚ã®ã‚³ãƒ¼ãƒ‰ã‚’è¨˜è¿°ã™ã‚‹å¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ Producer
yassan0627 2021/01/06
kafka

HDFS

flume

Flafka
ãƒªãƒ³ã‚¯
Flafka: Apache Flume Meets Apache Kafka for Event Processing - Cloudera Blog
The new integration between Flume and Kafka offers sub-second-latency event processing without the need for dedicated infrastructure. In this previous post you learned some Apache Kafka basics and explored a scenario for using Kafka in an online application. This post takes you a step further and highlights the integration of Kafka with Apache Hadoop, demonstrating both a basic ingestion capabilit
yassan0627 2021/01/06
kafka

HDFS

flume

Flafka
ãƒªãƒ³ã‚¯
Hadoop DistCpå®Ÿè·µã‚¬ã‚¤ãƒ‰2020å¹´ç‰ˆ - ç§‘å¦ã¨éžç§‘å¦ã®è¿·å®®
Hadoop DistCp (distributed copy, ã§ãƒã™ã¨ã—ãƒ¼ã´ãƒ¼ã€ã§ãƒã™ã¨ã“ã´ãƒ¼) ã¯ã€MapReduceã‚’ç”¨ã„ã¦Hadoopã‚¯ãƒ©ã‚¹ã‚¿é–“ã§ãƒ‡ãƒ¼ã‚¿ã‚³ãƒ”ãƒ¼ã™ã‚‹ãŸã‚ã®ãƒ„ãƒ¼ãƒ«ã§ã™ã€‚ä¿å®ˆé‹ç”¨ã—ã¦ã„ã‚‹å ´åˆã‚’é™¤ãã€ãŠãã‚‰ã2020å¹´ã«ãŠã„ã¦ã‚‚é‹ç”¨ä¸Šã®é¸æŠžè‚¢ã¨ã—ã¦æ®‹ã£ã¦ã„ã‚‹æœ€å¾Œã®MapReduceã®ãƒ„ãƒ¼ãƒ«ã§ã™ã€‚ã“ã®è¨˜äº‹ã§ã¯ã€DistCpã®ç´¹ä»‹ã¨å®Ÿè·µçš„ãªä½¿ã„æ–¹ã®åŸºæœ¬ã«ã¤ã„ã¦èª¬æ˜Žã—ã¦ã„ãã¾ã™ã€‚å†…å®¹ã¨ã—ã¦ã¯ä»¥ä¸‹ã®é€šã‚Šã§ã™ã€‚ Distcpã®æ¦‚è¦ã¨åŽŸç† å®Ÿè·µDistCp DistCpã«ãƒ‰ãƒ©ã‚¤ãƒ©ãƒ³ã¯ãªã„ ã‚³ãƒ”ãƒ¼ã¨ã‚¢ãƒƒãƒ—ãƒ‡ãƒ¼ãƒˆã®æŒ™å‹•ã®é•ã„ã‚’æŠ¼ã•ãˆã‚‹ ã‚¹ãƒŠãƒƒãƒ—ã‚·ãƒ§ãƒƒãƒˆã‚’å–å¾—ã™ã‚‹ ã‚½ãƒ¼ã‚¹ã¨å®›å…ˆã€ã©ã¡ã‚‰ã®ã‚¯ãƒ©ã‚¹ã‚¿ã§DistCpã‚’å®Ÿè¡Œã™ã‚‹ã‹ ç•°ãªã‚‹ãƒ¡ã‚¸ãƒ£ãƒ¼ãƒãƒ¼ã‚¸ãƒ§ãƒ³é–“ã§ã®ãƒ‡ãƒ¼ã‚¿è»¢é€ã«webhdfsã‚’ä½¿ã† -p ã‚ªãƒ—ã‚·ãƒ§ãƒ³ã®æŒ™å‹• 2ã¤ã®ã‚³ãƒ”ãƒ¼æˆ¦ç•¥: uniformizeã¨dynamic mapæ•°ã®èª¿æ•´ è»¢é€å¸¯åŸŸ
yassan0627 2020/11/09
æœ€é«˜ã®DistCpã®è³‡æ–™

hadoop

HDFS
ãƒªãƒ³ã‚¯
HDFS Migration from 2.7 to 3.3 and enabling Router Based Federation (RBF) in production #ACAH2020
yassan0627 2020/10/08
ApacheCon

hadoop

HDFS

ãƒ‡ãƒ¼ã‚¿

RBF
ãƒªãƒ³ã‚¯
Spark+AI Summit 2019 ã‚»ãƒƒã‚·ãƒ§ãƒ³ãƒã‚¤ãƒ©ã‚¤ãƒˆ (Spark Meetup Tokyo #1 - Spark+AI Summit 2019)
â– Spark Meetup Tokyo #1 - Spark+AI Summit 2019 ç™ºè¡¨è³‡æ–™ (2019/06/12) Spark+AI Summit 2019 ã‚»ãƒƒã‚·ãƒ§ãƒ³ãƒã‚¤ãƒ©ã‚¤ãƒˆ æ ªå¼ä¼šç¤¾NTTãƒ‡ãƒ¼ã‚¿ (NTT DATA) æŠ€è¡“é©æ–°çµ±æ‹¬æœ¬éƒ¨ çŒ¿ç”° æµ©è¼” / ç”°ä¸ æ£æµ© / éƒ½ç¯‰ æ£å®œ â€»ã‚¤ãƒ™ãƒ³ãƒˆæ¦‚è¦ https://spark-meetup-tokyo.connpass.com/event/131791/Read less
yassan0627 2020/07/07
ãƒ‡ãƒ¼ã‚¿

DeltaLake

hadoop

HDFS

spark
ãƒªãƒ³ã‚¯
HDFS dfs ã‚µãƒ–ã‚³ãƒžãƒ³ãƒ‰ã®ãƒãƒ¼ãƒˆã‚·ãƒ¼ãƒˆ
ä»¥å‰ã«å…¬é–‹ã—ã€ä¸€ç•ªã‚¢ã‚¯ã‚»ã‚¹ã•ã‚Œã¦ã„ã‚‹ hdfs dfs ã®ã‚µãƒ–ã‚³ãƒžãƒ³ãƒ‰ãƒšãƒ¼ã‚¸ã®æƒ…å ±ãŒå¤ã‹ã£ãŸã®ã§ã€Hadoopï¼“ç³»ã§æ¤œè¨¼ã—ã¦å…¨é¢çš„ã«æ›¸ãç›´ã—ã¾ã—ãŸã€‚ HDFS FSshell (hdfs dfs) ã‚³ãƒžãƒ³ãƒ‰ã®ãƒãƒ¼ãƒˆã‚·ãƒ¼ãƒˆã®PDFãƒ•ã‚¡ã‚¤ãƒ«ã§ã™ã€‚(Hadoop 3.1.1)ã€‚ ãƒ€ã‚¦ãƒ³ãƒãƒ¼ãƒ‰ã‚‚å¯èƒ½ã§ã™ã€‚ View Fullscreen ä½™åŠ›ãŒã‚ã‚Œã°ã€ç®¡ç†ç³»ã®ã‚³ãƒžãƒ³ãƒ‰(hdfs dfsadminç‰)ã‚‚è¿½ã£ã¦ä½œæˆã™ã‚‹ã‹ã‚‚ã—ã‚Œã¾ã›ã‚“ã€‚
yassan0627 2020/06/22
hadoop

HDFS

ãƒãƒ¼ãƒˆã‚·ãƒ¼ãƒˆ
ãƒªãƒ³ã‚¯
HDFSã®ãƒªãƒ¼ã‚¹ãƒªã‚«ãƒãƒªã€ãƒ–ãƒãƒƒã‚¯ãƒªã‚«ãƒãƒªã€ãŠã‚ˆã³ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ãƒªã‚«ãƒãƒªå‹•ä½œè©³è§£ (ãã®2) | by Sho Shimauchi | Cloudera Japan Official Blog
æŠ•ç¨¿æ—¥: 2015/03/18 æœ¬è¨˜äº‹ã¯ã€HDFSã®å†…éƒ¨å‹•ä½œã‚’èª¬æ˜Žã™ã‚‹ãƒ–ãƒã‚°ãƒã‚¹ãƒˆã®ç¬¬äºŒéƒ¨ã§ã‚ã‚Šã€Clouderaã®ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ãƒ»ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ã§Hadoopãƒ—ãƒã‚¸ã‚§ã‚¯ãƒˆã®ã‚³ãƒŸãƒƒã‚¿ã§ã‚ã‚‹Yongjun Zhangã«ã‚ˆã‚‹è¨˜äº‹ã‚’ç¿»è¨³ã—ãŸã‚‚ã®ã§ã™ã€‚åŽŸæ–‡ã«ã¤ã„ã¦ã¯ã“ã¡ã‚‰ã‚’ã”è¦§ãã ã•ã„ã€‚ HDFSã®é‡è¦ãªè¨è¨ˆè¦ä»¶ã®ã²ã¨ã¤ã¨ã—ã¦ã€é€£ç¶šçš„ã‹ã¤æ£ç¢ºãªå‹•ä½œã®ä¿è¨¼ãŒæŒ™ã’ã‚‰ã‚Œã‚‹ã€‚ãã®ãŸã‚ã€å•é¡ŒãŒç™ºç”Ÿã—ãŸã¨ãã«HDFSãŒã©ã®ã‚ˆã†ã«ãƒªã‚«ãƒãƒªã‚’è¡Œã†ã‹ç†è§£ã™ã‚‹ã®ã¯éžå¸¸ã«é‡è¦ã§ã‚ã‚‹ã¨è¨€ãˆã‚‹ã€‚æœ¬ãƒ–ãƒã‚°ã‚·ãƒªãƒ¼ã‚ºã®ãã®1ã§ã¯ã€ãƒªãƒ¼ã‚¹ãƒªã‚«ãƒãƒªã€ãƒ–ãƒãƒƒã‚¯ãƒªã‚«ãƒãƒªã«ã¤ã„ã¦è¿°ã¹ãŸã€‚æœ¬ãƒã‚¹ãƒˆã§ã¯ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ãƒªã‚«ãƒãƒªã«ã¤ã„ã¦èª¬æ˜Žã—ã‚ˆã†ã€‚ ãŠã•ã‚‰ã„ HDFSã§ã¯ãƒ•ã‚¡ã‚¤ãƒ«ãŒãƒ–ãƒãƒƒã‚¯ã«åˆ†å‰²ã•ã‚Œã€ãƒ•ã‚¡ã‚¤ãƒ«ã¸ã®ã‚¢ã‚¯ã‚»ã‚¹ã¯multi-reader, single-writerã‚»ãƒžãƒ³ãƒ†ã‚£ã‚¯ã‚¹ã«å¾“ã£ã¦ã„ã‚‹ã€‚éšœå®³è€æ€§ã®è¦ä»¶ã‚’æº€ãŸã™ãŸã‚ã€è¤‡æ•°ã®ãƒ–ãƒãƒƒã‚¯ãƒ¬ãƒ—ãƒªã‚«ãŒ
yassan0627 2020/06/05
HDFS

hadoop
ãƒªãƒ³ã‚¯
HDFSã®ãƒªãƒ¼ã‚¹ãƒªã‚«ãƒãƒªã€ãƒ–ãƒãƒƒã‚¯ãƒªã‚«ãƒãƒªã€ãŠã‚ˆã³ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ãƒªã‚«ãƒãƒªå‹•ä½œè©³è§£ (ãã®1) | by Sho Shimauchi | Cloudera Japan Official Blog
æŠ•ç¨¿æ—¥: 2015/03/03 æœ¬è¨˜äº‹ã¯ã€Cloudera ã®ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ãƒ»ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ Yongjun Zhang ã«ã‚ˆã‚‹è¨˜äº‹ã‚’ç¿»è¨³ã—ãŸã‚‚ã®ã§ã™ã€‚åŽŸæ–‡ã«ã¤ã„ã¦ã¯ã“ã¡ã‚‰ã‚’ã”è¦§ãã ã•ã„ã€‚ HDFSã®é‡è¦ãªè¨è¨ˆè¦ä»¶ã®ã²ã¨ã¤ã¨ã—ã¦ã€é€£ç¶šçš„ã‹ã¤æ£ç¢ºãªå‹•ä½œã®ä¿è¨¼ãŒæŒ™ã’ã‚‰ã‚Œã‚‹ã€‚ãƒãƒƒãƒˆãƒ¯ãƒ¼ã‚¯ã‚„ãƒŽãƒ¼ãƒ‰éšœå®³æ™‚ã«ã€HDFSã¸ã®æ›¸ãè¾¼ã¿ã®æ£ç¢ºã•ã‚’ä¿è¨¼ã™ã‚‹ã“ã¨ã¯è¤‡é›‘ãªå•é¡Œã®ã²ã¨ã¤ã¨è¨€ãˆã‚‹ã ã‚ã†ã€‚ã“ã‚Œã¯ã€ãƒªãƒ¼ã‚¹ãƒªã‚«ãƒãƒªã€ãƒ–ãƒãƒƒã‚¯ãƒªã‚«ãƒãƒªã€ã‚ã‚‹ã„ã¯ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ãƒªã‚«ãƒãƒªãƒ—ãƒã‚»ã‚¹ã«ã‚ˆã‚Šä¿è¨¼ã•ã‚Œã‚‹ã€‚ãã®ãŸã‚ã€ã„ã¤ã€ãªãœã“ã‚Œã‚‰ã®ãƒªã‚«ãƒãƒªãƒ—ãƒã‚»ã‚¹ãŒã‚³ãƒ¼ãƒ«ã•ã‚Œã‚‹ã®ã‹ã€ã¾ãŸãã‚Œã‚‰ãŒä½•ã‚’ã™ã‚‹ã¹ããªã®ã‹ã‚’ç†è§£ã™ã‚‹ã“ã¨ã¯ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ã«ã¨ã£ã¦ã‚‚é–‹ç™ºè€…ã«ã¨ã£ã¦ã‚‚ã€è‡ªèº«ã®HDFSã‚¯ãƒ©ã‚¹ã‚¿ãŒã©ã®ã‚ˆã†ã«å‹•ä½œã™ã‚‹ã®ã‹ç†è§£ã™ã‚‹æ‰‹åŠ©ã‘ã«ãªã‚‹ã¨æ€ã‚ã‚Œã‚‹ã€‚ ã“ã®è¨˜äº‹ã§ã¯ã€ã“ã‚Œã‚‰ã®ãƒªã‚«ãƒãƒªãƒ—ãƒã‚»ã‚¹ã‚’è©³ç´°ã«èª¬æ˜Žã™ã‚‹ã€‚ã¾ãšã¯HDFSã®æ›¸ãè¾¼ã¿ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã‚„ãã®ãƒªã‚«ãƒ
yassan0627 2020/06/05
HDFS

hadoop
ãƒªãƒ³ã‚¯
Hadoopã®ãƒãƒ¼ã‚¸ãƒ§ãƒ³æ··ç”¨ã¯å¯èƒ½?ã€€HDP 2.6.4 ã¨ã‚³ãƒŸãƒ¥ãƒ‹ãƒ†ã‚£ç‰ˆ Hadoop 3.2.1 ã«ãŠã‘ã‚‹HDFSã®äº’æ›æ€§èª¿æŸ»çµæžœã€€
ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10æœˆ1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ãƒ–ãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚LINEãƒ¤ãƒ•ãƒ¼ Tech Blog (English translation is available here) ãƒ¤ãƒ•ãƒ¼ã§ Hadoop ã®é‹ç”¨ãƒ»é–‹ç™ºã‚’ã—ã¦ã„ã‚‹æŽ ç‡®é³´ã§ã™ã€‚ç§ãŸã¡ã®ãƒãƒ¼ãƒ ã§ã¯ Hadoop 3.2.1 ã«å«ã¾ã‚Œã‚‹ Router Based Federation (RBF) ã§ Hadoop ã®ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£å•é¡Œã®è§£æ±ºã‚’è©¦ã¿ã¦ã„ã¾ã™ã€‚ã“ã®è¨˜äº‹ã§ã¯ RBF ã‚’å°Žå…¥ã™ã‚‹èƒŒæ™¯ã¨ Hadoop 3.2.1 ã¨æ—¢å˜ã®å®Ÿè¡Œç³»ã®äº’æ›æ€§ã‚’ç´¹ä»‹ã—ã¾ã™ã€‚ èƒŒæ™¯ãŠã‚ˆã³ç›®çš„ Hadoopã‚¯ãƒ©ã‚¹ã‚¿@ãƒ¤ãƒ•ãƒ¼ ãƒ¤ãƒ•ãƒ¼ã§ã¯ã€è‡ªç¤¾ãŒæä¾›ã—ã¦ã„ã‚‹å¤šç¨®å¤šæ§˜ãªã‚µãƒ¼ãƒ“ã‚¹ã®ãƒã‚°ã‚’åˆ†æžã—ã¦ã‚µãƒ¼ãƒ“ã‚¹ã®æ”¹å–„ã«å½¹ç«‹ã¦ã‚‹ãŸã‚ã€Hadoop ã‚¯ãƒ©ã‚¹ã‚¿ã‚’è¤‡æ•°é‹ç”¨ã—ã¦ã„ã¾ã™ã€‚ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ã‚¹ã‚¿ãƒƒã‚¯ã§ã¯ã€HD
yassan0627 2020/06/02
hadoop

HDFS
ãƒªãƒ³ã‚¯
Ozoneã«10å„„ãƒ•ã‚¡ã‚¤ãƒ«
Apache Hadoop Ozoneã¯åˆ†æ•£åž‹ã®ã‚ãƒ¼ãƒãƒªãƒ¥ãƒ¼ã‚¹ãƒˆã‚¢ã§ã‚ã‚Šã€å°ã•ãªãƒ•ã‚¡ã‚¤ãƒ«ã‹ã‚‰å¤§ããªãƒ•ã‚¡ã‚¤ãƒ«ã¾ã§ã®ä¸¡æ–¹ã‚’ç®¡ç†ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚Ozoneã¯ã€å°ã•ãªãƒ•ã‚¡ã‚¤ãƒ«ã«é–¢ã™ã‚‹HDFSã®ã‚¹ã‚±ãƒ¼ãƒ«ã®åˆ¶é™ã«å¯¾å‡¦ã™ã‚‹ãŸã‚ã«è¨è¨ˆã•ã‚Œã¾ã—ãŸã€‚HDFSã¯å¤§ããªãƒ•ã‚¡ã‚¤ãƒ«ã‚’æ ¼ç´ã™ã‚‹ãŸã‚ã«è¨è¨ˆã•ã‚Œã¦ãŠã‚Šã€HDFSã§ã®æŽ¨å¥¨ãƒ•ã‚¡ã‚¤ãƒ«æ•°ã¯ NameNode ã§3å„„å€‹ã§ã€ã“ã®åˆ¶é™ã‚’è¶…ãˆã¦ã®ã‚¹ã‚±ãƒ¼ãƒ«ã¯ã§ãã¾ã›ã‚“ã€‚ ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£ã‚’é”æˆã™ã‚‹ãŸã‚ã®Ozoneã®ä¸»ãªç‰¹å¾´ã¯ä»¥ä¸‹ã®é€šã‚Šã§ã™ã€‚ Ozoneã®åå‰ç©ºé–“ã¯ãƒãƒ¼ã‚«ãƒ«ã®RocksDBã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã«æ›¸ãè¾¼ã¾ã‚Œã€ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹ï¼ˆå…¨ã¦ã‚’ãƒ¡ãƒ¢ãƒªã«ä¿æŒã™ã‚‹ï¼‰ã¨ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£ï¼ˆã‚ã¾ã‚Šä½¿ã‚ã‚Œã¦ã„ãªã„ãƒ¡ã‚¿ãƒ‡ãƒ¼ã‚¿ã‚’ãƒ‡ã‚£ã‚¹ã‚¯ã«æ°¸ç¶šåŒ–ï¼‰ã®é–“ã®ãƒãƒ©ãƒ³ã‚¹ã‚’ã€ç°¡å˜ã«èª¿æ•´ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚åå‰ç©ºé–“ã¨ãƒ–ãƒãƒƒã‚¯ã‚¹ãƒšãƒ¼ã‚¹ã®ç®¡ç†ã¯ã€2ã¤ã®ç•°ãªã‚‹ãƒ‡ãƒ¼ãƒ¢ãƒ³ã€OzoneManager(OM)ã¨Storage
yassan0627 2020/05/29
ä»Šå¾ŒãŒæ¥½ã—ã¿ã€‚

HDFS

Apache Ozone

hadoop
ãƒªãƒ³ã‚¯
Hadoop HDFS FSImage | Grafana Labs
yassan0627 2020/05/23
prometheus

HDFS

k8s

hadoop
ãƒªãƒ³ã‚¯
GitHub - marcelmay/hadoop-hdfs-fsimage-exporter: Exports Hadoop HDFS content statistics to Prometheus
yassan0627 2020/05/23
HDFSã®ã‚¹ãƒ¢ãƒ¼ãƒ«ãƒ•ã‚¡ã‚¤ãƒ«å•é¡Œã¨ã‹ã«ä½¿ãˆãã†

prometheus

ç›£è¦–

HDFS

hadoop
ãƒªãƒ³ã‚¯
kubernetes-HDFS/charts/README.md at master Â· apache-spark-on-k8s/kubernetes-HDFS
yassan0627 2020/05/06
HDFS

k8s
ãƒªãƒ³ã‚¯
Ozoneã®ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯: CDPç”¨ Cloudera ã®æ¬¡ä¸–ä»£ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸
yassan0627 2020/04/15
hadoop

Apache Ozone

hive

HDFS
ãƒªãƒ³ã‚¯
1 2 3 æ¬¡ã®ãƒšãƒ¼ã‚¸