[B! hadoop] clavierã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

clavier id:clavier

hadoopã«é–¢ã™ã‚‹clavierã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (132)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Pythonã§å§‹ã‚ã‚‹MapReduceãƒ‡ãƒ¼ã‚¿å‡¦ç†ï¼šä¸ç´šè€…å‘ã‘ - Qiita
Deleted articles cannot be recovered. Draft of this article would be also deleted. Are you sure you want to delete this article? 1. ã¯ã˜ã‚ã« MapReduceã¯å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã®ãŸã‚ã®å¼·åŠ›ãªãƒ—ãƒã‚°ãƒ©ãƒŸãƒ³ã‚°ãƒ¢ãƒ‡ãƒ«ã§ã™ã€‚æœ¬è¨˜äº‹ã§ã¯ã€Google Colabç’°å¢ƒã§Pythonã‚’ä½¿ç”¨ã—ã¦MapReduceã®æ¦‚å¿µã‚’å¦ã³ã€åŠ¹çŽ‡çš„ãªãƒ‡ãƒ¼ã‚¿å‡¦ç†ãƒ‘ã‚¤ãƒ—ãƒ©ã‚¤ãƒ³ã‚’æ§‹ç¯‰ã™ã‚‹æ–¹æ³•ã‚’è§£èª¬ã—ã¾ã™ã€‚ MapReduceã®æ¦‚è¦ MapReduceã¯ä¸»ã«2ã¤ã®æ®µéšŽã‹ã‚‰æ§‹æˆã•ã‚Œã¦ã„ã¾ã™ï¼š Map: å…¥åŠ›ãƒ‡ãƒ¼ã‚¿ã‚’ key-value ãƒšã‚¢ã«å¤‰æ›ã™ã‚‹ Reduce: åŒã˜ã‚ãƒ¼ã‚’æŒã¤å€¤ã‚’ã¾ã¨ã‚ã¦å‡¦ç†ã™ã‚‹ ã“ã‚Œã‚‰ã®æ“ä½œã‚’çµ„ã¿åˆã‚ã›ã‚‹ã“ã¨ã§ã€å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿ã‚»ãƒƒãƒˆã‚’åŠ¹çŽ‡çš„ã«å‡¦ç†ã™ã‚‹ã“ã¨ãŒã§ãã¾ã™ã€‚ è¨˜äº‹ã®ç›®
clavier 2024/09/19
python

study

data

hadoop
ãƒªãƒ³ã‚¯
ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã®æ–°ã—ã„ã‚«ã‚¿ãƒï¼šOpen Table Formatã®ç´¹ä»‹ - æµæ²™æ²³éŽ®
ã¯ã˜ã‚ã« Open Table Formatã¯æ¬¡ä¸–ä»£ã®ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã®åŸºç›¤ã¨ãªã‚Šå¾—ã‚‹æŠ€è¡“ã§ã€å¾ã€…ã«å°Žå…¥äº‹ä¾‹(æœ«å°¾ã«åˆ—æŒ™)ãŒå¢—ãˆã¦ãã¦ã„ã‚‹ã‚‚ã®ã®ã€æ—¥æœ¬ã§ã®èªçŸ¥åº¦ã¯ç™ºå±•é€”ä¸Šãªå°è±¡ãŒã‚ã‚‹ã€‚æœ¬è¨˜äº‹ã§ã¯Open Table Formatç™»å ´ã®èƒŒæ™¯ã‚’ç´¹ä»‹ã™ã‚‹ã€‚åŸ·ç†ã«ã‚ãŸã£ã¦ã€Apache Iceberg: An Architectural Look Under the Coversã¨AWSã«ãŠã‘ã‚‹ Hudi/Iceberg/Delta Lake ã® ä½¿ã„ã©ã“ã‚ã¨é•ã„ã«ã¤ã„ã¦ã‚’ç‰¹ã«å‚è€ƒã«ã—ãŸã€‚ Open Table Formatã¨ã¯ï¼Ÿ Open Table Formatã¨ã¯ã€å¾“æ¥ã®ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã®æŠ€è¡“çš„ãªèª²é¡Œ&ãƒ¦ãƒ¼ã‚¹ã‚±ãƒ¼ã‚¹ã®è¦è«‹ã«å¿œãˆã‚‹å½¢ã§ç™»å ´ã—ãŸã€ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã«æœ€é©åŒ–ã•ã‚ŒãŸãƒ†ãƒ¼ãƒ–ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã‚’æŒ‡ã™æ¦‚å¿µã§ã€ä¸Šæ‰‹ãæ´»ç”¨ã™ã‚‹ã“ã¨ã§ã‚¯ã‚¨ãƒªãƒ—ãƒ©ãƒ³ãƒ‹ãƒ³ã‚°ã€å®Ÿè¡Œæ€§èƒ½ã®æœ€é©åŒ–ã€åŠ¹çŽ‡çš„ãªUpdateã‚„Deleteã€ã‚¿ã‚¤ãƒ
clavier 2023/09/26
data

ã‚ã¨ã§èªã‚€

hadoop

DataLake
ãƒªãƒ³ã‚¯
é«˜æ€§èƒ½åˆ†æ•£SQLã‚¨ãƒ³ã‚¸ãƒ³ã€ŒTrinoã€æœ€é€Ÿã‚¬ã‚¤ãƒ‰ - NTT Communications Engineers' Blog
ã“ã‚“ã«ã¡ã¯ã€‚ãªã‚“ã®å› æžœã‹NTTã‚³ãƒŸãƒ¥ãƒ‹ã‚±ãƒ¼ã‚·ãƒ§ãƒ³ã‚ºã®ã‚¨ãƒãƒ³ã‚¸ã‚§ãƒªã‚¹ãƒˆã‚’ã‚„ã£ã¦ã„ã‚‹è¥¿å¡šã§ã™ã€‚ ã“ã®è¨˜äº‹ã¯ã€NTT Communications Advent Calendar 2021 22æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ 5åˆ†ã§ã‚ã‹ã‚‹ã€ŒTrinoã€ ã€ŒTrinoã€ã¯ã€ç•°ãªã‚‹ãƒ‡ãƒ¼ã‚¿ã‚½ãƒ¼ã‚¹ã«å¯¾ã—ã¦ã‚‚é«˜é€Ÿã§ã‚¤ãƒ³ã‚¿ãƒ©ã‚¯ãƒ†ã‚£ãƒ–ã«åˆ†æžãŒã§ãã‚‹é«˜æ€§èƒ½åˆ†æ•£SQLã‚¨ãƒ³ã‚¸ãƒ³ã§ã™ã€‚ ä»¥ä¸‹ã®ç‰¹å¾´ã‚’æŒã£ã¦ãŠã‚Šã€ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚’æ”¯ãˆã‚‹é‡è¦ãªOSS(ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢)ã®1ã¤ã§ã™ã€‚ SQL-on-Anything: Hadoopã ã‘ã§ãªãå¾“æ¥ã®RDBMS(ãƒªãƒ¬ãƒ¼ã‚·ãƒ§ãƒŠãƒ«ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹)ã‚„NoSQLã¾ã§ã€æ¨™æº–SQL(ANSI SQL)ã«æº–æ‹ ã—ãŸã‚¢ã‚¯ã‚»ã‚¹ã‚’ãƒ¯ãƒ³ã‚¹ãƒˆãƒƒãƒ—ã«æä¾› ä¸¦åˆ—å‡¦ç†ã§ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—ã¦å®¹æ˜“ã«ã‚¹ã‚±ãƒ¼ãƒ«ã‚¢ãƒƒãƒ— ã—ã‹ã‚‚é«˜é€Ÿ(hiveã®æ•°åå€) Netflix, LinkedIn, Salesforce, Shopif
clavier 2022/01/12
sql

database

hadoop
ãƒªãƒ³ã‚¯
lambda-architecture.net
lambda-architecture.net 2024 è‘—ä½œæ¨©. ä¸è¨±è¤‡è£½ ãƒ—ãƒ©ã‚¤ãƒã‚·ãƒ¼ãƒãƒªã‚·ãƒ¼
clavier 2021/11/06
lambda

architecture

hadoop

spark

data

kafka
ãƒªãƒ³ã‚¯
The exabyte club: LinkedInâ€™s journey of scaling the Hadoop Distributed File System
clavier 2021/06/23
hadoop

Java
ãƒªãƒ³ã‚¯
ä¸¦åˆ—åˆ†æ•£å‡¦ç†åŸºç›¤ã®ã„ã¾ï½ž45åˆ†ã§å¦ã¶Hadoop/Spark/Kafka/ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤SWå…¥é–€ï½ž - ã‚»ãƒŸãƒŠãƒ¼ãƒ—ãƒã‚°ãƒ©ãƒ - ã‚ªãƒ¼ãƒ—ãƒ³ã‚½ãƒ¼ã‚¹ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹2020 Online/Kyoto
ä¸¦åˆ—åˆ†æ•£å‡¦ç†åŸºç›¤ã®ã„ã¾ï½ž45åˆ†ã§å¦ã¶Hadoop/Spark/Kafka/ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤SWå…¥é–€ï½ž 2020å¹´8æœˆ28æ—¥(é‡‘)Â 17:15Â ã€œÂ 18:00 OSSãƒ™ãƒ¼ã‚¹ã®åˆ†æ•£å‡¦ç†åŸºç›¤ã¨ã—ã¦Apache HadoopãŒèª•ç”Ÿã—ã¦10ä½™å¹´ãŒçµŒã¡ã€å¤§è¦æ¨¡ä¸¦åˆ—åˆ†æ•£å‡¦ç†ã®é ˜åŸŸã«ãŠã„ã¦ã€ã“ã‚Œã¾ã§ã«å¤šç¨®å¤šæ§˜ãªã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ãŒé–‹ç™ºã•ã‚Œã¦ãã¾ã—ãŸã€‚ æœ¬ã‚»ãƒƒã‚·ãƒ§ãƒ³ã§ã¯ã€ãã‚Œã‚‰ã®ã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ãŒã©ã®ã‚ˆã†ãªçµŒç·¯ã§èª•ç”Ÿã—ã€ã©ã®ã‚ˆã†ã«ä½¿ã‚ã‚Œã‚‹ã®ã‹ã‚’ãŠè©±ã—ã—ã¤ã¤ã€è¿‘å¹´æ³¨ç›®ã‚’é›†ã‚ã¦ã„ã‚‹ãƒ‡ãƒ¼ã‚¿åˆ†æžã‚’æŒ‡å‘ã—ãŸã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤SWã§ã‚ã‚‹Delta Lakeã«ã¤ã„ã¦ã‚‚ã”ç´¹ä»‹ã—ã¾ã™ã€‚
clavier 2020/08/30
hadoop

spark

kafka
ãƒªãƒ³ã‚¯
Uber Engineeringâ€™s Incremental Processing Framework on Hadoop
Youâ€™re seeing information for Japan . To see local features and services for another location, select a different city. Show more With the evolution of storage formats like Apache Parquet and Apache ORC and query engines like Presto and Apache Impala, the Hadoop ecosystem has the potential to become a general-purpose, unified serving layer for workloads that can tolerate latencies of a few minutes
clavier 2020/02/12
hadoop

engineering

Uber

hudi

spark
ãƒªãƒ³ã‚¯
Distributed Deep Learning with Chainer and Hadoop
Oracle Exadata Database Service(Dedicated Infrastructure)ï¼šã‚µãƒ¼ãƒ“ã‚¹æ¦‚è¦ã®ã”ç´¹ä»‹
clavier 2019/12/05
ã‚ã¨ã§èªã‚€

hadoop

chainer

æ©Ÿæ¢°å¦ç¿’

deeplearning

slide
ãƒªãƒ³ã‚¯
å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿æ´»ç”¨å‘ã‘ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤ã‚½ãƒ•ãƒˆã®ã“ã‚Œã¾ã§ã¨ã“ã‚Œã‹ã‚‰ï¼ˆNTTãƒ‡ãƒ¼ã‚¿ ãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ 2019 è¬›æ¼”è³‡æ–™ã€2019/09/05ï¼‰
1. Â© 2019 NTT DATA Corporation 1 Â© 2019 NTT DATA Corporation NTTãƒ‡ãƒ¼ã‚¿ ãƒ†ã‚¯ãƒŽãƒã‚¸ãƒ¼ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ 2019 å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿æ´»ç”¨å‘ã‘ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ¬ã‚¤ãƒ¤ã‚½ãƒ•ãƒˆã® ã“ã‚Œã¾ã§ã¨ã“ã‚Œã‹ã‚‰ 2019å¹´9æœˆ5æ—¥ æ ªå¼ä¼šç¤¾NTTãƒ‡ãƒ¼ã‚¿ ã‚·ã‚¹ãƒ†ãƒ æŠ€è¡“æœ¬éƒ¨ å‰ç”° è€•é™½ ç¦ä¹… ç¢ä¹Ÿ 2. Â© 2019 NTT DATA Corporation 2 â€»Deepãªè©±é¡Œã«èˆˆå‘³ãŒã‚ã‚‹æ–¹ã¯å½“ãƒãƒ¼ãƒ ãƒ¡ãƒ³ãƒã«ã‚ˆã‚‹éŽåŽ»ã®ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã§ã®ç™ºè¡¨è³‡æ–™ã‚‚ã”å‚ç…§ãã ã•ã„ã€‚ ( ã§æ¤œç´¢) æœ¬ã‚»ãƒƒã‚·ãƒ§ãƒ³ã§ãŠä¼ãˆã—ãŸã„ã“ã¨ Hadoopèª•ç”Ÿ(2006å¹´ã”ã‚)ã‹ã‚‰ã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ç³»OSSã‚’ä¸å¿ƒã¨ ã—ãŸæŠ€è¡“ã®ç™ºå±•ã¨ã€æ®‹å˜ã™ã‚‹æŠ€è¡“èª²é¡Œã«å¯¾ã™ã‚‹è¿‘å¹´å‡ºç¾ã—ã¦ ããŸãƒ—ãƒãƒ€ã‚¯ãƒˆã®ã‚¢ãƒ—ãƒãƒ¼ãƒã‚’è‹¥å¹²å¯„ã‚Šé“ã—ãªãŒã‚‰ç´¹ä»‹ã—ã¾ã™ã€‚ â€¢ ãŠè©±ã—ã—ãªã„å†…å®¹ â€¢ å€‹åˆ¥ã®æŠ€è¡“/ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã®â€Deepãªâ€è©±é¡Œ(â€»)
clavier 2019/11/17
hadoop

data

spark
ãƒªãƒ³ã‚¯
Java11å¯¾å¿œã»ã‹ã€Apache Hadoop 3.3.0ã®æ–°æ©Ÿèƒ½ã‚’ç´¹ä»‹
ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10æœˆ1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ãƒ–ãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚LINEãƒ¤ãƒ•ãƒ¼ Tech Blog ã“ã‚“ã«ã¡ã¯ã€ã€ŒHadoopã€é»’å¸¯ã®é¯µå‚(@ajis_ka)ã§ã™ã€‚æœ¬è¨˜äº‹ã§ã¯ã€Apache Hadoopã§æ¬¡ã«ãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã‚‹ãƒžã‚¤ãƒŠãƒ¼ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã§ã‚ã‚‹3.3.0ã§è¿½åŠ ã•ã‚Œã‚‹æ–°æ©Ÿèƒ½ã«ã¤ã„ã¦ç´¹ä»‹ã—ã¾ã™ã€‚æœ¬è¨˜äº‹ã§ã¯ã€ç‰¹ã«Hadoop Commonã¨HDFSï¼ˆHadoop Distributed FileSystemï¼‰ã®æ–°æ©Ÿèƒ½ã‚’é‡ç‚¹çš„ã«ç´¹ä»‹ã—ã¤ã¤ã€ãã‚Œã‚‰ã®æ©Ÿèƒ½ã«å¯¾ã™ã‚‹ãƒ¤ãƒ•ãƒ¼ã®è²¢çŒ®ã«ã¤ã„ã¦ã‚‚è§¦ã‚Œã¦ã„ãã¾ã™ã€‚ Hadoop 3.3.0 æ¦‚è¦ Hadoop 3.3.0ã¯ãŠãã‚‰ã2019å¹´ã®å¹´æœ«ã¾ã§ã«ãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã‚‹äºˆå®šã§ã€ã“ã®è¨˜äº‹ã‚’åŸ·ç†ã—ã¦ã„ã‚‹10æœˆ23æ—¥æ™‚ç‚¹ã§ã™ã§ã«1500ä»¶ä»¥ä¸Šã®ä¿®æ£ãŒå…¥ã£ã¦ã„ã¾ã™(Hadoop 3.2.0ã¯1089ä»¶)ã€‚ã¤ã¾ã‚Šã€
clavier 2019/11/12
hadoop

java
ãƒªãƒ³ã‚¯
æ–°ã—ã„ã‚°ãƒ©ãƒ•å‡¦ç†ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã€ŒSpark Graphã€ã¨ã¯ä½•ã‹ï¼Ÿã€€Apache Spark 2.4 & 3.0ã®æ–°æ©Ÿèƒ½ã‚’è§£èª¬ Part2
2019å¹´3æœˆ19æ—¥ã€Data Engineering MeetupãŒä¸»å‚¬ã™ã‚‹ã‚¤ãƒ™ãƒ³ãƒˆã€ŒData Engineering Meetup #1ã€ãŒé–‹å‚¬ã•ã‚Œã¾ã—ãŸã€‚ãƒ‡ãƒ¼ã‚¿ã®åŽé›†ã‚„ç®¡ç†ã€å‡¦ç†ã€å¯è¦–åŒ–ãªã©ã€ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ã«é–¢ã™ã‚‹æŠ€è¡“ã®æƒ…å ±ã‚’å…±æœ‰ã™ã‚‹æœ¬ã‚¤ãƒ™ãƒ³ãƒˆã€‚ãƒ‡ãƒ¼ã‚¿ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãƒªãƒ³ã‚°ã®æœ€å‰ç·šã§æ´»èºã™ã‚‹ã‚¨ãƒ³ã‚¸ãƒ‹ã‚¢ãŸã¡ãŒé›†ã„ã€è‡ªèº«ã®çŸ¥è¦‹ã‚’å…±æœ‰ã—ã¾ã™ã€‚ãƒ—ãƒ¬ã‚¼ãƒ³ãƒ†ãƒ¼ã‚·ãƒ§ãƒ³ã€ŒSpark 2.4 & 3.0 - What's next? - ã€ã«ç™»å£‡ã—ãŸã®ã¯ã€æ ªå¼ä¼šç¤¾ã‚¨ãƒŒãƒ»ãƒ†ã‚£ãƒ»ãƒ†ã‚£ãƒ»ãƒ‡ãƒ¼ã‚¿ã®çŒ¿ç”°æµ©è¼”æ°ã€‚è¬›æ¼”è³‡æ–™ã¯ã“ã¡ã‚‰ Accelerator Aware Scheduling çŒ¿ç”°æµ©è¼”æ°ï¼ˆä»¥ä¸‹ã€çŒ¿ç”°ï¼‰ï¼šBarrier Execution Modeã®ãŠè©±ã¯ã“ã“ã¾ã§ã§ã€æ¬¡ã¯Accelerator Aware Schedulingã§ã™ãã€‚æœ€è¿‘Project Hydrogenã®ä¸ã§ã¯ã“ã®æ©Ÿèƒ½ã®è°è«–ãŒæ´»ç™ºã§ã€S
clavier 2019/07/11
hadoop

spark
ãƒªãƒ³ã‚¯
Apache Hadoopã‚¨ã‚³ã‚·ã‚¹ãƒ†ãƒ ã«ãŠã‘ã‚‹ã€ç•°ãªã‚‹ãƒ•ã‚¡ã‚¤ãƒ«å½¢å¼ã¨ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã‚¨ãƒ³ã‚¸ãƒ³ã®ãƒ‘ãƒ•ã‚©ãƒ¼ãƒžãƒ³ã‚¹æ¯”è¼ƒ
è‘—è€…/Author: Zbigniew Branowski (Cern) åŽŸæ–‡/Originalï¼šhttp://blog.cloudera.com/blog/2017/02/performance-comparing-of-different-file-formats-and-storage-engines-in-hadoop-file-system/ Zbigniew Baranowskiã¯ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã‚·ã‚¹ãƒ†ãƒ ã®å°‚é–€å®¶ã§ã‚ã‚Šã€CERNã§ã‚»ãƒ³ãƒˆãƒ©ãƒ«ãƒ‡ãƒ¼ã‚¿ãƒ™ãƒ¼ã‚¹ã¨Hadoopãƒ™ãƒ¼ã‚¹ã®ã‚µãƒ¼ãƒ“ã‚¹ã‚’æä¾›ã€ã‚µãƒãƒ¼ãƒˆã—ã¦ã„ã‚‹ã‚°ãƒ«ãƒ¼ãƒ—ã®ãƒ¡ãƒ³ãƒãƒ¼ã§ã™ã€‚ ã“ã®ãƒ–ãƒã‚°ã¯ã‚‚ã¨ã‚‚ã¨CERNã®ã€ŒDatabases at CERNã€ãƒ–ãƒã‚°ã§å…¬é–‹ã•ã‚Œã¦ãŠã‚Šã€CERNã®è¨±å¯ã‚’å¾—ã¦ã“ã“ã§å…¬é–‹ã•ã‚Œã¦ã„ã¾ã™ã€‚ ãƒˆãƒ”ãƒƒã‚¯ã“ã®è¨˜äº‹ã§ã¯ã€Apache Hadoopã‚¨ã‚³ã‚·ã‚¹ãƒ†ãƒ ã§åˆ©ç”¨å¯èƒ½ãªã„ãã¤ã‹ã®ä¸€èˆ¬çš„ãªãƒ‡ãƒ¼ã‚¿ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã¨ã‚¹ãƒˆãƒ¬ãƒ¼
clavier 2019/06/21
hadoop

parquet

benchmark

filesystem

performance
ãƒªãƒ³ã‚¯
Partition Management in Hadoop | Cloudera Blog
Guest blog post written by Adir Mashiach In this post Iâ€™ll talk about the probl em of Hive tables with a lot of small partitions and files and describe my solution in details. A little background In my organization, â€Šwe keep a lot of our data in HDFS. Most of it is the raw data but a significant amount is the final product of many data enrichment processes. In order to manage all the data pipelines
clavier 2019/06/09
hadoop

data
ãƒªãƒ³ã‚¯
Marmaray: An Open Source Generic Data Ingestion and Dispersal Framework and Library for Apache Hadoop
Marmaray: An Open Source Generic Data Ingestion and Dispersal Framework and Library for Apache Hadoop Connecting users worldwide on our platform all day, every day requires an enormous amount of data management. When you consider the hundreds of operations and data science teams analyzing large sets of anonymous, aggregated data, using a variety of different tools to better understand and maintain
clavier 2019/05/26
hadoop

Uber
ãƒªãƒ³ã‚¯
DBEvents: A Standardized Framework for Efficiently Ingesting Data into Uber's Apache Hadoop Data Lake
Youâ€™re seeing information for Japan . To see local features and services for another location, select a different city. Show more Data / MLDBEvents: A Standardized Framework for Efficiently Ingesting Data into Uberâ€™s Apache Hadoop Data LakeMarch 14, 2019 / Global Keeping the Uber platform reliable and real-time across our global markets is a 24/7 business. People may be going to sleep in San Franc
clavier 2019/05/26
hadoop

uber

data
ãƒªãƒ³ã‚¯
ãƒŠãƒ“ã‚¿ã‚¤ãƒ ã‚¸ãƒ£ãƒ‘ãƒ³ã«é›†ã¾ã‚‹ç§»å‹•ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã®åˆ†æžåŸºç›¤
2019/04/25ã«é–‹å‚¬ã•ã‚ŒãŸã€Žãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤Developers Nightã€œ3ç¤¾3æ§˜åˆ†æžåŸºç›¤ã®å¤‰é·ã€œã€ã®è¬›æ¼”è³‡æ–™ã§ã™ã€‚
clavier 2019/05/09
hadoop

aws

ãƒ‡ãƒ¼ã‚¿åŸºç›¤
ãƒªãƒ³ã‚¯
HDFSã®ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£ã®é™ç•Œã‚’çªç ´ã™ã‚‹ãŸã‚ã®ã•ã¾ã–ã¾ãªå–ã‚Šçµ„ã¿ | Hadoop / Spark Conference Japan 2019 #hcj2019
HDFSã®ã‚¹ã‚±ãƒ¼ãƒ©ãƒ“ãƒªãƒ†ã‚£ã®é™ç•Œã‚’çªç ´ã™ã‚‹ãŸã‚ã®ã•ã¾ã–ã¾ãªå–ã‚Šçµ„ã¿ | Hadoop / Spark Conference Japan 2019 #hcj2019
clavier 2019/05/05
hadoop

hdfs
ãƒªãƒ³ã‚¯
æ›¸ç±æƒ…å ±: Architecting Modern Data Platforms: A Guide to Enterprise Hadoop at Scale 1st Edition
æ–°å¹´æ˜Žã‘ã¾ã—ã¦ãŠã‚ã§ã¨ã†ã”ã–ã„ã¾ã™ã€‚ æ–°å¹´æ—©ã€…ã«Clouderaã¨HortonworksãŒåˆä½µã¨ã„ã†ãƒ‹ãƒ¥ãƒ¼ã‚¹ãŒã‚ã‚Šã¾ã—ãŸãŒã€å¼•ãç¶šãé ‘å¼µã‚Šã¾ã™ã®ã§ä»Šå¹´ã‚‚ã‚ˆã‚ã—ããŠé¡˜ã„ã„ãŸã—ã¾ã™ã€‚ ã•ã¦ã€å¹´æœ«ã«ä½•å†Šã‹æŠ€è¡“æ›¸ã‚’ç©èªãƒªã‚¹ãƒˆã«å…¥ã‚Œã¦ã„ãŸã†ã¡ã®ä¸€å†Šã‚’ç´¹ä»‹ã—ã¾ã™ã€‚å‰²ã¨è©•åˆ¤ãŒè‰¯ã„ã‚ˆã†ã§ã€è±¡æœ¬ã®è‘—è€…ã®Tom Whiteã‹ã‚‰ã‚‚æŽ¨è–¦(?)ã•ã‚Œã¦ã„ã‚‹æ¨¡æ§˜ã€‚
clavier 2019/04/21
hadoop

book
ãƒªãƒ³ã‚¯
OASIS - Data Analysis Platform for Multi-tenant Hadoop Cluster
OASIS - Data Analysis Platform for Multi-tenant Hadoop Cluster OASIS is a web-based data analysis platform that enables employees at LINE to analyze data from their services in a multi-tenant Hadoop cluster. It provides features like query execution, result visualization, code execution in Scala/Python/R, scheduled execution, and result sharing. OASIS addresses security, stability, and feature req
clavier 2019/02/17
hadoop
ãƒªãƒ³ã‚¯
ãƒ¤ãƒ•ãƒ¼ç™ºã® OSS ã€Multiple-Dimension-Spreadï¼ˆMDSï¼‰ã®ç´¹ä»‹
ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã¯ã€2023å¹´10æœˆ1æ—¥ã«LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã«ãªã‚Šã¾ã—ãŸã€‚LINEãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ã®æ–°ã—ã„ãƒ–ãƒã‚°ã¯ã“ã¡ã‚‰ã§ã™ã€‚LINEãƒ¤ãƒ•ãƒ¼ Tech Blog ã¯ã˜ã‚ã« ã“ã‚“ã«ã¡ã¯ã€‚ ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ ãƒ‡ãƒ¼ã‚¿ï¼†ã‚µã‚¤ã‚¨ãƒ³ã‚¹ã‚½ãƒªãƒ¥ãƒ¼ã‚·ãƒ§ãƒ³çµ±æ‹¬æœ¬éƒ¨ãƒ‡ãƒ¼ã‚¿ãƒ—ãƒ©ãƒƒãƒˆãƒ•ã‚©ãƒ¼ãƒ æœ¬éƒ¨ãƒ‡ãƒ¼ã‚¿ãƒ‡ãƒªãƒãƒªãƒ¼éƒ¨ã®äº•å³¶ï¼†å¤§æˆ¸ã§ã™ã€‚ ä»Šå›žã¯ã€ãƒ¤ãƒ•ãƒ¼æ ªå¼ä¼šç¤¾ï¼ˆä»¥ä¸‹ãƒ¤ãƒ•ãƒ¼ï¼‰ãŒ OSS ã¨ã—ã¦å…¬é–‹ã—ãŸã‚«ãƒ©ãƒ ãƒŠã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ•ã‚¡ã‚¤ãƒ«ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã® Multiple-Dimension-Spread ã«ã¤ã„ã¦ã€é–‹ç™ºã®èƒŒæ™¯ã‚’äº¤ãˆã¦ç´¹ä»‹ã—ã¾ã™ã€‚ Multiple-Dimension-Spreadã¨ã¯ Multiple-Dimension-Spreadï¼ˆä»¥ä¸‹MDSï¼‰ã¯ãƒ¤ãƒ•ãƒ¼ãŒé–‹ç™ºã—ãŸã‚«ãƒ©ãƒ ãƒŠã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ãƒ•ã‚©ãƒ¼ãƒžãƒƒãƒˆã§ã™ã€‚ å¤§è¦æ¨¡ãªãƒ‡ãƒ¼ã‚¿ã‚’è“„ãˆã¦ãŠãä»•çµ„ã¿ã‚’æ¹–ã¨ã—ã¦æ‰ãˆãŸã‚‚ã®ã‚’ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã¨ã„ã„ã¾ã™ã€‚ MDS ã¯ãƒ‡ãƒ¼ã‚¿ãƒ¬ã‚¤ã‚¯ã«ãƒ‡ãƒ¼ã‚¿ã‚’ä¿å˜ã€åˆ©
clavier 2019/01/09
hadoop
ãƒªãƒ³ã‚¯
1 2 3 4 5 6 7 æ¬¡ã®ãƒšãƒ¼ã‚¸