Using Apache Parquet Data Files with CDH | 6.3.x | Cloudera Documentation
Apache Parquet is a columnar storage format available to any component in the Hadoop ecosystem, regardless of the data processing framework, data model, or programming language. The Parquet file format incorporates several features that support data warehouse-style operations: Columnar storage layout - A query can examine and perform calculations on all values for a column while reading only a sma
Cloudera Blog
We are excited to announce the acquisition of Octopai, a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. Cloudera’s mission since its inception has been to empower organizations to transform all their data to deliver trusted, valuable, and predictive insights. With AI and […] Read blog post
Dremel made simple with Parquet
Columnar storage is a popular technique to optimize analytical workloads in parallel RDBMs. The performance and compression benefits for storing and processing large amounts of data are well documented in academic literature as well as several commercial analytical databases. The goal is to keep I/O to a minimum by reading from a disk only the data required for the query. Using Parquet at Twitter,
RCFile，Parquet，ORCFile
ã“ã®2ヶ月ã§ï¼ŒCloudera/Twitter，Hortonworks ã‹ã‚‰ãã‚Œãžã‚Œåˆ¥ã®åˆ—指å‘ファイルフォーマットãŒå…¬é–‹ã•ã‚Œã¾ã—ãŸï¼ŽParquet 㨠ORCFile ã§ã™ï¼Ž ã“ã®è¨˜äº‹ã§ã¯ï¼Œã¾ãš RCFile ã®å¾©ç¿’ã‚’ã—ã¦ï¼Œãã®å¾Œ Parquet 㨠ORCFile ãã‚Œãžã‚Œã®å…±é€šç‚¹ã¨é•ã„ã‚’ãŠãŠã¾ã‹ã«è¦‹ã¦ã„ã“ã†ã¨æ€ã„ã¾ã™ï¼Žã‚³ãƒ¼ãƒ‰ãƒ¬ãƒ™ãƒ«ã®è©³ç´°ãªé•ã„ã«ã¤ã„ã¦ã¯ï¼Œæ¬¡å›žä»¥é™ã§è¦‹ã¦ã„ãã¾ã™ï¼Ž RCFile ã®å¾©ç¿’ RCFile ã¯ã€€Record Columnar File ã®ç•¥ã§ï¼ŒHive ã‹ã‚‰åˆ©ç”¨ã§ãるストレージフォーマットã§ã™ï¼Žç‰¹ã«ï¼ŒHDFS ã‚„ S3 ã¨ã„ã£ãŸåˆ†æ•£ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ä¸Šã§ãƒ‘フォーマンスãŒã§ã‚‹ã‚ˆã†ã«è¨è¨ˆã•ã‚Œã¦ã„ã¾ã™ï¼Ž HDFS/S3 ã¨ã„ã£ãŸã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã§ã¯ï¼ŒåŸºæœ¬çš„ã«ãƒ‡ãƒ¼ã‚¿ã‚’計算機間ã§åŒã˜è² è·ã«ãªã‚‹ã‚ˆã†ã«ãƒ‡ãƒ¼ã‚¿ã‚’分散é…ç½®ã—ã¾ã™ï¼Žã“ã®ãŸã‚，従æ¥ã®åˆ—指å‘ストレージフォーマットã®ã‚ˆã†ã«é©å½“ã«åˆ—毎ã«
Parquet Hadoop Summit 2013
Parquet is a columnar storage format for Hadoop data. It was developed by Twitter and Cloudera to optimize storage and querying of large datasets. Parquet provides more efficient compression and I/O compared to traditional row-based formats by storing data by column. Early results show a 28% reduction in storage size and up to a 114% improvement in query performance versus the original Thrift form
Twitter: データ分æžåŸºç›¤æ”¹å–„å–り組㿠- ワザノム| wazanova.jp
https://www.facebook.com/photo.php?v=10151697364230687&set=vb.9445547199&type=2&theater Twitterã®Analyticsインフラãƒãƒ¼ãƒ ãŒã€ãƒ‡ãƒ¼ã‚¿åˆ†æžåŸºç›¤ã®æ”¹å–„ã«å–り組んã§ããŸäº‹ä¾‹ã‚’紹介ã—ã¦ã„ã¾ã™ã€‚ 1) 背景 ４億tweet/日を発信 & 消費ã—ã¦ã„るユーザã®ã‚¢ã‚¯ãƒ†ã‚£ãƒ“ティをã€Twitter社内ã®å¤šãã®ãƒãƒ¼ãƒ ãŒãã‚Œãžã‚Œã®è¦³ç‚¹ & 様々ãªåˆ©ç”¨å½¢æ…‹ã§åˆ†æžãƒ‡ãƒ¼ã‚¿ã‚’å¿…è¦ã¨ã™ã‚‹ãŸã‚ã€é‡ãŠã‚ˆã³ãƒ‡ãƒ¼ã‚¿ã®ä¾å˜é–¢ä¿‚ãŒã€ç›¸å½“大ãã複雑ãªã‚‚ã®ã«ãªã£ã¦ã„る。Analyticsインフラã¯ã€1000ノードã‚ã‚‹Hadoopã®ã‚¯ãƒ©ã‚¹ã‚¿ã‚’ã„ãã¤ã‹ã‚‚ã¤è¦æ¨¡ã€‚ ストレージフットプリント & I/Oを減らã™ã ã‘ã§ãªãã€ä»–ã®æ–¹æ³•ã§ãƒ—ãƒã‚»ã‚¹ã‚¹ãƒ”ードをã‚ã’ã‚‹ã“ã¨ã«å–り組んã§ã„る。 2) Parquet （「Hadoop用ã®ã‚«ãƒ©ãƒ ナストレージフォー
Hadoop用カラムナストレージ「Parquetã€æ£å¼ç‰ˆã‚’TwitterãŒã‚ªãƒ¼ãƒ—ンソースã§å…¬é–‹
データを列方å‘ã«æ ¼ç´ã™ã‚‹ã“ã¨ã§èªã¿å‡ºã—性能をå‘上ã—ã€é«˜é€Ÿãªåˆ†æžã‚’実ç¾ã™ã‚‹æŠ€è¡“ã¯ã€ã€Œã‚«ãƒ©ãƒ 型データベースã€ã€Œã‚«ãƒ©ãƒ ナーストレージã€ã€Œã‚«ãƒ©ãƒ 型データストアã€ãªã©ã¨å‘¼ã°ã‚Œã¦æ³¨ç›®ã•ã‚Œã¦ã„ã¾ã™ã€‚ãã®æŠ€è¡“ã‚’Hadoopã®ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã«æŒãŸã›ã‚‹ã“ã¨ã§ã€Hadoopã§ã‚‚ã•ã‚‰ã«é«˜é€Ÿãªåˆ†æžã‚’å¯èƒ½ã«ã™ã‚‹ã€ŒParquetã€ãƒãƒ¼ã‚¸ãƒ§ãƒ³1.0ã‚’ã€TwitterãŒã‚ªãƒ¼ãƒ—ンソースã§å…¬é–‹ã—ã¾ã—ãŸã€‚ 公開ã—ãŸã®ã¯7月30æ—¥ã¨1カ月ã»ã©å‰ã®ã“ã¨ã§æ°—付ãã®ãŒå°‘々é…ã‹ã£ãŸã®ã§ã™ãŒã€ã»ã‹ã«æ—¥æœ¬èªžã®è¨˜äº‹ãŒè¦‹å½“ãŸã‚‰ãªã‹ã£ãŸã®ã§ç´¹ä»‹ã—ãŸã„ã¨æ€ã„ã¾ã™ã€‚ Parquetã¨ã¯ã©ã®ã‚ˆã†ãªã‚½ãƒ•ãƒˆã‚¦ã‚§ã‚¢ãªã®ã‹ã€Twitterã®ãƒ–ãƒã‚°ã‹ã‚‰å°‘ã—é•·ã‚ã®èª¬æ˜Žã‚’引用ã—ã¾ã—ょã†ã€‚ Parquet is an open-source columnar storage format for Hadoop. Its goal is to provide a state
1
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Read few parquet files at the same time in Spark
Cannot saveAsParquetFile from a RDD of case class
[SPARK-3368] Spark cannot be used with Avro and Parquet - ASF JIRA
Apache Spark User List - Kafka->HDFS to store as Parquet format
parquet-compatibility/parquet-testdata/tpch at master · Parquet/parquet-compatibility
parquet-compatibility/parquet-compat/src/test/java/parquet/compat/test/ConvertUtils.java at master · Parquet/parquet-compatibility
{{#tags}}- {{label}}
{{/tags}}