20160322 bdi
20160322 bdi 1. Spark as a Compiler + SQL Codegen Researches 2016.3.22 maropu@BDI 2. SparkSQL codegen h<ps://issues.apache.org/jira/browse/SPARK-Ââ€12795 •  Sparkers currently developing whole-Ââ€stage codegen under a JIRA Pcket, SPARK-Ââ€12795 –  a part of the Project Tungsten bringing Spark to bare-Ââ€metal –  fusing a sub-Ââ€tree of operators (stages) into a single efficient funcPon •  A quick overvie
「Hadoop/Spark Conference Japan 2016ã€ã§å°æ²¢å¥å²PMCãŒèªžã£ãŸâ€œYARNã®ã„ã¾â€ | gihyo.jp
「Hadoop/Spark Conference Japan 2016ã€ã§å°æ²¢å¥å²PMCãŒèªžã£ãŸâ€œYARNã®ã„ã¾â€ 2013å¹´10月ã«Hadoop 2.2ãŒæ£å¼ãƒªãƒªãƒ¼ã‚¹ã•ã‚Œã¦ä»¥æ¥ã€Hadoopã®ä¸–ç•Œã¯å¤§ãã変ã‚ã‚Šã¾ã—ãŸã€‚ãã‚Œã¾ã§ã®æ¦‚念ã§ã‚る「Hadoopï¼ HDFS＋MapReduceã€ãŒåˆ·æ–°ã•ã‚Œã€MapReduceãŒå—ã‘æŒã£ã¦ã„ãŸMapデータ処ç†ã¨ã‚¯ãƒ©ã‚¹ã‚¿ãƒªã‚½ãƒ¼ã‚¹ç®¡ç†ãŒåˆ†é›¢ã§ãるよã†ã«ãªã‚Šã¾ã—ãŸã€‚ã“ã‚Œã«ã‚ˆã‚Šã€Sparkãªã©MapReduce以外ã®ãƒ‡ãƒ¼ã‚¿å‡¦ç†ã‚¨ãƒ³ã‚¸ãƒ³ãŒHadoop上ã§åˆ©ç”¨å¯èƒ½ã«ãªã‚Šã¾ã—ãŸã€‚ ãã—ã¦Hadoopã®ã‚¯ãƒ©ã‚¹ã‚¿ãƒªã‚½ãƒ¼ã‚¹ç®¡ç†ã‚’è¡Œã†ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã¨ã—ã¦ã€æ–°ãŸãªãƒ‡ãƒ•ã‚¡ã‚¯ãƒˆã‚¹ã‚¿ãƒ³ãƒ€ãƒ¼ãƒ‰ã®åœ°ä½ã‚’確立ã—よã†ã¨ã—ã¦ã„ã‚‹ã®ãŒã€ŒApache Hadoop YARN（Yet Another Resource Negotiator： YARNâ ）â ã€ã§ã™ã€‚YARNã®ç™»å ´ã¯ä¸¦åˆ—分散処ç†ã«ã©ã‚“
Hadoopユーザー会ãŒ2月8æ—¥ã«ã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã€Sparkã®ä¸»è¦é–‹ç™ºè€…ãªã©ãŒè¬›æ¼”
日本Hadoopユーザー会ãŒ2016å¹´2月8æ—¥ã«æ±äº¬éƒ½å“å·åŒºã§ã€ã‚ªãƒ¼ãƒ—ンソースソフトウエア（OSS）ã®åˆ†æ•£å‡¦ç†ã‚½ãƒ•ãƒˆã€ŒHadoopã€ã‚„「Sparkã€ã®ãƒ¦ãƒ¼ã‚¶ãƒ¼ä¼šè°ã€ŒHadoop / Spark Conference Japan 2016ã€ã‚’開催ã™ã‚‹ï¼ˆå‹Ÿé›†ã‚µã‚¤ãƒˆï¼‰ã€‚å‚åŠ è²»ã¯ç„¡æ–™ã€‚ç±³Clouderaã‚„ç±³Databricksã®é–‹ç™ºè€…ã‚„ã€Hadoop/Sparkã®æ—¥æœ¬äººã‚³ãƒŸãƒƒã‚¿ã€ãƒ¦ãƒ¼ã‚¶ãƒ¼ä¼æ¥ãªã©ãŒè¬›æ¼”ã™ã‚‹ã€‚ 2009å¹´ã«ç¬¬1回ãŒé–‹å‚¬ã•ã‚ŒãŸåŒã‚«ãƒ³ãƒ•ã‚¡ãƒ¬ãƒ³ã‚¹ã¯ä»Šå›žãŒ6回目ã§ã€ä»Šå›žã‹ã‚‰ã€ŒHadoop Conference Japan 2016ã€ã¨ã€ŒSpark Conference Japan 2016ã€ã®ä½µå‚¬ã¨ãªã£ãŸã€‚Sparkã¯ã‚«ãƒªãƒ•ã‚©ãƒ«ãƒ‹ã‚¢å¤§å¦ãƒãƒ¼ã‚¯ãƒ¬ãƒ¼æ ¡ã®ã€ŒAMPLabã€ã‹ã‚‰ç”Ÿã¾ã‚ŒãŸOSSã§ã€Hadoopã®å¾Œç¶™ã‚½ãƒ•ãƒˆã¨ã—ã¦ç±³å›½ã ã‘ã§ãªã日本ã§ã‚‚利用ãŒåºƒãŒã‚Šå§‹ã‚ã¦ã„る。 カンファレンスã§ã¯Hadoo
YARNã€HDFSã€ãã—ã¦Sparkã®å°†æ¥åƒã¨ã¯ï¼šã€ŒHadoop/Spark Conã€åŸºèª¿è¬›æ¼”
å°åˆ·ã™ã‚‹ メールã§é€ã‚‹ テã‚スト HTML é›»åæ›¸ç± PDF ダウンãƒãƒ¼ãƒ‰ テã‚スト é›»åæ›¸ç± PDF クリップã—ãŸè¨˜äº‹ã‚’Myページã‹ã‚‰èªã‚€ã“ã¨ãŒã§ãã¾ã™ 日本Hadoopユーザー会ã¯2月8æ—¥ã€æ±äº¬éƒ½å“å·åŒºã§ã€ŒHadoop Conference Japan 2016ã€ã‚’開催ã—ãŸã€‚第6回目ã¨ãªã‚‹ä»Šå›žã®ã‚¤ãƒ™ãƒ³ãƒˆã§ã¯ã€ŒSpark Conference Japan 2016ã€ãŒåˆã‚ã¦ä½µå‚¬ã•ã‚Œã€ã‚ーノートã«ã¯Apache Sparkã®ä¸»è¦é–‹ç™ºè€…ã§ã‚ã‚‹Xin Reynoldæ°ã‚‚登壇。2016å¹´ã«ãƒªãƒªãƒ¼ã‚¹äºˆå®šã®Sparkã®æ¬¡æœŸãƒãƒ¼ã‚¸ãƒ§ãƒ³ã€ŒSpark 2.0ã€ã®æœ€æ–°æƒ…å ±ã‚’ç´¹ä»‹ã—ãŸã€‚今回ã®å‚åŠ ç™»éŒ²è€…æ•°ã¯1347人ã§ã€ãã®ã†ã¡63%ãŒåˆå‚åŠ ã ã¨ã„ã†ã€‚ã“ã®è¨˜äº‹ã§ã¯ã€åŒã‚¤ãƒ™ãƒ³ãƒˆã®ã‚ーノートã®æ§˜åをレãƒãƒ¼ãƒˆã™ã‚‹ã€‚ Apache Hadoopã¯ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã®åˆ†æ•£å‡¦ç†ã‚’è¡Œã†ã‚ªãƒ¼ãƒ—ソースã®ãƒŸãƒ‰ãƒ«ã‚¦ã‚§ã‚¢ã€‚分散ファイルシス
Hadoop / Spark Conference Japan 2016ã€2月8æ—¥ã«é–‹å‚¬ | gihyo.jp
日本Hadoopユーザー会ã¯ã€Apache Hadoopã€ãŠã‚ˆã³Apache Sparkã«é–¢ã™ã‚‹ã‚¤ãƒ™ãƒ³ãƒˆã€ŒHadoop Conference Japan 2016ã€ã€Œâ Spark Conference Japan 2016ã€ã‚’2月8日（月）ã«æ±äº¬ã€å¤§äº•ç”ºã®ãã‚…ã‚Šã‚ã‚“ã«ã¦é–‹å‚¬ã™ã‚‹ã€‚国内外ã®ã‚¨ã‚スパートã«ã‚ˆã‚‹Hadoopや並列分散処ç†ã«é–¢ã™ã‚‹æœ€æ–°ã®è¬›æ¼”やセッションãŒå¤šæ•°äºˆå®šã•ã‚Œã¦ã„ã‚‹ã€‚å…¥å ´ã¯ç„¡æ–™ã€‚å‚åŠ ç”³ã—è¾¼ã¿ã€ãƒ—ãƒã‚°ãƒ©ãƒ ã®å†…容ã¯ã‚¤ãƒ™ãƒ³ãƒˆãƒšãƒ¼ã‚¸ã‹ã‚‰ã€‚ 開催概è¦ã¯ä»¥ä¸‹ã®ã¨ãŠã‚Šã€‚
[SPARK-10324] MLlib 1.6 Roadmap - ASF JIRA
Following SPARK-8445, we created this master list for MLlib features we plan to have in Spark 1.6. Please view this list as a wish list rather than a concrete plan, because we don't have an accurate estimate of available resources. Due to limited review bandwidth, features appearing on this list will get higher priority during code review. But feel free to suggest new items to the list in comments
[SPARK-7078] Cache-aware binary processing in-memory sort - ASF JIRA
A cache-friendly sort algorithm that can be used eventually for: sort-merge join shuffle See the old alpha sort paper: http://research.microsoft.com/pubs/68249/alphasort.doc Note that state-of-the-art for sorting has improved quite a bit, but we can easily optimize the sorting algorithm itself later.
elliptium.net
elliptium.net 2024 著作権. ä¸è¨±è¤‡è£½ プライãƒã‚·ãƒ¼ãƒãƒªã‚·ãƒ¼
ランã‚ング
ランã‚ング
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
Five cool ways the JVM can run Apache Spark faster
Spark Pull Requests
Sparkã¨Hadoopã¯å‹ã ã¡ã§ã‚ã‚‹ã€æ•µã§ã¯ãªã„ | TechCrunch Japan
[SPARK-7079] Cache-aware external sort - ASF JIRA
Real-time Recommendations using Spark Comcast Labs
Deeplearning 4 j ã®ã‚¯ã‚¤ãƒƒã‚¯ã‚¹ã‚¿ãƒ¼ãƒˆã‚¬ã‚¤ãƒ‰ - Deeplearning4j: Open-source, Distributed Deep Learning for the JVM
Project Tungsten: Bringing Apache Spark Closer to Bare Metal
https://people.eecs.berkeley.edu/~keo/publications/nsdi15-final147.pdf
https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F13/projects/reports/project16_report.pdf
Apache Spark the Fastest Open Source Engine for Sorting a Petabyte
GitHub - databricks/spark-perf: Performance tests for Apache Spark
Spark at Twitter - Seattle Spark Meetup, April 2014
{{#tags}}- {{label}}
{{/tags}}