NetflixãŒOSS化ã—ãŸScala/Spark用ノートブックPolynoteを試ã—ã¦ã¿ã‚‹ - ãŸã‘ãžã†ç€•æ»ãƒ–ãƒã‚°
ã‚‚ã†1年以上å‰ã®è©±ã«ãªã‚Šã¾ã™ãŒã€NetflixãŒSpark対応ã®Scala用ノートブックPolynoteã‚’OSS化ã—ãŸã¨ã„ã†è©±ãŒã‚ã‚Šã¾ã—ãŸã€‚ netflixtechblog.com æ—¢å˜ã®ãƒŽãƒ¼ãƒˆãƒ–ックã§ã¯Scalaを使ã£ã¦ã„ã¦ã‚‚コード補完ãªã©ãŒã‚ã¾ã‚ŠåŠ¹ã‹ãªã„ã‚‚ã®ãŒå¤šãã€ã¾ã¨ã¾ã£ãŸã‚³ãƒ¼ãƒ‰ã‚’書ãã¨ãã¯çµå±€IDEを使ã†ã¨ã„ã†æ„Ÿã˜ã«ãªã‚ŠãŒã¡ãªã®ã§ã™ãŒã€Polynoteã¯Scalaを第一言語ã¨ã—ã¦ã‚µãƒãƒ¼ãƒˆã™ã‚‹çã—ã„ノートブックã§ã€ã‚³ãƒ¼ãƒ‰è£œå®Œãªã©ã®æ©Ÿèƒ½ã‚‚充実ã—ã¦ã„るよã†ãªã®ã§é…ã‚Œã°ã›ãªãŒã‚‰è©¦ã—ã¦ã¿ã¾ã—ãŸã€‚ インストール Sparkを使ã†å ´åˆã€ã¾ãšã¯å…ˆã«Sparkをインストールã—ã¦ãŠãå¿…è¦ãŒã‚ã‚Šã¾ã™ã€‚ã¨ã‚Šã‚ãˆãšãƒãƒ¼ã‚«ãƒ«ãƒ¢ãƒ¼ãƒ‰ã§å‹•ã‹ã™ã ã‘ã§ã‚ã‚Œã°Sparkã®ãƒªãƒªãƒ¼ã‚¹ãƒ‡ã‚£ã‚¹ãƒˆãƒªãƒ“ューションをダウンãƒãƒ¼ãƒ‰ã—ã¦é©å½“ãªãƒ‡ã‚£ãƒ¬ã‚¯ãƒˆãƒªã«å±•é–‹ã—ã¦ãŠãã ã‘ã§OKã§ã™ãŒã€Polynoteã¯å†…部的ã«spark-submitコ
Apache Spark用ã®ã‚¹ã‚¿ãƒ³ãƒ‰ã‚¢ãƒãƒ³JDBCドライム- ãŸã‘ãžã†ç€•æ»ãƒ–ãƒã‚°
Sparkã«JDBCã§ã‚¢ã‚¯ã‚»ã‚¹ã™ã‚‹ã«ã¯Thriftserverを入れãŸã‚ŠHive MetastoreãŒå¿…è¦ã ã£ãŸã‚Šã§è‰²ã€…é¢å€’ãªã®ã§ã™ãŒã€å˜ä½“ã§åˆ©ç”¨å¯èƒ½ãªæ–¹æ³•ã¯ãªã„ã®ã‹ãªã¨æ€ã£ã¦æŽ¢ã—ã¦ã¿ãŸã¨ã“ã‚ã€ä»¥ä¸‹ã®ã‚‚ã®ã‚’見ã¤ã‘ãŸã®ã§è©¦ã—ã¦ã¿ã¾ã—ãŸã€‚ github.com ã“ã®JDBCドライãƒã¯ä»¥ä¸‹ã®ã‚ˆã†ãªURLã§JDBC経由ã§SparkSQLを使ã†ã“ã¨ãŒã§ãã¾ã™ã€‚ com.zensolution.jdbc.spark:/Users/foobar/temp/console?format=csv&csv.header=true&csv.delimiter=; SQL内ã§ã‚¢ã‚¯ã‚»ã‚¹ã•ã‚Œã¦ã„るテーブルã¯ã‚¯ã‚¨ãƒªã®å®Ÿè¡Œå‰ã«ãƒ†ãƒ³ãƒãƒ©ãƒªãƒ“ューã¨ã—ã¦è‡ªå‹•çš„ã«ç™»éŒ²ã•ã‚Œã¾ã™ã€‚ãŸã¨ãˆã°ä»¥ä¸‹ã®ã‚ˆã†ãªSQLを実行ã—ãŸã¨ã—ã¾ã™ã€‚ SELECT * FROM people ã“ã®JDBCドライãƒã¯ã¾ãšã‚¯ã‚¨ãƒªã‚’パースã—ã€ã“ã®ã‚¯ã‚¨ãƒªã®å®Ÿè¡Œã«p
GitHub - iipc/awesome-web-archiving: An Awesome List for getting started with web archiving
ArchiveBox - A tool which maintains an additive archive from RSS feeds, bookmarks, and links using wget, Chrome headless, and other methods (formerly Bookmark Archiver). (In Development) archivenow - A Python library to push web resources into on-demand web archives. (Stable) ArchiveWeb.Page - A plugin for Chrome and other Chromium based browsers that lets you interactively archive web pages, repl
Spark Shellã‚’Scalaã§ä½¿ã†ã¨ãã®ãƒ¡ãƒ¢ - レンコン畑ã§ã¤ã‹ã¾ãˆã¦
ã¯ã˜ã‚㫠最近データã®æ•´å½¢ç‰ã«Spark Shellを使ã£ã¦ã„ã‚‹ã®ã§ã™ãŒã€ä½¿ã„方を忘れるã®ã§å‚™å¿˜éŒ²çš„ãªï½±ï¾šã§ã™ã€‚ Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.4 /_/ Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_222) ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã¯ã“ã‚“ãªæ„Ÿã˜ã§ã™ã€‚ データソースèªã¿è¾¼ã¿ CSV（エンコーディング指定・ヘッダã‹ã‚‰ã‚«ãƒ©ãƒ åを推定） val prods = spark.read.format("csv"). option("header", true). option("encoding", "shift_jis"). load("Prod.cs
Clojureã§Sparkã‚’ã¯ã˜ã‚ã‚‹ - euphonictechnologies’s diary
ãªãœSpark? ビッグデータã§ãƒ‡ãƒ¼ã‚¿ã‚µã‚¤ã‚¨ãƒ³ã‚¹ã§ãƒžã‚·ãƒ³ãƒ©ãƒ¼ãƒ‹ãƒ³ã‚°ã®ã‚¢ãƒ¼ãƒ†ã‚£ãƒ•ã‚£ã‚·ãƒ£ãƒ«ã‚¤ãƒ³ãƒ†ãƒªã‚¸ã‚§ãƒ³ã‚¹ã ã‹ã‚‰ã§ã™ã€‚ビッグデータ分æžã¯HadoopãŒãƒ‡ãƒ•ã‚¡ã‚¯ãƒˆã‚¹ã‚¿ãƒ³ãƒ€ãƒ¼ãƒ‰ã§ã™ã€ã¨ã„ã†ã“ã¨ã‚’最近入社ã—ãŸä¼šç¤¾ã§ç”Ÿã¾ã‚Œã¦åˆã‚ã¦çŸ¥ã‚Šã¾ã—ãŸã€‚ SparkãŒã‚ã‚Œã°MapReduceã ã‘ã§ã¯é›£ã—ã„分æžã‚„データ処ç†ã‚‚パパッã¨ã§ãã¦ã—ã¾ã„ã¾ã™ã€‚ ãªãœClojure? ç§ã¯OCamlãŒå¤§å¥½ãã§ã™ã€‚ã¤ã¾ã‚ŠJavaã¨ã‹ã¡ã‚‡ã£ã¨ã—ã‚“ã©ã„ã§ã™ã€‚ã—ã‹ã—Sparkã¯JVM言語ã‹Python(PySpark)を使ã†ã“ã¨å‰æã¨ãªã£ã¦ã„ã¾ã™ã€‚OCamlã¯æ®‹å¿µãªãŒã‚‰JVMã§ã¯å‹•ã‹ãªã„ã—Pythonã§ã‚‚ã‚ã‚Šã¾ã›ã‚“ã®ã§ä½¿ãˆã¾ã›ã‚“。 普通ã ã£ãŸã‚‰Javaã‹Scalaãªã®ã§ã™ãŒã€Javaを休日ã«ä½¿ã†ã®ã¯å‹˜å¼ã—ã¦æ¬²ã—ã„ã§ã™ã€‚final List<String> someString = new ArrayList<String>();ã£ã¦ã“
ãŠçŸ¥ã‚‰ã›
ランã‚ング
ランã‚ング
リリースã€éšœå®³æƒ…å ±ãªã©ã®ã‚µãƒ¼ãƒ“スã®ãŠçŸ¥ã‚‰ã›
最新ã®äººæ°—エントリーã®é…ä¿¡
j次ã®ãƒ–ックマーク
kå‰ã®ãƒ–ックマーク
lã‚ã¨ã§èªã‚€
eコメント一覧を開ã
oページを開ã
GitHub - W01fh4cker/cve-2022-33891: cve-2022-33891-poc
Accessing WARC files via SQL
demo-code/emr/eks/videos/external_metastores/hivejdbc.py at 321e0db99840e30078eca323dced241fdd6d9a76 · dacort/demo-code
GitHub - YUHAI0/spark-jq: A spark toolkit for half structured or structured format data's analytics
Simple ETL using Apache Spark & PostgreSQL
AmazonAthenaã§æ—¥æ™‚ãƒãƒƒãƒå‡¦ç†ã‚’ä½œæˆ â€“ TECH4ALL
GitHub - commoncrawl/cc-index-table: Index Common Crawl archives in tabular format
PySpark SQL Recipes PDF | FREE Programming PDF eBooks
GitHub - Cyb3rWard0g/HELK: The Hunting ELK
GitHub - commoncrawl/cc-pyspark: Process Common Crawl data with Python and Spark
GitHub - FavioVazquez/twitter_optimus_twint: Analyzing tweets with Twint, Optimus and Apache Spark.
Analyzing Tweets with NLP in minutes with Spark, Optimus and Twint
Dependency Injection with Spark
GitHub - polynote/polynote: A better notebook for Scala (and more)
Threat Hunting with Jupyter Notebooks — Part 3: Querying Elasticsearch via Apache Spark ✨
{{#tags}}- {{label}}
{{/tags}}