[アップデート] 1è¡ŒãŸã‚Šã¨ã‚‚コードã¯æ›¸ã‹ãªã„ï¼AWS Glueã§ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ETLãŒå¯èƒ½ã«ãªã‚Šã¾ã—㟠| DevelopersIO先日ã®ã‚¢ãƒƒãƒ—デート㧠AWS Glue ãŒã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚° ETL をサãƒãƒ¼ãƒˆã™ã‚‹ã‚ˆã†ã«ãªã‚Šã¾ã—ãŸï¼ AWS Glue now supports serverless streaming ETL New – Serverless Streaming ETL with AWS Glue ãªã«ãŒå¬‰ã—ã„ã®ã‹ ã¾ãšã€AWS Glue？ナニソレ？ã¨ã„ã†æ–¹ã¯ã€æ˜¯éžã€ä»¥ä¸‹ã®å†å…¥é–€ãƒ–ãƒã‚°ã‚’ãŠèªã¿ãã ã•ã„。 AWS Glue ã¯ãƒ•ãƒ«ãƒžãƒãƒ¼ã‚¸ãƒ‰ã§ã‚µãƒ¼ãƒãƒ¼ãƒ¬ã‚¹ãª ETL（抽出/変æ›/ãƒãƒ¼ãƒ‰ï¼‰ã‚µãƒ¼ãƒ“スã§ã™ãŒã€ã“ã‚Œã¾ã§ãƒ‡ãƒ¼ã‚¿ã‚½ãƒ¼ã‚¹ã¨ã—ã¦ã¯ S3ã€RDSã€Redshift・・・etc ã¨ã„ã£ãŸãƒ‡ãƒ¼ã‚¿ã‚¹ãƒˆãƒ¬ãƒ¼ã‚¸ã®ã¿ãŒå¯¾è±¡ã§ã—ãŸã€‚ãã®ãŸã‚ã€Kinesis Data Streams ã‚„ Apache Kafka ã¨ã„ã£ãŸã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ãƒ‡ãƒ¼ã‚¿ã«å¯¾ã—㦠Glue を組ã¿è¾¼ã‚“ã ETL 処ç†ã¯å‡ºæ¥ã¾ã›ã‚“ã§ã—ãŸã€‚ 従æ¥ã§ã‚ã‚Œã°ã€ä¾‹ãˆ
![[アップデート] 1è¡ŒãŸã‚Šã¨ã‚‚コードã¯æ›¸ã‹ãªã„ï¼AWS Glueã§ã‚¹ãƒˆãƒªãƒ¼ãƒŸãƒ³ã‚°ETLãŒå¯èƒ½ã«ãªã‚Šã¾ã—㟠| DevelopersIO](https://cdn-ak-scissors.b.st-hatena.com/image/square/909731178f06a3ebbfbb04bd46495a3f3e639863/height=288;version=1;width=512/https%3A%2F%2Fdevio2023-media.developers.io%2Fwp-content%2Fuploads%2F2019%2F04%2Faws-glue.png)
Pythonã§å¤§é‡ãƒ‡ãƒ¼ã‚¿å‡¦ç†ï¼ï¿½PySparkを用ã„ãŸãƒ‡ãƒ¼ã‚¿å‡¦ç†ã¨åˆ†æžã®ãã»ã‚“
Apache SparkãŒã‚¹ãƒ«ãƒ¼ãƒ—ットã¨ãƒ¬ã‚¤ãƒ†ãƒ³ã‚·ã‚’両立ã•ã›ãŸä»•çµ„ã¿ã¨æœ€æ–°å‹•å‘ã‚’ã€Sparkコミッタã¨ãªã£ãŸNTTデータ猿田æ°ã«èžã„ãŸï¼ˆå‰ç·¨ï¼‰
Asakusa on Spark - 急ãŒã°å›žã‚Œã€é¸ã¶ãªã‚‰è¿‘é“
『GMO プライベート DMP 㧠ビッグデータ解æžã‚’ã™ã‚‹ãŸã‚㫠アプリクラウド㧠Apache Spark ã®æ¤œè¨¼ã‚’ã—ã¦ã¿ãŸã€ã®è³‡æ–™ã‚’公開ã—ã¾ã—㟠- ã¹ã«ã‚„ã¾ã¶ã‚ã
ã‚·ãƒã‚¯ã§ã®åˆ©ç”¨æ–¹æ³•ã¨Sparkã¨ã®æ¯”較 // Speaker Deck
Stormã¨Kafkaã«ã‚ˆã‚‹ãƒªã‚¢ãƒ«ã‚¿ã‚¤ãƒ データ処ç†
Spark MLlibã§ã¯ã˜ã‚るスケーラブルãªæ©Ÿæ¢°å¦ç¿’
{{#tags}}- {{label}}
{{/tags}}