[B! lambda] tasukuchanã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

tasukuchan id:tasukuchan

lambdaã«é–¢ã™ã‚‹tasukuchanã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (4)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

AWS Lambdaã§Pythonå¤–éƒ¨ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã®Layerã‚’ä½œã‚‹å‰ã« - Qiita
çµè«– å…ˆäººã«ã‚ˆã£ã¦æ—¢ã«ä½œã‚‰ã‚Œã¦ãªã„ã‹ã€ã„ã£ãŸã‚“ç¢ºèªã—ã¦ã¿ã¾ã—ã‚‡ã†ã€‚ keithrozario/Klayers ä½¿ã„æ–¹ ãªãœã‹AWSå…¬å¼ã§LayerãŒæä¾›ã•ã‚Œã¦ã„ãªã„ãƒ©ã‚¤ãƒ–ãƒ©ãƒªã€pandasã‚’ä¾‹ã¨ã—ã¦ä½¿ã„ã¾ã™ã€‚ (Numpy + ScipyãŒã‚ã‚‹ãªã‚‰ã€pandasã‚‚ã‚ã£ã¦ãã‚Œã¦ã„ã„æ°—ãŒã—ã¦ã„ã¾ã™) 0. å‰æº–å‚™ AWS Lambdaã«ã‚¢ã‚¯ã‚»ã‚¹ã—ã¦ã€[é–¢æ•°ã®ä½œæˆ]ã‚’æŠ¼ã—ã€é©å½“ãªåå‰ã®é–¢æ•°ã‚’ä½œã‚Šã¾ã™ã€‚ ã€Œãƒ©ãƒ³ã‚¿ã‚¤ãƒ ã€ã¯Python3.8ã‚’é¸ã³ã¾ã™ã€‚ã€Œã‚¢ã‚¯ã‚»ã‚¹æ¨©é™ã€ã¯ãƒŽãƒ¼ã‚¿ãƒƒãƒã§OKã§ã™ã€‚ 1. pandasã‚’ã‚¤ãƒ³ãƒãƒ¼ãƒˆã—ã€ãƒ†ã‚¹ãƒˆã‚’ä½œæˆ import pandasã‚’è¿½åŠ ã—ã€[ä¿å˜]ã‚’æŠ¼ã—ã¾ã™ã€‚ [ãƒ†ã‚¹ãƒˆ]ã‚’æŠ¼ã—ã€ã€Œã‚¤ãƒ™ãƒ³ãƒˆåã€ã«é©å½“ãªåå‰ã‚’å…¥ã‚Œã¾ã™ã€‚ ä¸‹ã®JSONã¯ç‰¹ã«ã„ã˜ã‚‰ãªãã¦OKã§ã™ã€‚ ã“ã®æ™‚ç‚¹ã§ã‚‚ã†ä¸€åº¦[ãƒ†ã‚¹ãƒˆ]ã‚’æŠ¼ã™ã¨å¤±æ•—ã—ã¾ã™ã€‚pandasãŒç„¡ã„ã‚ˆã¨è¨€ã‚ã‚Œã¦ã„ã¾ã™ã€‚ ãã‚Œã¯ãã†ã¨ã„ã†æ„Ÿ
tasukuchan 2021/04/08
lambda

pandas
ãƒªãƒ³ã‚¯
Terraformã§ã€AWS Lambdaé–¢æ•°ã‚’ç™»éŒ²ã—ã¦å‹•ã‹ã—ã¦ã¿ã‚‹ - Qiita
What'sï¼Ÿ ã¯ã˜ã‚ã¦ã®AWS Lambdaã‚’è©¦ã—ã¦ã¿ãŸã„ AWS Lambdaã®ãƒ‡ãƒ—ãƒã‚¤ã¯Terraformã§è¡Œã„ãŸã„ ã¨ã„ã†ãŠé¡Œã§ã€‚ ã‚„ã£ã¦ãŠããŸã„ã“ã¨ã€å‹•ã‹ã—æ–¹ã¨ã—ã¦ã¯ AWS Lambdaé–¢æ•°ã¯ã€Terraformã§zipåœ§ç¸®ã—ã¦ã‚¢ãƒƒãƒ—ãƒãƒ¼ãƒ‰ã™ã‚‹ AWS Lambdaé–¢æ•°ã‹ã‚‰ãƒã‚°ã‚’å‡ºåŠ›ã™ã‚‹ Amazon CloudWatch Logsã¸å‡ºåŠ›ã™ã‚‹ã‚ˆã†ã«ãƒã‚°ã‚°ãƒ«ãƒ¼ãƒ—ã‚’ä½œæˆã™ã‚‹ AWS Lambdaé–¢æ•°ã‹ã‚‰ç’°å¢ƒå¤‰æ•°ã‚’èªã¿è¾¼ã‚€ ç’°å¢ƒå¤‰æ•°ã®æš—å·åŒ–ã«ã¯AWS KMSã®CMKã‚’ç”¨ã„ã‚‹ å‹•ä½œç¢ºèªã¯AWS CLIã§è¡Œã† ã¨ã„ã†ã¨ã“ã‚ã§ã€‚ ç’°å¢ƒ ä»Šå›žã®ç’°å¢ƒã¯ã€ã“ã¡ã‚‰ã§ã™ã€‚ $ terraform version Terraform v0.12.29 + provider.archive v1.3.0 + provider.aws v2.70.0 $ aws --version aws-cli/2.0
tasukuchan 2021/04/08
lambda

terraform
ãƒªãƒ³ã‚¯
AWS Glue ã‚’ Lambda ã‹ã‚‰èµ·å‹•ã—ã¦åˆ¶å¾¡ã™ã‚‹ (Python)
æ¦‚è¦ ã“ã¡ã‚‰ã®ãƒšãƒ¼ã‚¸ã§ä½¿ã„æ–¹ã‚’æŠŠæ¡ã—ãŸ AWS Glue ã‚’ã“ã¡ã‚‰ã®ãƒšãƒ¼ã‚¸ã§ä½¿ã„æ–¹ã‚’æŠŠæ¡ã—ãŸ AWS Lambda ã‹ã‚‰èµ·å‹•ã™ã‚‹ã‚ˆã†ã«ã™ã‚‹ã¨ã€å¤§è¦æ¨¡ãƒ‡ãƒ¼ã‚¿ã® ETL å‡¦ç†ã‚’ Job å¼•æ•°ã‚„ã‚¨ãƒ©ãƒ¼æ™‚ã®ãƒãƒ³ãƒ‰ãƒªãƒ³ã‚°ã‚’å«ã‚ã¦æŸ”è»Ÿã«è¡Œã†ã“ã¨ãŒã§ãã¾ã™ã€‚Glue ã¨ Lambda ã§åˆ©ç”¨ã™ã‚‹è¨€èªžã¯ã©ã¡ã‚‰ã‚‚ Python ã§ã‚ã‚‹ã¨ã—ã¦ã€ç°¡å˜ãªé€£æºæ–¹æ³•ã«ã¤ã„ã¦è¨˜è¼‰ã—ã¾ã™ã€‚ ETL å¯¾è±¡ã¨ãªã‚‹ S3 ãƒ‡ãƒ¼ã‚¿ã‚½ãƒ¼ã‚¹ã®æº–å‚™ ã“ã¡ã‚‰ã®ãƒšãƒ¼ã‚¸ã§ã¯ s3.json ã¨ã„ã†ä¸€ã¤ã®ãƒ•ã‚¡ã‚¤ãƒ«ã‚’ Crawler ã«æŒ‡å®šã—ã¾ã—ãŸã€‚ä»Šå›žã¯ä»¥ä¸‹ã®ã‚ˆã†ã«æ—¥ä»˜æ¯Žã«ãƒ•ã‚©ãƒ«ãƒ€åˆ†ã‘ã•ã‚Œã¦ãŠã‚Šã€å„æ—¥ä»˜ã®ãƒ•ã‚©ãƒ«ãƒ€ã«ã¯äºŒç¨®é¡žã®ãƒ•ã‚¡ã‚¤ãƒ«ãŒå˜åœ¨ã—ã¦ã„ã‚‹ã¨ã—ã¾ã™ã€‚ $ aws s3 ls --recursive s3://my-bucket-20171124/ 2017-12-03 00:56:39 140 20171201/filetype_a.j
tasukuchan 2021/04/07
glue

lambda
ãƒªãƒ³ã‚¯
AWS Lambdaã§Seleniumã¨ActiveRecordã‚’æ‰±ã† Â· ãƒ•ãƒãƒƒãƒ”ãƒ¼ãƒ‡ã‚£ã‚¹ã‚¯ã®æ®‹éª¸
æ°—ä»˜ã„ãŸã‚‰ã‚‚ã†9æœˆã§ã™ãã€‚ æœ€è¿‘ã€AWS Lambdaã§ã„ã‚ã„ã‚ã¨éŠã¶æ©Ÿä¼šãŒã‚ã£ãŸã®ã§ãƒ¡ãƒ¢ã¨ã—ã¦æ®‹ã—ã¾ã™ã€‚ ã¯ã˜ã‚ã« ã¨ã‚ã‚‹ã‚»ã‚ãƒ¥ãƒªãƒ†ã‚£ã‚²ãƒ¼ãƒ ã®é‹å–¶ç”¨ã«ã€Seleniumã§Webã‚¹ã‚¯ãƒ¬ã‚¤ãƒ”ãƒ³ã‚°ã‚’ã‚„ã£ã¦ã„ã‚‹Rubyã®ã‚¹ã‚¯ãƒªãƒ—ãƒˆã‚’EC2ã§é‹ç”¨ã—ã¦ã„ãŸã®ã§ã™ãŒã€Headless Chromeã‚’æ‰±ã†ãŸã‚å¤§é‡ã«èµ·å‹•ã™ã‚‹ã¨ãƒ¡ãƒ¢ãƒªé£Ÿã£ã¡ã‚ƒã†ã—ã€ã‚¹ã‚±ãƒ¼ãƒ«ã—ã‚ˆã†ã«ã‚‚EC2ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã®AutoScalingçµ„ã‚€ã®ã‚‚ã¡ã‚‡ã£ã¨ãªã‚ã€‚ã¨ã‹ã€ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã®èµ·å‹•ã¾ã§å¾…ã£ã¦ã‚‰ã‚Œãªã„ã‹ã‚‰ã‚ã‚‹ç¨‹åº¦å¤šã‚ã«ã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã‚’å®Ÿè¡Œã—ãŸã‚Šã™ã‚‹ã®ã‚‚ä½™åˆ†ã«ã‚³ã‚¹ãƒˆãŒã€‚ã€‚ã€‚ 1å®Ÿè¡Œã«15åˆ†ã‚‚ã‹ã‹ã‚‰ãªã„ã‚¹ã‚¯ãƒªãƒ—ãƒˆã ã—ã€ã¨ã„ã†ã“ã¨ã§Lambdaã«ç§»è¡Œã™ã‚‹ã“ã¨ã«ã—ã¾ã—ãŸã€‚ Lambda Layerã«ã¤ã„ã¦ Headless Chromeã‚’æ‰±ã†å ´åˆã€å˜ç´”ã«Functionã®ãƒ‡ãƒ—ãƒã‚¤ãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ã«ãƒã‚¤ãƒŠãƒªã‚’å«ã‚ã‚‹ã¨50MBã‚’è¶…ãˆã¦ã—ã¾ã†ãŸã‚ã€Lambda
tasukuchan 2020/09/14
lambda
ãƒªãƒ³ã‚¯
1