ãData Engineering Study #23 Data orchestration ç¹éãã®çºè¡¨è³æã§ã ã¤ãã³ããã¼ã¸: https://forkwell.connpass.com/event/310011/

AWS CodePipeline ã§ãµãã¼ãããã¦ããã¢ã¯ã·ã§ã³ã¿ã¤ãã« AWS Step Functions ãå«ã¾ããããã«ãªãã¾ãããããã«ãããã客æ§ã¯ããªãªã¼ã¹ããã»ã¹ã®ä¸é¨ã¨ãã¦è¤éãªã¯ã¼ã¯ããã¼ãç°¡åã«å¼ã³åºããã¨ãã§ãã¾ããæ°ããã¢ã¯ã·ã§ã³ã¿ã¤ãã«ãããCodePipeline Stages ã¯ãæ¡ä»¶ä»ãåå²ãã¨ã©ã¼å¦çãããã³éåæã¿ã¹ã¯ããµãã¼ããã Step Functions ã®ã¹ãã¼ããã·ã³ãããªã¬ã¼ããservice-integrations ãéãã¦ä»ã® AWS ã®ãµã¼ãã¹ãç°¡åã«å¼ã³åºããã¨ãã§ããããã«ãªãã¾ããã CodePipeline ã使ç¨ããã¨ãå®ç¾©ãããªãªã¼ã¹ã¢ãã«ã«åºã¥ãã¦ãé«éã§ä¿¡é ¼æ§ã®é«ãã¢ããªã±ã¼ã·ã§ã³ã¨ã¤ã³ãã©ã¹ãã©ã¯ãã£ã®æ´æ°ã®ãªãªã¼ã¹ããã»ã¹ãèªååãããã¨ãã§ãã¾ããStep Functions ãµã¼ãã¹ã¯ãè¤æ°ã® AWS ã®ãµ
AWS SAM ã® AWS Step Functions ãµãã¼ãã«ãããã¯ã¼ã¯ããã¼ãªã¼ã±ã¹ãã¬ã¼ã·ã§ã³ããµã¼ãã¼ã¬ã¹ã¢ããªã±ã¼ã·ã§ã³ã«è¿ éãã¤ç°¡åã«çµ±åã§ããããã«ãªãã¾ããããµã¼ãã¼ã¬ã¹ã¢ããªã±ã¼ã·ã§ã³æ§ç¯ç¨ã®ãªã¼ãã³ã½ã¼ã¹ãã¬ã¼ã ã¯ã¼ã¯ã§ãã AWS ãµã¼ãã¼ã¬ã¹ã¢ããªã±ã¼ã·ã§ã³ã¢ãã« (AWS SAM) ã®ç´æ¥ãµãã¼ãã«ãããã¢ããªã±ã¼ã·ã§ã³ã®ã¢ã¼ããã¯ãã£ãã³ã¼ãã¨å ±ã« Step Functions ã¹ãã¼ããã·ã³ã®ã¯ã¼ã¯ããã¼ãå®ç¾©ãã¦ããµã¼ãã¼ã¬ã¹ã¢ããªã±ã¼ã·ã§ã³ãããè¿ éã«æä¾ã§ããããã«ãªãã¾ããã AWS Step Functions ã使ç¨ããã¨ãå復åã®ãããµã¼ãã¼ã¬ã¹ã¯ã¼ã¯ããã¼ãæ§ç¯ãã¦ãAWS ã® AWS SAM 対å¿ãµã¼ãã¹ (AWS LambdaãAmazon SNSãAmazon DynamoDB ãªã©) ã使ç¨ããã¹ãã¼ããã·ã³ãã»ããã¢ããã§ãã¾ã
AWS Compute Blog AWS Step Functions support in Visual Studio Code The AWS Toolkit for Visual Studio Code has been installed over 115,000 times since launching in July 2019. We are excited to announce toolkit support for AWS Step Functions, enabling you to define, visualize, and create your Step Functions workflows without leaving VS Code. Version 1.8 of the toolkit provides two new commands in the
Express Workflows ã¯ãAWS Step Functions ã¯ã¼ã¯ããã¼ã®æ°ããã¯ã¼ã¯ããã¼ã¿ã¤ãã§ããããã«ãããAWS ã®ã³ã³ãã¥ã¼ãã£ã³ã°ããã¼ã¿ãã¼ã¹ãã¡ãã»ã¼ã¸ã³ã°ãµã¼ãã¹ãæ¯ç§ 100,000 件ãè¶ ããã¤ãã³ãã¬ã¼ãã§ã³ã¹ãå¹çãã調æ´ã§ãã¾ããExpress Workflow ã¯ãAmazon API Gateway ãä»ãã HTTP ãªã¯ã¨ã¹ããAWS Lambda ãªã¯ã¨ã¹ããAWS IoT ã«ã¼ã«ã¨ã³ã¸ã³ã®ã¢ã¯ã·ã§ã³ãããã³ Amazon EventBridge ã® 100 件以ä¸ã® AWS ããã³ SaaS ã¤ãã³ãã½ã¼ã¹ãªã©ã®ã¤ãã³ãã«å¿ãã¦ãèªåçã«éå§ããã¾ããExpress Workflows ã¯ãIoT ãã¼ã¿ã®åãè¾¼ã¿ãã¹ããªã¼ãã³ã°ãã¼ã¿ã®å¦çã¨å¤æã大éã®ãã¤ã¯ããµã¼ãã¹ãªã¼ã±ã¹ãã¬ã¼ã·ã§ã³ãªã©ã®å¤§éã®ã¤ãã³ãå¦çã¯ã¼ã¯ãã¼ãã«é©
AWS Step Functions ã§ãã¢ã¸ã¥ã©ã¼å¼ã§åå©ç¨å¯è½ãªã¯ã¼ã¯ããã¼ãä½æãããã¨ã«ãããããè¤éãªããã»ã¹ããªã¼ã±ã¹ãã¬ã¼ããããã¨ãã§ããããã«ãªãã¾ããã Step Functions ã§ã¯ãAWS ã®è¤æ°ã®ãµã¼ãã¹ããã«ããã¼ã¸ãåã¯ã¼ã¯ããã¼ã§èª¿æ´ã§ããããããã°ããã¢ããªã±ã¼ã·ã§ã³ããã«ãããã³æ´æ°ã§ãã¾ããçµç¹ã®ã¯ã¼ã¯ããã¼ä½¿ç¨ãæ¡å¤§ããã¨ãã¯ã¼ã¯ããã¼ã®æ§ç¯ããã¹ããå¤æ´ã¯è¤éã«ãªãã¾ãããã使ç¨ãããã¯ã¼ã¯ããã¼ã®ããã»ã¹ã¯ãå¤ãã®å ´åè¤æ°ã®å ´æã§å復ããã¾ããä½ãç°å¸¸ãèµ·ãã£ãå ´åã®å¤§è¦æ¨¡ã¯ã¼ã¯ããã¼ã®å®è¡ãã対å¦ããã大å¤ã«ãªãã¾ãã Step Functions ã¯ã¼ã¯ããã¼ããã¹ããããã¨ã«ãããããå°ããåç´ãªã¯ã¼ã¯ããã¼ããããã大è¦æ¨¡ã§è¤éãªã¯ã¼ã¯ããã¼ãæ§ç¯ãããã¨ãã§ãã¾ããã¯ã¼ã¯ããã¼ã«äººéã«ããã¬ãã¥ã¼ããã³æ¿èªãã¾ãã¯ã¢ã«ã¦ã³ãå ã®ãªã½
AWS Step Functions 㯠Amazon EMR ã¨çµ±åãããEMR ããã°ãã¼ã¿å¦çã¯ã¼ã¯ããã¼ã®æ§ç¯ã¨ç£è¦ã容æã«ãªãã¾ããã AWS Step Functions ã¯ãAmazon EMRãAmazon SageMakerãAWS Lambda ãªã©ã® AWS ãµã¼ãã¹ãå©ç¨ããå復åã®ããã¯ã¼ã¯ããã¼ãæ§ç¯ãããã¨ãå¯è½ã«ãã¾ããAmazon EMR ã¯ãæ¥çããªã¼ãããã¯ã©ã¦ããã¤ãã£ããªããã°ãã¼ã¿ãã©ãããã©ã¼ã ã§ãã大è¦æ¨¡ç°å¢ã§å¤§éã®ãã¼ã¿ãè¿ éãã¤ã³ã¹ãå¹çããå¦çã§ãã¾ããStep Functions 㨠Amazon EMR ã使ç¨ããã¨ãæå°éã®è¿½å ã³ã¼ããè¨è¿°ããªãããããã°ãã¼ã¿ã¯ã¼ã¯ããã¼ã調æ´ã§ãã¾ãã Amazon EMR 㨠AWS Step Functions ã使ç¨ããã¨ãAmazon EMR ã¹ãããã注æããä¾åé¢ä¿ã管çãã並è¡ã
AWS Step Functions ã¯åç並åå¦çã®ãµãã¼ããéå§ãã¾ããããã¼ã¿å¦çãã¿ã¹ã¯ã®èªååãªã©ã¨ãã£ãã¢ããªã±ã¼ã·ã§ã³ã®ã¯ã¼ã¯ããã¼ã®ããã©ã¼ãã³ã¹ã¨å¹çæ§ã«æé©åã§ãã¾ããåä¸ã®ã¿ã¹ã¯ã並è¡ãã¦å®è¡ãããã¨ã§ãä¸è²«ããå®è¡æéãå®ç¾ãããªã½ã¼ã¹ã®ä½¿ç¨çãæ¹åãã¦éç¨ã³ã¹ããç¯ç´ã§ãã¾ããStep Functions ã¯ãå ¥åã«å¿ãã¦ãªã½ã¼ã¹ãèªåçã«ã¹ã±ã¼ãªã³ã°ãã¾ãã Step Functions ã§ã¯ãAWS ã®è¤æ°ã®ãµã¼ãã¹ããã«ããã¼ã¸ãåã¯ã¼ã¯ããã¼ã§èª¿æ´ã§ããããããã°ããã¢ããªã±ã¼ã·ã§ã³ããã«ãããã³æ´æ°ã§ãã¾ããå¤ãã®çµç¹ã§ã¯ããªã½ã¼ã¹ãå¹ççã«ä½¿ç¨ããããã«ãããå¦çã¯ã¼ã¯ããã¼ã並è¡ãã¦å®è¡ãã¦ãã¾ãããã ããå¦çããã¢ã¤ãã ã®æ°ã¯ä¸æãªå ´åãå¤ãããã並åã¯ã¼ã¯ããã¼ã®å®è¡æéãäºæ¸¬ãããã¨ã¯å°é£ã§ãããããã®ä¸¦åã¯ã¼ã¯ããã¼ã®ãããã°ããã³æ§ç¯ã¯å°é£
Amazon Web Services ããã° AWS Step Functions 㨠AWS Glue ã使ç¨ã㦠Amazon DynamoDB ãã¼ãã«ã Amazon S3 ã«ã¨ã¯ã¹ãã¼ãããæ¹æ³ å¾æ¥ã® AWS ã®ããæ¹ã§ãAWS Glue ãã¼ã ã DynamoDB ãã¼ãã«ãããã¤ãã£ãã«èªã¿åã AWS Glue ã¯ãã¼ã©ããã³ AWS Glue ETL ã¸ã§ãã®æ©è½ããªãªã¼ã¹ããã¨ãã¯ãAWS ããã°ãã¼ã¿ããã°ã§ Goodreads ã¯ã©ã®ããã« Amazon DynamoDB ãã¼ãã«ã Amazon S3 ã«ãªããã¼ãããAmazon Athena ã使ç¨ãã¦ã¯ã¨ãªãå®è¡ããã®ããå ¬éãã¦ããä¸é±éãçµã£ã¦ãã¾ããã§ããããããã§ç§ã¯ããªããããããã¦ãã¾ãããã³ã¼ããããå°ãªãã¨ãããã¨ã¯ããã°ãããå°ãªããã¨ãæå³ãã¾ããå ã®ã¢ã¼ããã¯ãã£ã¯å°ãªãã¨ã 18
AWS Step Functions ã§ãã¤ãã³ãé§ååã®ãµã¼ãã¼ã¬ã¹ã¯ã¼ã¯ããã¼ã®æ§ç¯ã¨ç£è¦ãããè¿ éãã¤ç°¡åã«ãããã¯ã¼ã¯ããã¼å®è¡ã¤ãã³ãã®ãµãã¼ããéå§ããã¾ãããå®è¡ã¤ãã³ãéç¥ã¯ãCloudWatch Event ãéããã¯ã¼ã¯ããã¼ã®éå§ã¾ãã¯å®äºæã«ãAWS LambdaãAmazon SNSãAmazon Kinesisãããã㯠AWS Step Functions ãªã©ã®ãã¤ãã³ãã®èªåå¿çã®ã¿ã¼ã²ããã«èªåçã«é ä¿¡ããã¾ãã Step Functions ã® CloudWatch Event éç¥ã¯æ°åã§æå¹åã§ãã¾ããã客æ§ã«å¿ è¦ãªã®ã¯ãã¹ãã¼ããã·ã³ãéç¥ã®å¯¾è±¡ãããã³ã©ã®ãããªã¤ãã³ãã®å¤æ´ãéè¦ããç¹å®ããã ãã§ãã AWS Step Functions ã使ãã°ãã¢ããªã±ã¼ã·ã§ã³ã«æè»ãªã¯ã¼ã¯ããã¼èªååã追å ã§ãã¾ããã¯ã¼ã¯ããã¼ã®ã¹ãããã¯ãAWS L
ãã¼ã«ã«ãã¼ã¸ã§ã³ã® AWS Step Functions ã使ç¨ãã¦ãã¯ã¼ã¯ããã¼ãéçºããã³ãã¹ããããã¨ãã§ããããã«ãªãã¾ããã Step Functions ã®ãã¦ã³ãã¼ãå¯è½ãã¼ã¸ã§ã³ã§ãã AWS Step Functions Local ã§ã¯ãã客æ§ã®éçºç°å¢ã§å®è¡ããã¦ãã Step Functions ã®ãã¼ã¸ã§ã³ã使ç¨ãã¦ã¢ããªã±ã¼ã·ã§ã³ãéçºããã³ãã¹ãã§ãã¾ãããµã¼ãã¹ãã¤ã³ã¿ã¼ãããä¸ã§ã¯ãªããã¼ã«ã«ã§ä½¿ç¨ãããã¨ã«ãã£ã¦ãå ´åã«ãã£ã¦ã¯é度ãããããStep Functions ã®ç¶æ é·ç§»ãå°ãªããªãããµã³ãããã¯ã¹ã®å¶éãããç°¡åã«å¼·åã§ãã¾ãã AWS Step Functions Local ã¯ãJAR ããã±ã¼ã¸ããã³ Docker ã¤ã¡ã¼ã¸ã¨ãã¦ä½¿ç¨å¯è½ã§ããAWS Step Functions Local ã® Docker ã¤ã¡ã¼ã¸ã使ç¨ããã¨ãã
AWS Step Functions ãããã« 8 ã¤ã® AWS ãµã¼ãã¹ã¨çµ±åããã¯ã¼ã¯ããã¼ã®æ§ç¯ã®é«éåãä¿è·ã®ç°¡ç´ åãã¢ãã¿ãªã³ã°ã®å®¹æåãããã«æ¨ãé²ãã¾ããã AWS Step Functions ã使ãã°ãã¢ããªã±ã¼ã·ã§ã³ã«æè»ãªã¯ã¼ã¯ããã¼èªååã追å ã§ãã¾ããã¯ã¼ã¯ããã¼ã®ã¹ãããã¯ãAWS Lambda é¢æ°ãAmazon EC2ããªã³ãã¬ãã¹çãã©ãã«ã§ãåå¨ã§ãã¾ããä»åãAWS Step Functions ã¯ãAmazon ECSãAWS FargateãAmazon DynamoDBãAmazon SNSãAmazon SQSãAWS BatchãAWS GlueãAmazon SageMaker ã¨ãçµ±åããã¾ãããããã«ããããããå¦çã¸ã§ãã®èµ·åãã¡ãã£ã¢ã¾ãã¯ãã¼ã¿ãã¡ã¤ã«ã®å¦çãéç¥ã®éä¿¡ãæ©æ¢°å¦ç¿ã¯ã¼ã¯ããã¼ã®ãªã¼ã±ã¹ãã¬ã¼ããå¿ è¦ãªã¢ããªã±ã¼ã·ã§
ã©ã³ãã³ã°
ãç¥ãã
ã©ã³ãã³ã°
ãªãªã¼ã¹ãé害æ å ±ãªã©ã®ãµã¼ãã¹ã®ãç¥ãã
ææ°ã®äººæ°ã¨ã³ããªã¼ã®é ä¿¡
å¦çãå®è¡ä¸ã§ã
j次ã®ããã¯ãã¼ã¯
kåã®ããã¯ãã¼ã¯
lãã¨ã§èªã
eã³ã¡ã³ãä¸è¦§ãéã
oãã¼ã¸ãéã
{{#tags}}- {{label}}
{{/tags}}