Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(aws-scheduler-targets-alpha): Add SageMakerStartPipelineExecution Target #27457

Closed
1 of 2 tasks
filletofish opened this issue Oct 9, 2023 · 4 comments · Fixed by #28927
Closed
1 of 2 tasks

(aws-scheduler-targets-alpha): Add SageMakerStartPipelineExecution Target #27457

filletofish opened this issue Oct 9, 2023 · 4 comments · Fixed by #28927
Labels
@aws-cdk/aws-sagemaker Related to AWS SageMaker effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2

Comments

@filletofish
Copy link
Contributor

filletofish commented Oct 9, 2023

Describe the feature

Work to support L2 constructs for AWS Scheduler is in progress (#23394). See the approved RFC. RFC planned to add 12 templates targets, but only Lambda Invoke is currently implemented (#26575).

This issue tracks implementation of SageMakerStartPipelineExecution target to start Amazon SageMaker pipeline.

Use Case

Customers would like to use templated target SageMakerStartPipelineExecution to be able to start an Amazon SageMaker pipeline on schedule. L2 target construct should grant required permissions to the AWS Scheduler to start an Amazon SageMaker pipeline.

Proposed Solution

The proposed solution needs to be adopted to the recent examples of LambdaInvoke (https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-scheduler-targets-alpha/lib/lambda-invoke.ts).

Solution should also include unit and integration tests.

Class SageMakerStartPipelineExecution should:

  1. Grant Scheduler Execution Role permissions to start SageMaker pipeline via addTargetActionToRole
  2. Override bindBaseTargetConfig to return sageMakerPipelineParameters as part of ScheduleTargetConfig.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.99.1

Environment details (OS name and version, etc.)

MacOS

@filletofish filletofish added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Oct 9, 2023
@github-actions github-actions bot added the @aws-cdk/aws-sagemaker Related to AWS SageMaker label Oct 9, 2023
@pahud
Copy link
Contributor

pahud commented Oct 9, 2023

Thank you for all those feature requests and PRs!

@pahud pahud added p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Oct 9, 2023
@mergify mergify bot closed this as completed in #28927 Feb 13, 2024
mergify bot pushed a commit that referenced this issue Feb 13, 2024
…et (#28927)

This PR adds SageMakerStartPipelineExecution Target for EventBridge Scheduler.

Closes #27457

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

GavinZZ pushed a commit that referenced this issue Feb 22, 2024
…et (#28927)

This PR adds SageMakerStartPipelineExecution Target for EventBridge Scheduler.

Closes #27457

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@lorenzwalthert
Copy link

lorenzwalthert commented Aug 8, 2024

I am trying to use SageMakerStartPipelineExecution Target to create a EventBridge Rule to trigger a pipeline as described in the README of the Python package aws-cdk.aws-scheduler-targets-alpha:

import aws_cdk.aws_sagemaker as sagemaker

# pipeline: sagemaker.IPipeline


Schedule(self, "Schedule",
    schedule=ScheduleExpression.rate(Duration.minutes(60)),
    target=targets.SageMakerStartPipelineExecution(pipeline,
        pipeline_parameter_list=[targets.SageMakerPipelineParameter(
            name="parameter-name",
            value="parameter-value"
        )]
    )
)

However, I fail to understand how to create the pipeline object, which must be of class sagemaker.IPipeline according to the comment above. I looked in the docs and figured out how I can construct a Pipeline in CDK with sagemaker.CfnPipeline but that gives me an error since it's the wrong class. The documentation of IPipeline is not informative enough for me to learn how I can construct a IPipeline object. Looking through GitHub PRs and Issues I found out that the PR to support this feature had initially proposed to use CfnPipeline, but then, you decided to use the (in my understanding) placeholder sagemaker.IPipeline for now.

Can you clarify if the example code from the README is supposed to work with the current release of CDK? If yes, how can I create an instance of IPipeline? If not, it would be great if the official CDK distribution could support creation of the required construct. Otherwise, it seem like having a SageMakerStartPipelineExecution is of little value if it can't actually be used to schedule a pipeline execution. I also could not figure out how to use any combination of alpha and non-alpha CDK Python packages to achieve my goal (scheduling a Pipeline Execution with supplying parameters).

Side note: The sagemaker python sdk has support to attach EventBridge schedules to SageMaker Pipeline, however it can't take any parameter (so only pipeline executions with default parameters are possible), plus, it's not compatible with Local Mode, and the schedule does not make its way into the Pipeline definition, and creates the rules ad-hoc (before upsetting the pipeline), so if I create the pipeline definition with the sagemaker python sdk and then use this definition in cdk / cfn, I loose the schedule. Hence, there seems to be no way to schedule Pipeline Executions with non-default parameters with CDK or the sagemaker sdk.

I'd appreciate your help (maybe this needs a new issue?) and allow myself to tag the relevant people as described in the comment visibility warning @kaizencc, @pahud, @filletofish.

@gracelu0
Copy link
Contributor

gracelu0 commented Nov 7, 2024

Hi @lorenzwalthert , could you create a separate issue if this is still relevant? As we don't usually look at closed issues (I just happened to stumble across this). Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-sagemaker Related to AWS SageMaker effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants