Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-stepfunctions-tasks: workerType not dynamic from payload #32359

Open
1 task
ElearG opened this issue Dec 2, 2024 · 4 comments · May be fixed by #32453
Open
1 task

aws-stepfunctions-tasks: workerType not dynamic from payload #32359

ElearG opened this issue Dec 2, 2024 · 4 comments · May be fixed by #32453
Assignees
Labels
@aws-cdk/aws-stepfunctions-tasks bug This issue is a bug. effort/medium Medium work item – several days of effort needs-review p1

Comments

@ElearG
Copy link

ElearG commented Dec 2, 2024

Describe the bug

I want to implement a state machine that invokes a glue job synchronously and that at the time of invoking it, the type of worker and the number of workers are passed as parameters from the input (payload). It is verified that the implementation can be achieved at the console level, however, if I want to deploy it through CDK, it is not possible, because the value of "WorkerType" is a class of the WorkerType type and does not allow it to be placed dynamically from the input. A capture of the implementation in the console and the attempt at implementation in CDK are attached.

  • Only can set the workers' number dynamic from payload

CDK Repository Implementation
image

  • To make dynamic the workerType the workerType should look something like this:

image

- But here the error:

image

  • When implenet by console looks likes :

image

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Define in CDK the workerType to be dynamic from the payload input in the state machine, in other words, achieve the image but doing it with cdk.
I need that the argument "worker_type" not only accept a WorkerType Class that is a enum and only can put the define workers type insted of i need that also accept strings to pass "_sfn.JsonPath.string_at('$.glue_jobs_configs.executor_type')"in that argument

image

Current Behavior

Now, I can't put the workers type dynamic from payload input in the state machine because it only accept a class workerType that its a enum and only accepts the define workers.
If i pass a string that references the payload input it shows an error.

image
image

Reproduction Steps

In the step function task: GlueStartJobRun
Put the argument worker_type the value _sfn.JsonPath.string_at('$.glue_jobs_configs.executor_type')
worker_type= _sfn.JsonPath.string_at('$.glue_jobs_configs.executor_type')
image

Possible Solution

The argument worker_type accept also strings and dont validate that its only a class WorkerType

Additional Information/Context

No response

CDK CLI Version

2.171.1

Framework Version

No response

Node.js Version

20.11.1

OS

Windows 11

Language

Python

Language Version

3.11

Other information

No response

@ElearG ElearG added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 2, 2024
@ashishdhingra ashishdhingra added p1 investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Dec 3, 2024
@ashishdhingra ashishdhingra self-assigned this Dec 3, 2024
@ashishdhingra
Copy link
Contributor

ashishdhingra commented Dec 3, 2024

The code comment for WorkerType enum from aws-cdk-lib.aws_stepfunctions_tasks.WorkerConfigurationProperty states that If you need to use a WorkerType that doesn't exist as a static member, you can instantiate a WorkerTypeobject, e.g:WorkerType.of('other type').. However, this is incorrect since WorkerType is a enum, not a class. This might not be an overlook since per AWS Glue > Job runs, WorkerType accepts limited set of values. It might be a copy/paste error with excerpt taken from @aws-cdk/aws-glue-alpha > WorkerType class, which might be defined to support dynamic values (it's an experimental module).

Changing the type of property workerType in WorkerConfigurationProperty interface would be a breaking change and unsure if this should be done. Also, this property is used here in call to sfn.FieldUtils.renderObject(), which renders Parameters definition which is outputted in DefinitionString for AWS::StepFunctions::StateMachine resource in generated CFN template as something like - :states:::glue:startJobRun.sync","Parameters":{"JobName":"Test","WorkerType":"G.025X","NumberOfWorkers":3}}}}. There might not be a way to use escape hatch here. Perhaps introducing another property named workerTypeV2 which uses newly defined Worker Type class and deprecating old property workerType would be a feasible solution.

Needs review with the team.

@ashishdhingra ashishdhingra added effort/medium Medium work item – several days of effort needs-review and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Dec 3, 2024
@ElearG
Copy link
Author

ElearG commented Dec 3, 2024

Thanks for the reply. So I understand that a change must be made at the CDK level to make the workerType dynamic, right?

Is there another way to perform this operation?

I tried with the CallAwsService , Glue service, however I have problems with the integration_pattern, it does not accept synchronously (_sfn.IntegrationPattern.RUN_JOB)

Thanks for the support!

The code comment for WorkerType enum from aws-cdk-lib.aws_stepfunctions_tasks.WorkerConfigurationProperty states that If you need to use a WorkerType that doesn't exist as a static member, you can instantiate a WorkerTypeobject, e.g:WorkerType.of('other type').. However, this is incorrect since WorkerType is a enum, not a class. This might not be an overlook since per AWS Glue > Job runs, WorkerType accepts limited set of values. It might be a copy/paste error with excerpt taken from @aws-cdk/aws-glue-alpha > WorkerType class, which might be defined to support dynamic values (it's an experimental module).

Changing the type of property workerType in WorkerConfigurationProperty interface would be a breaking change and unsure if this should be done. Also, this property is used here in call to sfn.FieldUtils.renderObject(), which renders Parameters definition which is outputted in DefinitionString for AWS::StepFunctions::StateMachine resource in generated CFN template as something like - :states:::glue:startJobRun.sync","Parameters":{"JobName":"Test","WorkerType":"G.025X","NumberOfWorkers":3}}}}. There might not be a way to use escape hatch here. Perhaps introducing another property named workerTypeV2 which uses newly defined Worker Type class and deprecating old property workerType would be a feasible solution.

Needs review with the team.

@GavinZZ
Copy link
Contributor

GavinZZ commented Dec 10, 2024

Hello, thanks for creating this issue. I want to double check with you that you are able to workaround the NumberOfWorkers field and it's only the WorkerType property that doesn't work. If you have something like this class where you can use WorkerType.of(...), would that help solve your issue?

@ElearG
Copy link
Author

ElearG commented Dec 10, 2024

Hello, thanks, yeah the problem is only with WorkerType it's a enum not a class so "WorkerType.of" doesn't work. In the other hand NumberOfWorkers it's an integer type and its accept JsonPath.number_at normally so it works correct.
I think that if I can have a WorkerType that can accept "JsonPath.StringAt" it could solve my problem.
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-stepfunctions-tasks bug This issue is a bug. effort/medium Medium work item – several days of effort needs-review p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants