aws_batch_alpha: Role permissions error after upgrade from v2.73 to v2.79 #25675

paulharbulak-maxar · 2023-05-22T17:43:54Z

Describe the bug

After changing ComputeEnvironment, JobDefinition, and other related Batch constructs to ManagedEc2EcsComputeEnvironment, EcsJobDefinition, etc. due to breaking changes in v2.79, Batch jobs fail with a CloudWatch permissions error.

Expected Behavior

I expected any default IAM roles created by Batch constructs to have necessary permissions for the default logging configuration.

Current Behavior

Batch jobs fail with the following error: CannotStartContainerError: Error response from daemon: failed to initialize logging driver: failed to create Cloudwatch log stream: AccessDeniedException: User: <BatchContainerDefinitionExecutionRole ARN>.

Reproduction Steps

This is the section of code that is currently used to generate the Batch infrastructure producing the error:

batch_security_group = ec2.SecurityGroup(
    self, id="BatchSecurityGroup", vpc=vpc, allow_all_outbound=True
)
postgres.connections.allow_default_port_from(batch_security_group)
batch_environ = batch.ManagedEc2EcsComputeEnvironment(
    self,
    "BatchEnvironment",
    compute_environment_name=f"{stack_name}-env",
    security_groups=[batch_security_group],
    spot=True,
    vpc=vpc,
)

seeder_queue = batch.JobQueue(
    self,
    "BatchSeedsJobQueue",
    job_queue_name=f"{stack_name}-seeds",
    compute_environments=[
        batch.OrderedComputeEnvironment(
            compute_environment=batch_environ, order=1
        ),
    ],
    priority=1,
)

wfs_job_def = batch.EcsJobDefinition(
    self,
    "WFSBatchJobDefinition",
    job_definition_name=f"{stack_name}-meta",
    parameters={
        "tasks": "ALL",
    },
    container=batch.EcsEc2ContainerDefinition(
        self,
        "WFSBatchContainerDefinition",
        image=ecs.ContainerImage.from_asset('./metadata'),
        environment={
            "POSTGRES_HOST": postgres.db_instance_endpoint_address,
            "POSTGRES_PORT": postgres.db_instance_endpoint_port,
            "POSTGRES_DB_NAME": "postgres",
            "POSTGRES_PASS": postgres.secret.secret_value_from_json(
                "password"
            ).to_string(),
        },
        cpu=1,
        memory=Size.mebibytes(8192),
    ),
)

wfs_datasets = [
    "test a",
    "test b",
    "test c",
    "test d",
]

wfs_jobs = step.Parallel(self, "WFSParallelJobs")
for idx, ds in enumerate(wfs_datasets):
    wfs_jobs.branch(
        tasks.BatchSubmitJob(
            self,
            f"WFSJobSubmit_{idx}",
            job_name=f"WFSJobSubmit_{idx}",
            job_queue_arn=seeder_queue.job_queue_arn,
            job_definition_arn=wfs_job_def.job_definition_arn,
            container_overrides=tasks.BatchContainerOverrides(
                environment={"DATASETS": ds}
            ),
        )
    )

wfs_state_machine = step.StateMachine(
    self,
    "WFSStateMachine",
    definition=wfs_jobs,
)

wfs_target = events_targets.SfnStateMachine(
    machine=wfs_state_machine,
)

events.Rule(
    self,
    "MetaRule",
    schedule=events.Schedule.rate(Duration.days(2)),
    targets=[wfs_target],
)

Possible Solution

It appears as if the default container execution role of EcsEc2ContainerDefinition is not granted logs:CreateLogStream permissions. If you manually remove the container execution role from the job definition, the job will succeed. I'm guessing it gets the necessary permissions from the BatchEnvironmentInstanceProfileRole. However, if you set the EcsJobDefinition execution_role property to None it will still create and assign a default execution role, which will fail with the permissions error.

Additional Information/Context

Batch jobs created with JobDefinition/JobDefinitionContainer in v2.73 do not appear to use a default execution role, whereas batch jobs created in v2.79 do.

CDK CLI Version

2.79.1

Framework Version

No response

Node.js Version

16.20

OS

Ubuntu 22.04.2

Language

Python

Language Version

3.8

Other information

No response

The text was updated successfully, but these errors were encountered:

peterwoodworth · 2023-05-23T00:25:28Z

If you manually remove the container execution role from the job definition, the job will succeed

Do you mean with escape hatches, manually removing the role from being associated with the container?

With the current implementation, a role should always be created, so an execution role should always be specified using this construct. Do you mean that this is the issue, and that no role should be created in some circumstances?

aws-cdk/packages/@aws-cdk/aws-batch-alpha/lib/ecs-container-definition.ts

Line 512 in bbdb16a

    
           this.executionRole = props.executionRole ?? createExecutionRole(this, 'ExecutionRole');

paulharbulak-maxar · 2023-05-23T14:08:29Z

@peterwoodworth thanks for the response. What I did was manually create a new revision of the job definition, and in that revision remove the execution role associated with the container.

Do you mean that this is the issue, and that no role should be created in some circumstances?

I think always specifying an execution role is fine, but I would hope that the default role wouldn't cause permissions errors with the default logging.

wttat · 2023-05-31T15:27:58Z

face this problem as well, the default execution_role should be add for create cloudwatch logs

aws/aws-cdk#25675

comcalvi · 2023-06-29T21:57:58Z

@wttat @paulharbulak-maxar are your containers actually calling CreateLogStream through the CLI / SDK? How is the log stream being created?

I reproduced this by not specifying logging. I think I have a fix for this, but in the mean time you can unblock yourselves by specifying logging.

tam0ri · 2023-07-01T11:32:33Z

Reproduction

I could reproduce it on my end. If we don't specify logging property, which means we don't specify logDriver explicitly in job definition, Batch uses awslogs with log group /aws/batch/job by default.

https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#containerProperties

The log driver to use for the job. By default, AWS Batch enables the awslogs log driver.

@wttat @paulharbulak-maxar are your containers actually calling CreateLogStream through the CLI / SDK? How is the log stream being created?

In this context, CreateLogStream API is called by Docker daemon, because log driver is Docker's functionality. So user's containers don't call CloudWatch Logs API.

Problem

From #25466, executionRole has been always created by default. This means that underlying ECS tasks always use not EC2 instance role but the execution role for API calls by log driver.

However, if logging property is not specified, grantWrite for the execution role is never called. grantWrite is called in bind method.

aws-cdk/packages/@aws-cdk/aws-batch-alpha/lib/ecs-container-definition.ts

Lines 518 to 526 in dcf5352

    
           if (props.logging) { 
        
             this.logDriverConfig = props.logging.bind(this, { 
        
               ...this as any, 
        
               // TS! 
        
               taskDefinition: { 
        
                 obtainExecutionRole: () => this.executionRole, 
        
               }, 
        
             }); 
        
           }

aws-cdk/packages/aws-cdk-lib/aws-ecs/lib/log-drivers/aws-log-driver.ts

Line 119 in dcf5352

    
           this.logGroup.grantWrite(containerDefinition.taskDefinition.obtainExecutionRole());

Solution

I think that one possible solution is granting permission for default log group to the execution role when logging property is not specified. If this approach is acceptable, I'll submit PR later.

) Grant `CreateLogStream` to the job definition's execution role by default. Without this permission, jobs will fail if they produce any output, unless `logging` is specified. Closes #25675. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions · 2023-07-12T11:39:45Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

…#26288) Grant `CreateLogStream` to the job definition's execution role by default. Without this permission, jobs will fail if they produce any output, unless `logging` is specified. Closes aws#25675. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

paulharbulak-maxar added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 22, 2023

github-actions bot added the @aws-cdk/aws-batch Related to AWS Batch label May 22, 2023

peterwoodworth added p1 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels May 23, 2023

wttat added a commit to wttat/af2-batch-cdk that referenced this issue May 31, 2023

Role permissions error after cdk upgrade

ca6f04a

aws/aws-cdk#25675

npvisual mentioned this issue Jun 27, 2023

(batch): Add grant methods to batch constructs where necessary #16020

Open

comcalvi mentioned this issue Jul 7, 2023

fix(batch): grant execution role logs:CreateLogStream by default #26288

Merged

mergify bot closed this as completed in #26288 Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_batch_alpha: Role permissions error after upgrade from v2.73 to v2.79 #25675

aws_batch_alpha: Role permissions error after upgrade from v2.73 to v2.79 #25675

paulharbulak-maxar commented May 22, 2023

peterwoodworth commented May 23, 2023

paulharbulak-maxar commented May 23, 2023

wttat commented May 31, 2023

comcalvi commented Jun 29, 2023 •

edited

Loading

tam0ri commented Jul 1, 2023 •

edited

Loading

github-actions bot commented Jul 12, 2023

aws_batch_alpha: Role permissions error after upgrade from v2.73 to v2.79 #25675

aws_batch_alpha: Role permissions error after upgrade from v2.73 to v2.79 #25675

Comments

paulharbulak-maxar commented May 22, 2023

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

peterwoodworth commented May 23, 2023

paulharbulak-maxar commented May 23, 2023

wttat commented May 31, 2023

comcalvi commented Jun 29, 2023 • edited Loading

tam0ri commented Jul 1, 2023 • edited Loading

Reproduction

Problem

Solution

github-actions bot commented Jul 12, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

comcalvi commented Jun 29, 2023 •

edited

Loading

tam0ri commented Jul 1, 2023 •

edited

Loading