could you share more about the failing authentication issue ?

would be great if the CI tests would catch this, as it uses real S3 for testing...

@neolynx I saw the same issue and it was specific to EC2 IMDS (perhaps isolated to IMDSv2) combined with publish to S3. Maybe your CI tests publish to S3 but aren't using EC2 IMDS to obtain the access keys?

The error is reproducible on an EC2 instance (or a container image running on an EC2 instance) with IMDSv2 enabled, and no specific IAM policies required. The error happens before IAM policy policy evaluation. The issue is explained very well here:

aws/aws-sdk-go-v2#2370 (comment)

Steps To Reproduce

Using a container is recommended, but not required.

docker run --rm -ti ubuntu:22.04

The issue can be reproduced consistently.

# Prepare to install Aptly
apt-get update && apt-get -y install wget jq
wget -O /etc/apt/keyrings/aptly.asc https://www.aptly.info/pubkey.txt
cat <(echo "deb [signed-by=/etc/apt/keyrings/aptly.asc] http://repo.aptly.info/ci jammy main") > /etc/apt/sources.list.d/aptlyci.list

# actually install Aptly
apt-get update && apt-get -y install aptly

# create a repo and the default aptly.conf
aptly repo create test

# add S3PublishEndpoints to aptly.conf
cp ~/.aptly.conf ~/.aptly.conf.orig
jq '.S3PublishEndpoints={"test":{"region":"us-east-1","bucket":"dontcare"}}' ~/.aptly.conf.orig > ~/.aptly.conf

# attempt to publish repo
aptly publish repo -architectures=amd64 -distribution=test -skip-signing test s3:test:

The following error is observed

Warning: publishing from empty source, architectures list should be complete, it can't be changed after publishing (use -architectures flag)
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
ERROR: unable to publish: unable to publish file: error uploading /tmp/aptly1351411402/main_binary-amd64_Packages to S3: us-east-1:dontcare/: operation error S3: PutObject, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, not found, Signing

I would highlight this part of the error message in particular:

get credentials: failed to refresh cached credentials, no EC2 IMDS role found, not found, Signing

Despite the error message, EC2 IMDS is indeed available.

root@eb7e71c33e06:/# apt-get -y install cloud-utils
root@eb7e71c33e06:/# ec2metadata --instance-type
c6a.2xlarge

Version under test

root@eb7e71c33e06:/# aptly version
aptly version: 1.6.0~rc1+20241117171302.763b810c

root@eb7e71c33e06:/# apt-cache policy aptly
aptly:
  Installed: 1.6.0~rc1+20241117171302.763b810c