Using S3 sounds like a great way to set yourself up to be financially trolled hard. There are lots of unlikable companies people hold grudges against. Many many companies use S3 nowadays even on their corporate websites via CMSes for image hosting etc.
I notified the AWS security team. I suggested that they restrict the unfortunate S3 bucket name to protect their customers from unexpected charges, and to protect the impacted companies from data leaks. But they were unwilling to address misconfigurations of third-party products.
Anyone registering this bucket will either be
a) Screwed over by the same absurd costs OP is seeing, or
b) Maliciously using it to steal data
There is literally no “happy case” here. Despite being aware of this problem, Amazon isn’t even willing to stop people from registering it.
AWS has changed the bucket naming standards in the past, they could resolve the entire issue by just not allowing low entropy bucket names. I’m guessing the bucket in question was called something like “backup-data” or similar, while buckets /should/ be using 63 mostly random characters for a name.
The problem is made worse by the fact that bucket names are immutable, so if you leak a bucket name or if something collides with your bucket name you have ZERO RECOURSE aside from copying all of your data to a new bucket and deleting the old bucket.
Worse: it doesn’t matter what you name your bucket if you’re doing direct-to-S3 uploads from your front end, using pre-signed URLs. Or (less likely) serving anything directly from an S3 bucket.
Honestly, I think Amazon should seriously reconsider charging for auth failures. That, not bucket naming policy changes, is the fix here.
I would really be shocked to be charged for unauthorized requests, especially if this wasn’t mentioned in the documentation around choosing a bucket name. That seems extremely customer-hostile.
Does anyone know if competitors do the same? B2, R2, wasabi, linode object storage, DO spaces and similar?
If they don’t and Amazon continues to, this would seem like a (another?) good reason to avoid S3.
I’m honestly sort of amazed that s3 allows users to create manually named buckets and that their bucket naming rules allow for trivial names.
Bucket names must be between 3 (min) and 63 (max) characters long.
Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).
Bucket names must begin and end with a letter or number.
Bucket names must not contain two adjacent periods.
Bucket names must not be formatted as an IP address (for example, 192.168.5.4).
Using the above rules you top out on entropy at around 320~ bits which is good if you randomly generate the bucket name and use the maximum length, which /should/ be the best practice but… that is not what amazon docs suggest you do:
When naming a bucket, choose a name that is relevant to you or your business. Avoid using names associated with others. For example, you should avoid using AWS or Amazon in your bucket name.
If your application automatically creates buckets, choose a bucket naming scheme that is unlikely to cause naming conflicts. Ensure that your application logic will choose a different bucket name if a bucket name is already taken.
TLDR; if you use S3 and you aren’t treating your buckets names as a privileged credential you should expect that any adversarial actor will be able to punish you financially even if the rest of your access controls are correctly and securely configured.
edit:
It’d be a great basic improvement if the S3 bucket creation API defaulted to not requiring a name parameter and it instead randomly generated a high entropy bucket name. Additionally it could return a warning if the requested bucket name had a low entropy OR even require a force=1 type of parameter to create a bucket with a low entropy name.
P.S. I’m not an S3 user, I just briefly glanced over some of the documentation, the above may contain inaccuracies
A bucket is a subdomain, but with per-request billing instead of bandwidth-based billing. It looks like PUT requests are $0.005/1000 on AWS. Spitballing some numbers, let’s say a (rejected, like the author experienced) PUT request is 200 bytes. 1000 requests comes out to 200kB, and AWS charges come out to $0.025/MB = $25/GB if I’m trying to hammer your bucket to incur costs for you.
Comparatively, a $10/mo DigitalOcean Droplet gets billed for traffic overages at $0.01/GB. There’s a pretty massive scale difference here as far as costs go.
I’m just talking about the requirements for naming, like the op.
I’m very good with librarian paternalism here and nudging people away from low entropy names, but not allowing anything that complies with a domain name doesn’t sound like the correct solution here.
Yes, this would be much more secure, but also make for a very poor UX.
One of the common usage patterns for s3 is to copy data over. Imagine having to force users from ‘aws s3 sync mystuff mybucket ’ to enter either an env variable or an enormous token-like string.
And even if eventually it can be fixed with alias or nicknames it seems that indirection would not be the answer. I agree with the folks that see no inherent issue with the bucket name, if the charges for errors are tackled .
What I’m really surprised about is that I thought that bucket names were enforced to be unique, exactly like public domain names, so I wonder if I missed some changes to that rule?
S3 charges for unauthorized requests?? That’s crazy!
The house always wins.
Using S3 sounds like a great way to set yourself up to be financially trolled hard. There are lots of unlikable companies people hold grudges against. Many many companies use S3 nowadays even on their corporate websites via CMSes for image hosting etc.
Can think of many companies I’d like to punish, but that would just further enrich Amazon.
This is by far the craziest part IMO
Anyone registering this bucket will either be
a) Screwed over by the same absurd costs OP is seeing, or
b) Maliciously using it to steal data
There is literally no “happy case” here. Despite being aware of this problem, Amazon isn’t even willing to stop people from registering it.
AWS has changed the bucket naming standards in the past, they could resolve the entire issue by just not allowing low entropy bucket names. I’m guessing the bucket in question was called something like “backup-data” or similar, while buckets /should/ be using 63 mostly random characters for a name.
The problem is made worse by the fact that bucket names are immutable, so if you leak a bucket name or if something collides with your bucket name you have ZERO RECOURSE aside from copying all of your data to a new bucket and deleting the old bucket.
Jeff Barr has acknowledged the issue and is promising they are looking into it: https://twitter.com/jeffbarr/status/1785386554372042890.
Worse: it doesn’t matter what you name your bucket if you’re doing direct-to-S3 uploads from your front end, using pre-signed URLs. Or (less likely) serving anything directly from an S3 bucket.
Honestly, I think Amazon should seriously reconsider charging for auth failures. That, not bucket naming policy changes, is the fix here.
I would really be shocked to be charged for unauthorized requests, especially if this wasn’t mentioned in the documentation around choosing a bucket name. That seems extremely customer-hostile.
Does anyone know if competitors do the same? B2, R2, wasabi, linode object storage, DO spaces and similar?
If they don’t and Amazon continues to, this would seem like a (another?) good reason to avoid S3.
I’m honestly sort of amazed that s3 allows users to create manually named buckets and that their bucket naming rules allow for trivial names.
Using the above rules you top out on entropy at around 320~ bits which is good if you randomly generate the bucket name and use the maximum length, which /should/ be the best practice but… that is not what amazon docs suggest you do:
TLDR; if you use S3 and you aren’t treating your buckets names as a privileged credential you should expect that any adversarial actor will be able to punish you financially even if the rest of your access controls are correctly and securely configured.
edit:
It’d be a great basic improvement if the S3 bucket creation API defaulted to not requiring a name parameter and it instead randomly generated a high entropy bucket name. Additionally it could return a warning if the requested bucket name had a low entropy OR even require a
force=1type of parameter to create a bucket with a low entropy name.P.S. I’m not an S3 user, I just briefly glanced over some of the documentation, the above may contain inaccuracies
These are essentially the rules for domain names, right? Which is all a bucket is, a subdomain.
A bucket is a subdomain, but with per-request billing instead of bandwidth-based billing. It looks like PUT requests are $0.005/1000 on AWS. Spitballing some numbers, let’s say a (rejected, like the author experienced) PUT request is 200 bytes. 1000 requests comes out to 200kB, and AWS charges come out to $0.025/MB = $25/GB if I’m trying to hammer your bucket to incur costs for you.
Comparatively, a $10/mo DigitalOcean Droplet gets billed for traffic overages at $0.01/GB. There’s a pretty massive scale difference here as far as costs go.
I’m just talking about the requirements for naming, like the op.
I’m very good with librarian paternalism here and nudging people away from low entropy names, but not allowing anything that complies with a domain name doesn’t sound like the correct solution here.
At least Amazon doesn’t issue a cert for every domain that then can be looked up in cert registries… https://crt.sh/?q=s3.amazonaws.com
Yes, this would be much more secure, but also make for a very poor UX.
One of the common usage patterns for s3 is to copy data over. Imagine having to force users from ‘aws s3 sync mystuff mybucket ’ to enter either an env variable or an enormous token-like string.
And even if eventually it can be fixed with alias or nicknames it seems that indirection would not be the answer. I agree with the folks that see no inherent issue with the bucket name, if the charges for errors are tackled .
What I’m really surprised about is that I thought that bucket names were enforced to be unique, exactly like public domain names, so I wonder if I missed some changes to that rule?
I’m curious how other vendors deal with that - I’ve seen some claims various Azure services do the same, but not much detail.
Seems like Backblaze B2 (which also is the cheapest and has the same guarantees as S3) doesn’t.
At least according to this: https://www.reddit.com/r/backblaze/comments/16o4bed/charge_by_404_request/
Azure and GCP seem to charge.
You can use the requester pays option!
AWS’s docs: Using Requester Pays buckets for storage transfers and usage
According to this section, that feature wouldn’t have avoided the charges:
So, you can “make” the requestor pay, if they want to pay for it?