Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add endpoint options to Amazon S3 Storage #97

Merged
merged 1 commit into from
Mar 30, 2022

Conversation

zinrai
Copy link
Contributor

@zinrai zinrai commented Aug 8, 2021

I want to use an Amazon S3 compatible service.
I want to add an endpoint option to the storage_options of Amazon S3 Storage.

@shevron
Copy link
Contributor

shevron commented Sep 9, 2021

Hi,

Looks good - I'm just making sure that this has been tested with plain S3 as well - that is, that passing None as the endpoint URL causes boto to fall back to the default URL, and not go crazy in some sense. Unfortunately, I don't have the means of testing it myself right now.

@zinrai
Copy link
Contributor Author

zinrai commented Oct 31, 2021

I checked the behavior when set endpoint_url=None for boto3.

# docker pull datopian/giftless:0.5.0
0.5.0: Pulling from datopian/giftless
69692152171a: Pull complete
66a3c154490a: Pull complete
82a02637ec8f: Pull complete
d8b2fbd22335: Pull complete
e04b68c1294e: Pull complete
c82d0e158525: Pull complete
b9d75891fa63: Pull complete
86f9fad6fa7e: Pull complete
8493dceb4a6d: Pull complete
87e47e2283f7: Pull complete
c77b7c14ee90: Pull complete
25b75f059cd6: Pull complete
bfda31167a7a: Pull complete
dc37863108fb: Pull complete
f5426d00752f: Pull complete
Digest: sha256:62ece51318a7efc5b7efc76ce28d068b66c0ec90b47a55537ad118b45a765c00
Status: Downloaded newer image for datopian/giftless:0.5.0
docker.io/datopian/giftless:0.5.0
# docker run -d datopian/giftless:0.5.0
fab7a01309d63d02447bfd0ff124b54f24f0700ff7325ae9c537c3ec37b06040
# docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED         STATUS         PORTS      NAMES
fab7a01309d6   datopian/giftless:0.5.0   "tini uwsgi -- -s 12…"   6 seconds ago   Up 6 seconds   5000/tcp   vigilant_chebyshev
# docker exec -it fab7a01309d6 /bin/bash
giftless@fab7a01309d6:~$ python3
Python 3.7.10 (default, May 12 2021, 16:15:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.resource('s3', endpoint_url=None, region_name="ap-northeast-1", aws_access_key_id="*****", aws_secret_access_key="*****)
>>> bucket = s3.Bucket("giftless-boto3-test")
>>> print(bucket)
s3.Bucket(name='giftless-boto3-test')
>>> print([obj_summary.key for obj_summary in bucket.objects.all()])
['hoge.txt']
>>>
giftless@fab7a01309d6:~$ python3
Python 3.7.10 (default, May 12 2021, 16:15:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.client('s3', endpoint_url=None, region_name="ap-northeast-1", aws_access_key_id="*****", aws_secret_access_key="*****")
>>> response = s3.list_objects_v2(Bucket="giftless-boto3-test")
>>> for object in response['Contents']:
...     print(object['Key'])
...
hoge.txt
>>>

@rufuspollock
Copy link
Member

@zinrai this looks good and i'm merging. /cc @shevron hope that is ok.

@rufuspollock rufuspollock merged commit 442488e into datopian:master Mar 30, 2022
@rufuspollock rufuspollock self-assigned this Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants