Skip to content

Helper functions for navigating and getting data from an Amazon simple storage service (S3).

Notifications You must be signed in to change notification settings

roberto1648/AWS-S3-Helper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 

Repository files navigation

AWS S3 Helper

First open aws_helper.py in an editor and paste your access key and secret to ACCESS_KEY and SECRET, respectively.

To get a list of buckets

list_buckets(access_key=ACCESS_KEY, secret=SECRET)

To get a list of keys in a bucket with optional prefix and sufix in their name

e.g.,

list_keys( bucket_name, prefix='dir1/subdir/something', sufix='.json', access_key=ACCESS_KEY, secret=SECRET, )

If there are too many keys, use instead s3_bucket_keys_generator to not load all the keys at once. These functions use pagination to find all the keys.

Read a json object

dict = get_json_from_object( bucket_name, key="dir/subdir/some.json", access_key=ACCESS_KEY, secret=SECRET, )

Read a binary array

arr = get_numerical_array_from_object( bucket_name, key="dir/subdir/some.bin", access_key=ACCESS_KEY, secret=SECRET, res=4, from_to=(0, None), )

from_to allows downloading only a slice (arr = big_array(from_to[0]: from_to[1]). Use None to refer to the begining or the end, e.g., arr[:50] -> (None, 50). Input res refers to the byte resolution of the binary file.

About

Helper functions for navigating and getting data from an Amazon simple storage service (S3).

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages