Generate Presigned URLs #235
Description
Overview
It's nice to be able to generate presigned URLs for download/upload so you don't need to worry about passing around auth credentials.
It's often a little non-obvious how to do these with the downstream clients and each client handles the operation pretty differently so this feels like something that could fit nicely into cloudpathlib.
Proposed Interface/ implementation
I'd imagine the code for this looking something like the below implementation. The S3Path implementation works and is tested, the others (GSPath/AzureBlobPath) aren't actually tested but are about what the code would look like.
from datetime import datetime, timedelta
class CloudPath:
def generate_presigned_url(self, expire_seconds: int = 300):
raise NotImplementedError
class S3Path(CloudPath):
def generate_presigned_url(self, expire_seconds: int = 300):
object_key = "/".join(list(self.parts)[2:]) # Everything after the bucket name
url = self.client.client.generate_presigned_url(
"get_object",
Params={"Bucket": self.bucket, "Key": object_key},
ExpiresIn=expire_seconds,
)
class GSPath(CloudPath):
def generate_presigned_url(self, expire_seconds: int = 300):
object_key = "/".join(list(self.parts)[2:]) # Everything after the bucket name
gs_client = path.client.client
creds = gs_client.credentials
gs_bucket = path.client.client.get_bucket(path.bucket)
gs_blob = gs_bucket.blob(object_key)
url = gs_blob.generate_signed_url(
version="v4",
expiration=timedelta(seconds=expire_seconds),
service_account_email=creds.service_account_email,
access_token=creds.token,
method="GET"
)
return url
from azure.storage.blob import ResourceTypes, BlobSasPermissions, generate_blob_sas
class AzureBlobPath(CloudPath):
def generate_presigned_url(self, expire_seconds: int = 300):
object_key = "/".join(list(self.parts)[2:]) # Everything after the bucket name
az_client = self.client.client
sas_token = generate_blob_sas(
az_client.account_name,
container_name=self.container,
blob_name=object_key,
account_key=az_client.credential.account_key,
permission=BlobSasPermissions(read=True),
expiry=datetime.utcnow() + timedelta(seconds=expire_seconds)
)
url = f"https://{az_client.account_name}.blob.core.windows.net/{self.container}/{object_key}?{sas_token}"
return urlThe above implementation only handles downloads. Uploading via presigned urls/tokens is a bit weird across the different clouds but still doable. Happy to research that step more if it's of interest.
Happy to contribute the implementation, just want to make sure it's on track with the goals of the project.