Skip to content

feat: S3 Path Based on Data #17603

@bbernays

Description

@bbernays

Which problem is this feature request solving?

When syncing large amounts of data from an AWS organization I want to be able to structure the path based on my access patterns. If Most of my access patterns are based on AccountID I want to be able to have AccountID in the path, but because I am using the Org feature I would have to make a job and config for each account which not feasible because accounts are always being added and removed

Describe the solution you'd like

It would be helpful if I could define a path variable whose value would be determined by the data being synced.

For example if I am syncing EC2 instance data, I know that there are columns for region and account_id. I would want to set my path as:

{{.account_id}}/{{.region}}/{{UUID}}.{{FORMAT}}

And then the S3 plugin would extract account_id and region so the data would go to the right object.

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions