Skip to content

feat(ingest/s3): support path_specs of different S3 buckets in the same recipe #7514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hsheth2 merged 1 commit into from
Mar 15, 2023
Merged

feat(ingest/s3): support path_specs of different S3 buckets in the same recipe #7514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion gradle.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,3 @@ org.gradle.internal.repository.initial.backoff=1000

# Needed to publish to Nexus from a sub-module
gnsp.disableApplyOnlyOnRootProjectEnforcement=true

2 changes: 1 addition & 1 deletion metadata-ingestion/examples/recipes/csv_enricher_to_datahub_rest.dhub.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ source:
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
server: "http://localhost:8080"
2 changes: 1 addition & 1 deletion metadata-ingestion/examples/recipes/hana_to_datahub.dhub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ source:
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
server: "http://localhost:8080"
2 changes: 1 addition & 1 deletion metadata-ingestion/examples/recipes/mode_to_datahub.dhub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ source:
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
server: "http://localhost:8080"
2 changes: 1 addition & 1 deletion metadata-ingestion/examples/recipes/okta_to_datahub.dhub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ source:
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
server: "http://localhost:8080"
1 change: 0 additions & 1 deletion metadata-ingestion/examples/recipes/s3_to_file.dhub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,3 @@ sink:
type: "file"
config:
filename: "./s3_data_lake_mces.json"

2 changes: 1 addition & 1 deletion metadata-ingestion/examples/recipes/snowflake_to_datahub.dhub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ source:
email_domain: mycompany.com

classification:
enabled: True
enabled: True
2 changes: 1 addition & 1 deletion metadata-ingestion/examples/recipes/tableau_to_datahub.dhub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ source:
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
server: "http://localhost:8080"
1 change: 0 additions & 1 deletion metadata-ingestion/scripts/datahub_preflight.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,3 @@ fi


printf "\n\e[38;2;0;255;0m✅ Preflight was successful\e[38;2;255;255;255m\n"

11 changes: 0 additions & 11 deletions metadata-ingestion/src/datahub/ingestion/source/s3/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
from datahub.configuration.validate_field_rename import pydantic_renamed_field
from datahub.ingestion.source.aws.aws_common import AwsConnectionConfig
from datahub.ingestion.source.aws.path_spec import PathSpec
from datahub.ingestion.source.aws.s3_util import get_bucket_name
from datahub.ingestion.source.s3.profiling import DataLakeProfilerConfig

# hide annoying debug errors from py4j
Expand Down Expand Up @@ -92,16 +91,6 @@ def check_path_specs_and_infer_platform(
)
guessed_platform = guessed_platforms.pop()

# If platform is s3, check that they're all the same bucket.
if guessed_platform == "s3":
bucket_names = set(
get_bucket_name(path_spec.include) for path_spec in path_specs
)
if len(bucket_names) > 1:
raise ValueError(
f"All path_specs should reference the same s3 bucket. Got {bucket_names}"
)

# Ensure s3 configs aren't used for file sources.
if guessed_platform != "s3" and (
values.get("use_s3_object_tags") or values.get("use_s3_bucket_tags")
Expand Down
Loading