Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make repositories configurable for enterprise developers #9230

Merged
merged 4 commits into from
Nov 28, 2023
Merged

Make repositories configurable for enterprise developers #9230

merged 4 commits into from
Nov 28, 2023

Conversation

githendrik
Copy link
Contributor

Allow enterprise developers to build it behind corporate firewalls

Issue faced

At my company we're prohibited to pull from most public registries (Gradle, Maven, Npm, ...) directly and are forced to go via artifactory mirrors. Unfortunately Datahub doesn't currently support this.

I've dug into the code and propse the following changes to allow customisation of the docker builds. I've changed all Dockerfiles and the required build.gradle files.

How I now build the project is by setting the following GRADLE_OPTS:

GRADLE_OPTS_LIST=(
 "-Dorg.gradle.internal.plugins.portal.url.override=https://jfrog.enterprise.corp/artifactory/gradle-plugins-release-remote"
 "-Dorg.gradle.project.nodeDistBaseUrl=https://jfrog.enterprise.corp/artifactory/nodejs-dist-remote/"
 "-Dorg.gradle.project.alpineApkRepositoryUrl=https://jfrog.enterprise.corp/artifactory/alpine-remote"
 "-Dorg.gradle.project.debianAptRepositoryUrl=https://jfrog.enterprise.corp/artifactory/debian-deb-remote"
 "-Dorg.gradle.project.githubMirrorUrl=https://jfrog.enterprise.corp/artifactory/github-generic-remote"
 "-Dorg.gradle.project.mavenCentralRepositoryUrl=https://jfrog.enterprise.corp/artifactory/maven-remote"
 "-Dorg.gradle.project.apacheMavenRepositoryUrl=https://jfrog.enterprise.corp/artifactory/apache-maven-remote"
 "-Dorg.gradle.project.confluentMavenRepositoryUrl=https://jfrog.enterprise.corp/artifactory/confluent-maven-remote"
 "-Dorg.gradle.project.linkedinOpenSourceRepositoryUrl=https://jfrog.enterprise.corp/artifactory/linkedin-open-source-maven-remote"
 "-Dorg.gradle.project.pipMirrorUrl=https://jfrog.enterprise.corp/artifactory/api/pypi/pypi-remote/simple"
 "-Dorg.gradle.project.apacheDownloadUrl=https://jfrog.enterprise.corp/artifactory/apache-generic-remote"
)

Once these are set, I'm able to build the following gradle targets successfully:

:datahub-frontend:docker -x yarnTest -x yarnLint
:metadata-service:war:docker
:datahub-upgrade:docker
:docker:datahub-ingestion:docker
:docker:datahub-ingestion-base:docker
:metadata-jobs:mae-consumer-job:docker
:metadata-jobs:mce-consumer-job:docker
:docker:elasticsearch-setup:docker
:docker:kafka-setup:docker
:docker:mysql-setup:docker
:docker:postgres-setup:docker

When exiting the corporate network, I'm still able to build the same targets after unsetting the GRADLE_OPTS, so I assume everything will simply keep working for everyone else.

Please let me know if you see something missing (or, come to think of it, if I should add these configuration flags to a documentation page somewhere).

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • [] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Nov 13, 2023
david-leifker
david-leifker approved these changes Nov 28, 2023
View reviewed changes
Copy link
Collaborator

@david-leifker david-leifker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

@david-leifker david-leifker added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Nov 28, 2023
@david-leifker david-leifker merged commit 966cb17 into datahub-project:master Nov 28, 2023
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@david-leifker david-leifker david-leifker approved these changes

Assignees
No one assigned
Labels
devops PR or Issue related to DataHub backend & deployment ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green. product PR or Issue related to the DataHub UI/UX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@githendrik @david-leifker @hrichert