Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(docs-website): add vercel preview environment #7644

Merged
merged 18 commits into from
Mar 22, 2023
Merged

feat(docs-website): add vercel preview environment #7644

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4806cea
feat(docs-website): add vercel preview environments
hsheth2 Mar 20, 2023
156923b
config build dir in vercel
hsheth2 Mar 20, 2023
cc62487
add always update deployment
hsheth2 Mar 20, 2023
bc86a71
add deployments permission
hsheth2 Mar 20, 2023
a9de060
remove deployment stuff
hsheth2 Mar 20, 2023
fb1ece6
undo change
hsheth2 Mar 20, 2023
72e6abf
add back deployment stuff
hsheth2 Mar 21, 2023
0b6777a
enable debug
hsheth2 Mar 21, 2023
3539cca
comment out deploy stuff
hsheth2 Mar 21, 2023
9b96132
setup vercel app
hsheth2 Mar 21, 2023
b0670c4
add deps
hsheth2 Mar 21, 2023
9ab1a88
support yum install
hsheth2 Mar 21, 2023
c0b8968
test vercel again
hsheth2 Mar 21, 2023
16e1f93
add python3 install command
hsheth2 Mar 21, 2023
55e52e3
test out new python cmd
hsheth2 Mar 21, 2023
721433e
add requirements
hsheth2 Mar 21, 2023
d1fd642
Merge branch 'master' into vercel-deploy
hsheth2 Mar 21, 2023
153078e
Merge branch 'master' into vercel-deploy
hsheth2 Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/documentation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ jobs:
- name: Build Docs
run: |
./gradlew --info docs-website:build

- name: Deploy
if: github.event_name == 'push' && github.repository == 'datahub-project/datahub'
uses: peaceiris/actions-gh-pages@v3
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,4 @@ datahub-frontend/public/**
metadata-service/plugin/src/test/resources/sample-plugins/**

smoke-test/rollback-reports
.vercel
1 change: 1 addition & 0 deletions docs-website/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,4 @@ npm-debug.log*
yarn-debug.log*
yarn-error.log*
.vscode
.vercel
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need both this and above?

1 change: 1 addition & 0 deletions docs-website/generateDocsDir.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,7 @@ function copy_python_wheels(): void {
"docs/actions/sources",
"docs/actions/guides",
"metadata-ingestion/archived",
"metadata-ingestion/sink_docs",
"docs/what",
"docs/wip",
];
Expand Down
38 changes: 6 additions & 32 deletions docs-website/sidebars.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,3 @@
const fs = require("fs");

function list_ids_in_directory(directory, hardcoded_labels) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

if (hardcoded_labels === undefined) {
hardcoded_labels = {};
}

const files = fs.readdirSync(`../${directory}`).sort();
let ids = [];
for (const name of files) {
if (fs.lstatSync(`../${directory}/${name}`).isDirectory()) {
// Recurse into the directory.
const inner_ids = list_ids_in_directory(`${directory}/${name}`);
ids = ids.concat(inner_ids);
} else {
if (name.endsWith(".md")) {
const slug = name.replace(/\.md$/, "");
const id = `${directory}/${slug}`;

if (id in hardcoded_labels) {
label = hardcoded_labels[id];
ids.push({ type: "doc", id, label });
} else {
ids.push({ type: "doc", id });
}
}
}
}
return ids;
}

// note: to handle errors where you don't want a markdown file in the sidebar, add it as a comment.
// this will fix errors like `Error: File not accounted for in sidebar: ...`
module.exports = {
Expand Down Expand Up @@ -142,7 +111,12 @@ module.exports = {
],
},
{
Sinks: list_ids_in_directory("metadata-ingestion/sink_docs"),
Sinks: [
{
type: "autogenerated",
dirName: "metadata-ingestion/sink_docs",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come these are in a different place from the source docs?

dirName: "docs/generated/ingestion/sources"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source docs are generated by another build script, whereas these are raw md files

},
],
},
{
Transformers: [
Expand Down
25 changes: 19 additions & 6 deletions metadata-ingestion/scripts/install_deps.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,23 @@ set -euxo pipefail
if [ "$(uname)" == "Darwin" ]; then
brew install librdkafka
else
sudo apt-get update && sudo apt-get install -y \
librdkafka-dev \
python3-ldap \
libldap2-dev \
libsasl2-dev \
ldap-utils
sudo_cmd=""
if command -v sudo; then
sudo_cmd="sudo"
fi
Comment on lines +7 to +10
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if we care, but this will print the result of command -v sudo to stdout. Can either pipe into /dev/null or check [ -x "$(command -v sudo)" ] as per https://stackoverflow.com/questions/592620/how-can-i-check-if-a-program-exists-from-a-bash-script

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I didn't realize that

In this case I don't think we care because it's just the path to sudo and that script is only run in CI


if command -v yum; then
$sudo_cmd yum install -y \
librdkafka-devel \
openldap-devel \
cyrus-sasl-devel \
openldap-clients
else
$sudo_cmd apt-get update && $sudo_cmd apt-get install -y \
librdkafka-dev \
python3-ldap \
libldap2-dev \
libsasl2-dev \
ldap-utils
fi
fi
9 changes: 9 additions & 0 deletions vercel.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"buildCommand": "./gradlew :docs-website:build",
"github": {
"silent": true,
"autoJobCancelation": true
},
"installCommand": "amazon-linux-extras install python3.8 && py3=\"$(which python3)\" && rm \"$py3\" && ln \"$(which python3.8)\" \"$py3\" && ./metadata-ingestion/scripts/install_deps.sh && yum install -y gcc python38-devel",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come we're using python3.8 here? And what is running this command?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this command is being run by vercel, which is a third-party tool

"outputDirectory": "docs-website/build"
}