Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(Spark-smoke-test): add spark smoke test #4158

Merged
merged 40 commits into from
Feb 23, 2022
Merged

feat(Spark-smoke-test): add spark smoke test #4158

merged 40 commits into from
Feb 23, 2022

Conversation

MugdhaHardikar-GSLab
Copy link
Contributor

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

@MugdhaHardikar-GSLab
fix(spark-lineage): select mock server port dynamically for unit test
09441d3
@MugdhaHardikar-GSLab
refactor(spark-lineage): remove dependency of spark from McpEmitter
cac9e4b
@MugdhaHardikar-GSLab
fix(spark-lineage): fix checkstyle issues
6f64a20
@MugdhaHardikar-GSLab
fix(spark-lineage): restore consumers
8270c24
@MugdhaHardikar-GSLab
Merge branch 'linkedin:master' into master
dffa6c4
@MugdhaHardikar-GSLab
refactor(spark-lineage): enhance logging and documentation
1675dfe
@MugdhaHardikar-GSLab
Update McpEmitter.java
d6583fc
@MugdhaHardikar-GSLab
add debugging section
8e9291c
@MugdhaHardikar-GSLab
refactor(spark-lineage): enhance logs
7ba4ddf
@MugdhaHardikar-GSLab
Merge branch 'spark-logging' of https://github.com/MugdhaHardikar-GSL…
d396664 
…ab/datahub into spark-logging
@MugdhaHardikar-GSLab
Merge pull request #1 from MugdhaHardikar-GSLab/spark-logging
08b51d7 
refactor(spark-lineage): enhance logging and documentation
@MugdhaHardikar-GSLab
feat(spark-lineage): add spark integration smoke test
8ac3541
@MugdhaHardikar-GSLab
Update spark-smoke.yml
3a4680d
@MugdhaHardikar-GSLab
Update test_e2e.py
5f12d5b
@MugdhaHardikar-GSLab
Update PythonHdfsIn2HiveCreateInsertTable.json
27288c7
@MugdhaHardikar-GSLab
Update JavaHdfsIn2HiveCreateInsertTable.json
225bb89
@MugdhaHardikar-GSLab
Update build-and-test.yml
669e421
@MugdhaHardikar-GSLab
Update smoke.sh
0814f86
@MugdhaHardikar-GSLab
refactor(spark-lineage-smoke): move spark-smoke test to outer test
4b1bb57
@MugdhaHardikar-GSLab
modify permission for setup_spark_smoke_test.sh
97063ec
@MugdhaHardikar-GSLab
add requirement.txt installation
002a900
@MugdhaHardikar-GSLab
Update smoke.sh
c77ecc2
@MugdhaHardikar-GSLab
remove local test file
13abc22
@MugdhaHardikar-GSLab
Delete spark-aws.conf
c6f58f4
@MugdhaHardikar-GSLab
Delete junit.spark.smoke.xml
d39518b
@MugdhaHardikar-GSLab
remove unwanted files
0c51f79
@MugdhaHardikar-GSLab
Delete spark-smoke.yml
108bd59
@github-actions
Copy link

github-actions bot commented Feb 16, 2022 •
edited
Loading

Unit Test Results (build & test)

  70 files  ±0    70 suites  ±0   25m 41s ⏱️ + 12m 0s
611 tests ±0  552 ✔️ ±0  59 💤 ±0  0 ❌ ±0 

Results for commit 17cef08. ± Comparison against base commit c6cb549.

♻️ This comment has been updated with latest results.

shirshanka
shirshanka requested changes Feb 18, 2022
View reviewed changes
Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have a few questions.

smoke-test/spark-smoke-test/test-spark-lineage/gradlew Outdated
@@ -0,0 +1,188 @@
#!/usr/bin/env sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to add in a new gradle here? can we not use the module-based way of building this?

smoke-test/spark-smoke-test/smoke.sh Outdated
echo "--------------------------------------------------------------------"


datahub docker quickstart \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use the python way of starting the docker containers (like the other smoke tests)

@shirshanka shirshanka added the pending-submitter-response Issue/request has been reviewed but requires a response from the submitter label Feb 20, 2022
shirshanka
shirshanka approved these changes Feb 23, 2022
View reviewed changes
Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@shirshanka shirshanka merged commit ede31c4 into datahub-project:master Feb 23, 2022
@MugdhaHardikar-GSLab MugdhaHardikar-GSLab deleted the spark-smoke branch March 30, 2022 05:14
maggiehays pushed a commit to maggiehays/datahub that referenced this pull request Aug 1, 2022
@MugdhaHardikar-GSLab @maggiehays
feat(ci): add spark smoke test (datahub-project#4158)
7963057
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shirshanka shirshanka shirshanka approved these changes

Assignees
No one assigned
Labels
pending-submitter-response Issue/request has been reviewed but requires a response from the submitter
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MugdhaHardikar-GSLab @shirshanka