Adding YARNContainerFactory. This allows OpenWhisk to run actions on Apache Hadoop clusters. by SamHjelmfelt · Pull Request #4129 · apache/openwhisk

SamHjelmfelt · 2018-11-21T00:03:04Z

Thousands of organizations have Apache Hadoop clusters today. By implementing a YARNContainerFactory, OpenWhisk can run actions on Hadoop clusters. This will lower the barrier to adoption and expand the potential use cases for OpenWhisk.

Description

The YARNContainerFactory uses the Apache Hadoop Services REST API to create a single YARN service with a component for each action type. Both simple authentication and Kerberos/SPNEGO are supported. This was tested with Apache Hadoop 3.1.1.
https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html
The implemention is based on the MesosContainerFactory.

This was first implemented using Akka HTTP, but was rewritten to use the Apache HTTP client in order to suport SPNEGO.

There is a MockYARNRM in the tests directory. This mock RM simulates the YARN Resource Manager REST API and is used for the YARNContainerFactory tests.

Related issue and scope

I opened an issue to propose and discuss this change (#????)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
The website should be updated as well
I updated the documentation where necessary.

rabbah · 2018-11-21T00:04:37Z

🤯 this is excellent!

SamHjelmfelt · 2018-11-21T00:12:22Z

For anyone wanting to test this out, here is a quick guide for enabling Docker on YARN:
https://community.hortonworks.com/articles/226331/dockerized-yarn-services-quickstart.html

It was tested with HDP 3.0.1 from Hortonworks. Available here:
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/index.html

The steps are similar with a base Apache Hadoop installation (version 3.1.1 or higher).

rabbah

This looks generally straightforward to follow but there are some things to consider in the use of futures with synchronized and blocking operations. I'm curious to try this and will find some time to do that.

SamHjelmfelt · 2018-12-13T19:59:37Z

@rabbah Thanks for the feedback. I refactored the solution to use an Actor instead of synchronized blocks.

The TravisCI build errored due to:
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

rabbah · 2018-12-13T20:04:19Z

Nice! Thanks for improving the PR. I will go through the changes and restart Travis for you.

codecov-io · 2018-12-14T04:27:05Z

Codecov Report

Merging #4129 into master will decrease coverage by 5.04%.
The diff coverage is 79.91%.

@@            Coverage Diff            @@
##           master   #4129      +/-   ##
=========================================
- Coverage   85.75%   80.7%   -5.05%     
=========================================
  Files         163     168       +5     
  Lines        7594    7838     +244     
  Branches      502     525      +23     
=========================================
- Hits         6512    6326     -186     
- Misses       1082    1512     +430

Impacted Files	Coverage Δ
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	`94.57% <100%> (+0.04%)`	⬆️
.../org/apache/openwhisk/core/yarn/YARNRESTUtil.scala	`57.4% <57.4%> (ø)`
...cala/org/apache/openwhisk/core/yarn/YARNTask.scala	`70% <70%> (ø)`
...pache/openwhisk/core/yarn/YARNComponentActor.scala	`85.29% <85.29%> (ø)`
...che/openwhisk/core/yarn/YARNContainerFactory.scala	`87.5% <87.5%> (ø)`
...e/openwhisk/core/yarn/YARNContainerInfoActor.scala	`87.75% <87.75%> (ø)`
...core/database/cosmosdb/RxObservableImplicits.scala	`0% <0%> (-100%)`	⬇️
...core/database/cosmosdb/CosmosDBArtifactStore.scala	`0% <0%> (-95.18%)`	⬇️
...sk/core/database/cosmosdb/CosmosDBViewMapper.scala	`0% <0%> (-92.6%)`	⬇️
...whisk/core/database/cosmosdb/CosmosDBSupport.scala	`0% <0%> (-84.62%)`	⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6cc6ab4...69532fb. Read the comment docs.

SamHjelmfelt · 2018-12-14T19:11:50Z

The test coverage is missing some of the error handling and all the Kerberos/SPNEGO code. Is it possible to test Kerberos/SPNEGO with TravisCI?

rabbah · 2018-12-16T20:04:56Z

I don't know @SamHjelmfelt - not something I've tried. We do have a Jenkins job we can try once it's fully operational.

SamHjelmfelt · 2019-01-09T20:16:08Z

Here is a YARN sandbox that will simplify testing. It is a ~5GB docker image that contains YARN pre-installed and pre-configured. Just run this container and configure the invoker for a YARN RM at localhost:8088. Just use yarnquickstart-sample-hotfix.ini instead of yarnquickstart-sample.ini.
https://community.hortonworks.com/articles/232540/docker-on-yarn-sandbox.html.

Additionally, the following project has this pull request pre-configured with a single command quickstart. The relevant YARN configurations are commented out in the docker-whisk-controller.env file.
https://github.com/SamHjelmfelt/OpenWhisk-YarnDeployment.

rabbah · 2019-02-09T14:44:32Z

sorry @SamHjelmfelt i'm behind on this but i'm going through and regaining velocity on the prs so i'm hoping to get to it sooner rather than later.

…Apache Hadoop clusters. Rebased and squashed.

…decommissioning (YARN-8761)

SamHjelmfelt · 2019-02-15T21:15:51Z

Rebasing

rabbah · 2019-02-16T21:24:29Z

Thanks for reviewing this code with me and showing me a demo. Minor nits but LGTM generally.

SamHjelmfelt · 2019-02-20T02:45:07Z

@rabbah I believe I addressed everything you requested, but I am happy to make further improvements.

rabbah · 2019-02-23T14:05:27Z

@SamHjelmfelt this is a very neat addition!

…Apache Hadoop clusters. (apache#4129)

SamHjelmfelt force-pushed the master branch 2 times, most recently from e7753ca to d926b7d Compare November 21, 2018 17:30

rabbah reviewed Nov 30, 2018

View reviewed changes

rabbah added enhancement invoker review Review for this PR has been requested and yet needs to be done. labels Nov 30, 2018

SamHjelmfelt force-pushed the master branch 2 times, most recently from adb4079 to 4d57970 Compare December 17, 2018 23:21

rabbah self-assigned this Jan 31, 2019

chetanmeh reviewed Feb 15, 2019

View reviewed changes

Comment thread common/scala/src/main/resources/application.conf

SamHjelmfelt added 3 commits February 15, 2019 13:13

Adding YARNContainerFactory. This allows OpenWhisk to run actions on …

35dfff9

…Apache Hadoop clusters. Rebased and squashed.

Added support for invoker HA. Each creates its own YARN service

62706d6

Refactored to improve parallelism. Adding support for YARN container …

a7f7eed

…decommissioning (YARN-8761)

SamHjelmfelt force-pushed the master branch from 23f47ae to a7f7eed Compare February 15, 2019 21:15

SamHjelmfelt added 2 commits February 15, 2019 13:35

Accomodating changes to ContainerArgsConfig class

d22175c

Adding license headers

621bd3e

rabbah approved these changes Feb 16, 2019

View reviewed changes

rabbah added reviewed Review for this PR is finished. It is mergeable from a review's perspective. and removed review Review for this PR has been requested and yet needs to be done. labels Feb 18, 2019

rabbah removed the review Review for this PR has been requested and yet needs to be done. label Feb 18, 2019

Formating changes

4b94b3e

Added comments to application.conf

69532fb

rabbah merged commit 6e883f9 into apache:master Feb 23, 2019

BillZong pushed a commit to BillZong/openwhisk that referenced this pull request Nov 18, 2019

Adding YARNContainerFactory. This allows OpenWhisk to run actions on …

7cec5d7

…Apache Hadoop clusters. (apache#4129)

Uh oh!

Conversation

SamHjelmfelt commented Nov 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

Uh oh!

rabbah commented Nov 21, 2018

Uh oh!

SamHjelmfelt commented Nov 21, 2018

Uh oh!

rabbah left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SamHjelmfelt commented Dec 13, 2018

Uh oh!

rabbah commented Dec 13, 2018

Uh oh!

codecov-io commented Dec 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

SamHjelmfelt commented Dec 14, 2018

Uh oh!

rabbah commented Dec 16, 2018

Uh oh!

SamHjelmfelt commented Jan 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rabbah commented Feb 9, 2019

Uh oh!

Uh oh!

SamHjelmfelt commented Feb 15, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rabbah commented Feb 16, 2019

Uh oh!

SamHjelmfelt commented Feb 20, 2019

Uh oh!

rabbah commented Feb 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SamHjelmfelt commented Nov 21, 2018 •

edited

Loading

rabbah left a comment •

edited

Loading

codecov-io commented Dec 14, 2018 •

edited

Loading

SamHjelmfelt commented Jan 9, 2019 •

edited

Loading