[New Scheduler]Add CreationJobManager#5116
Merged
ningyougang merged 6 commits intoJun 7, 2021
Merged
Conversation
jiangpengcheng
force-pushed
the
add_creation_job_manager
branch
from
12e8fff to
1700c8d
Compare
Codecov Report
@@ Coverage Diff @@
## master #5116 +/- ##
==========================================
- Coverage 81.86% 75.75% -6.12%
==========================================
Files 225 227 +2
Lines 11423 11528 +105
Branches 501 497 -4
==========================================
- Hits 9352 8733 -619
- Misses 2071 2795 +724
Continue to review full report at Codecov.
|
ningyougang
reviewed
May 25, 2021
|
|
||
| private val topicPrefix = loadConfigOrThrow[String](ConfigKeys.kafkaTopicsPrefix) | ||
| private val topic = s"${topicPrefix}creationAck${schedulerInstanceId.asString}" | ||
| private val maxActiveAcksPerPoll = 128 |
Contributor
There was a problem hiding this comment.
In Scheduler.scala, this value is configurable.
Can we make it configurable as well?
Contributor
Author
There was a problem hiding this comment.
it seems other places are also hardcoded
ningyougang
reviewed
May 25, 2021
| dataManagementService: ActorRef)(implicit actorSystem: ActorSystem, logging: Logging) | ||
| extends Actor { | ||
| private implicit val ec: ExecutionContext = actorSystem.dispatcher | ||
| private val baseTimeout = loadConfigOrThrow[Int](ConfigKeys.schedulerInProgressJobRetentionSecond).seconds |
Contributor
There was a problem hiding this comment.
For time configuration, it is better to change it to loadConfigOrThrow[FiniteDuration]
Contributor
Author
There was a problem hiding this comment.
great catch, I will update
Contributor
|
LGTM with some small suggestions |
jiangpengcheng
force-pushed
the
add_creation_job_manager
branch
from
732e6e2 to
219c069
Compare
ningyougang
approved these changes
Jun 3, 2021
style95
approved these changes
Jun 3, 2021
KeonHee
approved these changes
Jun 4, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This module records container creation using a
CreationId, when a new container creation request is created, one entry with an uniqueCreationIdwill be created and saved in to ETCD, and will be deleted when creation is finished(or timeout waiting for creation complete ack).So that a
MemoryQueuecan query how many containers are being created from ETCD and then do containers scheduling based on this value.It can also help to trace/debug the procedure of a container creation.
Related issue and scope
My changes affect the following components
Types of changes
Checklist: