-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
src/patchset.py: Implement Patchset service #342
Conversation
00ef8cb
to
3dc76bc
Compare
Setup nodes
Patchset logs
Tarball logs
Check tarball
Check patchset node
|
aa29045
to
f4a6ebe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really appreciate posting such a detailed verification steps and all the log information. These will greatly help verifying this patch and also maybe set up an E2E suite for Patchwork-related pipeline changes. Let me deploy these changes and get back to you on Monday if some other questions come up
cb01cea
to
5301005
Compare
Submit patchset node
Patchset logs
Check tarball
Check node
|
5301005
to
544b19f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍
I suggested a minor change to align with current PR related to configuration loading
if checkout_node["state"] == "running": | ||
self.log.info( | ||
f"Patchset node {patchset_node['id']} is waiting " | ||
f"for checkout node {checkout_node['id']} to complete", | ||
) | ||
continue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, now I understand why you are finding all the running nodes on every event.
According to the state machine and design, checkout
node should only have children after its state changes to available
.
Instead of manually creating checkout
and patchset
nodes, we can listen to available checkout
node events in the patchset service and create a node automatically from the service itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, for every patchset
node I should create a new checkout
node and wait for it to become available
? It will pause the queue for quite some time and seems to be quite inefficient.
However it gave me another idea. Ideally I want to apply patchset to whatever latest commit is available on a tree. Is it possible to query latest available checkout node?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's what I am suggesting. You can listen to available checkout nodes and create patchset nodes from the service directly.
See, for instance, you can use the below filter here to get available checkout node events:
def _setup(self, *args):
return self._api_helper.subscribe_filters({
"kind": "checkout",
"state": "available",
})
Then you can get checkout
node from the event just like https://github.com/kernelci/kernelci-pipeline/blob/main/src/tarball.py#L130 and then you can create patchset
node from it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it should be done outside of patchset
service, right? In an another service that will prepare checkout
node and then pass it to patchset
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I think the greatest challenge with checkout nodes so is picking the right one. I'm not sure how to determine an appropriate version to apply a patch and then correctly query a right node. If we have checkouts for 6.4
, 6.8
etc how to pick the right version. Maybe we should pick the latest stable and next
branch and create two patchset nodes based on them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In an another service that will prepare checkout node and then pass it to patchset?
The pipeline service trigger
is already creating checkout nodes whenever a new commit is detected for a particular kernel revision. As I have not been involved in the patchset-related discussions, I am not aware of all the requirements, but do you think you can use checkout
nodes created by the trigger
service?
Once the trigger service creates a checkout
node, the tarball
service will create and push a tarball and change the node's status to available
. Then you can use the below filter in the patchset
service and create a patchset node from it:
def _setup(self, *args):
return self._api_helper.subscribe_filters({
"op": "updated",
"name": "checkout",
"state": "available",
})
def _run(self, _sub_id):
self.log.info("Listening for new trigger events")
self.log.info("Press Ctrl-C to stop.")
while True:
checkout_node = self._api_helper.receive_event_node(sub_id)
patchset_node = { create_patchset_node_from_checkout() }
patchset_node = self._api.node.add(patchset_node)
self._process_patchset(checkout_node, patchset_node)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we discussed with @gctucker in kernelci-pipeline#295, patchset
service only processed patchset nodes created by another service (patchwork<>kernelci connector). So patchset service should tream these nodes as created on-demand and process them accordingly.
I am open to suggestions on how to link existing checkout nodes and freshly created patchset node. I think it's a topic for a larger discussion on how we want to test kernel patches in general.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JenySadadia @yurinnick I agree it will be an important next step to find the best way to connect patchset
nodes with checkout
ones created by the trigger
service. Let keep the scope of this PR to just processing patchset
nodes created by on-demand - with an assumption for an already existing parent checkout node (like in the verification steps posted in this PR).
I created #490 to track this task. @JenySadadia Please let me know if there something else that should be done before starting integration of this service.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should merge this PR now and improvements can be discussed as follow-ups.
@yurinnick Please rebase the PR. |
544b19f
to
f5936c8
Compare
Rebased. |
f5936c8
to
3629a49
Compare
Done:
Todo:
|
Sorry for jumping into this discussion with naive questions, but why not extend the As it turns out, This would of course need some thinking around how to configure patchsets in |
@a-wai good point! I think we could fold some parts of the Let's keep Patchwork integration as a separate service for now and revisit this idea once the patchset processing gets stableish - I created #491 to track this. |
Patchset service process patchset nodes: - Wait for parent checkout node to be available - Download checkout node tarball - Apply patches and calculate patchset hash - Upload new tarball Signed-off-by: Nikolay Yurin <[email protected]>
3629a49
to
0f096bc
Compare
Rebased and fixed copyright header |
@a-wai this is what we can iterate to. I'd prefer |
@JenySadadia Please have a look at merging this PR if there are no blockers (you're the sole Pipeline maintainer) |
Let's give it a go to the staging first. |
Updated staging TOML and started
|
Patchset service process patchset nodes: