Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src/patchset.py: Implement Patchset service #342

Merged
merged 1 commit into from
Mar 28, 2024

Conversation

yurinnick
Copy link
Contributor

Patchset service process patchset nodes:

  • Wait for parent checkout node to be available
  • Download checkout node tarball
  • Apply patches and calculate patchset hash
  • Upload new tarball

@yurinnick yurinnick marked this pull request as draft October 21, 2023 09:15
@yurinnick yurinnick added the staging-skip Don't test automatically on staging.kernelci.org label Oct 21, 2023
@yurinnick
Copy link
Contributor Author

yurinnick commented Oct 21, 2023

Setup nodes

$ echo '{
  "name":"checkout",
  "path": ["checkout"],
  "revision":{
    "tree":"mainline",
    "url":"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
    "branch":"master",
    "commit":"2a987e65025e2b79c6d453b78cb5985ac6e5eb26"
  }
}' | ./kci node submit
...
$ echo '{
  "name":"patchset",
  "path": [
    "checkout",
    "patchset"
  ],
  "parent": "6533024fc2a20de21962f5c6",
  "revision":{
    "tree":"mainline",
    "url":"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
    "branch":"master",
    "commit":"2a987e65025e2b79c6d453b78cb5985ac6e5eb26"
  },
  "artifacts": {
    "0001-patch.patch": "https://patchwork.kernel.org/project/linux-mm/patch/2ea7cbce80ac7c62e90cbfb9653a7972f902439f.1697816692.git.raghavendra.kt@amd.com/raw"
  }
}' | ./kci node submit
...

Patchset logs

kernelci-pipeline-patchset | 10/21/2023 10:33:31 AM UTC [INFO] Downloading checkout tarball, url: http://172.17.0.1:8002/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b.tar.gz
kernelci-pipeline-patchset | 10/21/2023 10:33:57 AM UTC [INFO] Applying patch 0001-patch.patch, url: https://patchwork.kernel.org/project/linux-mm/patch/2ea7cbce80ac7c62e90cbfb9653a7972f902439f.1697816692.git.raghavendra.kt@amd.com/raw
kernelci-pipeline-patchset | 10/21/2023 10:33:59 AM UTC [INFO] Making tarball linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487
kernelci-pipeline-patchset | 10/21/2023 10:33:59 AM UTC [INFO] set -e
kernelci-pipeline-patchset | cd /home/kernelci/data/src/linux-patchset/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b/
kernelci-pipeline-patchset | tar --create --transform "s/^/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487\//" * | gzip > /home/kernelci/data/output/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487.tar.gz
kernelci-pipeline-patchset |
kernelci-pipeline-patchset | 10/21/2023 10:34:29 AM UTC [INFO] Tarball created
kernelci-pipeline-patchset | 10/21/2023 10:34:29 AM UTC [INFO] Uploading linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487.tar.gz
kernelci-pipeline-patchset | 10/21/2023 10:34:29 AM UTC [INFO] Connected (version 2.0, client OpenSSH_8.4p1)
kernelci-pipeline-patchset | 10/21/2023 10:34:29 AM UTC [INFO] Authentication (publickey) successful!
kernelci-pipeline-patchset | 10/21/2023 10:34:30 AM UTC [INFO] Upload complete: http://172.17.0.1:8002/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487.tar.gz

Tarball logs

kernelci-pipeline-tarball | 10/21/2023 08:53:25 AM UTC [INFO] Updating repo for mainline
kernelci-pipeline-tarball | From https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
kernelci-pipeline-tarball |  * branch                      HEAD       -> FETCH_HEAD
kernelci-pipeline-tarball | From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
kernelci-pipeline-tarball |  * branch                      HEAD       -> FETCH_HEAD
kernelci-pipeline-tarball | HEAD is now at 9c5d00cb7b6b Merge tag 'perf-tools-fixes-for-v6.6-2-2023-10-20' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools
kernelci-pipeline-tarball | 10/21/2023 08:53:31 AM UTC [INFO] Repo updated
kernelci-pipeline-tarball | 10/21/2023 08:53:31 AM UTC [INFO] Making tarball linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b.tar.gz
kernelci-pipeline-tarball | 10/21/2023 08:53:31 AM UTC [INFO] set -e
kernelci-pipeline-tarball | cd /home/kernelci/data/src/linux
kernelci-pipeline-tarball | git archive --format=tar --prefix=linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b/ HEAD | gzip > ../../output/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b.tar.gz
kernelci-pipeline-tarball |
kernelci-pipeline-tarball | 10/21/2023 08:54:04 AM UTC [INFO] Tarball created
kernelci-pipeline-tarball | 10/21/2023 08:54:04 AM UTC [INFO] Uploading /home/kernelci/data/output/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b.tar.gz
kernelci-pipeline-tarball | 10/21/2023 08:54:05 AM UTC [INFO] Upload complete: http://172.17.0.1:8002/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b.tar.gz

Check tarball

$ wget http://localhost:8002/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487tar.gz
$ tar -tf ~/Downloads/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487tar.gz | head
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/COPYING
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/CREDITS
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/.gitignore
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/ABI/
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/ABI/README
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/ABI/obsolete/
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/ABI/obsolete/o2cb
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/ABI/obsolete/procfs-i8k
linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487/Documentation/ABI/obsolete/sysfs-bus-iio

Check patchset node

./kci node get 6533a8f2c2a20de21962f5f4 | jq
{
  "id": "6533a8f2c2a20de21962f5f4",
  "kind": "node",
  "name": "patchset",
  "path": [
    "checkout",
    "patchset"
  ],
  "group": null,
  "revision": {
    "tree": "mainline",
    "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
    "branch": "master",
    "commit": "2a987e65025e2b79c6d453b78cb5985ac6e5eb26",
    "describe": "v6.6-rc6-285-g9c5d00cb7b6b",
    "version": {
      "version": 6,
      "patchlevel": 6,
      "sublevel": null,
      "extra": "-rc6-285-g9c5d00cb7b6b",
      "name": null
    },
    "patchset": "cd372fb85148700fa88095e3492d3f9f5beb43e555e5ff26d95f5a6adc36f8e6"
  },
  "parent": "6533024fc2a20de21962f5c6",
  "state": "available",
  "result": null,
  "artifacts": {
    "tarball": "http://172.17.0.1:8002/linux-mainline-master-v6.6-rc6-285-g9c5d00cb7b6b-cd372fb851487.tar.gz"
  },
  "data": null,
  "created": "2023-10-21T10:33:22.738000",
  "updated": "2023-10-21T10:34:30.975000",
  "timeout": "2023-10-21T16:33:22.738000",
  "holdoff": "2023-10-21T10:44:30.968000",
  "owner": "admin",
  "user_groups": []
}

@yurinnick yurinnick force-pushed the patchset-service branch 3 times, most recently from aa29045 to f4a6ebe Compare October 21, 2023 10:55
@yurinnick yurinnick marked this pull request as ready for review October 30, 2023 20:02
src/patchset.py Outdated Show resolved Hide resolved
Copy link
Contributor

@pawiecz pawiecz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really appreciate posting such a detailed verification steps and all the log information. These will greatly help verifying this patch and also maybe set up an E2E suite for Patchwork-related pipeline changes. Let me deploy these changes and get back to you on Monday if some other questions come up

src/tarball.py Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
src/tarball.py Outdated Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
@yurinnick
Copy link
Contributor Author

yurinnick commented Feb 8, 2024

Submit patchset node

$ cat patchset_node.json
{
  "name":"patchset",
  "path": [
    "checkout",
    "patchset"
  ],
  "parent": "65bd7cfdb656782e895218c3",
  "artifacts": {
    "0001-patch.patch": "https://patchwork.kernel.org/project/kvm/patch/[email protected]/raw"
  }
}
$ kci node submit ./patchset_node.json

Patchset logs

02/08/2024 09:29:23 PM UTC [INFO] Listening for new trigger events
02/08/2024 09:29:23 PM UTC [INFO] Press Ctrl-C to stop.
02/08/2024 09:29:24 PM UTC [INFO] Waiting 30 seconds for a new nodes...
02/08/2024 09:29:54 PM UTC [DEBUG] Found patchset nodes: [{'id': '65c547ca316ff4bd1ddf9395', 'kind': 'node', 'name': 'patchset', 'path': ['checkout', 'patchset'], 'group': None, 'parent': '65bd7cfdb656782e895218c3', 'state': 'running', 'result': None, 'artifacts': {'0001-patch.patch': 'https://patchwork.kernel.org/project/kvm/patch/[email protected]/raw'}, 'data': None, 'debug': None, 'created': '2024-02-08T21:29:46.243000', 'updated': '2024-02-08T21:29:46.243000', 'timeout': '2024-02-09T03:29:46.243000', 'holdoff': None, 'owner': 'admin', 'user_groups': []}]
02/08/2024 09:29:54 PM UTC [INFO] Processing patchset node: 65c547ca316ff4bd1ddf9395
02/08/2024 09:29:54 PM UTC [INFO] Downloading checkout tarball, url: http://172.17.0.1:8002/linux-mainline-master-v6.8-rc2-397-g56897d51886f.tar.gz
02/08/2024 09:33:37 PM UTC [INFO] Applying patch 0001-patch.patch, url: https://patchwork.kernel.org/project/kvm/patch/[email protected]/raw
02/08/2024 09:33:42 PM UTC [DEBUG] Hashable content:
--- a/arch/x86/kvm/mmu/mmu.c
/n+++ b/arch/x86/kvm/mmu/mmu.c
/n+	/*
/n+	 * Pre-check for a relevant mmu_notifier invalidation event prior to
/n+	 * acquiring mmu_lock.  If there is an in-progress invalidation and the
/n+	 * kernel allows preemption, the invalidation task may drop mmu_lock
/n+	 * and yield in response to mmu_lock being contended, which is *very*
/n+	 * counter-productive as this vCPU can't actually make forward progress
/n+	 * until the invalidation completes.  This "unsafe" check can get false
/n+	 * negatives, i.e. KVM needs to re-check after acquiring mmu_lock.
/n+	 *
/n+	 * Do the pre-check even for non-preemtible kernels, i.e. even if KVM
/n+	 * will never yield mmu_lock in response to contention, as this vCPU is
/n+	 * *guaranteed* to need to retry, i.e. waiting until mmu_lock is held
/n+	 * to detect retry guarantees the worst case latency for the vCPU.
/n+	 */
/n+	if (mmu_invalidate_retry_gfn_unsafe(vcpu->kvm, fault->mmu_seq, fault->gfn)) {
/n+		kvm_release_pfn_clean(fault->pfn);
/n+		return RET_PF_RETRY;
/n+	}
/n+
/n--- a/include/linux/kvm_host.h
/n+++ b/include/linux/kvm_host.h
/n+
/n+/*
/n+ * This lockless version of the range-based retry check *must* be paired with a
/n+ * call to the locked version after acquiring mmu_lock, i.e. this is safe to
/n+ * use only as a pre-check to avoid contending mmu_lock.  This version *will*
/n+ * get false negatives and false positives.
/n+ */
/n+static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm,
/n+						   unsigned long mmu_seq,
/n+						   gfn_t gfn)
/n+{
/n+	/*
/n+	 * Use READ_ONCE() to ensure the in-progress flag and sequence counter
/n+	 * are always read from memory, e.g. so that checking for retry in a
/n+	 * loop won't result in an infinite retry loop.  Don't force loads for
/n+	 * start+end, as the key to avoiding infinite retry loops is observing
/n+	 * the 1=>0 transition of in-progress, i.e. getting false negatives
/n+	 * due to stale start+end values is acceptable.
/n+	 */
/n+	if (unlikely(READ_ONCE(kvm->mmu_invalidate_in_progress)) &&
/n+	    gfn >= kvm->mmu_invalidate_range_start &&
/n+	    gfn < kvm->mmu_invalidate_range_end)
/n+		return true;
/n+
/n+	return READ_ONCE(kvm->mmu_invalidate_seq) != mmu_seq;
/n+}

02/08/2024 09:33:42 PM UTC [DEBUG] Patch 0001-patch.patch hash: 96560b86f7238970362dc0f4220cddfc0c437e1f605f61eef8e074c9b37ef4c2
02/08/2024 09:33:42 PM UTC [DEBUG] Patchset hash 4f3428d5f61f53d0eabb4c30fe02ef47b9a7d7f4305017318d1c2116da6b8b96
02/08/2024 09:33:42 PM UTC [INFO] Making tarball linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5
02/08/2024 09:33:42 PM UTC [INFO] set -e
cd /home/kernelci/data/src/linux-patchset/linux-mainline-master-v6.8-rc2-397-g56897d51886f/
tar --create --transform "s/^/linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5\//" * | gzip > /home/kernelci/data/output/linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5.tar.gz

02/08/2024 09:38:53 PM UTC [INFO] Tarball created
02/08/2024 09:38:53 PM UTC [INFO] Uploading linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5.tar.gz
02/08/2024 09:38:53 PM UTC [INFO] Connected (version 2.0, client OpenSSH_8.4p1)
02/08/2024 09:38:54 PM UTC [INFO] Authentication (publickey) successful!
02/08/2024 09:39:04 PM UTC [INFO] Upload complete: http://172.17.0.1:8002/linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5.tar.gz
02/08/2024 09:39:05 PM UTC [INFO] Waiting 30 seconds for a new nodes...

Check tarball

$ tar -tf ~/Downloads/linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5.tar.gz | head
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/COPYING
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/CREDITS
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/.gitignore
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/ABI/
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/ABI/README
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/ABI/obsolete/
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/ABI/obsolete/o2cb
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/ABI/obsolete/procfs-i8k
linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5/Documentation/ABI/obsolete/sysfs-bus-iio

Check node

{
  "id": "65c8a3583cfe47352d9c1fa4",
  "kind": "node",
  "name": "patchset",
  "path": [
    "checkout",
    "patchset"
  ],
  "group": null,
  "parent": "65bd7cfdb656782e895218c3",
  "state": "available",
  "result": null,
  "artifacts": {
    "0001-patch.patch": "https://patchwork.kernel.org/project/kvm/patch/[email protected]/raw",
    "tarball": "http://172.17.0.1:8002/linux-mainline-master-v6.8-rc2-397-g56897d51886f-4f3428d5f61f5.tar.gz"
  },
  "data": {
    "kernel_revision": {
      "tree": "mainline",
      "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
      "branch": "master",
      "commit": "56897d51886fa7e9f034ff26128eb09f1b811594",
      "describe": "v6.8-rc2-397-g56897d51886f",
      "version": {
        "version": "6",
        "patchlevel": "8",
        "extra": "-rc2-397-g56897d51886f"
      },
      "patchset": "4f3428d5f61f53d0eabb4c30fe02ef47b9a7d7f4305017318d1c2116da6b8b96"
    }
  },
  "debug": null,
  "created": "2024-02-11T10:37:12.696000",
  "updated": "2024-02-11T10:38:27.664000",
  "timeout": "2024-02-11T16:37:12.696000",
  "holdoff": "2024-02-11T10:48:27.659000",
  "owner": "admin",
  "user_groups": []
}

Copy link
Contributor

@pawiecz pawiecz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

I suggested a minor change to align with current PR related to configuration loading

src/patchset.py Outdated Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
src/patchset.py Outdated Show resolved Hide resolved
src/patchset.py Show resolved Hide resolved
src/patchset.py Show resolved Hide resolved
Comment on lines +276 to +287
if checkout_node["state"] == "running":
self.log.info(
f"Patchset node {patchset_node['id']} is waiting "
f"for checkout node {checkout_node['id']} to complete",
)
continue
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, now I understand why you are finding all the running nodes on every event.
According to the state machine and design, checkout node should only have children after its state changes to available.
Instead of manually creating checkout and patchset nodes, we can listen to available checkout node events in the patchset service and create a node automatically from the service itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, for every patchset node I should create a new checkout node and wait for it to become available? It will pause the queue for quite some time and seems to be quite inefficient.
However it gave me another idea. Ideally I want to apply patchset to whatever latest commit is available on a tree. Is it possible to query latest available checkout node?

Copy link
Collaborator

@JenySadadia JenySadadia Mar 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's what I am suggesting. You can listen to available checkout nodes and create patchset nodes from the service directly.
See, for instance, you can use the below filter here to get available checkout node events:

def _setup(self, *args):
        return self._api_helper.subscribe_filters({
            "kind": "checkout",
            "state": "available",
        })

Then you can get checkout node from the event just like https://github.com/kernelci/kernelci-pipeline/blob/main/src/tarball.py#L130 and then you can create patchset node from it.

Copy link
Contributor Author

@yurinnick yurinnick Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it should be done outside of patchset service, right? In an another service that will prepare checkout node and then pass it to patchset?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I think the greatest challenge with checkout nodes so is picking the right one. I'm not sure how to determine an appropriate version to apply a patch and then correctly query a right node. If we have checkouts for 6.4, 6.8 etc how to pick the right version. Maybe we should pick the latest stable and next branch and create two patchset nodes based on them.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In an another service that will prepare checkout node and then pass it to patchset?

The pipeline service trigger is already creating checkout nodes whenever a new commit is detected for a particular kernel revision. As I have not been involved in the patchset-related discussions, I am not aware of all the requirements, but do you think you can use checkout nodes created by the trigger service?
Once the trigger service creates a checkout node, the tarball service will create and push a tarball and change the node's status to available. Then you can use the below filter in the patchset service and create a patchset node from it:

    def _setup(self, *args):
        return self._api_helper.subscribe_filters({
            "op": "updated",
            "name": "checkout",
            "state": "available",
        })
    def _run(self, _sub_id):
        self.log.info("Listening for new trigger events")
        self.log.info("Press Ctrl-C to stop.")

        while True:
              checkout_node = self._api_helper.receive_event_node(sub_id)
              patchset_node = { create_patchset_node_from_checkout() }
              patchset_node = self._api.node.add(patchset_node)
              self._process_patchset(checkout_node, patchset_node)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed with @gctucker in kernelci-pipeline#295, patchset service only processed patchset nodes created by another service (patchwork<>kernelci connector). So patchset service should tream these nodes as created on-demand and process them accordingly.

I am open to suggestions on how to link existing checkout nodes and freshly created patchset node. I think it's a topic for a larger discussion on how we want to test kernel patches in general.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JenySadadia @yurinnick I agree it will be an important next step to find the best way to connect patchset nodes with checkout ones created by the trigger service. Let keep the scope of this PR to just processing patchset nodes created by on-demand - with an assumption for an already existing parent checkout node (like in the verification steps posted in this PR).

I created #490 to track this task. @JenySadadia Please let me know if there something else that should be done before starting integration of this service.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should merge this PR now and improvements can be discussed as follow-ups.

@JenySadadia
Copy link
Collaborator

@yurinnick Please rebase the PR.

@yurinnick
Copy link
Contributor Author

Rebased.

@yurinnick
Copy link
Contributor Author

Done:

  • Rebase
  • Pass all args into self._service_config instead of assigning individual values
  • Rename configs into global_configs and args into service_config to better reflect their purpose
  • Fix patchset service usage of tarball config
  • Move patchset global variables into patchset config section
  • Fix checkout dir path

Todo:

  • Patchset self._name is still tarball, need to fix this
  • Checkout nodes querying

@a-wai
Copy link
Contributor

a-wai commented Mar 7, 2024

Sorry for jumping into this discussion with naive questions, but why not extend the tarball service so it can generate checkout nodes with configured patchset(s) applied?

As it turns out, kbuild nodes currently listen for checkout nodes, so I'm fearing that adding another type of basically the same thing (i.e. provide a kernel source tree) could introduce unneeded complexity.

This would of course need some thinking around how to configure patchsets in pipeline.yaml, but it feels more intuitive to me (remember I don't have any background information/history wrt. this feature, though)

@pawiecz
Copy link
Contributor

pawiecz commented Mar 21, 2024

but why not extend the tarball service so it can generate checkout nodes with configured patchset(s) applied? (snip)

@a-wai good point! I think we could fold some parts of the patchset service in its current form into tarball service like you suggested and convert/reduce this one to a ~trigger-patchset service as a next step.

Let's keep Patchwork integration as a separate service for now and revisit this idea once the patchset processing gets stableish - I created #491 to track this.

Patchset service process patchset nodes:
- Wait for parent checkout node to be available
- Download checkout node tarball
- Apply patches and calculate patchset hash
- Upload new tarball

Signed-off-by: Nikolay Yurin <[email protected]>
@yurinnick
Copy link
Contributor Author

Rebased and fixed copyright header

@yurinnick
Copy link
Contributor Author

@a-wai this is what we can iterate to. I'd prefer patchset service to pick up existing node for the next tree and build against it.

@pawiecz
Copy link
Contributor

pawiecz commented Mar 27, 2024

@JenySadadia Please have a look at merging this PR if there are no blockers (you're the sole Pipeline maintainer)

@JenySadadia
Copy link
Collaborator

@JenySadadia Please have a look at merging this PR if there are no blockers (you're the sole Pipeline maintainer)

Let's give it a go to the staging first.

@JenySadadia JenySadadia removed the staging-skip Don't test automatically on staging.kernelci.org label Mar 27, 2024
@JenySadadia
Copy link
Collaborator

Updated staging TOML and started patchset service. Staging looks OK:

today at 14:56:4403/28/2024 09:26:44 AM UTC [INFO] Listening for new trigger events
today at 14:56:4403/28/2024 09:26:44 AM UTC [INFO] Press Ctrl-C to stop.
today at 14:56:4703/28/2024 09:26:47 AM UTC [INFO] Waiting 30 seconds for a new nodes...

@JenySadadia JenySadadia added this pull request to the merge queue Mar 28, 2024
Merged via the queue into kernelci:main with commit 0a2fe9c Mar 28, 2024
3 checks passed
@yurinnick yurinnick deleted the patchset-service branch April 18, 2024 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants