GitHub Runner start delays #1218
-
|
One of my builds is using a standard-8 GitHub runner (8 core, 32GB memory as far as I understand), but today I am seeing long delays before one is started in te ubicloud GitHub Runners dashboard. My jobs in GitHub actions say: "Waiting for a runner to pick up this job..." and it has been 4-5 minutes before one is started on a few occassions today already. Yesterday these jobs were starting quickly. There is currently no reported downtime or issues on GitHub's APIs, so I'm led to believe this might be an issue on Ubicloud's side? I am using Hetzner as the provider. Any ideas, or has anyone else seen this? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 6 replies
-
|
Hello @sean-stage, When a new workflow job is triggered, GitHub sends us a webhook event. We keep a pool of runners ready to serve customer requests quickly. However, depending on the load, the runner pool may become depleted, requiring the provision of a new one upon request. We anticipate jobs to start within 15-20 seconds if a runner is selected from the pool; otherwise, it can take 50-60 seconds. If GitHub sends the webhook on time, the process should not take 4-5 minutes. Sometimes, GitHub may fail to send webhook events. We have an open support ticket with GitHub, and their engineering team is aware of and working on the issue. If you share your workflow run details with us at [email protected], we can investigate further. You can find the run ID in the URL after opening the details of a specific run. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @enescakir , I suspect then it may have been a case of GitHub not sending a webhook payload, or it was severely delayed. If I notice it again I'll grab the run ID and send it along to that email address. Thanks again. |
Beta Was this translation helpful? Give feedback.
-
|
I'm finding that some of build never even start—is this known by the Ubicloud team? @sean-stage |
Beta Was this translation helpful? Give feedback.
-
|
Hey @dphuang2 , We're still seeing a few cases of this. The message tends to be @enescakir , do you have any updates on progress of this issue with GitHub? We're seeing great success with using Ubicloud for our github actions runs, but these few isolated cases of jobs not picking up an executor are holding us back a bit. We would love to change all of our repositories over to use ubicloud but want to wait for this issue to be ironed out first. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dphuang2 and @sean-stage, In response to this, we're currently revising our listening logic to use polling instead of solely relying on webhooks. We anticipate that these changes will be deployed within this week. Additionally, we're experiencing compatibility issues with the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the info. I'll check whether the jobs are using max-parallel or not too. It's great that you'll be using a polling mechanism to help offset any issues on github side via webhooks - thanks! |
Beta Was this translation helpful? Give feedback.
-
|
We've been observing this too. Is there anything the Ubicloud team can do to improve the throughput? Increasing the pool size: both the static size for the whole system, but also dynamically based on, say, time of day/week, to match anticipated levels of traffic at a particular point in time? |
Beta Was this translation helpful? Give feedback.
-
|
We've noticed an increase in failure rates for API calls to GitHub, along with some delays in webhook deliveries. After we register the runner to GitHub, seems GitHub has issues with job assignments. Although there's no active incident reported, unofficial data from https://downdetector.com/status/github/ suggests others are experiencing some problems. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! |
Beta Was this translation helpful? Give feedback.
Hello @sean-stage,
Thanks for trying out Ubicloud!
When a new workflow job is triggered, GitHub sends us a webhook event. We keep a pool of runners ready to serve customer requests quickly. However, depending on the load, the runner pool may become depleted, requiring the provision of a new one upon request. We anticipate jobs to start within 15-20 seconds if a runner is selected from the pool; otherwise, it can take 50-60 seconds.
If GitHub sends the webhook on time, the process should not take 4-5 minutes.
Sometimes, GitHub may fail to send webhook events. We have an open support ticket with GitHub, and their engineering team is aware of and working on the issue.
If you share your workf…