Sorry for asking but how will this ensure that we don't run into IP limits?

Sorry for asking but how will this ensure that we don't run into IP limits?

It doesn't. But we don't have access to Excavator's logs any more at https://scoop.r15.ch

According to GitHub's docs, we can make 1000 requests per hour. This will be fine for the Main bucket, but for Extras, we will have to split the Action to run on 1000 manifests in one run and on 246 manifests in a second run (total 1246 manifests in Extras as of now).

Could we run GitHub Actions twice a day?

Once for odd files, another for even files?

@rasa @rashil2000 any idea how they count requests?

For example would checkver be 1 request, 32 bit download be 1 request and 64 bit download be one request?

In that case, we might face issues on Main repo as well.

Not all manifests will use GitHub for checkver and downloading, right? I'm not sure if we can count something like this.

@rashil2000 depending on how they count even if 500 use them, we might come close to the limit

I'm not sure if we can count something like this.

Would a grep like command work to count the number of lines present in all files with github.com or checkver: "github" and ignore those starting with the word homepage?

We will have to create an elaborate and ugly bash script then 😥 and use that instead of kiddong/Scoop-GitHubActions

Another option is to ignore all Github checkver and updates for the main excavator, and implement a rate limited excavator for Github API dependent manifests that start a new run every other or third hour depending on number of manifests.

Or add a checkver interval option, with say, hourly, daily, weekly, monthly options, and use that as a selector for the excavator.

It doesn't. But we don't have access to Excavator's logs any more at scoop.r15.ch

That was an error on my side. If the DNS record gets updated the logs should be available again.

Logs are back up at https://scoop.r15.ch/ :)

remote:   [email protected]:ScoopInstaller/Extras.git        
remote: error: GH006: Protected branch update failed for refs/heads/master.        
remote: error: At least 1 approving review is required by reviewers with write access.        
To github.com:lukesampson/scoop-extras.git
 ! [remote rejected]     master -> master (protected branch hook declined)
error: failed to push some refs to '[email protected]:lukesampson/scoop-extras.git'

Pushing updates ...
hub push origin master
remote: error: GH006: Protected branch update failed for refs/heads/master.        
remote: error: At least 1 approving review is required by reviewers with write access.        
To github.com:ScoopInstaller/Main.git
 ! [remote rejected]     master -> master (protected branch hook declined)
error: failed to push some refs to '[email protected]:ScoopInstaller/Main.git'

So what's wrong? (and also too many 429...)

Only a little remark :
uses: kidonng/Scoop-GithubActions@stable
Scoop-GithubActions should be clone in ScoopInstaller organisation.
Last update for this repo is 2020.

It looks like Excavator was indeed trying to push changes, but ran into some branch protection restrictions that were recently changed by @rasa I think

It looks like Excavator was indeed trying to push changes, but ran into some branch protection restrictions that were recently changed by @rasa I think

Excavator won't work if the master branch is protected. It could be changed to create PRs though. (which would result in a lot of spam PRs 😄)
Or my account can be whitelisted (Excavator could also be changed to use the ScoopBot Account)

I'll turn off branch protection, for now. Once we work out the kinks, I will re-enable it.

Excavator could also be changed to use the ScoopBot Account

I think this will be better in the long run

@r15ch13 can we, for the time being, prioritize giving another account Excavator access (ScoopBot)?

We are facing multiple problems (lptstr/winfetch#112, ScoopInstaller/Extras#7258, ScoopInstaller/Extras#7263, ScoopInstaller/Extras#7265, #2847) with updates - mostly where download URLs are static but version numbers change at the source, so hash mismatch occurs. 😢

I'm currently working on adding a parameter to the Excavator workflow, let's say X, which will check every X'th manifest depending on the current hour as the starting index. I've updated the cron to run 6 times in a day. If X is 3, it will go something like this:

(assuming a bucket has around 1000 manifests)

00:20 - every third manifest, starting at index 0
04:20 - every third manifest, starting at index 1
08:20 - every third manifest, starting at index 2
12:20 - every third manifest, starting at index 0
16:20 - every third manifest, starting at index 1
20:20 - every third manifest, starting at index 2

This means that each manifest will get checked twice a day for updates. Running 300 GitHub API calls every hour (3 calls per manifest - total around 900) ensures that we don't face rate limiting.

Is this approach fine?

@niheaven is there a way to pass multiple app names to the checkver.ps1 and auto-pr.ps1 utilities?

auto-pr just direct push or make PR for every updated manifest, and checkver supports * and ? wildcards.

You can carefully split manifests by, e.g., names (a*, b*, etc.), and then check them separately.

Each part should have similar number of manifests.

You can carefully split manifests by, e.g., names (a*, b*, etc.), and then check them separately.

Each part should have similar number of manifests.

This won't be possible alphabetically, because some letters (like q, z) will have very less manifests.

I think we can add an additional parameter to checkver.ps1, where it accepts an array of appnames?

That way I can filter manifests based on index and pass them to auto-pr.ps1.

I think tweaking the Exacavtor to

1. Use authenticated requests, and
2. pause based on the X-RateLimit-* HTTP headers
   is the best course of action.

See ScoopInstaller/Scoop#3912 and https://github.com/rasa/scoop-directory/blob/9ed3a43e0ee0b408d423f99be28c7a301f53675f/maintenance/github-crawler.py#L170 for example.

See also ScoopInstaller/Excavator#5 (comment)

I completely agree. That'll be the best idea.

Current situation of using GitHub Actions for Excavator workflow

I experimented a lot with this today. Logs can be seen here - https://github.com/rashil2000/Extras/actions/workflows/excavator.yml

Scoop's checkver now uses authenticated GitHub API requests. But there are other problems:

• On Linux runners, we don't get rate-limited. However, 80% of the checkver calls (whether to GitHub or to somewhere else) fail with SSL errors. This can be solved by using Windows runners, see chore(ci): Use Windows Actions to avoid SSL errors #3047, chore(ci): Use Windows Actions to avoid SSL errors Extras#7465, chore(ci): Use Windows Actions to avoid SSL errors Versions#337.
• On Windows runners, there are no SSL errors, but we get heavily rate-limited. We need to pause based on X-RateLimit-* headers here. @rasa can you help with this?

We need to pause based on X-RateLimit-* headers here. @rasa can you help with this?

@rashil2000 Yes.

I think there was some problem with GitHub yesterday...

Without making any changes, I let the Excavator workflow run at its scheduled times (6 times per day), for a total of 6 runs in my fork of Extras and 3 in my fork of Main. Check the logs here - https://github.com/rashil2000/Extras/actions/workflows/excavator.yml and https://github.com/rashil2000/Main/actions/workflows/excavator.yml.

There are no (403) Rate-Limiting errors now.

This PR is now ready for review.

@r15ch13 once this is merged, you can disable your personal pipelines :)

3 more runs on Extras and 3 more on Main overnight, still no errors. Merging this now.

@r15ch13 please disable ScoopInstaller/Excavator to prevent any possible conflicts

okay, I have disabled Excavator for Main, Extras and Versions 👍🏽

@r15ch13 If the Excavator is still running for Java, you can disable it there too.

I have one more observation to state.

• On Windows runners, there are no SSL errors, but we get heavily rate-limited. We need to pause based on X-RateLimit-* headers here.

These are not rate limiting errors. The 403 errors I saw are actually because of the GitHub API's hard limit of 1000 requests per hour. You can observe this if you try to run the excavator workflow more than once within an hour span.

The only way I can think around this is the manifest-splitting solution I mentioned earlier - as the number of manifests in the buckets increases, we will have to split checkver process.