Skip to content

feat: GitHub plugin scalability improvements #16478

@pilvikala

Description

@pilvikala

Which problem is this feature request solving?

Large organizations run into rate limiting issues with PAT.

Quote from a customer:
"At a scale of 20,000 - 40,000 repos actively using GitHub Actions you can easily burn through even the GitHub app limits of 15K calls per hour."

We should consider some of the options suggested below to improve the GitHub plugin so it would work for these large orgs.

Describe the solution you'd like

  • Use conditional requests that don’t count under the hard rate limit. Requires caching the information somewhere
  • Use incremental syncs when it’s possible to limit data based on dates
  • Listen to Webhooks events and sync data from them (similar to event based syncs in AWS). Would be easier to do for managed syncs since we need to expose a server that listens to Webhooks

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions