Skip to content

Github Source Plugin does not retrieve last api page #17453

@komodo472

Description

@komodo472

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Branches over the last 100th are not recorded

Expected Behavior

All branches are recorded

CloudQuery (redacted) config

kind: source
spec:
  name: "github"
  path: "cloudquery/github"
  version: "v8.2.1"
  # all tables: https://www.cloudquery.io/docs/plugins/sources/aws/tables
  tables:
    - github_repositories
    - github_repository_branches
  skip_tables:
    - github_repository_dependabot_alerts
    - github_releases
    - github_repository_dependabot_secrets
    - github_repository_sboms
    - github_repository_keys
  destinations: ["postgresql"]
  spec:
    access_token: "${GITHUB_PERSONAL_ACCESS_TOKEN}"

Steps To Reproduce

Run a sync with a repo that has over 100 branches.

CloudQuery (redacted) logs

No errors in log

CloudQuery version

5.11.0

Additional Context

Because Page is set to NextPage before checking if it’s equal to LastPage, LastPage will never be queried.

This issue appears to be present in most of the Github fetch functions.

opts.ListOptions.Page = resp.NextPage
if opts.ListOptions.Page == resp.LastPage {
break
}

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions