Skip to content

feat!: Skip archived repositories by default, rename skip_archived_repos to include_archived_repos #17339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kodiakhq merged 2 commits into from
Mar 27, 2024
Merged

feat!: Skip archived repositories by default, rename skip_archived_repos to include_archived_repos #17339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db5b66b
feat: Skip archived repositories by default
erezrokah Mar 26, 2024
74801b7
Merge branch 'main' into feat/skip_archived_by_default
kodiakhq[bot] Mar 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions plugins/source/github/client/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ func New(ctx context.Context, logger zerolog.Logger, spec Spec) (schema.ClientMe
repos: spec.Repos,
}
c.logger.Info().Msg("Discovering repositories")
orgRepositories, err := c.discoverRepositories(ctx, spec.DiscoveryConcurrency, spec.Orgs, spec.Repos, spec.SkipArchivedRepos)
orgRepositories, err := c.discoverRepositories(ctx, spec.DiscoveryConcurrency, spec.Orgs, spec.Repos, spec.IncludeArchivedRepos)
if err != nil {
return nil, fmt.Errorf("failed to discover repositories: %w", err)
}
Expand All @@ -155,19 +155,19 @@ func servicesForClient(c *github.Client) GithubServices {
}
}

func (c *Client) filterArchivedRepos(repos []*github.Repository) []*github.Repository {
func (c *Client) removeArchivedRepos(repos []*github.Repository) []*github.Repository {
filtered := []*github.Repository{}
for _, repo := range repos {
if repo.GetArchived() {
c.logger.Info().Msgf("Skipping archived repository %q", repo.GetFullName())
c.logger.Debug().Msgf("Skipping archived repository %q", repo.GetFullName())
continue
}
filtered = append(filtered, repo)
}
return filtered
}

func (c *Client) discoverRepositories(ctx context.Context, discoveryConcurrency int, orgs []string, repos []string, skipArchivedRepos bool) (map[string][]*github.Repository, error) {
func (c *Client) discoverRepositories(ctx context.Context, discoveryConcurrency int, orgs []string, repos []string, includeArchivedRepos bool) (map[string][]*github.Repository, error) {
orgRepos := make(map[string][]*github.Repository)
orgReposLock := sync.Mutex{}
errorGroup, gtx := errgroup.WithContext(ctx)
Expand All @@ -184,8 +184,8 @@ func (c *Client) discoverRepositories(ctx context.Context, discoveryConcurrency
if err != nil {
return err
}
if skipArchivedRepos {
repos = c.filterArchivedRepos(repos)
if !includeArchivedRepos {
repos = c.removeArchivedRepos(repos)
}
orgRepositories = append(orgRepositories, repos...)

Expand Down
4 changes: 2 additions & 2 deletions plugins/source/github/client/schema.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions plugins/source/github/client/spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ type Spec struct {
Concurrency int `json:"concurrency,omitempty" jsonschema:"default=10000"`
// Controls the number of parallel requests to GitHub when discovering repositories, a negative value means unlimited.
DiscoveryConcurrency int `json:"discovery_concurrency,omitempty" jsonschema:"default=1"`
// Skip archived repositories when discovering repositories.
SkipArchivedRepos bool `json:"skip_archived_repos,omitempty"`
// Include archived repositories when discovering repositories.
IncludeArchivedRepos bool `json:"include_archived_repos,omitempty"`
}

type EnterpriseSettings struct {
Expand Down
2 changes: 1 addition & 1 deletion plugins/source/github/docs/_configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ spec:
# # Optional parameters
# concurrency: 10000 # Optional. Number of concurrent requests to GitHub API. Default is 10000.
# discovery_concurrency: 1 # Optional. Number of concurrent requests to GitHub API during discovery phase. Default 1.
# skip_archived_repos: false # Optional. Skip archived repositories. Default false.
# include_archived_repos: false # Optional. Include archived repositories in the sync. Default false.
```

See [tables](/docs/plugins/sources/github/tables) for a full list of available tables.
Expand Down
4 changes: 2 additions & 2 deletions plugins/source/github/docs/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,6 @@ This is the (nested) spec used by GitHub Source Plugin
By default the plugin discovers repositories one organization at a time. You can increase `discovery_concurrency` to discover multiple organizations in parallel, or use a negative value to discover all organizations in parallel.
Please note that it's possible to hit GitHub API rate limits when using a high value for `discovery_concurrency`.

- `skip_archived_repos` (`bool`) (default: `false`)
- `include_archived_repos` (`bool`) (default: `false`)

By default archived repositories are included in the sync. To skip archived repositories set `skip_archived_repos` to `true`.
By default archived repositories are not included in the sync. To include archived repositories set `include_archived_repos` to `true`.