Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage Refactor #3

Closed
12 tasks done
ikreymer opened this issue Oct 9, 2021 · 0 comments
Closed
12 tasks done

Storage Refactor #3

ikreymer opened this issue Oct 9, 2021 · 0 comments

Comments

@ikreymer
Copy link
Member

ikreymer commented Oct 9, 2021 •
edited
Loading

  • Add default vs custom (s3) storage
  • K8S: All storages correspond to secrets
  • K8S: Default storages inited via helm
  • K8S: Custom storage results in custom secret (per archive)
  • K8S: Don't add secret per crawl config
  • API for changing storage per archive
  • Docker: default storage just hard-coded from env vars (only one for now)
  • Validate custom storage via aiobotocore before confirming
  • Data Model: remove usage from users
  • Data Model: support adding multiple files per crawl for parallel crawls
  • Data Model: track completions for parallel crawls
  • Data Model: initial support for tags per crawl, add collection as 'coll' tag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ikreymer