Skip to content

feat: add storage helpers to crawler & context#192

Merged
vdusek merged 13 commits intomasterfrom
add-storage-helpers
Jun 18, 2024
Merged

feat: add storage helpers to crawler & context#192
vdusek merged 13 commits intomasterfrom
add-storage-helpers

Conversation

@vdusek
Copy link
Copy Markdown
Collaborator

@vdusek vdusek commented Jun 18, 2024

Description

  • Add start requests option to BasicCrawler
  • Add storage-related helpers get_data, push_data, and export_to to BasicCrawler and BasicContext.
  • It also unifies the requests argument (closer to the user we are more flexible: request: str | BaseRequestData | Request, further away from the user we use only: request: Request)

Related issues

Testing

  • New unit tests were implemented.

Checklist

  • Changes are described in the CHANGELOG.md
  • CI passed

@github-actions github-actions bot added this to the 92nd sprint - Tooling team milestone Jun 18, 2024
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Jun 18, 2024
@vdusek vdusek force-pushed the add-storage-helpers branch from a33bc53 to 42bfa35 Compare June 18, 2024 12:27
@vdusek vdusek marked this pull request as ready for review June 18, 2024 12:51
@vdusek vdusek requested a review from janbuchar June 18, 2024 12:54
@vdusek vdusek requested a review from janbuchar June 18, 2024 15:10
Copy link
Copy Markdown
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vdusek vdusek merged commit f8f4066 into master Jun 18, 2024
@vdusek vdusek deleted the add-storage-helpers branch June 18, 2024 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

2 participants