fix(session): ignore unsupported Playwright cookie fields#1335
Closed
enigmAsad wants to merge 3 commits intoapify:masterfrom
Closed
fix(session): ignore unsupported Playwright cookie fields#1335enigmAsad wants to merge 3 commits intoapify:masterfrom
enigmAsad wants to merge 3 commits intoapify:masterfrom
Conversation
Playwright recently added internal fields like _crHasCrossSiteAncestor and partitionKey to cookie objects, which are not accepted by Crawlee's SessionCookies.set() method. This commit filters out any cookie keys starting with _ and removes partitionKey explicitly to prevent a TypeError. Fixes issue apify#1333"
….Adds a unit test in tests/unit/sessions/test_cookies.py to ensure that unsupported Playwright cookie fields like `_crHasCrossSiteAncestor` and `partitionKey` are filtered out correctly. Verifies that SessionCookies.set() does not raise errors when such fields are present in input cookies. Builds upon previous fix in set_cookies_from_playwright_format().
Collaborator
|
Hey, @enigmAsad Thank you for your PR and interest in Crawlee. I really appreciate it. Due to the fact that the _crHasCrossSiteAncestor key for Playwright is not described in the Playwright-python data models - https://github.com/microsoft/playwright-python/blob/main/playwright/_impl/_api_structures.py I would prefer to explicitly ignore any unknown parameters that may be passed to the cookie constructor. This would help avoid similar issues in the future. I have implemented this in PR #1336. |
Author
|
"Thanks @Mantisus for the update! That makes perfect sense, happy to see the issue resolved in a more general way. Glad I could help flag it!" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
### Problem
Crawlee currently crashes when importing cookies from Playwright due to unsupported fields like
partitionKeyand_crHasCrossSiteAncestor, which are Chromium-specific and not part of the standard cookie format.Error:
TypeError: SessionCookies.set() got an unexpected keyword argument 'partitionKey'## Root Cause
Recent versions of Playwright (especially with Chromium) include internal browser metadata fields in the cookie dictionary. Crawlee's
SessionCookies.set()method does not accept these extra keys.Affected fields:
partitionKey_crHasCrossSiteAncestor## Fix
Modified the
set_cookies_from_playwright_format()method insessions/_cookies.pyto:_partitionKeyif presentThis ensures compatibility and future-proofs against additional Chromium-internal fields.
### Tests
Added a unit test in tests/unit/sessions/test_cookies.py to verify that cookies with unsupported fields are cleaned and do not raise exceptions when passed to set_cookies_from_playwright_format().
### Environment
### Related Issue
Closes #1333