Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeEncodeError: 'latin-1' codec can't encode characters in position 13-15: ordinal not in range(256) #6633

Open
Helena152 opened this issue Dec 9, 2024 · 5 comments

Comments

@Helena152
Copy link

C:\Users\USERNAME>gallery-dl https://www.fanbox.cc/@msmspc --verbose
[gallery-dl][debug] Version 1.28.1 - Executable (stable/windows)
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.19045
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.3
[gallery-dl][debug] Configuration Files ['%USERPROFILE%\gallery-dl.conf', 'C:\Users\USERNAME\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.fanbox.cc/@msmspc'
[fanbox][debug] Using FanboxCreatorExtractor for 'https://www.fanbox.cc/@msmspc'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.fanbox.cc:443
[fanbox][error] An unexpected error occurred: UnicodeEncodeError - 'latin-1' codec can't encode characters in position 13-15: ordinal not in range(256). Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[fanbox][debug]
Traceback (most recent call last):
File "gallery_dl\job.pyc", line 151, in run
File "gallery_dl\extractor\fanbox.pyc", line 59, in items
File "gallery_dl\extractor\fanbox.pyc", line 340, in _pagination_creator
File "gallery_dl\extractor\common.pyc", line 174, in request
File "requests\sessions.pyc", line 589, in request
File "requests\sessions.pyc", line 703, in send
File "requests\adapters.pyc", line 667, in send
File "urllib3\connectionpool.pyc", line 789, in urlopen
File "urllib3\connectionpool.pyc", line 495, in _make_request
File "urllib3\connection.pyc", line 440, in request
File "urllib3\connection.pyc", line 354, in putheader
File "http\client.pyc", line 1225, in putheader
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 13-15: ordinal not in range(256)

Setup: using standalone executable, as for unknown reason any installations attempted via py -3 install commands are not recognized as existing (typing in gallery-dl in command prompt after the installation and without the standalone executable results in the command not being recognized).
No problems when using it on kemono sites.
Changed system locale and checked on the utf-8 beta feature thingy too, as I saw some references to locale and encoding from past issues. Problem persists even after computer restart.
I have basically zero proficiency in programming and python, so I will need detailed step-by-step instructions (as in go look for this in this folder, paste this command in command prompt, etc) to even acquire additional information if needed.

@mikf
Copy link
Owner

mikf commented Dec 9, 2024

I think you are setting some weird - non-latin1 encodable - cookie or general header values for fanbox in your two (why?) config files.

What do your fanbox cookie settings look like? (censor any sensitive values)


as for unknown reason any installations attempted via py -3 install commands are not recognized as existing (typing in gallery-dl in command prompt after the installation and without the standalone executable results in the command not being recognized

You need to either update PATH or use py -m gallery_dl ...: See #6624

@Helena152
Copy link
Author

Now it says something about cloudflare challenge or something (for both with and without the py -m thing) instead of problem with encodable thingy, and I literally just came back from work so I did not change anything until this time.

C:\Users\USERNAME>py -m gallery_dl https://www.fanbox.cc/@memechan --verbose
[gallery-dl][debug] Version 1.28.1
[gallery-dl][debug] Python 3.12.8 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.3
[gallery-dl][debug] Configuration Files ['%USERPROFILE%\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.fanbox.cc/@memechan'
[fanbox][debug] Using FanboxCreatorExtractor for 'https://www.fanbox.cc/@memechan'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.fanbox.cc:443
[urllib3.connectionpool][debug] https://api.fanbox.cc:443 "GET /post.paginateCreator?creatorId=memechan HTTP/11" 403 None
[fanbox][warning] Cloudflare challenge
[fanbox][error] HttpError: '403 Forbidden' for 'https://api.fanbox.cc/post.paginateCreator?creatorId=memechan'
[fanbox][debug]
Traceback (most recent call last):
File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\gallery_dl\job.py", line 151, in run
for msg in extractor:
^^^^^^^^^
File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\gallery_dl\extractor\fanbox.py", line 59, in items
for content_body, post in self.posts():
^^^^^^^^^^^^
File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\gallery_dl\extractor\fanbox.py", line 340, in _pagination_creator
urls = self.request(url, headers=self.headers).json()["body"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\gallery_dl\extractor\common.py", line 244, in request
raise exception.HttpError(msg, response)
gallery_dl.exception.HttpError: '403 Forbidden' for 'https://api.fanbox.cc/post.paginateCreator?creatorId=memechan'

I was originally using --cookies pathtotheexportedcookies.txt command, but I manually added them in the config file just now too (my config file was originally missing cf_clearance and __cf_bm, if that is of any relevance. I do not remember when the config file first appeared or how I got it, but I am absolutely sure I did not make it myself).

I don't know anything about two config files, I only see one in the same folder as the gallery-dl.exe.

"fanbox": {
"metadata": true,
"directory": ["{user[name]}", "{date:%y.%m.%d} {title}"],
"filename": "{filename}.{extension}",
"postprocessors": [
{
"name": "metadata",
"event": "post",
"filename": "{date:%y.%m.%d} {title}.txt",
"mode": "custom",
"format": "{content}"
}
],
"cookies": {
"FANBOXSESSID": "numbers_numbersandletters",
"cf_clearance": "longlineofnumberslettersanddashes",
"__cf_bm": "shorterlineofnumberslettersanddashes"
}
}

@Helena152
Copy link
Author

I reinstalled python 3.12.8, making sure to check the box that says add to PATH before finishing installation. Now gallery-dl command is accepted even after I delete the standalone executable, but the cloudflare challenge thing still persists. I just tested it with another website (subscribestar) and it worked fine.

@Helena152
Copy link
Author

Exported cookies .txt file content:

Netscape HTTP Cookie File

http://curl.haxx.se/rfc/cookie_spec.html

This is a generated file! Do not edit.

.fanbox.cc TRUE / TRUE 1738049376 p_ab_id 0
.fanbox.cc TRUE / TRUE 1738049376 p_ab_id_2 5
.fanbox.cc TRUE / TRUE 1738049376 p_ab_d_id 309357378
.fanbox.cc TRUE / TRUE 1736380166 FANBOXSESSID numbers_numbersandletters
.fanbox.cc TRUE / TRUE 1768348166 privacy_policy_agreement 0
.fanbox.cc TRUE / TRUE 1768348166 privacy_policy_notification 0
.fanbox.cc TRUE / TRUE 1733789966 __cf_bm shorterlineofnumberslettersanddashes
.fanbox.cc TRUE / TRUE 1765324167 cf_clearance longlineofnumberslettersanddashes

@mikf
Copy link
Owner

mikf commented Dec 11, 2024

.fanbox.cc TRUE / TRUE 1765324167 cf_clearance longlineofnumberslettersanddashes

Good. You need this cf_clearance cookie (you might need to update it again) together with the user-agent header of your web browser in orderf to bypass "Cloudflare challenge" pages.

See #5791

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants