Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] PRAW exception: received 400 HTTP response #947

Open
3 tasks done
nascentt opened this issue Feb 19, 2024 · 20 comments
Open
3 tasks done

[BUG] PRAW exception: received 400 HTTP response #947

nascentt opened this issue Feb 19, 2024 · 20 comments
Labels
bug Something isn't working

Comments

@nascentt
Copy link

nascentt commented Feb 19, 2024

  • I am reporting a bug.
  • I am running the latest version of BDfR
  • I have read the Opening an issue

Description

When trying to use bdfr I get the error "PRAW exception: received 400 HTTP response" for all downloads.
I presume it's related to me being forced to reset my password on reddit, but as bdfr uses an app client id and there being nothing in the documentation about needing to change credentials after a password reset, I don't understand why it's now not working.

Command

python3 -m bdfr download --authenticate C:\bdfr\ --sort new --user nascentt --submitted

Environment (please complete the following information)

  • OS: Windows 10
  • Python version: python3 --version
    Python 3.9.13

Logs

[2024-02-19 19:17:38,330 - bdfr.connector - ERROR] - User nascentt failed to be retrieved due to a PRAW exception: received 400 HTTP response

Edit:
Ok if I remove --authenticate it works (until it hits a rate limit). So there seems to be a bug where bdfr needs to reauthenticate but it won't.

@nascentt nascentt added the bug Something isn't working label Feb 19, 2024
@drebob24
Copy link

drebob24 commented Feb 19, 2024

Have you recently had to change your reddit password?

If so you need to re-authenticate. Just delete the "user_token = fakeusertoken1234" line in the default_config file (make a copy of the file first as a backup just in case) and run another command with --authenticate which then bdfr will prompt you with a link.

@nascentt
Copy link
Author

nascentt commented Feb 19, 2024

Thanks for the assistance.

Yes, I was forced to reset the password by Reddit.However I did try deleting the line in my config with user-token and reran the download authenticate just in case but it continues to error out with a JSON decode error. Again no error without --authenticate

@Serene-Arc
Copy link
Owner

What is the new error?

@nascentt
Copy link
Author

nascentt commented Feb 20, 2024

if i remove the user_token line:

Traceback (most recent call last):
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\__main__.py", line 222, in <module>
    cli()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\__main__.py", line 117, in cli_download
    reddit_downloader = RedditDownloader(config, [stream])
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\downloader.py", line 41, in __init__
    super(RedditDownloader, self).__init__(args, logging_handlers)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\connector.py", line 63, in __init__
    self._setup_internal_objects()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\connector.py", line 80, in _setup_internal_objects
    self.create_reddit_instance()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\connector.py", line 139, in create_reddit_instance
    oauth2_authenticator = OAuth2Authenticator(
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\oauth2.py", line 21, in __init__
    self._check_scopes(wanted_scopes)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\oauth2.py", line 31, in _check_scopes
    known_scopes = [scope for scope, data in response.json().items()]
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Downloading nascentt
cmd: python3
args: -m  bdfr  download  --authenticate   C:\bdfr\ --sort new --log C:\bdfr\LOG_FILES\20240220010101\20240220010101.log --user nascentt --submitted
both: python3 -m  bdfr  download  --authenticate   C:\bdfr\ --sort new --log C:\bdfr\LOG_FILES\20240220010101\20240220010101.log --user nascentt--submitted
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\__main__.py", line 222, in <module>
    cli()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1042, in main
    args = _expand_args(args)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\utils.py", line 571, in _expand_args
    matches = glob(arg, recursive=glob_recursive)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\glob.py", line 22, in glob
    return list(iglob(pathname, recursive=recursive))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\glob.py", line 51, in _iglob
    if os.path.isdir(dirname):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\genericpath.py", line 42, in isdir

@ymgenesis
Copy link

ymgenesis commented Feb 20, 2024

I think this happened to me once. If I recall there's two config locations, one where bdfr itself is installed, and one in your users folder. I'm on Mac, but the user location is in the readme. I think I had to remove the user token line in both configs, then it prompted me with a url to login via a browser the next time I used --authenticate.

This may help (try without reinstalling first, not sure if it's needed):
#884 (comment)

Hopefully that works!

@nascentt
Copy link
Author

Many thanks for trying to help ymgenesis. Apparently I have 7 of these default_config.cfg files, what a mess.

"C:\Users\nascent\AppData\Roaming\Python\Python39\site-packages\bdfr\default_config.cfg"
"C:\Users\nascent\AppData\Roaming\Python\Python310-32\site-packages\bdfr\default_config.cfg"
"C:\Users\nascent\AppData\Roaming\Python\Python310\site-packages\bdfr\default_config.cfg"
"C:\Users\nascent\AppData\Roaming\Python\config\default_config.cfg"
"C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\default_config.cfg"
"C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\config\default_config.cfg"
"C:\Users\nascent\AppData\Local\BDFR\bdfr\default_config.cfg"

No 'Lib\site-packages\bdfr\default_config.cfg' as linked.

I ensured the user_token line was removed from all of them , but --authenticate continued to error out as above.

So I then tried pip install --upgrade --force-reinstall bdfr
after about 20 minutes it completed, and again --authenticate caused the above error.
So then I deleted all the above directories and then ran pip install --upgrade --force-reinstall bdfr - like mentioned in #884
20 minutes later, --authenticate still errors out.

Insane to me this #884 bug has been open for a year without resolution. Seems a lost cause.

I really miss v1 when bdfr didn't depend on python.

@Serene-Arc
Copy link
Owner

Well that's not true because the BDFR has always been written in python. Also that bug is partly from pipx, which is not the same. You can manually specify a configuration file path if you're not sure which one is being used.

@nascentt
Copy link
Author

I just meant it was a lot easier to use when it was an executable.
Anyway I've tried using --config to a specific config.cfg file and it still crashes into a trace

Traceback (most recent call last):
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)

If I remove --authenticate but use --config it works.

@Serene-Arc
Copy link
Owner

Providing part of the traceback is as useless as providing none. Can you provide the logs?

@nascentt
Copy link
Author

I already posted a full traceback the same day.
Not sure why the outstanding issue isn't fixed. #884 (comment)

Traceback (most recent call last):
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\__main__.py", line 222, in <module>
    cli()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\click\decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\__main__.py", line 117, in cli_download
    reddit_downloader = RedditDownloader(config, [stream])
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\downloader.py", line 41, in __init__
    super(RedditDownloader, self).__init__(args, logging_handlers)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\connector.py", line 63, in __init__
    self._setup_internal_objects()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\connector.py", line 80, in _setup_internal_objects
    self.create_reddit_instance()
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\connector.py", line 139, in create_reddit_instance
    oauth2_authenticator = OAuth2Authenticator(
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\oauth2.py", line 21, in __init__
    self._check_scopes(wanted_scopes)
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bdfr\oauth2.py", line 31, in _check_scopes
    known_scopes = [scope for scope, data in response.json().items()]
  File "C:\Users\nascent\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

@Serene-Arc
Copy link
Owner

A traceback is not the logs. Hence why I asked for the logs. They, among other things, say which configuration file you're using, which will be useful in helping you resolve this problem. They will also tell me what version of the BDFR you are running, and other important information, which is why there is a section for the logs in your bug report that currently only has a single line, which is not what was asked. Hence me asking for them now.

If you're unhappy it's not fixed, you're more than welcome to submit a PR that does so.

@nascentt
Copy link
Author

you're more than welcome to submit a PR that does so.

Nice. If I could fix it I'd have already done so.

If you want me to upload a log please direct me to the log you want me to upload.

@Serene-Arc
Copy link
Owner

This is an open-source project. If you are unhappy with the state of it, you can learn and contribute to it and get it to a place where you are satisfied. Else you can fork it and do what you want. If you're unable or unwilling to do that, you'll have to wait for someone else, such as me, to do it. That'll happen when I have the time and the motivation, since it's a project I do entirely in my free time and without compensation. Otherwise, something such as GitPay may be helpful, as you can provide a monetary bounty for having the issue solved.

Right at the top of this page, you ticked the box that said you'd read the document Opening an issue. That and the README contain clear instructions on what log file to upload, which is why it's linked right at the top, and why you're supposed to actually read it before ticking that box. It will direct you to the file to upload.

@nascentt
Copy link
Author

nascentt commented Feb 22, 2024

Not sure what warrants the attitude, but regardless I'll continue to provide information to help fix this bug.

The Opening an issue page states:

When opening an issue about a bug, please provide the full log file for the run in which the bug occurred. This log file is named log_output.txt in the configuration folder Check the README for information on where this is.

The readme states:

C:\Users\<User>\AppData\Local\BDFR\bdfr

this contains a file last updated [2021-05-30] other than that I see nothing other than 0 byte files, and one single 4kb file with the traceback I posted above.

Fingers crossed a python developer can help you if you don't have time to work on the project anymore, it's a great project and provides an amazing service when it works.

edit:
ok i deleted all bdfr directories uninstalled and reinstalled bdfr, same issue keeps happening. latest log:
bdfr download C:\temp\ --subreddit 'Python, all, mindustry' -L 10 --authenticate
log_output.txt

@Serene-Arc
Copy link
Owner

Is this the development version?

@nascentt
Copy link
Author

I just use the default version the readme says to use: python3 -m pip install bdfr --upgrade

@Serene-Arc
Copy link
Owner

Can you install the development version with python3 -m pip install https://github.com/aliparlakci/bulk-downloader-for-reddit.git

@nascentt
Copy link
Author

python3 -m pip install https://github.com/aliparlakci/bulk-downloader-for-reddit.git
Collecting https://github.com/aliparlakci/bulk-downloader-for-reddit.git
  Downloading https://github.com/aliparlakci/bulk-downloader-for-reddit.git
     / 438.9 kB 2.0 MB/s 0:00:00
  ERROR: Cannot unpack file C:\Users\nascent\AppData\Local\Temp\pip-unpack-9ckplgfn\bulk-downloader-for-reddit.git (downloaded from C:\Users\nascent\AppData\Local\Temp\pip-req-build-c7uofazv, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of C:\Users\nascent\AppData\Local\Temp\pip-req-build-c7uofazv

@Serene-Arc
Copy link
Owner

Sorry it's python3 -m pip install git+https://github.com/aliparlakci/bulk-downloader-for-reddit.git

@kungfoome
Copy link

Sorry it's python3 -m pip install git+https://github.com/aliparlakci/bulk-downloader-for-reddit.git

This worked for me as I was having the same issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants