Skip to content

Commit c19571d

Browse files
illia-villia-v
authored
Merge commit from fork
* Prevent decompression bomb for zstd in Python 3.14 * Add experimental `decompress_iter` for Brotli * Update changes for Brotli * Add `GzipDecoder.decompress_iter` * Test python-hyper/brotlicffi#207 * Pin Brotli * Add `decompress_iter` to all decoders and make tests pass * Pin brotlicffi to an official release * Revert changes to response.py * Add `max_length` parameter to all `decompress` methods * Fix the `test_brotlipy` session * Unset `_data` on gzip error * Add a test for memory usage * Test more methods * Fix the test for `stream` * Cover more lines with tests * Add more coverage * Make `read1` a bit more efficient * Fix PyPy tests for Brotli * Revert an unnecessarily moved check * Add some comments * Leave just one `self._obj.decompress` call in `GzipDecoder` * Refactor test params * Test reads with all data already in the decompressor * Prevent needless copying of data decoded with `max_length` * Rename the changed test * Note that responses of unknown length should be streamed too * Add a changelog entry * Avoid returning a memory view from `BytesQueueBuffer` * Add one more note to the changelog entry
1 parent 816fcf0 commit c19571d

File tree

8 files changed

+621
-154
lines changed

8 files changed

+621
-154
lines changed

‎CHANGES.rst‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,25 @@
1+
2.6.0 (TBD)
2+
==================
3+
4+
Bugfixes
5+
--------
6+
7+
- Fixed a security issue where streaming API could improperly handle highly
8+
compressed HTTP content ("decompression bombs") leading to excessive resource
9+
consumption even when a small amount of data was requested. Reading small
10+
chunks of compressed data is safer and much more efficient now.
11+
12+
.. caution::
13+
- If urllib3 is not installed with the optional `urllib3[brotli]` extra, but
14+
your environment contains a Brotli/brotlicffi/brotlipy package anyway, make
15+
sure to upgrade it to at least Brotli 1.2.0 or brotlicffi 1.2.0.0 to
16+
benefit from the security fixes and avoid warnings. Prefer using
17+
`urllib3[brotli]` to install a compatible Brotli package automatically.
18+
19+
- If you use custom decompressors, please make sure to update them to
20+
respect the changed API of ``urllib3.response.ContentDecoder``.
21+
22+
123
2.5.0 (2025-06-18)
224
==================
325

‎docs/advanced-usage.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,8 @@ When using ``preload_content=True`` (the default setting) the
6666
response body will be read immediately into memory and the HTTP connection
6767
will be released back into the pool without manual intervention.
6868

69-
However, when dealing with large responses it's often better to stream the response
69+
However, when dealing with responses of large or unknown length,
70+
it's often better to stream the response
7071
content using ``preload_content=False``. Setting ``preload_content`` to ``False`` means
7172
that urllib3 will only read from the socket when data is requested.
7273

‎docs/user-guide.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,8 +145,8 @@ to a byte string representing the response content:
145145
print(resp.data)
146146
# b"\xaa\xa5H?\x95\xe9\x9b\x11"
147147
148-
.. note:: For larger responses, it's sometimes better to :ref:`stream <stream>`
149-
the response.
148+
.. note:: For responses of large or unknown length, it's sometimes better to
149+
:ref:`stream <stream>` the response.
150150

151151
Using io Wrappers with Response Content
152152
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

‎noxfile.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
def tests_impl(
1616
session: nox.Session,
1717
extras: str | None = None,
18+
extra_dependencies: list[str] | None = None,
1819
# hypercorn dependency h2 compares bytes and strings
1920
# https://github.com/python-hyper/h2/issues/1236
2021
byte_string_comparisons: bool = False,
@@ -33,8 +34,9 @@ def tests_impl(
3334
implementation_name, release_level, _is_gil_enabled = session_python_info.split(" ")
3435
free_threading = _is_gil_enabled == "False"
3536

36-
# brotlicffi does not support free-threading
37-
extras = "socks,zstd,h2" if free_threading else "socks,brotli,zstd,h2"
37+
if extras is None:
38+
# brotlicffi does not support free-threading
39+
extras = "socks,zstd,h2" if free_threading else "socks,brotli,zstd,h2"
3840

3941
# Install deps and the package itself.
4042
session.run_install(
@@ -45,6 +47,8 @@ def tests_impl(
4547
dependency_group,
4648
*(f"--extra={extra}" for extra in (extras.split(",") if extras else ())),
4749
)
50+
if extra_dependencies:
51+
session.install(*extra_dependencies)
4852
# Show the uv version.
4953
session.run("uv", "--version")
5054
# Print the Python version, bytesize and free-threading status.
@@ -130,8 +134,12 @@ def test_brotlipy(session: nox.Session) -> None:
130134
'brotlicffi' that we still don't blow up.
131135
"""
132136
session.env["UV_PROJECT_ENVIRONMENT"] = session.virtualenv.location
133-
session.install("brotlipy")
134-
tests_impl(session, extras="socks", byte_string_comparisons=False)
137+
tests_impl(
138+
session,
139+
extras="socks",
140+
extra_dependencies=["brotlipy"],
141+
byte_string_comparisons=False,
142+
)
135143

136144

137145
def git_clone(session: nox.Session, git_url: str) -> None:

‎pyproject.toml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ dynamic = ["version"]
4343

4444
[project.optional-dependencies]
4545
brotli = [
46-
"brotli>=1.0.9; platform_python_implementation == 'CPython'",
47-
"brotlicffi>=0.8.0; platform_python_implementation != 'CPython'"
46+
"brotli>=1.2.0; platform_python_implementation == 'CPython'",
47+
"brotlicffi>=1.2.0.0; platform_python_implementation != 'CPython'"
4848
]
4949
zstd = [
5050
"backports.zstd>=1.0.0; python_version<'3.14'",
@@ -158,6 +158,7 @@ filterwarnings = [
158158
'''default:ssl\.PROTOCOL_TLSv1_1 is deprecated:DeprecationWarning''',
159159
'''default:ssl\.PROTOCOL_TLSv1_2 is deprecated:DeprecationWarning''',
160160
'''default:ssl NPN is deprecated, use ALPN instead:DeprecationWarning''',
161+
'''default:Brotli >= 1.2.0 is required to prevent decompression bombs\.:urllib3.exceptions.DependencyWarning''',
161162
# https://github.com/SeleniumHQ/selenium/issues/13328
162163
'''default:unclosed file <_io\.BufferedWriter name='/dev/null'>:ResourceWarning''',
163164
# https://github.com/SeleniumHQ/selenium/issues/14686

0 commit comments

Comments
 (0)