Skip to content

Commit 191dfb1

Browse files
Vincent1-pythonVincent1-python
committed
esp32: Merge branch 'master' into idf6.2_esp32s31.
Signed-off-by: Vincent1-python <[email protected]>
2 parents 0f85a3a + f67ab9e commit 191dfb1

45 files changed

Lines changed: 1231 additions & 60 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ports_esp32.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,14 +34,14 @@ jobs:
3434
- esp32_build_cmod_spiram_s2
3535
- esp32_build_s3_c3
3636
- esp32_build_c2_c5_c6
37-
- esp32_build_p4
37+
- esp32_build_h2_p4
3838
- esp32_build_s31
3939
#exclude:
4040
# Exclude some jobs on the oldest IDF version, to save resources
4141
#- idf_ver: *oldest
4242
# ci_func: esp32_build_c2_c5_c6
4343
#- idf_ver: *oldest
44-
# ci_func: esp32_build_p4
44+
# ci_func: esp32_build_h2_p4
4545
runs-on: ubuntu-latest
4646
steps:
4747
- uses: actions/checkout@v7

‎docs/develop/writingtests.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,11 @@ If you run your tests, this test should appear in the test output:
4646
Tests are run by comparing the output from the test target against the output from CPython.
4747
So any test should use print statements to indicate test results.
4848

49+
When writing tests for name or string-related functionality, please add both English/ASCII
50+
as well as non-English/non-ASCII text and include Unicode examples.
51+
Please do add comments in English explaining the meaning and intent of the Unicode text.
52+
This help ensure Unicode support is tested and verified across different platforms.
53+
4954
For tests that can't be compared to CPython (i.e. micropython-specific functionality),
5055
you can provide a ``.py.exp`` file which will be used as the truth for comparison.
5156

‎docs/library/builtins.rst‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,35 @@ Functions and types
2525

2626
|see_cpython| `python:bytes`.
2727

28+
.. method:: bytes.decode(encoding='utf-8', errors='strict')
29+
30+
Decode the bytes object to a string using the specified *encoding*.
31+
32+
MicroPython supports the following encodings:
33+
34+
- ``'utf-8'`` or ``'utf8'`` - UTF-8 encoding (default)
35+
- ``'ascii'`` - ASCII encoding (subset of UTF-8)
36+
37+
The *errors* parameter controls how decoding errors are handled:
38+
39+
- ``'strict'`` - Raise a ``UnicodeError`` on invalid UTF-8 (default)
40+
- ``'ignore'`` - Skip invalid bytes (requires ``MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS``)
41+
- ``'replace'`` - Replace invalid bytes with U+FFFD '�' (requires ``MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS``)
42+
43+
.. note::
44+
Error handler support depends on build configuration. On constrained
45+
systems, only ``'strict'`` mode may be available.
46+
47+
Example::
48+
49+
>>> b'\xc2\xa9 2024'.decode('utf-8') # © symbol
50+
'© 2024'
51+
>>> b'hello\xffworld'.decode('utf-8', 'ignore') # Skip invalid bytes
52+
'helloworld'
53+
54+
Raises ``LookupError`` if the encoding is not supported, or
55+
``UnicodeError`` if the data contains invalid UTF-8 and ``errors='strict'``.
56+
2857
.. function:: callable()
2958

3059
.. function:: chr()
@@ -144,6 +173,35 @@ Functions and types
144173

145174
.. class:: str()
146175

176+
.. method:: str.encode(encoding='utf-8')
177+
178+
Encode the string to bytes using the specified *encoding*.
179+
180+
MicroPython supports the following encodings:
181+
182+
- ``'utf-8'`` or ``'utf8'`` - UTF-8 encoding (default)
183+
- ``'ascii'`` - ASCII encoding (subset of UTF-8)
184+
185+
Example::
186+
187+
>>> '© 2024'.encode('utf-8') # Copyright symbol
188+
b'\xc2\xa9 2024'
189+
190+
Raises ``LookupError`` if the encoding is not supported.
191+
192+
.. method:: str.center(width)
193+
194+
Return a centered string of length *width*. Padding is done using spaces.
195+
196+
When Unicode support is enabled (``MICROPY_PY_BUILTINS_STR_UNICODE``), this
197+
method counts Unicode characters rather than bytes, ensuring proper alignment
198+
for multi-byte UTF-8 characters.
199+
200+
Example::
201+
202+
>>> 'café'.center(10) # é is 2 bytes in UTF-8
203+
' café '
204+
147205
.. function:: sum()
148206

149207
.. function:: super()

‎docs/reference/constrained.rst‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -251,7 +251,26 @@ instances so the process of eliminating Unicode can be painless.
251251
b = b'the quick brown fox' # A bytes instance
252252
253253
Where it is necessary to convert between strings and bytes the :meth:`str.encode`
254-
and the :meth:`bytes.decode` methods can be used. Note that both strings and bytes
254+
and the :meth:`bytes.decode` methods can be used. MicroPython validates the
255+
encoding parameter and only supports UTF-8 and ASCII. The :meth:`bytes.decode`
256+
method also supports error handlers (``'ignore'`` and ``'replace'``) for handling
257+
invalid UTF-8, when enabled in the build configuration.
258+
259+
For memory-conscious applications processing untrusted data, using the ``'ignore'``
260+
error handler can be more efficient than ``'strict'`` mode (the default), as it
261+
avoids raising exceptions while still recovering valid text::
262+
263+
# Strict mode (default) raises an error on invalid UTF-8
264+
try:
265+
s = data.decode('utf-8')
266+
except UnicodeError:
267+
# Handle error
268+
pass
269+
270+
# Ignore mode skips invalid bytes (more memory-efficient)
271+
s = data.decode('utf-8', 'ignore')
272+
273+
Note that both strings and bytes
255274
are immutable. Any operation which takes as input such an object and produces
256275
another implies at least one RAM allocation to produce the result. In the
257276
second line below a new bytes object is allocated. This would also occur if ``foo``

‎docs/reference/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,5 +31,6 @@ implementation and the best practices to use them.
3131
packages.rst
3232
asm_thumb2_index.rst
3333
filesystem.rst
34+
unicode_support.rst
3435
pyboard.py.rst
3536
micropython2_migration.rst

‎docs/reference/unicode_support.rst‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
.. _unicode_support:
2+
3+
Unicode Support
4+
===============
5+
6+
MicroPython provides Unicode support for strings, with the level of support
7+
depending on the build configuration.
8+
9+
Terminology
10+
-----------
11+
12+
This document uses the following Unicode terms:
13+
14+
- **Code point**: a single Unicode value in the range U+0000 to U+10FFFF, for
15+
example U+0041 ``A`` or U+1F600 😀. MicroPython strings are sequences of
16+
code points.
17+
- **Character**: informally used to mean a code point. Be aware that a
18+
user-perceived character (a *grapheme*) may consist of several code points,
19+
such as a base letter followed by combining marks.
20+
- **Byte**: a single 8-bit value. In UTF-8 each code point is stored as one to
21+
four bytes (see below).
22+
23+
Operations such as ``len()``, indexing and slicing act on code points, not on
24+
graphemes or display width, so a base letter followed by a combining mark counts
25+
as two code points.
26+
27+
Character Encoding
28+
------------------
29+
30+
MicroPython uses UTF-8 encoding for all strings. When Unicode support is enabled
31+
(``MICROPY_PY_BUILTINS_STR_UNICODE``), strings can contain any valid Unicode code
32+
point from U+0000 to U+10FFFF.
33+
34+
ASCII characters (0-127) are stored in a single byte, making them as memory-efficient
35+
as on systems without Unicode support. Multi-byte UTF-8 code points use 2-4 bytes
36+
depending on the code point:
37+
38+
- U+0000 to U+007F: 1 byte (ASCII)
39+
- U+0080 to U+07FF: 2 bytes
40+
- U+0800 to U+FFFF: 3 bytes
41+
- U+10000 to U+10FFFF: 4 bytes
42+
43+
Encoding and Decoding
44+
----------------------
45+
46+
The :meth:`bytes.decode` and :meth:`str.encode` methods support the following encodings:
47+
48+
- UTF-8 (``'utf-8'`` or ``'utf8'``)
49+
- ASCII (``'ascii'``)
50+
51+
Other encodings (such as ``'latin-1'``, ``'utf-16'``, etc.) are not supported and
52+
will raise ``LookupError``.
53+
54+
Example::
55+
56+
>>> '日本語'.encode('utf-8')
57+
b'\xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e'
58+
>>> b'\xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e'.decode('utf-8')
59+
'日本語'
60+
61+
Error Handling
62+
~~~~~~~~~~~~~~
63+
64+
When decoding bytes that contain invalid UTF-8 sequences, the ``errors`` parameter
65+
of :meth:`bytes.decode` controls the behavior:
66+
67+
- ``'strict'`` (default): Raise ``UnicodeError``
68+
- ``'ignore'``: Skip invalid bytes (requires ``MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS``)
69+
- ``'replace'``: Replace invalid bytes with U+FFFD � (requires ``MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS``)
70+
71+
Example::
72+
73+
>>> # Strict mode (default) raises an error
74+
>>> b'hello\xffworld'.decode('utf-8')
75+
UnicodeError: invalid UTF-8
76+
77+
>>> # Ignore mode skips invalid bytes
78+
>>> b'hello\xffworld'.decode('utf-8', 'ignore')
79+
'helloworld'
80+
81+
>>> # Replace mode substitutes replacement character
82+
>>> b'hello\xffworld'.decode('utf-8', 'replace')
83+
'hello�world'
84+
85+
For memory-conscious applications, consider using ``'ignore'`` mode when processing
86+
untrusted or partially corrupted data, as it avoids raising exceptions while still
87+
recovering valid text.
88+
89+
The same ``errors`` handling applies when decoding any bytes-like object, including
90+
via the ``str()`` constructor (for example ``str(buf, 'utf-8', 'replace')`` where
91+
``buf`` is a ``bytes``, ``bytearray``, ``memoryview`` or ``array`` object).
92+
93+
94+
String Methods
95+
--------------
96+
97+
When Unicode support is enabled, string methods operate on code points rather than bytes:
98+
99+
- :meth:`str.center` - Counts code points for width calculation
100+
- ``len(s)`` - Returns number of code points (not bytes)
101+
- String indexing and slicing work on code-point boundaries
102+
- No support for display width calculations (East Asian width, combining characters, etc.)
103+
104+
Example::
105+
106+
>>> s = 'Hello 世界'
107+
>>> len(s) # 8 code points
108+
8
109+
>>> len(s.encode()) # 12 bytes
110+
12
111+
>>> s.center(12) # Centered by code-point count
112+
' Hello 世界 '
113+
114+
String Formatting
115+
-----------------
116+
117+
The ``%c`` format specifier and ``{:c}`` format code support full Unicode:
118+
119+
- Accepts code points from 0 to 0x10FFFF
120+
- Properly encodes multi-byte UTF-8 code points
121+
- Raises ``ValueError`` for invalid code points
122+
123+
Example::
124+
125+
>>> '%c' % 65 # ASCII
126+
'A'
127+
>>> '%c' % 0x03B1 # Greek α
128+
'α'
129+
>>> '%c' % 0x1F600 # Emoji 😀
130+
'😀'
131+
>>> '{:c}'.format(0x4E2D) # Chinese 中
132+
'中'
133+
134+
>>> # Invalid code point
135+
>>> '%c' % 0x110000
136+
ValueError: %c arg not in range(0x110000)
137+
138+
F-strings also support the ``:c`` format code::
139+
140+
>>> code_point = 0x2665 # Heart suit ♥
141+
>>> f'I {code_point:c} Python'
142+
'I ♥ Python'
143+
144+
Build Configuration
145+
-------------------
146+
147+
Unicode features are controlled by several build-time flags in ``mpconfigport.h``:
148+
149+
``MICROPY_PY_BUILTINS_STR_UNICODE``
150+
Enable Unicode string support. When enabled, strings can contain any valid
151+
Unicode character and string operations work on character boundaries rather
152+
than byte boundaries.
153+
154+
Default: Enabled at ``MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES`` and above.
155+
156+
``MICROPY_PY_BUILTINS_STR_UNICODE_CHECK``
157+
Enable UTF-8 validation during string operations. When disabled, string
158+
operations may produce incorrect results with invalid UTF-8 sequences.
159+
160+
Default: Follows ``MICROPY_PY_BUILTINS_STR_UNICODE`` setting.
161+
162+
``MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS``
163+
Enable the ``'ignore'`` and ``'replace'`` error handlers for
164+
:meth:`bytes.decode`. When enabled, invalid UTF-8 bytes can be either
165+
skipped (``'ignore'``) or replaced with U+FFFD (``'replace'``).
166+
167+
Default: Enabled at ``MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES`` and above.
168+
169+
Example Configuration
170+
~~~~~~~~~~~~~~~~~~~~~
171+
172+
For a constrained port with limited flash, disable error handlers::
173+
174+
#define MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS (0)
175+
176+
For a port with more resources, enable all Unicode features::
177+
178+
#define MICROPY_CONFIG_ROM_LEVEL (MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES)
179+
// This automatically enables:
180+
// - MICROPY_PY_BUILTINS_STR_UNICODE
181+
// - MICROPY_PY_BUILTINS_BYTES_DECODE_ERRORS
182+
183+
Limitations
184+
-----------
185+
186+
MicroPython's Unicode support has some limitations compared to CPython:
187+
188+
- Only UTF-8 and ASCII encodings are supported
189+
- No support for Unicode normalization
190+
- No locale-aware string operations
191+
- The ``errors`` parameter accepts only positional arguments (not keyword arguments)
192+
- String methods like ``upper()``, ``lower()``, etc. work correctly only for ASCII
193+
- The MicroPython interactive REPL and ``input()`` function currently have limited
194+
Unicode support. The line editor is unaware of the displayed width of characters:
195+
wide characters (for example many CJK characters) take two terminal columns, while a
196+
grapheme cluster (a base code point plus combining marks, or an emoji sequence) can
197+
span several code points yet occupy a single column. Because editing tracks code
198+
points rather than displayed columns, line-editing keys such as backspace and the
199+
left/right arrows may leave the cursor misaligned with the text shown on screen.
200+
A workaround is to place the Unicode text in a UTF-8 encoded MicroPython script and
201+
run it using ``mpremote run <script.py>``.

‎ports/esp32/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ This is a port of MicroPython to the Espressif ESP32 series of
55
microcontrollers. It uses the ESP-IDF framework and MicroPython runs as
66
a task under FreeRTOS.
77

8-
Currently supports ESP32, ESP32-C2 (aka ESP8684), ESP32-C3, ESP32-C5, ESP32-C6,
8+
Currently supports ESP32, ESP32-C2 (aka ESP8684), ESP32-C3, ESP32-C5, ESP32-C6, ESP32-H2,
99
ESP32-P4, ESP32-S2 and ESP32-S3. ESP8266 is supported by a separate MicroPython port.
1010

1111
Supported features include:

‎ports/esp32/boards/ESP32_GENERIC_H2/board.json‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"deploy": [
3+
"../deploy.md"
4+
],
5+
"deploy_options": {
6+
"flash_offset": "0"
7+
},
8+
"docs": "",
9+
"features": [
10+
"BLE"
11+
],
12+
"images": [
13+
"esp32h2_devkitmini.jpg"
14+
],
15+
"mcu": "esp32h2",
16+
"product": "ESP32-H2",
17+
"thumbnail": "",
18+
"url": "https://www.espressif.com/en/products/modules",
19+
"vendor": "Espressif"
20+
}

‎ports/esp32/boards/ESP32_GENERIC_H2/mpconfigboard.cmake‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
include(boards/mpconfigboard_esp32h2_common.cmake)

‎ports/esp32/boards/ESP32_GENERIC_H2/mpconfigboard.h‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
// This configuration is for a generic ESP32H2 board with 4MiB (or more) of flash.
2+
3+
#define MICROPY_HW_BOARD_NAME "ESP32H2 module"
4+
#define MICROPY_HW_MCU_NAME "ESP32-H2"
5+
6+
// Disable WiFi/WLAN (ESP32-H2 does not have WiFi)
7+
#define MICROPY_PY_NETWORK_WLAN (0)
8+
#define MICROPY_PY_ESPNOW (0)
9+
10+
// Enable UART REPL for modules that have an external USB-UART and don't use native USB.
11+
#define MICROPY_HW_ENABLE_UART_REPL (1)

0 commit comments

Comments
 (0)