esp32: Support Esp-idf6.2 and ESP32S31#19421
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19421 +/- ##
=======================================
Coverage 98.51% 98.51%
=======================================
Files 177 177
Lines 22992 22992
=======================================
Hits 22651 22651
Misses 341 341 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
f220bed to
d988983
Compare
|
With regard to the failure of esp32 port / build_idf (master, esp32_build_cmod_spiram_s2) (pull_request), I must declare that this CI is related to Btree, which cannot be compiled because of the compiler, but the firmware can see that the compilation is successful. I took a look at the IDF version of CI in magic change and will change it back later. |
e21521d to
2fb77cc
Compare
548ba58 to
de9cee9
Compare
de9cee9 to
f44abbb
Compare
|
I don't quite understand the CI of check code size/build (pull_request), but my code is correct, but it uses the old version of the lock file and adds the newlib I have eliminated. I don't know where to start. |
Signed-off-by: Vincent1-python <[email protected]>
f44abbb to
5e5ef31
Compare
|
Basically completed, please review. @dpgeorge |
Signed-off-by: Vincent1-python <[email protected]>
Signed-off-by: Vincent1-python <[email protected]>
|
Wouldn't it make more sense to split this into two PRs (ESP-IDF update and S31 support)? I've had a quick look but there are so many odd changes that make this harder to review than it should be. The only review comment I can make with some certainty is that the ESP32S31 does support the RV32 Zcmp extension, like the ESP32P4 (as they probably share the same CPU core). Please update Also, did you actually run the native emitter/viper tests on this board? (And I guess this will have the same response as when I asked the same question for the P4 :)) |
I have already made a statement about these changes, that is, I use Espressif's official warehouse esp-vision to make the changes. Of course, I don't rule out that Espressif's warehouse is strange or lacks certain changes, but I am also trying to fix it, and there may be some omissions. Just ask me to fix it. After all, this PR is a long-term project, so I am not in a hurry. |
As for the native emitter/viper tests you mentioned, I just tried it before joining RV32 Zcmp extension. The code and results are as follows. |
Thanks! For the P4 we ended up finding at the last minute that the emitter for this specific CPU core required a forced cache flush, which could have been caught earlier :) It's nice to know that change is still valid. For the future, you can also run the native test suite via [1] If you're on Linux, that is. Replace u0 with the appropriate device name if it doesn't appear as /dev/ttyUSB0 (u1 for /dev/ttyUSB1, a0 for /dev/ttyACM0, etc.). If you're on Windows I'm afraid I have no idea on how to select the COM port it binds to. |
|
After the extension of RV32 Zcmp has just been added, the test results are as follows. |
Ok, I'm going to solve PCNT soon. I'll try it when I solve it. |
fc57ca6 to
8253245
Compare
Signed-off-by: Vincent1-python <[email protected]> py: Fix build with no error detail reporting enabled. This commit fixes two build errors occurring when building any modern port with error detail reporting is not enabled (ie. `MICROPY_ERROR_REPORTING` is set to `MICROPY_ERROR_REPORTING_NONE` when building). Signed-off-by: Alessandro Gatti <[email protected]> tests/feature_check/target_info.py: Work with error reporting disabled. This commit updates the target info reporting script to work correctly even when error reporting is disabled, effectively skipping any error message attached to exceptions. The code expected the exception message to be an empty string, but it is `None` instead, making the script fail and the test runner not able to execute tests. Signed-off-by: Alessandro Gatti <[email protected]> tests/run-tests.py: Update tests skip list for no detailed errors. This commit updates the list of tests to skip if the error message level is "none", adding `extmod/asyncio_gather_notimpl.py`. `extmod/asyncio_gather_notimpl.py` in its current form depends on runtime exceptions to have different error messages to identify certain situations. Since when the interpreter is configured to report errors with no details except for the exception type, this test cannot be relied upon in this case. Signed-off-by: Alessandro Gatti <[email protected]> tests/basics: Fix tests with no error details. This commit updates `basics/deque_micropython`, `basics/generator_pend_throw`, `basics/int_64_basics`, and `basics/subclass_native_exc_new` tests to also pass even if the interpreter was built with no detailed error messages. Originally the tests assumed the error messages level was always higher than `MICROPY_ERROR_REPORTING_NONE`, and thus expected a particular string pattern to appear in certain errors' output. Signed-off-by: Alessandro Gatti <[email protected]> tests/extmod: Fix tests with no error details. This commit updates `extmod/asyncio_cancel_self`, `extmod/cryptolib_aes12_ctr`, `extmod/ssl_cadata`, `extmod/ssl_keycert`, and `extmod/ssl_keycert_pkcs8` tests to also pass even if the interpreter was built with no detailed error messages. Originally the tests assumed the error messages level was always higher than `MICROPY_ERROR_REPORTING_NONE`, and thus expected a particular string pattern to appear in certain errors' output. Signed-off-by: Alessandro Gatti <[email protected]> tests/float/math: Fix tests with no error details. This commit updates `float/math_fun` and `float/math_fun_special` tests to also pass even if the interpreter was built with no detailed error messages. Originally the tests assumed the error messages level was always higher than `MICROPY_ERROR_REPORTING_NONE`, and thus expected a particular string pattern to appear in certain errors' output. Signed-off-by: Alessandro Gatti <[email protected]> tests/stress: Fix tests with no error details. This commit updates `stress/bytecode_limit`, `stress/qstr_limit`, and `stress/qstr_limit_str_modulo` tests to also pass even if the interpreter was built with no detailed error messages. Originally the tests assumed the error messages level was always higher than `MICROPY_ERROR_REPORTING_NONE`, and thus expected a particular string pattern to appear in certain errors' output. Signed-off-by: Alessandro Gatti <[email protected]> tests/micropython: Fix tests with no error details. This commit updates `micropython/import_mpy_invalid`, `micropython/import_mpy_native`, and `micropython/viper_error` tests to also pass even if the interpreter was built with no detailed error messages. Originally the tests assumed the error messages level was always higher than `MICROPY_ERROR_REPORTING_NONE`, and thus expected a particular string pattern to appear in certain errors' output. Signed-off-by: Alessandro Gatti <[email protected]> tools/ci.sh: Add new Unix target with no error messages. This commit adds a new CI target for the Unix port, which is the `standard` variant being built with no error messages (as in, exceptions won't report anything but the exception type). Even though it won't cover all possible tests failing with no error messages (some occur only in the `coverage` build variant), this should be a good start for preventing new tests to not assume a particular error message level. To make things automated, the new target is also added to the regular CI jobs set in the Unix port's GitHub workflow file. Signed-off-by: Alessandro Gatti <[email protected]> tests/extmod/vfs_blockdev_invalid.py: Catch only expected exceptions. This allows MemoryError to propagate through and skip the test, eg on esp8266 boards. Signed-off-by: Damien George <[email protected]> tests/extmod_hardware/machine_sdcard_dma_align.py: Allow skip on esp32. Some esp32 boards (eg ESP32_GENERIC_C3) require extra arguments to the SDCard constructor and fail with a ValueError if they aren't provided. Signed-off-by: Damien George <[email protected]> py/dynruntime.mk: Refactor Picolibc probing for RISC-V targets. This commit moves all Picolibc probing done for RISC-V targets into one single place. Both RV32 and RV64 target sections performed the same standard library probing checks to see whether to use Picolibc or Newlib. Those two directive blocks have been merged in one single place outside of the main target default definition assignments. This should also make things easier when adding Picolibc support to other targets if the need arises, since Picolibc architecture support is extensive enough to cover all supported MicroPython targets. Signed-off-by: Alessandro Gatti <[email protected]> py/dynruntime.mk: Let natmods be built with Clang. This commit modifies the build rules for native modules in order to remove the dependence on GCC for creating native MPY files. Whilst the Unix port of MicroPython can be built with Clang by overriding the `CC` variable, natmods require a bit more work. GCC builds compilers that are tailored for a single architecture, but Clang takes the opposite approach, so a single binary may target more than one architecture. Architecture selection is, by definition, not compatible between those two compilers. These changes attempt to make things easier to handle when using Clang. Native modules can now be built with something like this: make CC=clang ARCH=<arch> CFLAGS_EXTRA='--target=<clang-target-arch>' So, for example building an x86 native module the command line will look something like this: make CC=clang ARCH=x86 CFLAGS_EXTRA='--target=i686-unknown-linux-gnu' Clang and GCC, however, have different tolerances for deviations from the chosen C standard. Whilst GCC doesn't really mind whether a typedef is defined multiple times as long as it is defined to the same value, Clang does raise a warning which is then interpreted as an error. Unfortunately #ifdef/#ifndef does not work with typedefs, and the way native modules are built meant that `py/mpconfig.h` would first include the native module's generated configuration file and then proceed with the rest of the configuration. However, both files attempt to provide aliases for both `mp_int_t` and `mp_uint_t`, and that doesn't really work. Those definitions aren't going to be emitted any longer by the linker, preventing this issue from occurring in the first place. Signed-off-by: Alessandro Gatti <[email protected]> tools/mpy_ld.py: Align the trampoline if requested. This commit updates the entry point trampoline generation, making sure that the requested text segment alignment is taken into account. x86 and x64 object files generated by GCC have no forced section alignment for the text segment, so the trampoline only needs to be generated once as addresses won't move around when the MPY file is assembled. This is not the case for Clang-generated objects, which have a section alignment of 4, and so the address calculation needs to take this into account. As this is still experimental, the trampoline calculation hasn't been generalised into one single block of code. If no section alignment is requested then the old calculation is performed. Signed-off-by: Alessandro Gatti <[email protected]> tools/mpy_ld.py: Add support for R_ARM_GOT_PREL relocations. This commit introduces support for the R_ARM_GOT_PREL relocation, found in the text segment of certain Arm native modules. Until now this was not needed, since the only supported compiler for linking native modules was GCC, which did not seem to ever generate such a relocation. With the recent work in making Clang a supported compiler as well, it was quickly found out that such a compiler actually does generate such a relocation type. This relocation seem to be enough to at least make `examples/natmod/features0` link with Clang targeting `armv7m`, and then run the output native module on an appropriate interpreter running under the QEMU MPS2-AN500 target. Signed-off-by: Alessandro Gatti <[email protected]> examples/natmod/btree: Fix building with Clang toolchains. This commit fixes building the `btree` module using Clang rather than using GCC, on x86, x64, and ArmV7 targets. The Clang standard library (libc) implementation of `memset` for x86 and x64 depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed for `x86` and `x64` targets. On a more general note, Clang also required some additional flags to disable an extra warning that GCC did not seem to raise. On Arm targets, `memset` is not a builtin of the compiler toolchain, so it has to be fetched from the runtime support library. Signed-off-by: Alessandro Gatti <[email protected]> examples/natmod/deflate: Fix building with Clang toolchains. This commit fixes building the `deflate` module using Clang rather than using GCC, on x86, x64, and ArmV7 targets. The Clang standard library (libc) implementation of `memset` for x86 and x64 depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed for `x86` and `x64` targets. On Arm targets, `memset` is not a builtin of the compiler toolchain, so it has to be fetched from the runtime support library. Signed-off-by: Alessandro Gatti <[email protected]> examples/natmod/features2: Fix building with Clang toolchains. This commit fixes building the `features2` module using Clang rather than using GCC. Parts of the floating point support code for Clang may end up in libc.a rather than its builtins support library. This is the case for the armv7m target, so we have to force linking symbols from libc.a in that case. Depending on the toolchain, this may or may not build successfully, but the location of the `roundf` symbol should not move across toolchains. This makes it work on armv6m, rv32imc, and possibly on armv7m too. The latter depends on whether Clang's builtins support library not relying on features that are unsupported by `tools/mpy_ld.py`. Signed-off-by: Alessandro Gatti <[email protected]> examples/natmod/framebuf: Fix building with Clang toolchains. This commit fixes building the `framebuf` module using Clang rather than using GCC, on x86, x64, and ArmV7 targets. The Clang standard library (libc) implementation of `memset` for x86 and x64 depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed for `x86` and `x64` targets. On Arm targets, `memset` is not a builtin of the compiler toolchain, so it has to be fetched from the runtime support library. Signed-off-by: Alessandro Gatti <[email protected]> examples/natmod/re: Fix building with Clang toolchains. This commit fixes building the `re` module using Clang rather than using GCC, on x86, x64, and ArmV7 targets. The Clang standard library (libc) implementation of `memset` for x86 and x64 depends on functions that have a non-empty data section, which is not currently supported. Therefore we provide our own `memset` implementation that is good enough to let linking succeed for `x86` and `x64` targets. On Arm targets, `memset` is not a builtin of the compiler toolchain, so it has to be fetched from the runtime support library. Signed-off-by: Alessandro Gatti <[email protected]> tools/ci.sh: Use an i686 cross-compiler for Unix/x86 bit builds. This commit forces usage of an i686 cross-compiler for 32-bits x86 Unix port builds, instead of relying on the "-m32" flag passed to the currently installed GCC version. Before this change it was not possible to build an x86 native module on anything but an x86/x64 machine, since the scripts used the generic "gcc" compiler installed on the system the code was built on. Recent Ubuntu versions (at least five years old now) provide a 32-bits x86 cross-compiler on all supported machines ("gcc-i686-linux-gnu" and "g++-i686-linux-gnu"), and since the CI uses Ubuntu as its base OS, that compiler is now used for x86 builds. Installing said cross-compiler, though, conflicts with the "gcc-multilib" and "g++-multilib" packages that are installed on the CI image as part of the 32-bits jobs setup procedure. This means that all 32-bits x86 builds have to be migrated to the new cross-compiler as well. Signed-off-by: Alessandro Gatti <[email protected]> unix: Remove MICROPY_FORCE_32BIT flag usage. This commit removes support for forced 32-bits builds of the Unix port, as it only worked on x64 machines and x86 usage is low enough these days to allow making changes on how the latter target is built. Passing such flag to the Unix makefile will stop the build, as the flag itself will be removed from MicroPython entirely (right now only the MinGW Windows builds use it) in subsequent commits. CI build scripts had to be updated to manually pass the necessary flags that were once implied by MICROPY_FORCE_32BIT, for all targets that are meant to build 32-bits binaries. Finally, documentation for the Unix port has been updated with a section explaining how to make x86 builds on x64 hosts work again. Given that the 32-bits builds now use a cross-compiler, then other machines for which such a compiler is available can build x86 binaries too (eg. AArch64 or RISC-V 64). Signed-off-by: Alessandro Gatti <[email protected]> py/py.mk: Move MICROPY_FORCE_32BIT to the Windows port's Makefile. This commit moves the definition of the "MICROPY_FORCE_32BIT" build flag away from the core's Makefile definition, migrating it to the Windows port's Makefile instead. Both Windows and Unix ports used this flag, which was intrinsically x64 specific. Since the Unix port is no longer using it and the Windows port is currently limited to x86/x64 builds it makes more sense to move that flag's support away from MicroPython's core. This assures that MinGW builds will still operate as usual without any changes. Support for this flag will probably be removed from the Windows port (and therefore from MicroPython) when Windows/Arm builds will show up. Signed-off-by: Alessandro Gatti <[email protected]>
2f5f640 to
6b2808a
Compare
Signed-off-by: Vincent1-python <[email protected]>
ae4404e to
aad487b
Compare
Signed-off-by: Vincent1-python <[email protected]>
aad487b to
b61b58e
Compare
Signed-off-by: Vincent1-python <[email protected]>
71a8232 to
de691b9
Compare
Signed-off-by: Vincent1-python <[email protected]>
Signed-off-by: Vincent1-python <[email protected]>
Signed-off-by: Vincent1-python <[email protected]>
06755d9 to
0f85a3a
Compare
Signed-off-by: Vincent1-python <[email protected]>
137a7ba to
191dfb1
Compare
projectgus
left a comment
There was a problem hiding this comment.
@Vincent1-python I'm afraid that while the CI is failing and there are merge conflicts this isn't ready for review. If you need help with specific problems in the CI or rebasing then please let us know.
Wouldn't it make more sense to split this into two PRs (ESP-IDF update and S31 support)?
Yes, there are too many changes here for us to review this in one PR. Please split it up into two as @agatti suggests.
We need ESP-IDF 6.2 to first work on our existing boards, and then we can look at adding support for ESP32S31 boards after that.
Generative AI
Please edit the PR description to answer this mandatory question, with reference to our policy.
I've left one inline comment but we can't review this properly until after you deal with the major issues.
| IDF_OLDEST_VER: &oldest "v5.3" | ||
| IDF_NEWEST_VER: &newest "v5.5.2" | ||
| IDF_OLDEST_VER: &oldest "master" | ||
| IDF_NEWEST_VER: &newest "master" |
There was a problem hiding this comment.
We use this in CI so we don't break support for older ESP-IDF accidentally. Turning this check off is not an option.
Summary
Support Esp-idf6.2 and ESP32S31.
Mainly refer to the support of https://github.com/espressif/esp-vision warehouse for ESPIDF6+ and ESP32S31.
Testing
ESP32 ok
ESP32C2 ok
ESP32C3 ok
ESP32C5 ok
ESP32C6 ok
ESP32S2 ok
ESP32S3 ok
ESP32P4 ok
ESP32S31 ok
Trade-offs and Alternatives
Not support:
I2S
ESPNOW
BLE
Btree
ADC(ESP32S31)
TouchPad(ESP32S31)
......
Some of them are not supported, but there is no serious problem in normal use, so you can consider supporting them separately.
Generative AI
I did not use generative AI tools when creating this PR.
I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.