Add c_module() manifest function for user C modules#18229
Conversation
|
Code size report: |
|
This is an aggressively sensible change, and as much as I'm bought into the spaghetti of CMake tooling I've built to handle C modules... doing it this way makes so, so much more sense. 👍 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #18229 +/- ##
==========================================
- Coverage 98.51% 98.51% -0.01%
==========================================
Files 177 180 +3
Lines 22927 22982 +55
Branches 0 5 +5
==========================================
+ Hits 22586 22640 +54
Misses 341 341
- Partials 0 1 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
adb0ccf to
d90efde
Compare
78942fc to
12b4538
Compare
12b4538 to
5efe8e7
Compare
acf76fe to
fdffeeb
Compare
|
Note on subpackage example fix: Found what appears to be a bug in This went unnoticed because:
Fixed in commit fdffeeb2f6. Could be a copy-paste error from when the example was added in 2023, though haven't confirmed this with the original author or other maintainers yet. |
fdffeeb to
8cc793b
Compare
8cc793b to
043688d
Compare
043688d to
8cc793b
Compare
|
Did a couple of small cleanups on top of this: The new manifest processing logic added to The Quick sanity check on all four affected ports using the |
I don't see either of these changes in the latest PR here. Maybe it wasn't pushed properly? |
0f1df30 to
340d7e8
Compare
68f122a to
01136ba
Compare
|
Just a note that CI is failing here. |
Add c_module() manifest function for user C modules MBM-PR: 18229 MBM-URL: micropython#18229 Signed-off-by: Andrew Leech <[email protected]>
|
@andrewleech Feel free to request a review when you get a chance to look at this again. |
2ee27a9 to
61f4ad3
Compare
Add c_module() manifest function for user C modules MBM-PR: 18229 MBM-URL: micropython#18229 Signed-off-by: Andrew Leech <[email protected]>
61f4ad3 to
6b43ce8
Compare
Add a new c_module() function to the manifest API that lets a manifest declare which user C module directories should be included in the build. This pairs with the build system changes that pick up the resulting paths and feed them into USER_C_MODULES. The function validates the path exists, is a directory, and contains a micropython.mk or micropython.cmake, and supports the usual \$(MPY_DIR), \$(BOARD_DIR), \$(PORT_DIR), \$(MPY_LIB_DIR) substitutions. Unresolved \$(VAR) tokens raise a clear error rather than failing later. makemanifest.py gains --list-c-modules which prints one path per line so the build system can pick them up (newline output handles paths that contain whitespace on the cmake side). Signed-off-by: Andrew Leech <[email protected]>
When MICROPY_FROZEN_MANIFEST is set, the build system runs makemanifest.py with --list-c-modules to extract C module paths from the manifest and appends them to USER_C_MODULES so they go through the same code path as user modules specified on the command line. The Make side lives in py/manifest.mk (included from py/py.mk) and the CMake side lives in py/manifest.cmake (included from py/usermod.cmake). Both share defaulting for the MICROPY_MANIFEST_* substitution variables so a manifest can reference $(MPY_DIR) etc. consistently on either side. The make-side extraction is gated on the build target needing it and on micropython-lib being initialised, and surfaces makemanifest.py errors via .SHELLSTATUS. The cmake side uses FATAL_ERROR to match. Signed-off-by: Andrew Leech <[email protected]>
Move the MICROPY_USER_FROZEN_MANIFEST resolution to run right after the board config (rather than near the end of CMakeLists.txt) so it's applied before py/usermod.cmake is included; py/manifest.cmake (invoked from usermod.cmake) needs the final MICROPY_FROZEN_MANIFEST value to extract c_module() paths. Also quote FROZEN_MANIFEST when forwarding it from the Makefile to cmake so paths containing whitespace round-trip correctly. Signed-off-by: Andrew Leech <[email protected]>
Quote FROZEN_MANIFEST when passing it through IDFPY_FLAGS so paths containing whitespace round-trip correctly to the cmake configure step; py/manifest.cmake (auto-included via py/usermod.cmake) needs the resolved value to extract c_module() paths from the manifest. Signed-off-by: Andrew Leech <[email protected]>
The micropython.cmake referenced examplemodule.c but the source file is named modexamplepackage.c. The subpackage example wasn't built via cmake before, so the typo went unnoticed. Signed-off-by: Andrew Leech <[email protected]>
Document the new c_module() function in the manifest reference, including the FROZEN_MANIFEST prerequisite, additive interaction with USER_C_MODULES, and the make-side whitespace limitation. Add a corresponding section to the cmodules developer guide so the feature is discoverable from both entry points. Signed-off-by: Andrew Leech <[email protected]>
Add a manifest_test.py per port (rp2, stm32, esp32) that pulls in the example user C modules via c_module(), and wire up ci.sh and the stm32 workflow so each port exercises both the c_module() manifest entry and the legacy USER_C_MODULES command-line variable. Signed-off-by: Andrew Leech <[email protected]>
The unix coverage variant set USER_C_MODULES from mpconfigvariant.mk to build the example user C modules into the test binary. Move that setting into the coverage manifest as a c_module() call to exercise the new manifest API end-to-end as part of the standard coverage test build. Signed-off-by: Andrew Leech <[email protected]>
6b43ce8 to
45cf384
Compare
Summary
This adds a
c_module()function to the manifest system so projects can specify user C module directories directly in manifest.py files.The
USER_C_MODULESapproach requires all C modules to exist within a single parent directory that gets scanned for modules. The newc_module()function provides two key benefits: it consolidates both Python and C modules in a single manifest location, and it allows each C module to be located anywhere in the codebase rather than requiring them all in one flat parent directory.Example usage:
The implementation adds early manifest processing during build configuration in both Make and CMake build systems. C module paths are extracted via
makemanifest.py --list-c-modulesand appended to the existingUSER_C_MODULESlist, maintaining full backward compatibility. Multiplec_module()calls are supported, and the same$(VAR)path substitution works as with other manifest functions. Invalid paths fail the build immediately with clear error messages.I consolidated all manifest processing into a new
py/manifest.mkfile that handles variables, C module extraction, and frozen content generation. This eliminated duplication between py.mk and mkrules.mk.Testing
Tested on unix (coverage variant), esp32, rp2, and stm32 ports with the example C modules. All ports build successfully and modules are available at runtime.
Binary equivalence testing confirmed that builds using
c_module()produce identical binaries to those usingUSER_C_MODULESon the command line. Verified on rp2 (RPI_PICO_W), stm32 (PYBV11), and unix (coverage) - all sections match byte-for-byte on embedded targets, with unix showing only expected gcov metadata differences.During rp2 testing I found and fixed a bug where
FROZEN_MANIFESToverrides weren't restoring the user manifest before usermod.cmake inclusion, causingc_module()calls to be silently ignored. The fix ensures all three methods (c_module() in board manifest, FROZEN_MANIFEST override, USER_C_MODULES) work correctly.CI now tests
c_module()on unix, esp32, rp2, and stm32 ports. The existing stm32 pyboard CI job still usesUSER_C_MODULESto verify backward compatibility.Trade-offs and Alternatives
No runtime impact - this is purely build-time functionality. Build time overhead is minimal since manifest parsing was already happening for frozen modules.
One intentional behavioral difference:
USER_C_MODULESscans a directory and includes all modules with micropython.mk/micropython.cmake files, whilec_module()requires explicit listing of each module directory. This gives precise control over which modules are included rather than pulling in everything from a directory scan.