Skip to content

Add c_module() manifest function for user C modules#18229

Open
andrewleech wants to merge 8 commits into
micropython:masterfrom
andrewleech:manifest_c_module
Open

Add c_module() manifest function for user C modules#18229
andrewleech wants to merge 8 commits into
micropython:masterfrom
andrewleech:manifest_c_module

Conversation

@andrewleech

@andrewleech andrewleech commented Oct 9, 2025

Copy link
Copy Markdown
Contributor

Summary

This adds a c_module() function to the manifest system so projects can specify user C module directories directly in manifest.py files.

The USER_C_MODULES approach requires all C modules to exist within a single parent directory that gets scanned for modules. The new c_module() function provides two key benefits: it consolidates both Python and C modules in a single manifest location, and it allows each C module to be located anywhere in the codebase rather than requiring them all in one flat parent directory.

Example usage:

c_module("$(MPY_DIR)/examples/usercmodule/cexample")
c_module("$(BOARD_DIR)/drivers/sensor")

The implementation adds early manifest processing during build configuration in both Make and CMake build systems. C module paths are extracted via makemanifest.py --list-c-modules and appended to the existing USER_C_MODULES list, maintaining full backward compatibility. Multiple c_module() calls are supported, and the same $(VAR) path substitution works as with other manifest functions. Invalid paths fail the build immediately with clear error messages.

I consolidated all manifest processing into a new py/manifest.mk file that handles variables, C module extraction, and frozen content generation. This eliminated duplication between py.mk and mkrules.mk.

Testing

Tested on unix (coverage variant), esp32, rp2, and stm32 ports with the example C modules. All ports build successfully and modules are available at runtime.

Binary equivalence testing confirmed that builds using c_module() produce identical binaries to those using USER_C_MODULES on the command line. Verified on rp2 (RPI_PICO_W), stm32 (PYBV11), and unix (coverage) - all sections match byte-for-byte on embedded targets, with unix showing only expected gcov metadata differences.

During rp2 testing I found and fixed a bug where FROZEN_MANIFEST overrides weren't restoring the user manifest before usermod.cmake inclusion, causing c_module() calls to be silently ignored. The fix ensures all three methods (c_module() in board manifest, FROZEN_MANIFEST override, USER_C_MODULES) work correctly.

CI now tests c_module() on unix, esp32, rp2, and stm32 ports. The existing stm32 pyboard CI job still uses USER_C_MODULES to verify backward compatibility.

Trade-offs and Alternatives

No runtime impact - this is purely build-time functionality. Build time overhead is minimal since manifest parsing was already happening for frozen modules.

One intentional behavioral difference: USER_C_MODULES scans a directory and includes all modules with micropython.mk/micropython.cmake files, while c_module() requires explicit listing of each module directory. This gives precise control over which modules are included rather than pulling in everything from a directory scan.

@github-actions

github-actions Bot commented Oct 9, 2025

Copy link
Copy Markdown

Code size report:

Reference:  examples/natmod/re: Fix building with Clang toolchains. [1454c3e]
Comparison: unix: Migrate coverage variant to use c_module() in manifest. [merge of 45cf384]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@Gadgetoid

Copy link
Copy Markdown
Contributor

This is an aggressively sensible change, and as much as I'm bought into the spaghetti of CMake tooling I've built to handle C modules... doing it this way makes so, so much more sense. 👍

@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Oct 11, 2025
@codecov

codecov Bot commented Oct 11, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.51%. Comparing base (1454c3e) to head (45cf384).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #18229      +/-   ##
==========================================
- Coverage   98.51%   98.51%   -0.01%     
==========================================
  Files         177      180       +3     
  Lines       22927    22982      +55     
  Branches        0        5       +5     
==========================================
+ Hits        22586    22640      +54     
  Misses        341      341              
- Partials        0        1       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andrewleech andrewleech force-pushed the manifest_c_module branch 2 times, most recently from 78942fc to 12b4538 Compare December 10, 2025 05:10
@andrewleech andrewleech changed the title py,tools: Add c_module() manifest function for user C modules Add c_module() manifest function for user C modules Dec 10, 2025
@projectgus projectgus self-requested a review January 28, 2026 23:49
@andrewleech andrewleech force-pushed the manifest_c_module branch 6 times, most recently from acf76fe to fdffeeb Compare February 2, 2026 23:57
@andrewleech

Copy link
Copy Markdown
Contributor Author

Note on subpackage example fix: Found what appears to be a bug in examples/usercmodule/subpackage/micropython.cmake - it references examplemodule.c but the actual file is modexamplepackage.c. The Make version (micropython.mk) has the correct filename.

This went unnoticed because:

  1. The subpackage cmake file isn't included in the parent examples/usercmodule/micropython.cmake
  2. Existing CI builds either use Make (correct filename in .mk) or use the parent cmake (which excludes subpackage)
  3. Direct inclusion via c_module("path/to/subpackage") is the first time this cmake file actually gets processed

Fixed in commit fdffeeb2f6. Could be a copy-paste error from when the example was added in 2023, though haven't confirmed this with the original author or other maintainers yet.

@andrewleech

Copy link
Copy Markdown
Contributor Author

Did a couple of small cleanups on top of this:

The new manifest processing logic added to py/usermod.cmake has been extracted out into its own py/manifest.cmake, parallel to py/manifest.mk. usermod.cmake just includes it now, so the file that ports reference stays unchanged. It was a bit odd having manifest-processing logic buried inside usermod.cmake with no obvious way to find it.

The py/mkrules.mk include $(TOP)/py/manifest.mk was also redundant - all ports include py.mk before mkrules.mk, and manifest.mk has an include guard (MANIFEST_MK_INCLUDED), so that second include was always a no-op. Removed it.

Quick sanity check on all four affected ports using the manifest_test.py files with c_module() entries - unix/coverage, PYBV11 (stm32), RPI_PICO_W (rp2), and ESP32_GENERIC (espressif idf container) - verified via nm on the ELF / cmake "Found User C Module(s)" output confirming all three modules linked in. On unix the modules also work at runtime: cexample.add_ints(1, 2) returns 3, cppexample.cppfunc(3, 4) returns (7, 'hellocpp'), and example_package is importable.

@dpgeorge

Copy link
Copy Markdown
Member

The new manifest processing logic added to py/usermod.cmake has been extracted out into its own py/manifest.cmake,

The py/mkrules.mk include $(TOP)/py/manifest.mk was also redundant - all ports include py.mk before mkrules.mk, and manifest.mk has an include guard (MANIFEST_MK_INCLUDED), so that second include was always a no-op. Removed it.

I don't see either of these changes in the latest PR here. Maybe it wasn't pushed properly?

@andrewleech andrewleech force-pushed the manifest_c_module branch 2 times, most recently from 0f1df30 to 340d7e8 Compare May 23, 2026 01:09
@andrewleech andrewleech force-pushed the manifest_c_module branch 3 times, most recently from 68f122a to 01136ba Compare May 24, 2026 19:30
@dpgeorge

dpgeorge commented Jun 5, 2026

Copy link
Copy Markdown
Member

Just a note that CI is failing here.

andrewleech added a commit to andrewleech/micropython that referenced this pull request Jun 22, 2026
Add c_module() manifest function for user C modules

MBM-PR: 18229
MBM-URL: micropython#18229
Signed-off-by: Andrew Leech <[email protected]>
@projectgus projectgus removed their request for review July 1, 2026 00:27
@projectgus

Copy link
Copy Markdown
Contributor

@andrewleech Feel free to request a review when you get a chance to look at this again.

@andrewleech andrewleech force-pushed the manifest_c_module branch 3 times, most recently from 2ee27a9 to 61f4ad3 Compare July 1, 2026 08:39
andrewleech added a commit to andrewleech/micropython that referenced this pull request Jul 1, 2026
Add c_module() manifest function for user C modules

MBM-PR: 18229
MBM-URL: micropython#18229
Signed-off-by: Andrew Leech <[email protected]>
Add a new c_module() function to the manifest API that lets a manifest
declare which user C module directories should be included in the build.
This pairs with the build system changes that pick up the resulting paths
and feed them into USER_C_MODULES.

The function validates the path exists, is a directory, and contains a
micropython.mk or micropython.cmake, and supports the usual \$(MPY_DIR),
\$(BOARD_DIR), \$(PORT_DIR), \$(MPY_LIB_DIR) substitutions. Unresolved
\$(VAR) tokens raise a clear error rather than failing later.

makemanifest.py gains --list-c-modules which prints one path per line so
the build system can pick them up (newline output handles paths that
contain whitespace on the cmake side).

Signed-off-by: Andrew Leech <[email protected]>
When MICROPY_FROZEN_MANIFEST is set, the build system runs makemanifest.py
with --list-c-modules to extract C module paths from the manifest and
appends them to USER_C_MODULES so they go through the same code path as
user modules specified on the command line.

The Make side lives in py/manifest.mk (included from py/py.mk) and the
CMake side lives in py/manifest.cmake (included from py/usermod.cmake).
Both share defaulting for the MICROPY_MANIFEST_* substitution variables
so a manifest can reference $(MPY_DIR) etc. consistently on either side.
The make-side extraction is gated on the build target needing it and on
micropython-lib being initialised, and surfaces makemanifest.py errors
via .SHELLSTATUS. The cmake side uses FATAL_ERROR to match.

Signed-off-by: Andrew Leech <[email protected]>
Move the MICROPY_USER_FROZEN_MANIFEST resolution to run right after the
board config (rather than near the end of CMakeLists.txt) so it's applied
before py/usermod.cmake is included; py/manifest.cmake (invoked from
usermod.cmake) needs the final MICROPY_FROZEN_MANIFEST value to extract
c_module() paths. Also quote FROZEN_MANIFEST when forwarding it from the
Makefile to cmake so paths containing whitespace round-trip correctly.

Signed-off-by: Andrew Leech <[email protected]>
Quote FROZEN_MANIFEST when passing it through IDFPY_FLAGS so paths
containing whitespace round-trip correctly to the cmake configure step;
py/manifest.cmake (auto-included via py/usermod.cmake) needs the
resolved value to extract c_module() paths from the manifest.

Signed-off-by: Andrew Leech <[email protected]>
The micropython.cmake referenced examplemodule.c but the source file is
named modexamplepackage.c. The subpackage example wasn't built via cmake
before, so the typo went unnoticed.

Signed-off-by: Andrew Leech <[email protected]>
Document the new c_module() function in the manifest reference, including
the FROZEN_MANIFEST prerequisite, additive interaction with USER_C_MODULES,
and the make-side whitespace limitation. Add a corresponding section to
the cmodules developer guide so the feature is discoverable from both
entry points.

Signed-off-by: Andrew Leech <[email protected]>
Add a manifest_test.py per port (rp2, stm32, esp32) that pulls in the
example user C modules via c_module(), and wire up ci.sh and the stm32
workflow so each port exercises both the c_module() manifest entry and
the legacy USER_C_MODULES command-line variable.

Signed-off-by: Andrew Leech <[email protected]>
The unix coverage variant set USER_C_MODULES from mpconfigvariant.mk to
build the example user C modules into the test binary. Move that setting
into the coverage manifest as a c_module() call to exercise the new
manifest API end-to-end as part of the standard coverage test build.

Signed-off-by: Andrew Leech <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

py-core Relates to py/ directory in source

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants