pyoxidizer: rewrite build logic to use PyO3

This commit finishes the port of PyOxidizer to PyO3. This entails a lot of changes. The gist of it is we write out a PyO3 configuration file to configure that crate. Then we retool pyembed and various Rust project building code to use that config file and PyO3's configuration mechanisms. There are a lot of changes in here.
indygreg · Sep 6, 2021 · dfd7b66 · dfd7b66
1 parent 48c97a6
commit dfd7b66
Show file tree

Hide file tree

Showing 24 changed files with 286 additions and 286 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/oxidized-importer/Cargo.toml b/oxidized-importer/Cargo.toml
@@ -14,7 +14,6 @@ crate-type = ["cdylib"]
 [dependencies]
 anyhow = "1.0"
 lazy_static = "1.4"
-python3-sys = "0.6.0"
 
 [dependencies.python-packed-resources]
 version = "0.8.0-pre"

diff --git a/oxidized-importer/src/lib.rs b/oxidized-importer/src/lib.rs
@@ -7,7 +7,7 @@
 // due to unbound symbols. So we suppress clippy as a workaround.
 
 #[cfg(not(feature = "cargo-clippy"))]
-use python3_sys as pyffi;
+use pyo3::ffi as pyffi;
 
 #[cfg(not(feature = "cargo-clippy"))]
 #[no_mangle]

diff --git a/pyembed/Cargo.toml b/pyembed/Cargo.toml
@@ -26,12 +26,10 @@ links = "pyembed"
 [dependencies]
 # Update documentation in lib.rs when new dependencies are added.
 anyhow = "1.0"
-cpython = "0.6.0"
 dunce = "1.0"
 jemalloc-sys = { version = "0.3", optional = true }
 memmap = "0.7"
 once_cell = "1.7"
-python3-sys = "0.6.0"
 
 [dependencies.snmalloc-sys]
 version = "0.2"
@@ -86,14 +84,10 @@ allocator-snmalloc = ["snmalloc-sys"]
 #
 # This crate links against whatever Python is picked up by the cpython crate
 # using its default link semantics.
-build-mode-default = [
-    "cpython-link-default",
-]
+build-mode-default = []
 
 # Build this crate in isolation, without using PyOxidizer.
-build-mode-standalone = [
-    "cpython-link-unresolved-static",
-]
+build-mode-standalone = []
 
 # Build this crate by executing a `pyoxidizer` executable to build
 # required artifacts.
@@ -106,26 +100,8 @@ build-mode-pyoxidizer-exe = []
 build-mode-prebuilt-artifacts = []
 
 build-mode-extension-module = [
-    "cpython/extension-module",
+    "pyo3/extension-module",
 ]
 
 # Build mode for running tests.
-build-mode-test = [
-    "cpython-link-default",
-]
-
-# Build the cpython crate in unresolved static mode (we will provide a static
-# library to link against).
-cpython-link-unresolved-static = [
-    "cpython/py-link-mode-unresolved-static",
-    "cpython/python3-sys",
-    "cpython/no-auto-initialize",
-]
-
-# Build the cpython with default link mode control. It will pick up the prebuilt
-# libpython from PYTHON_SYS_EXECUTABLE or from PATH.
-cpython-link-default = [
-    "cpython/py-link-mode-default",
-    "cpython/python3-sys",
-    "cpython/no-auto-initialize",
-]
+build-mode-test = []
diff --git a/pyembed/docs/oxidized_importer_getting_started.rst b/pyembed/docs/oxidized_importer_getting_started.rst
@@ -49,7 +49,7 @@ If you don't have a Python 3.9 ``python3`` executable in your ``PATH``, you
 will need to tell the Rust build system which ``python3`` executable to use to
 help derive the build configuration for the Python extension::
 
-   $ PYTHON_SYS_EXECUTABLE=/path/to/python3.9 cargo build
+   $ PYO3_PYTHON=/path/to/python3.9 cargo build
 
 Using
 =====

diff --git a/pyembed/docs/pyembed_controlling_python.rst b/pyembed/docs/pyembed_controlling_python.rst
@@ -36,7 +36,7 @@ use it:
 
 Since CPython's API relies on static variables (sadly), if you really wanted
 to, you could call out to CPython C APIs directly (probably via the
-bindings in the ``python3-sys`` crate) and they would interact with the
+bindings in the ``pyo3`` crate) and they would interact with the
 interpreter started by the ``pyembed`` crate. This is all ``unsafe``, of course,
 so tread at your own peril.
 

diff --git a/pyembed/docs/pyembed_crate_configuration.rst b/pyembed/docs/pyembed_crate_configuration.rst
@@ -20,9 +20,8 @@ The features are described in the following sections.
 This is the default build mode. It is enabled by default.
 
 This build mode uses default Python linking behavior and feature detection
-as implemented by the ``cpython`` and ``python3-sys`` crates. It will attempt
-to find a ``python`` in ``PATH`` or from the ``PYTHON_SYS_EXECUTABLE``
-environment variable and dynamically link against it.
+as implemented by the ``pyo3``. It will attempt to find a ``python`` in
+``PATH`` or from the ``PYO3_PYTHON`` environment variable and link against it.
 
 This is the default mode for convenience, as it enables the ``pyembed`` crate
 to build in the most environments. However, the built binaries will have a
@@ -69,40 +68,14 @@ artifacts.
 
 Do not attempt to invoke ``pyoxidizer`` or find artifacts it would have
 built. It is possible to build the ``pyembed`` crate in this mode if
-the ``rust-cpython`` and ``python3-sys`` crates can find a Python
-interpreter. But, the ``pyembed`` crate may not be usable or work in
-the way you want it to.
+the ``pyo3`` crate can find a Python interpreter. But, the ``pyembed``
+crate may not be usable or work in the way you want it to.
 
 This mode is intended to be used for performing quick testing on the
 ``pyembed`` crate. It is quite possible that linking errors will occur
 in this mode unless you take additional actions to point Cargo at
 appropriate libraries.
 
-``cpython-link-unresolved-static``
-----------------------------------
-
-Configures the link mode of the ``cpython`` crate to use a static
-``pythonXY`` library without resolving the symbol at its own build
-time. The ``pyembed`` crate or a crate building it will need to emit
-``cargo:rustc-link-lib=static=pythonXY`` and any
-``cargo:rustc-link-search=native={}`` lines to specify an explicit
-``pythonXY`` library to link against.
-
-This is the link mode used to produce self-contained binaries containing
-``libpython`` and ``pyembed`` code.
-
-``cpython-link-default``
-------------------------
-
-Configures the link mode of the ``cpython`` crate to use default
-semantics. The crate's build script will find a pre-built Python
-library by querying the ``python`` defined by ``PYTHON_SYS_EXECUTABLE``
-or found on ``PATH``. See the ``cpython`` crate's documentation for
-more.
-
-This link mode should be used when linking against an existing ``libpython``
-that can be found by the ``cpython`` crate's build script.
-
 .. _pyembed_build_artifacts:
 
 Build Artifacts

diff --git a/pyembed/src/conversion.rs b/pyembed/src/conversion.rs
@@ -101,6 +101,9 @@ pub fn osstring_to_bytes<'p>(py: Python<'p>, s: OsString) -> &'p PyAny {
     }
 }
 
+/// Convert a Rust path to a Python object.
+///
+/// TODO remove and use PyO3 conversion traits instead.
 pub fn path_to_pyobject<'p>(py: Python<'p>, path: &Path) -> PyResult<&'p PyAny> {
     let encoding_ptr = unsafe { pyffi::Py_FileSystemDefaultEncoding };
 

diff --git a/pyembed/src/lib.rs b/pyembed/src/lib.rs
@@ -32,9 +32,8 @@ serves as a high-level interface for running code in the interpreter.
 
 # Dependencies
 
-Under the hood, `pyembed` makes direct use of the `python3-sys` crate for
-low-level Python FFI bindings as well as the `cpython`/`pyo3` crate for higher-level
-interfacing.
+Under the hood, `pyembed` makes direct use of the `pyo3` crate for
+low-level Python FFI bindings as well as higher-level interfacing.
 
 **It is an explicit goal of this crate to rely on as few external dependencies
 as possible.** This is because we want to minimize bloat in produced binaries.
@@ -75,18 +74,6 @@ to build required artifacts.
 `PyOxidizer` out-of-band. In this mode, the `PYOXIDIZER_ARTIFACT_DIR`
 environment variable can refer to the directory containing build artifacts
 that this crate needs. If not set, `OUT_DIR` will be used.
-
-The exist mutually exclusive `cpython-link-*` features to control how
-the `cpython`/`python3-sys` crates are built.
-
-`cpython-link-unresolved-static` instructs to leave the Python symbols
-as unresolved. This crate will provide a static library providing the
-symbols.
-
-`cpython-link-default` builds `cpython` with default link mode control.
-That crate's build script will attempt to find a `libpython` from the
-`python` defined by `PYTHON_SYS_EXECUTABLE` or present on `PATH`.
-
 */
 
 #[allow(unused)]

diff --git a/pyoxidizer/Cargo.toml b/pyoxidizer/Cargo.toml
@@ -61,6 +61,10 @@ walkdir = "2"
 which = "4"
 zstd = "0.9"
 
+[dependencies.pyo3-build-config]
+version = "0.14.5"
+default-features = false
+
 [dependencies.python-packaging]
 version = "0.11.0-pre"
 path = "../python-packaging"

diff --git a/pyoxidizer/docs/pyoxidizer_faq.rst b/pyoxidizer/docs/pyoxidizer_faq.rst
@@ -65,17 +65,17 @@ this wasn't perceived as a significant annoyance.
 ==================================================================
 
 This is due to a dependent crate insisting that a Python executable
-exist on ``PATH``. Set the ``PYTHON_SYS_EXECUTABLE`` environment
-variable to the path of a Python 3.7 executable and try again. e.g.::
+exist on ``PATH``. Set the ``PYO3_PYTHON`` environment variable to
+the path of a Python 3.8+ executable and try again. e.g.::
 
    # UNIX
-   $ export PYTHON_SYS_EXECUTABLE=/usr/bin/python3.7
+   $ export PYO3_PYTHON=/usr/bin/python3.9
    # Windows
-   $ SET PYTHON_SYS_EXECUTABLE=c:\python37\python.exe
+   $ SET PYO3_PYTHON=c:\python39\python.exe
 
 .. note::
 
-   The ``pyoxidizer`` tool should take care of setting ``PYTHON_SYS_EXECUTABLE``
+   The ``pyoxidizer`` tool should take care of setting ``PYO3_PYTHON``
    and prevent this error. If you see this error and you are building with
    ``pyoxidizer``, it is a bug that should be reported.
 

diff --git a/pyoxidizer/docs/pyoxidizer_history.rst b/pyoxidizer/docs/pyoxidizer_history.rst
@@ -39,6 +39,24 @@ Bug Fixes
   has been replaced to automatically assume the use of ``MACOSX_DEPLOYMENT_TARGET``.
   A warning message that this will possibly lead to build failures is printed. (#414)
 
+Backwards Compatibility Notes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* The run-time Rust code for interfacing with the Python interpreter now uses
+  the [PyO3](https://github.com/PyO3/pyo3) crate instead of
+  [cpython](https://github.com/dgrunwald/rust-cpython). The code port was quite
+  extensive and while we believe all important run-time functionality is
+  backwards compatible, there are possibly subtle differences in behavior. Please
+  file GitHub issues to report any undesired changes in behavior.
+* Development workflows relying on specifying the ``PYTHON_SYS_EXECUTABLE``
+  environment variable have changed to use ``PYO3_PYTHON``, as the environment
+  variable has changed between the ``cpython`` and ``pyo3`` crates.
+* The ``pyembed`` crate no longer has ``cpython-link-unresolved-static`` and
+  ``cpython-link-default`` Cargo features. Autogenerated Rust projects also no
+  longer have ``cpython-link-unresolved-static`` and ``cpython-link-default``
+  features (which existed as proxies to these features in the ``pyembed``
+  crate).
+
 Other Relevant Changes
 ^^^^^^^^^^^^^^^^^^^^^^
 
@@ -47,6 +65,18 @@ Other Relevant Changes
   target. Previously, we would allow the use of the SDK, only to likely encounter
   a hard-to-debug compile error. If the SDKs version does not meet the desired
   minimum version, a warning message is printed but the build proceeds. (#431)
+* The ``pyembed`` crate (which built binaries use to interface with an embedded
+  Python interpreter) now uses the ``pyo3`` crate instead of ``cpython`` to
+  interface with Python APIs.
+* Nightly Cargo features are no longer required on Windows. (Courtesy of PyO3
+  giving us complete control over how Python is linked.)
+* The mechanism by which built binaries link against ``libpython`` has been
+  significantly refactored. Before, the ``cpython`` crate would link against
+  a partial ``libpython`` in many configurations and the ``pyembed`` crate
+  would *complete* the linking with a library defined by PyOxidizer. With the
+  PyO3 crate supporting a configuration file to configure all attributes of
+  linking, the PyO3 crate now fully links against ``libpython`` and ``pyembed``
+  doesn't care about linking ``libpython`` at all.
 
 .. _version_0_17_0:
 

diff --git a/pyoxidizer/docs/pyoxidizer_rust_cargo_source_checkouts.rst b/pyoxidizer/docs/pyoxidizer_rust_cargo_source_checkouts.rst
@@ -69,10 +69,10 @@ In its default configuration, a Python 3.9 executable needs to be found on
 ``No python interpreter found of version 3.*`` error at build time.
 
 To work around this, add a ``python3.9`` or ``python3`` executable to
-``PATH`` or run ``cargo build`` with the ``PYTHON_SYS_EXECUTABLE`` environment
+``PATH`` or run ``cargo build`` with the ``PYO3_PYTHON`` environment
 variable pointing to a specific Python 3 executable. e.g.
 
-    $ PYTHON_SYS_EXECUTABLE=/path/to/python3.9 cargo build -p pyembed
+    $ PYO3_PYTHON=/path/to/python3.9 cargo build -p pyembed
 
 ``oxidized-importer`` Crate
 ===========================
@@ -83,4 +83,3 @@ needed for :ref:`oxidized_importer`.
 
 Because this crate is a thin shim, the caveats that apply to building
 ``pyembed`` apply to it as well.
-