Skip to content

An iterative plan to produce multi-Python packages

Python “wheels” are currently the standard (and the easiest) way to distribute Python packages, esp. via Python's standard platform, PyPI. Python standard library has also dedicated tools to produce such ‘wheels’ (eg., setuptools) and the ‘wheels’ can be installed easily via pip. For further details, see Python Packaging User Guide and PEP 427.

The conceptual build process for BA is the following:

                  +--------------+
                  |Core libraries|
                  +-------+------+
                          |
                    +-----v----+
                    |Core tests|
                    +-----+----+
                          |
      +-------------------+-----------------+
      |                                     |
  +---v---+                          +------v-------+
  |  GUI  |                          | Python wheel |
  +---+---+                          +------+-------+
      |                                     |
 +----v----+                       +--------v---------+
 |GUI tests|                       |Python wheel tests|
 +----+----+                       +------------------+
      |
+-----v-----+
|GUI package|
+-----------+

The diagram indicates that there is a possibility for a multi-stage parallel build process in GitLab CI; see eg., https://about.gitlab.com/blog/2020/12/10/basics-of-gitlab-ci-updated.

Currently this possibility is not leveraged due to the structure of the code base and the CMake scripts.

An iterative plan

In the following, an iterative plan is proposed to restore the mechanism for building Python-wheel as soon as possible.

Step 1) Add the CMake modules to the current scripts to build Python packages for a single given Python version; that is, the GUI package will be built as before (with the SWIG API), and thereafter, a Python wheel is produced for the given version. The tests are done only for the GUI version. The complete build process must be repeated for as many Python versions as needed.

For this step, we can re-use a modified version of CMake modules in cmake/multipython.

The CMake commands will be like the following:

  • Build a Python package for the default Python version on the system
cmake .. -DBA_PY_PACKAGE=ON
  • Build a Python package with a given Python platform (which has its own CMake config files)
cmake .. -DBA_PY_PACKAGE=ON -DBA_PY_PLATFORM="/opt/Python3.7"

Step 2) Separate the Python-related parts of the codebase into a new module, say PyTools. The rest of the core will be strictly Python-free. This seclusion of the Python-related code has many advantages, besides the better robustness and modularity; see Step 3). In this step we gain also the Fabio-related functionalities.

Note that the code for this step is also implemented before.

The Python-dependent parts of BA codebase are currently the following:

Base/Py/PyCore
Base/Py/PyUtils
Sample/Multilayer/PyImport
Device/Data/OutputData::getArray
Device/Histo/Histogram1D::binCentersNumpy 
Device/Histo/Histogram1D::binValuesNumpy 
Device/Histo/Histogram1D::binErrorsNumpy

Step 3) Modify the CMake scripts to perform the following:
3a) Build core libraries once, without the SWIG API.
3b) Build the PyTools library for the default version of Python (this will be fast since the Python-dependent code is small).
3c) Build the GUI and link with the core and the PyTools libraries.
3d) The core libraries can be reused as many times as required to build the API for any version of Python via SWIG, since merely the SWIG wrapper must be compiled and linked with the corresponding library. For each version of Python, PyTools must be also rebuilt, however that's very fast.

Note that the GUI library can also be linked to any Python-version of PyTools since this library remains binary-compatible across different Python versions. That means, via this procedure, we also obtain the GUI package for any given version of Python for free.

As shown in the conceptual diagram above, this process can be effectively parallelized.

The CMake commands will be like the following:

cmake ..  # configuration phase
cmake --build .. --target Core  # => 3a,3b

The GUI package can be built using the core libraries stored e.g. at ../lib/core:

cmake --build .. --target GUI -DBA_CORE_LIB="../lib/core"  # => 3c

The Python wheels for any version can be built using the core libraries stored e.g. at ../lib/core:

cmake --build .. -DBA_PY_PACKAGE=ON -DBA_CORE_LIB="../lib/core"  # => 3d

or

cmake --build .. -DBA_PY_PACKAGE=ON -DBA_PY_PLATFORM="/opt/Python3.7" -DBA_CORE_LIB="../lib/core"  # => 3d
Edited by Ammar Nejati