Python “wheels” are currently the standard (and the easiest) way to distribute Python packages, esp. via Python's standard platform, PyPI. Python standard library has also dedicated tools to produce such ‘wheels’ (eg., `setuptools`) and the ‘wheels’ can be installed easily via `pip`. For further details, see [Python Packaging User Guide](https://packaging.python.org) and [PEP 427](https://www.python.org/dev/peps/pep-0427). The conceptual build process for BA is the following: ``` +--------------+ |Core libraries| +-------+------+ | +-----v----+ |Core tests| +-----+----+ | +-------------------+-----------------+ | | +---v---+ +------v-------+ | GUI | | Python wheel | +---+---+ +------+-------+ | | +----v----+ +--------v---------+ |GUI tests| |Python wheel tests| +----+----+ +------------------+ | +-----v-----+ |GUI package| +-----------+ ``` The diagram indicates that there is a possibility for a multi-stage parallel build process in GitLab CI; see eg., <https://about.gitlab.com/blog/2020/12/10/basics-of-gitlab-ci-updated>. Currently this possibility is not leveraged due to the structure of the code base and the CMake scripts. ## An iterative plan In the following, an iterative plan is proposed to restore the mechanism for building Python-wheel as soon as possible. Step 1) Add the CMake modules to the current scripts to build Python packages for a _single_ given Python version; that is, the GUI package will be built as before (with the SWIG API), and thereafter, a Python wheel is produced for the given version. The tests are done only for the GUI version. The _complete_ build process must be repeated for as many Python versions as needed. For this step, we can re-use a modified version of CMake modules in `cmake/multipython`. The CMake commands will be like the following: - Build a Python package for the default Python version on the system ``` cmake .. -DBA_PY_PACKAGE=ON ``` - Build a Python package with a given Python platform (which has its own CMake config files) ``` cmake .. -DBA_PY_PACKAGE=ON -DBA_PY_PLATFORM="/opt/Python3.7" ``` Step 2) Separate the Python-related parts of the codebase into a new module, say `PyTools`. The rest of the _core_ will be strictly Python-free. This seclusion of the Python-related code has many advantages, besides the better robustness and modularity; see Step 3). In this step we gain also the Fabio-related functionalities. Note that the code for this step is also implemented before. The Python-dependent parts of BA codebase are currently the following: ``` Base/Py/PyCore Base/Py/PyUtils Sample/Multilayer/PyImport Device/Data/OutputData::getArray Device/Histo/Histogram1D::binCentersNumpy Device/Histo/Histogram1D::binValuesNumpy Device/Histo/Histogram1D::binErrorsNumpy ``` Step 3) Modify the CMake scripts to perform the following: 3a) Build core libraries _once_, _without_ the SWIG API. 3b) Build the `PyTools` library for the _default_ version of Python (this will be fast since the Python-dependent code is small). 3c) Build the GUI and link with the core and the `PyTools` libraries. 3d) The core libraries can be _reused_ as many times as required to build the API for any version of Python via SWIG, since merely the SWIG wrapper must be compiled and linked with the corresponding library. For each version of Python, `PyTools` must be also rebuilt, however that's very fast. Note that the GUI library can also be linked to any Python-version of `PyTools` since this library remains _binary-compatible_ across different Python versions. That means, via this procedure, we also obtain the GUI package for any given version of Python for free. As shown in the conceptual diagram above, this process can be effectively parallelized. The CMake commands will be like the following: ``` cmake .. # configuration phase cmake --build .. --target Core # => 3a,3b ``` The GUI package can be built using the core libraries stored e.g. at `../lib/core`: ``` cmake --build .. --target GUI -DBA_CORE_LIB="../lib/core" # => 3c ``` The Python wheels for any version can be built using the core libraries stored e.g. at `../lib/core`: ``` cmake --build .. -DBA_PY_PACKAGE=ON -DBA_CORE_LIB="../lib/core" # => 3d ``` or ``` cmake --build .. -DBA_PY_PACKAGE=ON -DBA_PY_PLATFORM="/opt/Python3.7" -DBA_CORE_LIB="../lib/core" # => 3d ```