Cppyy insights

This issue is referencing branch cppyy5.

How it works:

With cppyy you include headerfiles (cppyy.include) and load libraries (cppyy.load_library).

The symbols are not accessible directly after calling cppppyy.include, because cppyy is implemented in a lazy loading style.Therefore each symbol which is wanted first needs to manually be adressed by the developer. This can be done e.g by just typing

from cppyy.gbl import a_bornagain_symbol

or by calling

getattr(cppyy.gbl,"a_bornagain_symbol")

After this step it is possible to use the symbol by calling:

my_instance = cppyy.gbl.a_bornagain_symbol()
my_instance.do_sth()

Migrating from SWIG:

SWIG provides interface files where every header needed is declared in.

// libBornAgainBase.i
...
%{
#include <heinz/Complex.h>
#include "Base/Types/ICloneable.h"
#include "Base/Types/Span.h"
#include "Base/Const/Units.h"
#include "Base/Util/ThreadInfo.h"
#include "Base/Axis/Bin.h"
#include "Base/Axis/VariableBinAxis.h"
#include "Base/Axis/ConstKBinAxis.h"
#include "Base/Axis/CustomBinAxis.h"
#include "Base/Axis/FixedBinAxis.h"
#include "Base/Axis/PointwiseAxis.h"
#include "Base/Axis/Frame.h"
#include "Base/Vector/RotMatrix.h"
%}
...

We don't need these files anymore. Instead: We add a new cmake variable named api_files for every libBornagain Module that has a cmake library target (Base,Device,Param,...). This variable is a list of file paths to the header files listed in the .i files.

This variable is used in the add_library_to_wheel cmake function. Here we use a tool provided by cppyy. The cppyy-generator parses all header files provided by the api_files and outputs the structure as a JSON file. Every file is getting created in the root directory of the python package (e.g libBornAgainbase.map). Additionally right now every header file is getting copied into a include directory into the python package.

(This tool also depends on libclang provided by pip, please before running, change the path to libclang.so in MakePythonWheel.cmake line 162 to the path of the installed libclang pip package (or add cmake script that automates finding it))

Using the JSON files prevents the need for importing every needed symbol manually. On package loading all JSON files get parsed and every found symbol is getting mapped from cppyy.gbl to the module bornagain This is done in function map_symbols_to_module(pkg) listed below:

#initializer.py
...
def map_symbols_to_module(pkg):
    api_files = [
        'libBornAgainBase.map',
        'libBornAgainDevice.map',
        'libBornAgainFit.map',
        'libBornAgainParam.map',
        'libBornAgainResample.map',
        'libBornAgainSample.map',
        'libBornAgainSim.map'
    ]

    for file in api_files:
        with open(os.path.join(ROOT_PATH,file),'r') as fp:
            header_files = json.load(fp)
        lib_name = file.split("BornAgain")[1].split(".")[0]
        for header_file in header_files:
            if header_file["kind"] != "file":
                raise RuntimeError(f"Cannot load ${pkg} doe to broken python mapping.")
            cppyy.include(os.path.join(ROOT_PATH,'include',lib_name,header_file["name"]))
            for symbol in header_file["children"]:
                name = symbol["name"]
                entity = getattr(cppyy.gbl, name)
                if getattr(entity, "__module__", None) == "cppyy.gbl":
                    setattr(entity, "__module__", pkg)
                setattr(sys.modules[pkg], name, entity)

Because of this it is now possible for users of the library bornagain to write

my_instance = bornagain.a_bornagain_symbol()
my_instance.do_sth()

instead of this

my_instance = cppyy.gbl.a_bornagain_symbol()
my_instance.do_sth()

Things which still would need to be done:

  • API consistency: Cppyy respects the C++ structure including namespaces. There might be some symbols which were mapped to the global python package space using SWIG, while with Cppyy they are still inside a namespace.
  • Lazy-Loading: Check in detail which header files need to be included to the python package? If header files in api_files depend on on other headers, these need to be available on the user system.
  • 3rd-party headers: Check what would be the best way of including ROOT,heinz,formfactor headers for the user system.
  • Pythonization: In the current state special functions implemented in SWIGs .i files were ignored, these can be reimplemented in either C++ or Python. (See https://cppyy.readthedocs.io/en/latest/pythonizations.html)
  • Import from Python: The BornAgain UI uses SWIG, investigate the best way to update this part

Building using cppyy:

  1. pip install cppyy libclang
  2. Edit MakePythonWheel.cmake line 162 change the path to libclang.so to your python package path
  3. cmake .. -DBA_PY_PACKAGE=On
  4. make
  5. bash var/mk_py_package.sh
  6. pip install PyPackage/py310

Performance Testing(n=10;upper table:cppyy, lower table: swig):

Additionally here are some performance tests. Here we test the performance of every Examples/ff Python example and run each test 10 times for SWIG and for Cppyy. The upper table displays the results taken from testing with Cppyy. The lower table displays the result from testing with SWIG.

5182ee Dodecahedron.py PlatonicTetrahedron.py Pyramid4.py Pyramid2.py Icosahedron.py PlatonicOctahedron.py sim_det_box.py Sphere.py Cone.py TruncatedSphere.py Box.py SawtoothRippleBox.py HemiEllipsoid.py Spheroid.py Prism3.py HorizontalCylinder.py TruncatedCube.py Pyramid6.py Bipyramid4.py sim_demo_1quadrants.py CantellatedCube.py Pyramid3.py sim_demo_4quadrants.py Prism6.py SasCosineRipple.py Cylinder.py EllipsoidalCylinder.py TruncatedSpheroid.py
mean 3.34276207099997 2.79965125049998 2.9494933856 3.2128743100001 3.42447132090001 2.9514600080001 1.94268009860002 2.86845716739999 3.52410304689988 3.02792742360007 2.66751606119997 2.75923403730003 3.60219025469996 2.6577893274 2.71643187529994 2.89672972909998 3.21509325319998 3.32063746380004 3.71075087849995 2.12205279120003 3.17704765809995 3.12622730379999 1.94553996479999 2.84665271119998 1.96892243560005 2.7138803368 2.7722023324 2.99571711659996
stddev 0.183479613661811 0.0157522524407603 0.0754508130795409 0.0644149973903588 0.0421313359430174 0.0170348802772594 0.0759302817753841 0.0288650934989848 0.096161792682904 0.0254099561863507 0.0157398345254834 0.071215110665438 0.129192283547371 0.0181602897734817 0.0116492576666151 0.0112433320359328 0.0231683681251344 0.0524262838836981 0.721763731810366 0.259458725188499 0.0207627682444578 0.107578590917981 0.0585982695537285 0.09192908264215 0.0684311845397714 0.0619145027507019 0.0653505360339707 0.019881574249568
min 3.25147535999986 2.77798475000009 2.89483780299997 3.10936834800032 3.38180723699998 2.93210847400042 1.89370528900008 2.82383491600012 3.4336364589999 2.98819764400014 2.64380353499973 2.68312021700012 3.41236439399972 2.63060792700026 2.69781963300011 2.87450098299996 3.17756714200004 3.27686823800013 3.41070553000009 1.93292018600005 3.13636459000008 3.04572813499999 1.88416838299963 2.72446736900019 1.89590959200041 2.65678512300019 2.66136586399989 2.9718713530001
median 3.27634443750003 2.79822265200005 2.93523093649992 3.20551380699999 3.40516495249994 2.94852265000009 1.92023844200003 2.86518547749984 3.49389550599994 3.03417437550024 2.66610279349993 2.75110228999984 3.59259246850024 2.65648042199996 2.71892888050002 2.89783980750008 3.21604763999994 3.30581252399998 3.425347504 2.06241059550007 3.17461127999991 3.0667604570001 1.92938834350025 2.85115747750001 1.95446966949999 2.69905524299998 2.77945494299979 2.98969381200027
max 3.85353895700018 2.82168345900027 3.15749878099996 3.30661556700034 3.49671947500019 2.97726379000005 2.1465252869998 2.9233807999999 3.77731765099998 3.05855282499988 2.69935877000034 2.89184001000012 3.83685331799961 2.68301155300014 2.73776733800014 2.91465403699976 3.25482642899988 3.46527290199992 5.70077428500008 2.80785408800011 3.21478926000009 3.30033949900007 2.07688465899992 3.00379815899987 2.0772680340001 2.87453830100003 2.86163482199981 3.02449532199989
Return SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS FAILURE SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS FAILURE SUCCESS SUCCESS FAILURE SUCCESS FAILURE SUCCESS SUCCESS SUCCESS
5f3e18 Dodecahedron.py PlatonicTetrahedron.py Pyramid4.py Pyramid2.py Icosahedron.py PlatonicOctahedron.py sim_det_box.py Sphere.py Cone.py TruncatedSphere.py Box.py SawtoothRippleBox.py HemiEllipsoid.py Spheroid.py Prism3.py HorizontalCylinder.py TruncatedCube.py Pyramid6.py Bipyramid4.py sim_demo_1quadrants.py CantellatedCube.py Pyramid3.py sim_demo_4quadrants.py Prism6.py SasCosineRipple.py Cylinder.py EllipsoidalCylinder.py TruncatedSpheroid.py
mean 1.33910755699999 0.88913060409991 0.97804873630007 1.18590325140003 1.43971411640009 0.984567696800013 0.215120034300026 0.897078821799914 1.50119466080005 1.19519759189993 0.976996539700031 0.805872025100007 1.62873919459998 0.7833775393 0.773625292600082 0.956954904500026 1.3206725606 1.34510699779989 1.50885546939999 0.229150938500015 1.21842722850001 1.14006286999993 0.219661601399866 0.837591796100105 0.257127258499986 0.789909253199994 0.760700090100045 1.08353628489999
stddev 0.0657868978195177 0.0409571095476205 0.026486323085047 0.0226635140628365 0.0290509182583103 0.00466116685949051 0.00214356420992682 0.0197516007750506 0.0232197812898962 0.167820328095021 0.113021022629153 0.0438235491533582 0.107940913725801 0.0455375676308709 0.0221928215159933 0.0185721750980553 0.0736947907411737 0.0239421299534376 0.0735165791927896 0.00770748973247399 0.0244223547974128 0.0595726198787645 0.00570838150373798 0.0908829112436658 0.0258726909223686 0.0467979702431285 0.0951260752253945 0.0352262815394364
min 1.29730209900026 0.83759829399969 0.944563796000239 1.14614947100017 1.4078203009999 0.976636436000263 0.212385473000268 0.886041432999718 1.462825694 1.05291545599994 0.771590670000023 0.757785057000092 1.49128805700002 0.702586600999894 0.754246425000019 0.9361728519998 1.23760222200008 1.32537343600006 1.45611311199991 0.2218253369997 1.18483498000023 1.07453590500018 0.214044098999693 0.760331080000015 0.228234089000125 0.737394703000064 0.689436440999998 1.03258319299994
median 1.3085305269999 0.883955139499904 0.972828197500121 1.18542926349983 1.43261282200001 0.984181313000136 0.215173004000008 0.891574006499923 1.5101589840001 1.14589764849984 0.993287979000343 0.811826828999983 1.60946032799984 0.786317079500122 0.763643775500214 0.95951596000009 1.29645137800003 1.3375826864999 1.48231149599997 0.228345743500086 1.21988593899982 1.14492504550003 0.218358169999874 0.798540487000082 0.25178545599988 0.776457526500053 0.715087202999939 1.08081771550019
max 1.46816353500026 0.94710312899997 1.03392899699975 1.22076283200022 1.4995120829999 0.993908022000142 0.218156477000321 0.952401965999798 1.52724493400001 1.53960662099962 1.116568981 0.883826251000301 1.86206379300029 0.841355298000053 0.817448339000293 0.999137444000098 1.46788753500005 1.40791563199991 1.70147421699994 0.249385637000159 1.26458005099994 1.24753483799987 0.233237967999685 1.01253183600011 0.305922982000084 0.864672011000039 0.976842465000118 1.15262895499973
Return SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS FAILURE SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS FAILURE SUCCESS SUCCESS FAILURE SUCCESS FAILURE SUCCESS SUCCESS SUCCESS
Diff -2.00365451399998 -1.91052064640007 -1.97144464929993 -2.02697105860007 -1.98475720449992 -1.96689231120008 -1.7275600643 -1.97137834560008 -2.02290838609983 -1.83272983170014 -1.69051952149994 -1.95336201220002 -1.97345106009998 -1.8744117881 -1.94280658269986 -1.93977482459995 -1.89442069259999 -1.97553046600015 -2.20189540909996 -1.89290185270002 -1.95862042959993 -1.98616443380006 -1.72587836340012 -2.00906091509987 -1.71179517710007 -1.92397108360001 -2.01150224229996 -1.91218083169997

Conclusion: The overall execution time using cppyy increases by about 2 second. This may be preventable using precompiled header files instead of raw headers.

This can be confirmed by just timing (n=10) the following snippet:

import bornagain as ba
from bornagain import ba_plot as bp, deg, nm, std_samples, std_simulations

The average execution time here is: 2.281655096600025s for only importing and initializing the package

After optimizing the includes the average execution time of an example running with cppyy dropped to a difference of 0.5-1.2 seconds

ad4815 Dodecahedron.py PlatonicTetrahedron.py Pyramid4.py Pyramid2.py Icosahedron.py PlatonicOctahedron.py sim_det_box.py Sphere.py Cone.py TruncatedSphere.py Box.py SawtoothRippleBox.py HemiEllipsoid.py Spheroid.py Prism3.py HorizontalCylinder.py TruncatedCube.py Pyramid6.py Bipyramid4.py sim_demo_1quadrants.py CantellatedCube.py Pyramid3.py sim_demo_4quadrants.py Prism6.py SasCosineRipple.py Cylinder.py EllipsoidalCylinder.py TruncatedSpheroid.py
mean 2.7432242713999586 2.2942829109997547 2.212913983400358 2.158841763399869 2.4482925276999596 2.0143637971003043 0.9436566752996441 1.9345536543003619 2.57817558650022 2.1418041516000814 1.7328445611998178 1.7251811526997698 2.483882754200022 1.7113314901003833 1.764314533900142 1.955152703400381 2.3516601445997365 2.53405819089985 2.542381649600247 0.958024651300002 2.2179335812999854 2.119817284700184 0.9846616708000511 1.8152433689998362 0.9621647766998649 1.7726254217997848 1.7386188380996828 2.074690428699796
stddev 0.09772338081821835 0.1305187071699955 0.17208703616454193 0.008340797222271793 0.013240390893544557 0.017826647319886067 0.007026990349737903 0.07782495660926973 0.0846964704349635 0.07191854562455996 0.0252134057633042 0.020153887590855946 0.010444989540119435 0.012228664253190835 0.013059259899296266 0.019237531799044274 0.08988192323110519 0.123089812296123 0.07520776348905402 0.014701060431923178 0.010376916189618798 0.03382659107541675 0.05646214131987946 0.03251544118773943 0.011910564018835822 0.04081371683260542 0.02352384164071929 0.027802253212069354
min 2.5925702109998383 2.181972809999934 1.9937848720001057 2.14497561799908 2.42678908400012 1.994971076001093 0.9363064449989906 1.8678805610015843 2.470006072000615 2.057979470999271 1.7123730429993884 1.7027911429995584 2.4733432730008644 1.6912163790002523 1.7487849649987766 1.936145146999479 2.250936939000894 2.375713180001185 2.4782671590000973 0.9424734610001906 2.199907707999955 2.0909683509999013 0.9451685809999617 1.766386830000556 0.9451708370015695 1.7181745469988527 1.7015240929995343 2.0356048710000323
median 2.732213489500282 2.2323821694999424 2.1942466145001163 2.1616675394989215 2.4482395249997353 2.007381235000139 0.9427481414995782 1.8999355850000939 2.5516317279998475 2.1152195835002203 1.7242460809993645 1.7214851335002095 2.480652154999916 1.7115339760002826 1.7664641840001423 1.9485987620000742 2.340649215999292 2.537840595500711 2.508009766500436 0.9530009684995093 2.2212865319997945 2.113158977999774 0.971683494500212 1.8237247124989153 0.9672634009993999 1.7763549009996495 1.7381522329997097 2.0693590839991884
max 2.86522587699983 2.5902267830006167 2.5045696170000156 2.1690352850000636 2.472951366999041 2.0409056409989716 0.9615123019993916 2.0847997090004355 2.7089876840000215 2.2573730849999265 1.7890444149998075 1.7659327490000578 2.506958127998587 1.7267154210003355 1.7852245430003677 1.997139844999765 2.519646131000627 2.785485782998876 2.716632544999811 0.9835127089991147 2.234965348001424 2.208626780000486 1.141757367999162 1.8551722379997955 0.9776715649986727 1.8388769660014077 1.772564135999346 2.1164182760003314
Return SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS FAILURE SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS SUCCESS FAILURE SUCCESS SUCCESS FAILURE SUCCESS FAILURE SUCCESS SUCCESS SUCCESS
  • The examples failing still use an older python api which provides the module 'bornplot' instead of 'ba_plot'
Edited by Ludwig Jaeck