Line data    Source code

       1           0 : \cond NEVER
       2             : Distributed under the MIT License.
       3             : See LICENSE.txt for details.
       4             : \endcond
       5             : # Writing Python Bindings {#spectre_writing_python_bindings}
       6             : 
       7             : \tableofcontents
       8             : 
       9             : ## CMake and Directory Layout
      10             : 
      11             : To allow users to analyze output from simulations and take advantage of
      12             : SpECTRE's data structures and functions in python, bindings must sometimes be
      13             : written. SpECTRE uses [pybind11](https://pybind11.readthedocs.io/)
      14             : to aid with generating the bindings. The C++ code for the bindings should
      15             : generally go in a `Python` subdirectory. For example, the bindings for the
      16             : DataStructures library would go in `src/DataStructures/Python/`. SpECTRE
      17             : provides the `spectre_python_add_module` CMake function to make adding a new
      18             : python module, be it with or without bindings, easy.  The python bindings are
      19             : built only if `-D BUILD_PYTHON_BINDINGS=ON` is passed when invoking cmake
      20             : (enabled by default).
      21             : You can specify the Python version, interpreter and libraries used for compiling
      22             : and testing the bindings by setting the `-D Python_EXECUTABLE` to an absolute
      23             : path such as `/usr/bin/python3`.
      24             : 
      25             : The function `spectre_python_add_module` takes as its first argument the module,
      26             : in our case `DataStructures`. Optionally, a list of `SOURCES` can be passed to
      27             : the CMake function. If you specify `SOURCES`, you must also specify a
      28             : `LIBRARY_NAME`. A good `LIBRARY_NAME` is the name of the C++ library for which
      29             : bindings are being built prefixed with `Py`, e.g. `PyDataStructures`. If the
      30             : Python module will only consist of Python files, then the `SOURCES` option
      31             : should not be specified. Python files that should be part of the module can be
      32             : passed with the keyword `PYTHON_FILES`. Finally, the `MODULE_PATH`
      33             : named argument can be passed with a string that is the path to where the module
      34             : should be. For example, `MODULE_PATH "submodule0/submodule1/"` would mean the
      35             : module is accessed from python using
      36             : `import spectre.submodule0.submodule1.MODULE_NAME`.
      37             : 
      38             : Here is a complete example of how to call the `spectre_python_add_module`
      39             : function:
      40             : 
      41             : \code
      42             : spectre_python_add_module(
      43             :   Extra
      44             :   LIBRARY_NAME "PyExtraDataStructures"
      45             :   MODULE_PATH "DataStructures/"
      46             :   SOURCES Bindings.cpp MyCoolDataStructure.cpp
      47             :   PYTHON_FILES CoolPythonDataStructure.py
      48             :   )
      49             : \endcode
      50             : 
      51             : The library that is added has the name `PyExtraDataStructures`. Make sure to
      52             : call `spectre_python_link_libraries` for every Python module that compiles
      53             : `SOURCES`. For example,
      54             : 
      55             : \code
      56             : spectre_python_link_libraries(
      57             :   PyExtraDataStructures
      58             :   PRIVATE
      59             :   ExtraDataStructures
      60             :   pybind11::module
      61             :   )
      62             : \endcode
      63             : 
      64             : You may also call `spectre_python_add_dependencies` for Python modules that
      65             : have `SOURCES`, e.g.
      66             : 
      67             : \code
      68             : spectre_python_add_dependencies(
      69             :   PyExtraDataStructures
      70             :   PyDataStructures
      71             :   )
      72             : \endcode
      73             : 
      74             : Note that these functions will skip adding or configure any C++ libraries if
      75             : the `BUILD_PYTHON_BINDINGS` flag is `OFF`.
      76             : 
      77             : ## Writing Bindings
      78             : 
      79             : Once a python module has been added you can write the actual bindings. You
      80             : should structure your bindings directory to reflect the structure of the library
      81             : you're writing bindings for. For example, say we want bindings for `DataVector`
      82             : and `Matrix` then we should have one source file for each class's bindings
      83             : inside `src/DataStructures/Python`. The functions that generate the bindings
      84             : should be in the `py_bindings` namespace and have a reasonable name such as
      85             : `bind_datavector`. There should be a file named `Bindings.cpp` which calls all
      86             : the `bind_*` functions. The `Bindings.cpp` file is quite simple and should
      87             : `include <pybind11/pybind11.h>`, forward declare the `bind_*` functions, and
      88             : then have a `PYBIND11_MODULE` function. For example,
      89             : 
      90             : \code{.cpp}
      91             : #include <pybind11/pybind11.h>
      92             : 
      93             : namespace py = pybind11;
      94             : 
      95             : namespace py_bindings {
      96             : void bind_datavector(py::module& m);
      97             : }  // namespace py_bindings
      98             : 
      99             : PYBIND11_MODULE(_Pybindings, m) {
     100             :   py_bindings::bind_datavector(m);
     101             : }
     102             : \endcode
     103             : 
     104             : Note that the library name is passed to `PYBIND11_MODULE` and is prefixed
     105             : with an underscore. The underscore is important and the library name must be the
     106             : same that is passed as `LIBRARY_NAME` to `spectre_python_add_module` (see
     107             : above).
     108             : 
     109             : The `DataVector` bindings serve as an example with code comments on how to write
     110             : bindings for a class. There is also extensive documentation available directly
     111             : from [pybind11](https://pybind11.readthedocs.io/).
     112             : 
     113             : If you are binding a library full of similarly structured free functions, such
     114             : as libraries in `src/PointwiseFunctions/`, you can bind all functions directly
     115             : in the `Bindings.cpp` file to avoid unnecessary boilerplate code. See
     116             : `src/PointwiseFunctions/GeneralRelativity/Python/Bindings.cpp` for an example.
     117             : 
     118             : \note Exceptions should be allowed to propagate through the bindings so that
     119             : error handling via exceptions is possible from python rather than having the
     120             : python interpreter being killed with a call to `abort`.
     121             : 
     122             : ## Testing Python Bindings and Code
     123             : 
     124             : All the python bindings must be tested. SpECTRE uses the
     125             : [unittest](https://docs.python.org/3/library/unittest.html) framework
     126             : provided as part of python. To register a test file with CMake use the
     127             : SpECTRE-provided function `spectre_add_python_test` passing as the first
     128             : argument the test name (e.g. `"Unit.DataStructures.Python.DataVector"`), the
     129             : file as the second argument (e.g. `Test_DataVector.py`), and a semicolon
     130             : separated list of labels as the last (e.g. `"unit;datastructures;python"`).
     131             : All the test cases should be in a single class so that the python unit testing
     132             : framework will run all test functions on a single invocation to avoid startup
     133             : cost.
     134             : 
     135             : Below is an example of registering a python test file for bindings:
     136             : 
     137             : \snippet tests/Unit/DataStructures/CMakeLists.txt example_add_pybindings_test
     138             : 
     139             : Python code that does not use bindings must also be tested. You can register the
     140             : test file using the `spectre_add_python_test` CMake function with the same
     141             : signature as shown above.
     142             : 
     143             : ## Using The Bindings
     144             : 
     145             : See \ref spectre_using_python "Using SpECTRE's Python"
     146             : 
     147             : ## Notes:
     148             : 
     149             : - All python libraries are dynamic/shared libraries.
     150             : - Exceptions should be allowed to propagate through the bindings so that
     151             :   error handling via exceptions is possible from python rather than having the
     152             :   python interpreter being killed with a call to `abort`.
     153             : - All function arguments in Python bindings should be named using `py::arg`.
     154             :   See the Python bindings in `IO/H5/` for examples. Using the named arguments in
     155             :   Python code is optional, but preferred when it makes code more readable.
     156             :   In particular, use the argument names in the tests for the Python bindings so
     157             :   they are being tested as well.
     158             : 
     159             : ## Guidelines for writing command-line interfaces (CLIs)
     160             : 
     161             : - List all CLI endpoints in `support/Python/__main__.py`.
     162             : - Follow the recommendations in the
     163             :   [click](https://click.palletsprojects.com/en/8.1.x/) documentation.
     164             : - Split your code into free functions that know nothing about the CLI and can
     165             :   just as well be called independently from Python, and the CLI commands that
     166             :   call the functions. Test both.
     167             : - Take only input files that the script operates on as positional arguments
     168             :   (like H5 data files or YAML input files) and everything else as options.
     169             : - Choose option names and shorthands consistent with other CLI endpoints in the
     170             :   repository. For example, H5 subfile names are specified with '--subfile-name'
     171             :   / '-d' and output files are specified with '--output' / '-o'. Look at other
     172             :   CLI endpoints before making choices for option names.
     173             : - Never read or write files to or from "default" locations. Instead, take all
     174             :   input files as arguments and write all output files to locations specified
     175             :   explicitly by the user. This is important so users are not afraid of moving
     176             :   and renaming files, and are not left wondering where the script wrote its
     177             :   output. Examples:
     178             :   - Don't try to read a file like "spectre.out" from the current directory just
     179             :     because it might be there by convention. Instead, add an argument or option
     180             :     like `--out-filename` so the user can specify it.
     181             :   - Don't write a file like "plot.pdf" to the current directory without telling
     182             :     the user. Instead, add an option like `--output` / `-o` for the user to
     183             :     specify explicitly so they know exactly where output is written to.
     184             : - Operate on files instead of directories when possible. For example, prefer
     185             :   taking many H5 volume data files as arguments instead of the directory that
     186             :   contains them. This helps with operating on H5 files in segments or other
     187             :   subdirectory structures. Passing many files to a script is easy for the user
     188             :   by using a glob (note: don't take the glob as a string argument, take the
     189             :   expanded list of files directly using `click.argument(..., nargs=-1,
     190             :   type=click.Path(...))`).
     191             : - Never overwrite or delete files without prompting the user or asking them to
     192             :   run with `--force`.
     193             : - When the input to a script is empty, [gracefully degrade to a
     194             :   noop](https://click.palletsprojects.com/en/8.1.x/arguments/#variadic-arguments).
     195             : - When the user did not specify an option, print possible values for it and
     196             :   return instead of raising an exception. For example, print the subfile names
     197             :   in an H5 file if no subfile name was specified. This allows the user to make
     198             :   selections incrementally.
     199             : - When the user did not specify an output file, write the output to `sys.stdout`
     200             :   if possible instead of raising an exception. This allows the user to use pipes
     201             :   and chain commands if they want, or add a quick `-o` option to write to a
     202             :   file.
     203             : - Always use Python's `logging` module over plain `print` statements. This
     204             :   allows the user to control the verbosity.