Line data Source code
1 0 : \cond NEVER
2 : Distributed under the MIT License.
3 : See LICENSE.txt for details.
4 : \endcond
5 : # Writing Python Bindings {#spectre_writing_python_bindings}
6 :
7 : \tableofcontents
8 :
9 : ## CMake and Directory Layout
10 :
11 : To allow users to analyze output from simulations and take advantage of
12 : SpECTRE's data structures and functions in python, bindings must sometimes be
13 : written. SpECTRE uses [pybind11](https://pybind11.readthedocs.io/)
14 : to aid with generating the bindings. The C++ code for the bindings should
15 : generally go in a `Python` subdirectory. For example, the bindings for the
16 : DataStructures library would go in `src/DataStructures/Python/`. SpECTRE
17 : provides the `spectre_python_add_module` CMake function to make adding a new
18 : python module, be it with or without bindings, easy. The python bindings are
19 : built only if `-D BUILD_PYTHON_BINDINGS=ON` is passed when invoking cmake
20 : (enabled by default).
21 : You can specify the Python version, interpreter and libraries used for compiling
22 : and testing the bindings by setting the `-D Python_EXECUTABLE` to an absolute
23 : path such as `/usr/bin/python3`.
24 :
25 : The function `spectre_python_add_module` takes as its first argument the module,
26 : in our case `DataStructures`. Optionally, a list of `SOURCES` can be passed to
27 : the CMake function. If you specify `SOURCES`, you must also specify a
28 : `LIBRARY_NAME`. A good `LIBRARY_NAME` is the name of the C++ library for which
29 : bindings are being built prefixed with `Py`, e.g. `PyDataStructures`. If the
30 : Python module will only consist of Python files, then the `SOURCES` option
31 : should not be specified. Python files that should be part of the module can be
32 : passed with the keyword `PYTHON_FILES`. Finally, the `MODULE_PATH`
33 : named argument can be passed with a string that is the path to where the module
34 : should be. For example, `MODULE_PATH "submodule0/submodule1/"` would mean the
35 : module is accessed from python using
36 : `import spectre.submodule0.submodule1.MODULE_NAME`.
37 :
38 : Here is a complete example of how to call the `spectre_python_add_module`
39 : function:
40 :
41 : \code
42 : spectre_python_add_module(
43 : Extra
44 : LIBRARY_NAME "PyExtraDataStructures"
45 : MODULE_PATH "DataStructures/"
46 : SOURCES Bindings.cpp MyCoolDataStructure.cpp
47 : PYTHON_FILES CoolPythonDataStructure.py
48 : )
49 : \endcode
50 :
51 : The library that is added has the name `PyExtraDataStructures`. Make sure to
52 : call `spectre_python_link_libraries` for every Python module that compiles
53 : `SOURCES`. For example,
54 :
55 : \code
56 : spectre_python_link_libraries(
57 : PyExtraDataStructures
58 : PRIVATE
59 : ExtraDataStructures
60 : pybind11::module
61 : )
62 : \endcode
63 :
64 : You may also call `spectre_python_add_dependencies` for Python modules that
65 : have `SOURCES`, e.g.
66 :
67 : \code
68 : spectre_python_add_dependencies(
69 : PyExtraDataStructures
70 : PyDataStructures
71 : )
72 : \endcode
73 :
74 : Note that these functions will skip adding or configure any C++ libraries if
75 : the `BUILD_PYTHON_BINDINGS` flag is `OFF`.
76 :
77 : ## Writing Bindings
78 :
79 : Once a python module has been added you can write the actual bindings. You
80 : should structure your bindings directory to reflect the structure of the library
81 : you're writing bindings for. For example, say we want bindings for `DataVector`
82 : and `Matrix` then we should have one source file for each class's bindings
83 : inside `src/DataStructures/Python`. The functions that generate the bindings
84 : should be in the `py_bindings` namespace and have a reasonable name such as
85 : `bind_datavector`. There should be a file named `Bindings.cpp` which calls all
86 : the `bind_*` functions. The `Bindings.cpp` file is quite simple and should
87 : `include <pybind11/pybind11.h>`, forward declare the `bind_*` functions, and
88 : then have a `PYBIND11_MODULE` function. For example,
89 :
90 : \code{.cpp}
91 : #include <pybind11/pybind11.h>
92 :
93 : namespace py = pybind11;
94 :
95 : namespace py_bindings {
96 : void bind_datavector(py::module& m);
97 : } // namespace py_bindings
98 :
99 : PYBIND11_MODULE(_Pybindings, m) {
100 : py_bindings::bind_datavector(m);
101 : }
102 : \endcode
103 :
104 : Note that the library name is passed to `PYBIND11_MODULE` and is prefixed
105 : with an underscore. The underscore is important and the library name must be the
106 : same that is passed as `LIBRARY_NAME` to `spectre_python_add_module` (see
107 : above).
108 :
109 : The `DataVector` bindings serve as an example with code comments on how to write
110 : bindings for a class. There is also extensive documentation available directly
111 : from [pybind11](https://pybind11.readthedocs.io/).
112 :
113 : If you are binding a library full of similarly structured free functions, such
114 : as libraries in `src/PointwiseFunctions/`, you can bind all functions directly
115 : in the `Bindings.cpp` file to avoid unnecessary boilerplate code. See
116 : `src/PointwiseFunctions/GeneralRelativity/Python/Bindings.cpp` for an example.
117 :
118 : \note Exceptions should be allowed to propagate through the bindings so that
119 : error handling via exceptions is possible from python rather than having the
120 : python interpreter being killed with a call to `abort`.
121 :
122 : ## Testing Python Bindings and Code
123 :
124 : All the python bindings must be tested. SpECTRE uses the
125 : [unittest](https://docs.python.org/3/library/unittest.html) framework
126 : provided as part of python. To register a test file with CMake use the
127 : SpECTRE-provided function `spectre_add_python_test` passing as the first
128 : argument the test name (e.g. `"Unit.DataStructures.Python.DataVector"`), the
129 : file as the second argument (e.g. `Test_DataVector.py`), and a semicolon
130 : separated list of labels as the last (e.g. `"unit;datastructures;python"`).
131 : All the test cases should be in a single class so that the python unit testing
132 : framework will run all test functions on a single invocation to avoid startup
133 : cost.
134 :
135 : Below is an example of registering a python test file for bindings:
136 :
137 : \snippet tests/Unit/DataStructures/CMakeLists.txt example_add_pybindings_test
138 :
139 : Python code that does not use bindings must also be tested. You can register the
140 : test file using the `spectre_add_python_test` CMake function with the same
141 : signature as shown above.
142 :
143 : ## Using The Bindings
144 :
145 : See \ref spectre_using_python "Using SpECTRE's Python"
146 :
147 : ## Notes:
148 :
149 : - All python libraries are dynamic/shared libraries.
150 : - Exceptions should be allowed to propagate through the bindings so that
151 : error handling via exceptions is possible from python rather than having the
152 : python interpreter being killed with a call to `abort`.
153 : - All function arguments in Python bindings should be named using `py::arg`.
154 : See the Python bindings in `IO/H5/` for examples. Using the named arguments in
155 : Python code is optional, but preferred when it makes code more readable.
156 : In particular, use the argument names in the tests for the Python bindings so
157 : they are being tested as well.
158 :
159 : ## Guidelines for writing command-line interfaces (CLIs)
160 :
161 : - List all CLI endpoints in `support/Python/__main__.py`.
162 : - Follow the recommendations in the
163 : [click](https://click.palletsprojects.com/en/8.1.x/) documentation.
164 : - Split your code into free functions that know nothing about the CLI and can
165 : just as well be called independently from Python, and the CLI commands that
166 : call the functions. Test both.
167 : - Take only input files that the script operates on as positional arguments
168 : (like H5 data files or YAML input files) and everything else as options.
169 : - Choose option names and shorthands consistent with other CLI endpoints in the
170 : repository. For example, H5 subfile names are specified with '--subfile-name'
171 : / '-d' and output files are specified with '--output' / '-o'. Look at other
172 : CLI endpoints before making choices for option names.
173 : - Never read or write files to or from "default" locations. Instead, take all
174 : input files as arguments and write all output files to locations specified
175 : explicitly by the user. This is important so users are not afraid of moving
176 : and renaming files, and are not left wondering where the script wrote its
177 : output. Examples:
178 : - Don't try to read a file like "spectre.out" from the current directory just
179 : because it might be there by convention. Instead, add an argument or option
180 : like `--out-filename` so the user can specify it.
181 : - Don't write a file like "plot.pdf" to the current directory without telling
182 : the user. Instead, add an option like `--output` / `-o` for the user to
183 : specify explicitly so they know exactly where output is written to.
184 : - Operate on files instead of directories when possible. For example, prefer
185 : taking many H5 volume data files as arguments instead of the directory that
186 : contains them. This helps with operating on H5 files in segments or other
187 : subdirectory structures. Passing many files to a script is easy for the user
188 : by using a glob (note: don't take the glob as a string argument, take the
189 : expanded list of files directly using `click.argument(..., nargs=-1,
190 : type=click.Path(...), required=True)`).
191 : Note that Click recommends to avoid `required=True` here but to [gracefully
192 : degrade to a noop](https://click.palletsprojects.com/en/8.1.x/arguments/#variadic-arguments)
193 : instead, but that's confusing for the user.
194 : - Never overwrite or delete files without prompting the user or asking them to
195 : run with `--force`.
196 : - When the user did not specify an option, print possible values for it and
197 : return instead of raising an exception. For example, print the subfile names
198 : in an H5 file if no subfile name was specified. This allows the user to make
199 : selections incrementally.
200 : - When the user did not specify an output file, write the output to `sys.stdout`
201 : if possible instead of raising an exception. This allows the user to use pipes
202 : and chain commands if they want, or add a quick `-o` option to write to a
203 : file.
204 : - Always use Python's `logging` module over plain `print` statements. This
205 : allows the user to control the verbosity.
|
