The Nerva-Rowwise Python manual

Before installing the Python bindings, ensure the following C++ libraries are available on your system:

These libraries must be available as header files (and, for MKL, also as linkable libraries) at build time.

The build system locates dependencies using the following environment variables:

If the ONEAPI_ROOT variable is set, the MKL path will be inferred as $ONEAPI_ROOT/mkl/latest. If you are on Windows, setting ONEAPI_ROOT is required to find the Intel OpenMP runtime (libiomp5).

If you built the C++ library using CMake, you can optionally have FMT, Eigen, and pybind11 downloaded automatically via CMake’s FetchContent mechanism. These will be placed in the _deps subdirectory of the build directory.

To use the CMake-resolved dependencies, set the following environment variable:

This tells setup.py to look for the headers in:

Refer to the CMake install section in the C++ documentation for details.

A C++17-compatible compiler is required. The installation has been tested with

The setup script uses OpenMP and links against MKL; make sure these are available on your platform.

The bindings are compatible with Python 3.12 and newer. Make sure your pip corresponds to a supported version of Python.

To install the Python bindings, navigate to the python subdirectory of the repository and run:

This will build the native extension and install the nerva Python module. If any required environment variables are missing or incorrectly set, setup.py will raise an informative error.

The tool mlp.py can be used for training multilayer perceptrons. An example invocation of the mlp.py tool is

This will train a CIFAR-10 model using an MLP consisting of three sparse layers with activation functions ReLU, ReLU and no activation. A script prepare_data.py is available in the data directory that can be used to download the dataset, flatten it and store it in .npz format. See the section Preparing data for details.

The output may look like this:

The tool inspect_npz.py can be used to inspect the contents of a dataset stored in .npz format.

An example invocation of the inspect_npz.py tool is

The output may look like this:

With the command line option --shapes-only a summary can be obtained:

The Nerva-Rowwise Python Library uses 32-bit floats as its number type. The C++ library also supports 64-bit floats.

The most important files in the nerva module are given in the table below.

To a limited extent, the C++ data structures can be accessed in Python. In the file loss_test.py it is demonstrated how to modify the weight matrices and bias vectors of dense layers via the _layer attribute:

The weight matrices of sparse layers are not yet fully exposed to Python.

The class StochasticGradientDescentAlgorithm can be used to train a neural network. It takes as input a multilayer perceptron, a dataset, a loss function, a learning rate scheduler, and a struct containing options like the number of epochs. The main loop looks like this:

In every epoch, the dataset is divided into a number of batches. This is handled by the DataLoader, that creates batches X of a given batch size, with corresponding targets T (i.e. the expected outputs). Each batch goes through the three steps of stochastic gradient descent:

The default storage format used in the Nerva libraries is the NumPy NPZ format, see numpy.lib.format. The reason for choosing this format is portability between C++ and Python implementations. A file in .npz format can be used to store a dictionary of arrays in a compressed format.

The mlp.py tool requires training and testing data to be stored in .npz format. To help with this, a script is provided to download and preprocess datasets commonly used in experiments, including MNIST and CIFAR-10.

The script is located at ../../../python/data/prepare_data.py and can be run from the command line.

The mlp.py tool has options --load-weights and --save-weights for loading and saving the weights and bias vectors of an MLP, and options --load-data and --save-data for loading and saving a dataset in NPZ format. The keys in the dictionary for the weight matrices and bias vectors of linear layers are W1, W2, …​ and b1, b2, …​. The keys for the training data plus targets are Xtrain and Ttrain, while for the test data plus targets we use Xtest and Ttest.

The mlp.py tool supports saving and loading both datasets and model parameters using the NumPy .npz format. This ensures compatibility between Python and C++ implementations by storing everything in a standard dictionary of arrays.

The .npz file for datasets contains the following keys:

The .npz file for model parameters stores each layer’s weights and biases under the keys:

These arrays use standard NumPy formats and can be inspected or manipulated easily in Python using numpy.load() and numpy.savez().

In textbooks and tutorials, the training of a neural network is usually explained in terms of individual examples. But in order to achieve high performance, it is absolutely necessary to use mini-batches. On Wikipedia this is explained as follows:

To support mini-batches, the Nerva-Rowwise Python Library defines all equations that play a role in the execution of a neural network in matrix form, including the backpropagation equations, see the paper Batch Matrix-form Equations and Implementation of Multilayer Perceptrons. For the latter, many neural network frameworks rely on Automatic differentiation, see also [1]. We use explicit backpropagation equations to implement truly sparse layers and to provide an instructive resource for those studying neural network execution.

The performance of training a neural network largely depends on the calculation of matrix products during the backpropagation step of linear layers. In order to do this efficiently, the Intel Math Kernel library (MKL) is used. Currently, this dependency is hard coded, but there are plans to make this optional. To experiment with other implementations, like SYCL or BLAS, a global setting is used that is discussed in the next section.

Experiments with sparse neural networks have shown that the performance can be negatively influenced by subnormal numbers. The example program subnormal_numbers.cpp illustrates the problem. The table below is the result of the following experiment. The dot product of two large vectors of floating-point numbers is computed. One vector is filled with random values between 0 and 1, and the other with powers of 10, ranging from 1 to 1e−45. For values larger than 1e−35, the time needed for this calculation is about 0.044 seconds. For smaller values we end up in the range of subnormal numbers. This causes the runtime to increase more than eight-fold to 0.37 seconds. In our experiments we observed that when layers with high sparsity are used, it may happen that subnormal values appear in weight matrices, and their amount increases every epoch.

On Google Groups this problem is discussed. A possible solution is to instruct the compiler to flush subnormal values to zero. But there doesn’t seem to be a portable way to achieve this. In the Nerva-Rowwise Python Library different solutions have been tried. One of them is to periodically flush weights in the subnormal range to zero using the --clip command line option of the mlp tool. In [2] the problem of subnormal numbers is discussed.

Since we are dealing with a programming context, we say that the support of a sparse matrix refers to the set of positions (or indices) in the matrix that are explicitly stored. Elements inside the support can have a non-zero value. Elements outside the support have the value zero by definition.

Sparse matrices in the Nerva-Rowwise Python Library are stored in CSR format. This matrix representation stores arrays of column and row indices to define the support, plus an array of the corresponding values. CSR matrices are unstructured sparse matrices, meaning they have non-zero elements located at arbitrary positions. Alternatively, there are structured sparse matrices, take for example butterfly matrices [5].

Sparse evolutionary training (SET) is a method for efficiently training sparse neural networks, see e.g. [6]. The idea behind this method is to start the training with a random sparse topology, and to periodically prune and regrow some of the weights.

In SET, the sparsity is not divided evenly over the sparse layers. Instead, small layers are assigned a higher density than larger ones. In [6] formula (3), Erdős–Rényi graph topology is suggested to calculate the densities of the sparse layers given a desired overall density of the sparse layers combined. In the Nerva-Rowwise Python Library this is implemented in the function compute_sparse_layer_densities, see layer_algorithms.h. The original Python implementation can be found here, along with several other sparse initialization strategies. In the tool mlp.py the option --overall-density is used for assigning Erdős–Rényi densities to the sparse layers. See [mlp_output] for an example of this. The overall density of 0.05 is converted into densities [0.042382877, 0.06357384, 1.0] for the individual layers.

Pruning weights is about removing parameters from a neural network, see also Pruning (artificial_neural_network). In our context removing parameters is about removing elements from the support of a sparse weight matrix. The effect of this is that the values corresponding to these elements are zeroed.

Growing weights is about adding parameters to a neural network. In our context adding parameters is about adding elements to the support of a sparse weight matrix.

In the Nerva-Rowwise Python Library, the classes prune_function and grow_function are used to represent generic pruning and growing strategies:

In the command line tool mlp.py the user can select specific implementations of these prune and grow functions. They are called at the start of each epoch of training via an attribute regrow_function that applies pruning and growing to the sparse layers of an MLP. See also the [on_start_epoch] event.

In [7] we report on some of our experiments with sparse neural networks.

An example of a dynamic sparse training experiment is

At the start of every epoch 20% of the weights is pruned, and the same number of weights is added back at different locations. The output may look like this:

As an example, we will explain how the loss function SoftmaxCrossEntropyLoss is added to the nerva Python module.

Note that the Python class derives from the C++ class. In the same file, an entry to the function parse_loss_function should be added.