Fault-tolerance with PFASST: node failures
==========================================

In this project, we explore PFASST's potential to deal with node failures.
We derive different strategies which allow PFASST to continue after one time-step has failed.
Failure injectation as well as the different strategies are contained in ``emulate_hard_faults`` and the modified controller ``allinclusive_classic_nonMPI_hard_faults`` allows faults to appear before a fine sweep.
We test our ideas for two simple toy problems and two more complex show cases.
This project contains the code for the publication `Toward fault-tolerant parallel-in-time integration with PFASST <https://arxiv.org/abs/1510.08334>`_ of pySDC v2,
while the original code can be found under `pySDC: Fault-tolerant PFASST <https://doi.org/10.5281/zenodo.32765>`_.
Note that due to the long runtime, the results are not generated via Travis. Only the visualization (and therefore the existence of the data files) is tested.

Propagation of a single node failure
------------------------------------

We start by analyzing the propagation and containment of a single fault at step 7, iteration 7, see ``hard_faults_detail``.
We do this for the heat and advection equation, both in 1D.
Four different strategies are tested:

- ``SPREAD``: a simple restart from scratch, i.e. the node/time-step is restarted by copying u0 to all quadrature nodes
- ``SPREAD_PREDICT``: in addition to copying u0, we also do multiple SDC sweeps on the coarse level
- ``INTERP``: instead of copying u0, we interpolate the values at the quadrature nodes by taking the next and the following time-step into account
- ``INTERP_PREDICT``: in addition to interpolation, we also do coarse SDC sweeps

The results are plotted using ``postproc_hard_faults_detail``.

.. include:: doc_node_failure_hard_faults_detail.rst

Node failures at different steps and iterations
-----------------------------------------------

The next step is to check how faults impact the convergence of PFASST at different steps and iterations.
We systematically study this in ``hard_faults_test``, where for the heat and the advection equation each combination of step and iteraston is tested separately.
Heat maps generated by ``postproc_hard_faults_test`` then show how many more iterations are required to converge.

.. include:: doc_node_failure_hard_faults_test.rst


The Boussinesq test case
------------------------

A first, more complex test case is the semi-implicit 2D Boussinesq system (order-coarsening only), see ``boussinesq_example``.
We inject faults randomly with a rate of 3%, i.e. in 3% of all fine sweeps a node fails.
To ensure comparability, we define these fails a priori, so that each run has to deal with the same failures.
The ``postproc_boussinesq`` script visualizes the results.

.. include:: doc_node_failure_boussinesq.rst

The Gray-Scott test case
------------------------

Another test case is the 1D Gray-Scott reaction diffusion model, see ``grayscott_example``.
With spatial coarsening enabled, this application plays well with PFASST and the goal of this part is to see how fault injections and recovery impact the convergence.
We again inject faults randomly, but in a pre-defined way with a rate of 3%.
The ``postproc_grayscott`` script visualizes the results. Also, the script ``animate_convergence`` can be used to create an animated convergence plot (and can be easily adapted for other examples, too).

.. include:: doc_node_failure_grayscott.rst