\section{Parallel Computation}
\label{sec:parallelcomputation}
This section explains how \Dumux can be used
on multicore / multinode systems.

There are different concepts and methods for parallel programming, which are
often grouped in \textit{shared-memory} and \textit{distributed-memory}
approaches. The parallelization in \Dumux is based on the model supported by Dune which is currently based on
\textit{Message Passing Interface} (MPI) (distributed-memory approach).

The main idea behind the MPI parallelization is the concept of \textit{domain
decomposition}. For parallel simulations, the computational domain is split into
subdomains and one process (\textit{rank}) is used to solve the local problem of each
subdomain. During the global solution process, some data exchange between the
ranks/subdomains is needed. MPI is used to send data to other ranks and to receive
data from other ranks. The domain decomposition in Dune is handled by the grid managers.
The grid is partitioned and distributed on several nodes. Most grid managers contain own domain decomposition methods to split the
computational domain  into subdomains. Some grid managers also support external
tools like METIS, ParMETIS, PTScotch or ZOLTAN for partitioning.
On the other hand, linear algebra types such as matrices and vectors
do not know that they are in a parallel environment. Communication is then handled by the components of the
parallel solvers. Currently, the only parallel solver backend is \texttt{Dumux::AMGBiCGSTABBackend}, a parallel AMG-preconditioned
BiCGSTAB solver.

In order for \Dumux simulation to run in parallel, an
MPI library (e.g. OpenMPI, MPICH or IntelMPI) implementation
must be installed on the system.

Furthermore, we note that the parallel AMG preconditioner of \texttt{dune-istl}
defaults to an iterative SSOR coarse grid solver if no direct solver is found on your system. Unfortunately,
the iterative solver has a very high and hard-coded tolerance as a termination criterion, which will not solve
the coarse grid system with sufficient accuracy for typical problems in \Dumux. We therefore recommend
to install one of the direct solver libraries supported by \texttt{dune-istl}. This is either UMFPack contained
in SuiteSparse, or SuperLU, see Section~\ref{sec:listofexternallibs}.

\subsection{Prepare a Parallel Application}
Not all parts of \Dumux can be used in parallel. In order to switch to the parallel \texttt{Dumux::AMGBiCGSTABBackend}
solver backend include the respective header

\begin{lstlisting}[style=DumuxCode]
#include <dumux/linear/amgbackend.hh>
\end{lstlisting}

Second, the linear solver must be switched to the AMG backend

\begin{lstlisting}[style=DumuxCode]
using LinearSolver = AMGBiCGSTABBackend<LinearSolverTraits<GridGeometry>>;
\end{lstlisting}

and the application must be recompiled. The parallel \texttt{Dumux::AMGBiCGSTABBackend} instance has to be
constructed with a \texttt{Dune::GridView} object and a mapper, in order to construct the
parallel index set needed for communication.

\begin{lstlisting}[style=DumuxCode]
auto linearSolver = std::make_shared<LinearSolver>(leafGridView, gridGeometry->dofMapper());
\end{lstlisting}

\subsection{Run a Parallel Application}
The starting procedure for parallel simulations depends on the chosen MPI library.
Most MPI implementations use the \textbf{mpirun} command

\begin{lstlisting}[style=Bash]
mpirun -np <n_cores> <executable_name>
\end{lstlisting}

where \textit{-np} sets the number of cores (\texttt{n\_cores}) that should be used for the
computation. On a cluster you usually have to use a queuing system (e.g. slurm) to
submit a job. Check with your cluster administrator how to run parallel applications on the cluster.

\subsection{Handling Parallel Results}
For serial computations, \Dumux produces single vtu-files as default output format.
During a simulation, one VTU file is written for every output step.
In the parallel case, one VTU file for each step and processor is created.
For parallel computations, an additional variable \texttt{"process rank"} is written
into the file. The process rank allows the user to inspect the subdomains
after the computation. The parallel VTU files are combined in a single pvd file
like in sequential simulations that can be opened with e.g. ParaView.
