Debugging with GDB
==================

Avocado has two different types of GDB support that complement each
other:

* Transparent execution of executables inside the GNU Debugger. This
  takes standard and possibly unmodified tests that uses the
  :mod:`avocado.utils.process` APIs for running processes. By using a
  command line option, the executable is run on GDB. This allows the user
  to interact with GDB, but to the test itself, things are pretty much
  transparent.
* The :mod:`avocado.utils.gdb` APIs that allows a test to interact with GDB,
  including setting a executable to be run, setting breakpoints or any
  other types of commands. This requires a test written with that
  approach and API in mind.

.. tip:: Even though this section describes the use of the Avocado GDB
   features, which allow live debugging of binaries inside Avocado
   tests, it's also possible to debug some application offline by
   using tools such as `rr <http://rr-project.org>`_.  Avocado ships
   with an example wrapper script (to be used with ``--wrapper``) for
   that purpose.


Transparent Execution of Executables
------------------------------------

This feature adds a few command line options to the Avocado ``run``
command::

  $ avocado run --help
  ...
  GNU Debugger support:

    --gdb-run-bin EXECUTABLE[:BREAKPOINT]
                          Run a given executable inside the GNU debugger,
                          pausing at a given breakpoint (defaults to "main")
    --gdb-prerun-commands EXECUTABLE:COMMANDS
                          After loading an executable in GDB, but before
                          actually running it, execute the GDB commands in the
                          given file. EXECUTABLE is optional, if omitted
                          COMMANDS will apply to all executables
    --gdb-coredump {on,off}
                          Automatically generate a core dump when the inferior
                          process received a fatal signal such as SIGSEGV or
                          SIGABRT
  ...

To get started you want to use ``--gdb-run-bin``, as shown in the example bellow.

Example
~~~~~~~

The simplest way is to just run
``avocado run --gdb-run-bin=doublefree examples/tests/doublefree.py``, which
wraps each executed executable with name ``doublefree`` inside GDB server and
stops at the executable entry point.

Optionally you can specify single breakpoint using
``--gdb-run-bin=doublefree:$breakpoint`` (eg: ``doublefree:1``) or just
``doublefree:`` to stop only when an interruption happens (eg: SIGABRT).

It's worth mentioning that when breakpoint is not reached, the test finishes
without any interruption. This is helpful when you identify regions where you
should never get in your code, or places which interests you and you can run
your code in production and GDB variants. If after a long time you get to this
place, the test notifies you and you can investigate the problem. This is
demonstrated in ``examples/tests/doublefree_nasty.py`` test. To unveil the
power of Avocado, run this test using::

    avocado run --gdb-run-bin=doublefree: examples/tests/doublefree_nasty.py --gdb-prerun-commands examples/tests/doublefree_nasty.py.data/gdb_pre --mux-yaml examples/tests/doublefree_nasty.py.data/iterations.yaml

which executes 100 iterations of this test while setting all breakpoints from
the ``examples/tests/doublefree_nasty.py.data/gdb_pre`` file (you can specify
whatever GDB supports, not only breakpoints).

As you can see this test usually passes, but once in a while it gets into
the problematic area. Imagine this is very hard to spot (dependent on HW
registers, ...) and this is one way to combine regular testing and the
possibility of debugging hard-to-get parts of your code.

Caveats
~~~~~~~

Currently, when using the Avocado GDB plugin, that is, when using the
`--gdb-run-bin` option, there are some caveats you should be aware of:

* It is not currently compatible with Avocado's `--output-check-record` feature
* There's no way to perform proper input to the process, that is, manipulate its `STDIN`
* The process `STDERR` content is mixed with the content generated by `gdbserver` on its
  own `STDERR` (because they are in fact, the same thing)

But, you can still depend on the process `STDOUT`, as exemplified by this fictional
test::

 from avocado import Test
 from avocado.utils import process

 class HelloOutputTest(Test):

     def test(self):
         result = process.run("/path/to/hello", ignore_status=True)
         self.assertIn("hello\n", result.stdout)

If run under GDB or not, `result.stdout` behavior and content is expected to be the same.

Reasons for the caveats
~~~~~~~~~~~~~~~~~~~~~~~

There are a two basic reasons for the mentioned caveats:

* The architecture of Avocado's GDB feature
* GDB's own behavior and limitations

When using the Avocado GDB plugin, that is, `--gdb-run-bin`, Avocado runs a `gdbserver` instance
transparently and controls it by means of a `gdb` process. When a given event happens, say a
breakpoint is reached, it disconnects its own `gdb` from the server, and allows the user to use
a standard `gdb` to connect to the `gdbserver`. This provides a natural and seamless user experience.

But, `gdbserver` has some limitations at this point, including:

* Not being able to set a controlling `tty`
* Not separating its own `STDERR` content from the application being run

These limitations are being addressed both on Avocado and GDB, and will be resolved in future Avocado
versions.

Workaround
~~~~~~~~~~

If the application you're running as part of your test can read input from alternative
sources (including devices, files or the network) and generate output likewise, then
you should not be further limited.

GDB support and avocado-virt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Another current limitation is the use of `avocado-virt` and `avocado` GDB support.

The supported API for transparent debugging is currently limited to
:func:`avocado.utils.process.run`, and does not cover advanced uses of the
:class:`avocado.utils.process.SubProcess` class. The `avocado-virt`
extension, though, uses :class:`avocado.utils.process.SubProcess` class to
execute `qemu` in the background.

This limitation will be addressed in future versions of `avocado` and `avocado-virt`.


:mod:`avocado.utils.gdb` APIs
-----------------------------

Avocado's GDB module, provides three main classes that lets a test writer
interact with a `gdb` process, a `gdbserver` process and also use the GDB
remote protocol for interaction with a remote target.

Please refer to :mod:`avocado.utils.gdb` for more information.

Example
~~~~~~~

Take a look at ``examples/tests/modify_variable.py`` test::

    def test(self):
        """
        Execute 'print_variable'.
        """
        path = os.path.join(self.workdir, 'print_variable')
        app = gdb.GDB()
        app.set_file(path)
        app.set_break(6)
        app.run()
        self.log.info("\n".join(app.read_until_break()))
        app.cmd("set variable a = 0xff")
        app.cmd("c")
        out = "\n".join(app.read_until_break())
        self.log.info(out)
        app.exit()
        self.assertIn("MY VARIABLE 'A' IS: ff", out)

You can see that instead of running the executable using
``process.run`` we invoke :class:`avocado.utils.gdb.GDB`. This allows
us to automate the interaction with the GDB in means of setting
breakpoints, executing commands and querying for output.

When you check the output (``--show-job-log``) you can see that despite
declaring the variable as 0, ff is injected and printed instead.
