Metadata-Version: 1.1
Name: exactly
Version: 0.8.9.5
Summary: Tests a command line program by executing it in a temporary sandbox directory and inspecting its result.
Home-page: https://github.com/emilkarlen/exactly/wiki
Author: Emil Karlen
Author-email: emil@member.fsf.org
License: GPLv3+
Description: Tests a command line program by executing it in a temporary sandbox directory and inspecting its result.
        
        Or tests properties of existing files, directories etc.
        
        
        Supports individual test cases and test suites.
        
        Exactly has a `Wiki
        <https://github.com/emilkarlen/exactly/wiki>`_.
        It also has a built in help system,
        which can, among other things,
        generate this `Reference manual
        <http://htmlpreview.github.io/?https://raw.githubusercontent.com/wiki/emilkarlen/exactly/Reference.html>`_.
        
        
        .. contents::
        
        
        TEST CASES
        ========================================
        
        A test case is written as a plain text file.
        
        
        Testing stdin, stdout, stderr, exit code
        ------------------------------------------------------------
        
        The following checks that your new ``my-contacts-program`` reads a contact list from stdin,
        and is able to find the email of a person::
        
            [setup]
        
            stdin = -file some-test-contacts.txt
        
            [act]
        
            my-contacts-program get-email-of --name 'Pablo Gauss'
        
            [assert]
        
            exit-code == 0
        
            stdout equals <<EOF
            pablo@gauss.org
            EOF
        
            stderr empty
        
        
        If the file 'contacts.case' contains this test case, then Exactly can execute it::
        
        
            > exactly contacts.case
            PASS
        
        
        "PASS" means that all assertions were satisfied.
        
        This test assumes that
        
         * the system under test - ``my-contacts-program`` - is is found in the same directory as the test case file
         * the file "some-test-contacts.txt" (that is referenced from the test case) is found in the same directory as the test case file
        
        The ``home`` and ``act-home`` instructions
        can be used to change the directories where Exactly looks for files referenced from the test case.
        
        
        Testing side effects on files and directories
        ------------------------------------------------------------
        
        When the execution of a test case starts,
        the current directory is set to a temporary directory.
        This gives the test case a sandbox where it can create and manipulate files.
        
        The sandbox - and all files within it - are removed when the execution ends.
        
        
        The following tests a program that classifies
        files as either good or bad, by moving them to the
        appropriate directory::
        
            [setup]
        
            dir input
            dir output/good
            dir output/bad
        
            file input/a.txt = <<EOF
            GOOD contents
            EOF
        
            file input/b.txt = <<EOF
            bad contents
            EOF
        
            [act]
        
            classify-files-by-moving-to-appropriate-dir GOOD input/ output/
        
            [assert]
        
            dir-contents input empty
        
            exists -file output/good/a.txt
            dir-contents output/good num-files == 1
        
            exists -file output/bad/b.txt
            dir-contents output/bad num-files == 1
        
        
        ``file`` and ``dir`` makes files in the current directory (by default).
        
        
        Using predefined source files
        ------------------------------------------------------------
        
        The *home directory structure* is directories containing
        predefined files involved in a test case:
        
        *act-home*
         Location of the program file being tested
        
        *home*
          Location of arbitrary test resources
        
        
        Both of them defaults to the directory
        that contains the test case file,
        but can be changed via ``[conf]``.
        
        
        There are options for making paths relative to them,
        and also to the temporary sandbox directories.
        
        ``-rel-home`` refers to the *home* directory,
        and ``-rel-act`` to the temporary directory
        that is the current directory at the start of the execution::
        
        
            [conf]
        
            act-home = ../bin/
        
            home     = data/
        
            [setup]
        
            copy  -rel-home input.txt  -rel-act actual.txt
        
            [act]
        
            filter-lines "text to find" actual.txt
        
            [assert]
        
            contents -rel-act actual.txt
                     equals
                     -file -rel-home expected.txt
        
        
        These "relativity" options have defaults designed to minimize the
        need for them.
        The following case does the same thing as the one above::
        
            [conf]
        
            act-home = ../bin/
        
            home     = data/
        
            [setup]
        
            copy input.txt actual.txt
        
            [act]
        
            filter-lines "text to find" actual.txt
        
            [assert]
        
            contents actual.txt
                     equals
                     -file expected.txt
        
        
        Testing and transforming the contents of files
        ------------------------------------------------------------
        
        Use ``contents`` to test the contents of a file,
        or a transformed version of it,
        by applying a "string transformer".
        
        Such a "string transformer" may be given a name
        using the ``def`` instruction
        to make the test easier to read.
        
        The following case
        tests that "timing lines" are output as part of a log file "log.txt".
        
        The challenge is that the (fictive) log file contains
        non-timing lines that the test is not interested in,
        and that timing lines contains a time stamp of the form
        "NN:NN", who's exact value also is not interesting.
        
        A "string transformer" is used to extract all timing lines
        and to replace "NN:NN" time stamps with the constant string ``TIMESTAMP``::
        
        
            [setup]
        
            def line-matcher       IS_TIMING_LINE     = regex ^timing
        
            def string-transformer REPLACE_TIMESTAMPS = replace [0-9]{2}:[0-9]{2} TIMESTAMP
        
            def string-transformer GET_TIMING_LINES   = select IS_TIMING_LINE | REPLACE_TIMESTAMPS
        
            [act]
        
            program-that-writes-log-file
        
            [assert]
        
            contents log.txt
                     -transformed-by GET_TIMING_LINES
                     equals <<EOF
            timing TIMESTAMP begin
            timing TIMESTAMP preprocessing
            timing TIMESTAMP validation
            timing TIMESTAMP execution
            timing TIMESTAMP end
            EOF
        
        
        The ``-transformed-by`` option does not modify the tested file,
        it just applies the assertion to a transformed version of it.
        
        
        
        Using external helper programs
        ------------------------------------------------------------
        
        External programs can with help with setup and assertions etc.
        
        Exactly can run executable files, shell commands  and programs in the OS PATH,
        using ``run``, ``$``, ``%``.
        
        The following case shows some examples, but doesn't make sense tough::
        
            [setup]
        
            run my-setup-helper-program first "second arg"
        
            run % mysql -uu -pp -hlocalhost -Dd --batch --execute "create table my_table(id int)"
        
            $ touch file
        
            file root-files.txt = -stdout-from $ ls /
        
            [act]
        
            $ echo ${PATH}
        
            [assert]
        
            run my-assert-helper-program
        
            $ test -f root-files.txt
        
            stdout -from
                   % echo 'Interesting output'
                   equals
            <<EOF
            Interesting output
            EOF
        
            [cleanup]
        
            run % mysql -uu -pp -hlocalhost -Dd --batch --execute "drop table my_table"
        
        
        A program executed in ``[assert]`` becomes an assertion that depends on the exit code.
        
        
        Program values can be defined for reuse using ``def`` and run using ``@``::
        
            [setup]
        
            def program RUN_MYSQL   = % mysql -uu -pp -hlocalhost -Dd
            def program EXECUTE_SQL = @ RUN_MYSQL --skip-column-names --batch --execute
        
        
            run @ EXECUTE_SQL "create table my_table(id int)"
        
            [act]
        
            system-under-test
        
            [assert]
        
            stdout -from
                   @ EXECUTE_SQL "select * from my_table"
                   ! empty
        
            [cleanup]
        
            run @ EXECUTE_SQL "drop table my_table"
        
        
        Testing source code files
        -------------------------
        
        The ``actor`` instruction can specify an interpreter to test a source code file::
        
            [conf]
        
            actor = -file python
        
            [act]
        
            my-python-program.py 'an argument' second third
        
            [assert]
        
            stdout equals
            <<EOF
            Argument: an argument
            Argument: second
            Argument: third
            EOF
        
        
        Print output from the tested program
        ------------------------------------
        
        
        If ``--act`` is used, the output of the "act" phase (the "action to check")
        will become the output of ``exactly`` -
        stdout, stderr and exit code
        ::
        
        
            [setup]
        
            dir  a-dir
            file a-file
        
            [act]
        
            $ ls
        
            [assert]
        
            stdout num-lines == 314
        
        ::
        
            > exactly --act my-test.case
            a-dir
            a-file
        
        
        The test case is executed in a temporary sandbox, as usual,
        but assertions are ignored.
        
        
        Testing existing OS environment - tests without ``[act]``
        ----------------------------------------------------------------------
        
        A test case does not need to have an ``[act]`` phase.
        
        For example, to just check that the names of some SQL files are correct::
        
            [assert]
        
            def path SQL_DIR = -rel-here sql
        
            exists -dir @[SQL_DIR]@
        
        
            'sql/ must only contain sql files'
        
            dir-contents @[SQL_DIR]@
                         -selection ! name *.sql
                         empty
        
        
        ORGANIZING TESTS
        ========================================
        
        File inclusion
        ------------------------------------
        
        Test case contents can be included from external files::
        
            [setup]
        
            including my-dir-symbols.def
        
            including my-common-setup-and-cleanup.xly
        
        
        
        Test suites
        ------------------------------------
        
        
        Tests can be grouped in suites::
        
        
            first.case
            second.case
        
        or::
        
            [cases]
        
            helloworld.case
            *.case
            **/*.case
            
        
            [suites]
        
            sub-suite.suite
            *.suite
            pkg/suite.suite
            **/*.suite
        
        
        
        If the file ``my-suite.suite`` contains this text, then Exactly can run it::
        
            > exactly suite my-suite.suite
            ...
            OK
        
        
        The result of a suite can be reported as
        simple progress information,
        or JUnit XML.
        
        
        Suites can contain test case functionality that is common
        to all cases in the suite. For example::
        
        
            [cases]
        
            *.case
        
            [conf]
        
            act-home = ../bin/
        
            [setup]
        
            def string CONF_FILE = my.conf
        
            file @[CONF_FILE]@ =
            <<EOF
            common = configuration
            EOF
        
        
        The common functionality is included in each test case.
        
        
        MORE EXAMPLES
        ========================================
        
        The ``examples/`` directory of the source distribution contains more examples.
        
        
        INSTALLING
        ========================================
        
        
        Exactly is written in Python and does not require any external libraries.
        
        Exactly requires Python >= 3.5 (not tested on earlier version of Python 3).
        
        Use ``pip`` or ``pip3`` to install::
        
            > pip install exactly
        
        or::
        
            > pip3 install exactly
        
        The program can also be run from a source distribution::
        
            > python3 src/default-main-program-runner.py
        
        
        DEVELOPMENT STATUS
        ========================================
        
        
        Current version is fully functional, but some syntax and semantics is inconsistent:
        
        * Some instructions allow arguments to span multiple lines, some do not.
        * Most instructions interpret symbol references in arguments, some do not.
        * Support for escapes characters in strings is missing.
        
        Incompatible changes to syntax and semantics may occur in every release until v 1.0.
        
        
        Comments are welcome!
        
        
        Future development
        ------------------------------------
        
        More functionality is needed, smaller and larger.
        Including (but not limited to):
        
        * More string transformers
        * New type "string matcher"
        * Possibility to use "program" values in more places, e.g. in ``[act]``
        * Improved string character escaping
        * Separate sets of environment variables for "action to check" and other processes
        * Possibility to set stdin for processes other than the "action to check"
        * Command to list symbols defined by a test case
        * Variables - corresponding to symbol definitions -
          but for variable values
        * Macros and functions
        * Python library for running cases and suites from within Python as a DSEL
        
        
        AUTHOR
        ========================================
        
        
        Emil Karlén
        
        emil@member.fsf.org
        
        
        THANKS
        ========================================
        
        
        The Python IDE
        `PyCharm
        <https://www.jetbrains.com/pycharm/>`_
        from
        `JetBrains
        <https://www.jetbrains.com/>`_
        has greatly helped the development of this software.
        
        
        DEDICATION
        ========================================
        
        
        Aron Karlén
        
        Tommy Karlsson
        
        Götabergsgatan 10, lägenhet 4
        
Keywords: test case suite check assert script shell console command line program execute sandbox
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Systems Administration
Classifier: Environment :: Console
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
