###
QUICKSTART GUIDE TO THE NEW FLASH TEST

1. Check out a working copy from the flashTest repository:

   If you're on flash, which hosts the subversion repository, just type:

     svn co file:///home/svn/repos/flashTest/trunk flashTest

   If you're logged into another computer:

     svn co svn+ssh://flash.uchicago.edu:/home/svn/repos/flashTest/trunk flashTest

   (if you're using 'svn' as a wrapper for 'svn_real', try the above lines
    with 'svn_real' substituted for 'svn')

   A directory "flashTest" should appear in your current working directory.

2. Edit the "configTemplate" file and re-save it as "config":

   You have to tell FlashTest a few pieces of information, which will be stored
   in a file called "config"

   Go into your new flashTest directory and open the file "configTemplate" in
   your favorite text editor. There are some notes at the top of the file about
   how each of the variables is used. The variables are assigned values at the
   bottom of the file.

   For now, you need only supply a value for the variable "pathToFlash". This
   should be an absolute path to the top-level directory of a working copy of
   the FLASH code. FlashTest sets up, compiles, runs, and tests Flash problems,
   so without FLASH, there can be no FlashTest.

   The dummy value value in the template suggests a location, but you should
   make whatever changes are appopriate.

   When you have filled in the value, re-save the file as "config". DO NOT save
   the file as "configTemplate"! At run-time, FlashTest looks for a file called
   "config". "configTemplate" is only there to get you started.

3. Edit "exeScriptTemplate" (if necessary) and re-save it as "exeScript":

   Eventutally, after you've got FlashTest running and have used it to compile
   a FLASH executable, it will look for a file called "exeScript" which tells
   it how that executable is to be invoked. This will be a call to "mpirun"
   with varying options depending on your platform.

   Open the file "exeScriptTemplate" and check to see if your platform is
   represented. Uncomment the necessary lines, make whatever other changes are
   necessary, and re-save. Do not worry about replacing the values in angle
   brackets. Throughout FlashTest, in the "config" file, in the "exeScript" and
   in the "test.info" files, values in angle brackets will be filled in by
   flashTest.py during run time.

   Once again, be sure to save the new file as "exeScript". Do not save it as
   "exeScriptTemplate".

   *****************************************************************************
   To continue from this point, you must supply FlashTest with a "test.info"
   file. You can either write one yourself or, if FlashTest is already running
   somewhere, you can make use of an already-extant "test.info" file.

   To write your own "test.info" file, please proceed to step 4.
   To use an already-extant "test.info" file, please skip to step 4a.
   *****************************************************************************

4. Write your own "test.info" file:

   FlashTest is almost ready. It knows where FLASH is and it knows how to run a
   FLASH executable. Now you only need to tell FlashTest what kind of a test you
   want to run and what parameters to pass to that test. This can be done either
   directly from the command line or via a "test.info" file. In most cases using
   a "test.info" file will be easier.

   Before writing the "test.info" file, we must think about where it's going to
   live. The nature of a "test.info" file is determined partly by its content
   and partly by where it is.

   The directory called "infoFiles" will hold all the "test.info" files. At this
   point in a new install, this directory should be empty.

   But now try running the script:

     ./createTestDirectories.py

   The output should show that some sub-directories were created inside the
   "infoFiles" directory.

   Now 'cd' into the "infoFiles" directory, and 'ls'. You should see a single
   directory listed, whose name is the first element of your computer's fully-
   qualified hostname as returned by the "hostname" command. For example, if
   your computer's fully-qualified hostname is "flash.uchicago.edu", the name
   of the new directory should be "flash"

   Now 'cd' into this new directory. You are now at the default root of the info
   files tree. Unless 'pathToInfoTree' is otherwise specified in your "config",
   FlashTest will assume that any paths to "test.info" files supplied by the
   user are rooted at this directory.

   Do another 'ls'. You should see a few more directory names, one for every
   test currently supported by FlashTest. Whenever you pass a path to a "test.
   info" file to FlashTest, it MUST begin with one of these names. In general,
   if you invoke flashTest.py with (for example):

     ./flashTest.py MyTest/mySubDiretory

   Three conditions must be satisfied:

       i. There must be a directory under flashTest/infoFiles/[hostname] called
          "MyTest"

      ii. There must be a class "MyTest" defined in "Tests.py"

     iii. There must exist a "test.info" file thus:
            flashTest/infoFiles/[hostname]/MyTest/1d/test.info

   The "createTestDirectories.py" script makes note of all test classes defined
   in "Tests.py" and creates appropriate directories in flashTest/infoFiles. If
   at any point you were to create a new test class, it would be very easy to
   just create an appropriately-named directory at the correct location using
   "mkdir", but you can always use "createTestDirectories.py" again to create
   these directories for you if you prefer. The script will never overwrite
   existing data, but will only create directories for test classes that do not
   yet have representative directories underneath the "infoFiles" directory.
   After using it, you will know that conditions i and ii above will always be
   satisfied.

   One of the directories just created by the script will be "UnitTestUG".
   'cd' into this directory. Once you are inside a directory named for a test
   class, you may create a "test.info" file. However, for now, make one more
   subdirectory first.

     mkdir 1d

   and 'cd' into it. Your current working directory should now be:

     .../flashTest/infoFiles/[hostname]/UnitTestUG/1d

   From this location, start your text editor and write your "test.info" file.
   Like the "config" file, the "test.info" file is a newline-delimited list of
   key value pairs which follow the pattern "key: value".

   For this first "test.info" file, enter the following:   

     setupName: unitTest/UG
     setupOptions: -auto -1d
     numProcs: 1

   Save the file as "test.info". The file must be saved under this name. Then
   navigate back up to the top-level FlashTest directory.

   *****************
   **  IMPORTANT  **
   *****************
   When you reach step 5 of these instructions, please use "UnitTestUG/1d" as
   the value for [shortPathToInfo]

   *****************************************************************************
   From here you can either read about making use of already-extant "test.info"
   files in step 4a, or you can proceed directly to step 5 if you want to run
   FlashTest with the "test.info" file you just created.
   *****************************************************************************

4a. Make use of an already extant "test.info" file

   If FlashTest is already in use on a computer that is accessible to you, and
   you want to run some of the same tests that are already being run elsewhere,
   FlashTest can be configured to look for "test.info" files in any location,
   not just in the "infoFiles" directory of your local working copy.

   To do this, open up your "config" file again, and un-comment the line:

     pathToInfoTree:

   If the "test.info" files that you would like to access are part of the same
   file-system as your working copy of FlashTest, the value of 'pathToInfoTree'
   should be an absolute path to these files.

     pathToInfoTree: /abs/path/to/infoFiles/[some-hostname]

   If the "test.info" files that you would like to access are on a remote
   computer, the value of 'pathToInfoTree' should be the name of the remote
   host, followed by a colon, followed by an absolute path to the files.

     pathToInfoTree: [remote-host]:/abs/path/to/infoFiles/[some-hostname]

   ********************
   Two important notes!
   ********************

   I. In both of these cases, the final element of the path to the "test.info"
   files is the directory *immediately below which* are directories named for
   classes of tests. That is, an "ls" of one of the above paths should produce
   a result like:

     Comparison UnitTestParticles UnitTestUG

   If the setup of FlashTest from which you are using "test.info" files was
   configured in the standard way, your path should end with:

     ...infoFiles/[some-hostname]

   where "some-hostname" is a directory named for the computer for which the
   "test.info" files inside are intended.

   But whether you are working from this same computer or not is not an issue.
   You are free to "steal" the "test.info" files for your own use. The above
   path merely points to the directory containing directories named for
   FlashTest test classes.


   II. If you want to retrieve "test.info" files from a remote computer, your
   ssh-keys must be correctly set up in advance so that FlashTest can connect
   to and read from the appropriate directory without requiring an interactive
   password prompt.

   Once 'pathToInfoTree' is correctly configured, re-save "config" and exit
   your editor.

   **********************
   **  ONE LAST THING  **
   **********************
   In the next step, step 5, the place-holder [shortPathToInfo] should be
   replaced by a relative (short) path to the test.info file in which you're
   interested. The first element of this short path should be the name of a
   FlashTest test class (Comparison, UnitTestUG, etc.) and the last element
   should be the name of the directory containing the "test.info" file.

   For example, you might replace [shortPathToInfo] with:

     Comparison/Sod/pm3/hdf5/parallel/1d

5. Run flashTest.py

   First try invoking the script with no arguments. You'll see some brief notes
   on usage and a list of supported tests. Take a look at what they have to say.

     ./flashTest.py

   Now let's start FlashTest for real. Invoke the script again, followed by the
   "-v" option (so that we can see output from FlashTest as it's running) and
   the relative path from the info file root to "test.info" file we want to use.
   Please refer back to step 4 or 4a for just what this should be.

     ./flashTest.py -v [shortPathToInfo]

   Note that we don't have to supply a "-z" (path to FLASH source) parameter,
   since we've already defined this in our "config".

   As FlashTest runs it generates a logfile. The output you see on your screen
   (thanks to the '-v' option) is a real-time record of what is going into this
   logfile as it is generated. There may be periodic pauses in the output from
   FlashTest, for example while the FLASH executable is being compiled and when
   the executable is being run, but if all goes well you should eventually see
   the output "End of Logfile."

6. Examine the output by hand

   From the top-level FlashTest directory, 'cd' into the newly-created directory
   "output", and 'ls'. There should be a single directory inside, named after
   your computer's hostname. 'cd' into this and 'ls' again.

   You should see again see a single directory, this time in the form of a date.
   This is an "invocation directory" and contains all data generated by a single
   call to the flashTest.py script. 'cd' into it and 'ls' again.

   You should see two files "errors" and "flash_test.log", and a lone directory.
   This is a "build directory", and it contains all data resulting from the
   execution of the instructions in a single "test.info" file. An invocation
   directory can contain an arbitrary number of build directories, but since we
   only pointed flashTest.py to one "test.info" file, it only generated one.

   Please take note of the name of this directory. It is a representation of the
   location of the "test.info" file that originally generated it. This is why
   the location of your "test.info" files is important. They should be located
   inside directories that indicate their nature. This makes it very easy to
   find the output related to a specific "test.info" file and, just as
   importantly, to find the original "test.info" file that generated a certain
   build directory.

   Now "cd" inside this build directory and "ls". You'll see several text-files
   including "setup_output" and "gmake_output", which hold the results of the
   FLASH setup and compilation stages.

   Additionally, assuming FLASH ran without crashing, there will be some number
   of directories (possibly only one) inside the build directory. Each of these
   is called a "run directory" and contains the output from a single run of a
   FLASH executable against a single parfile. If you "cd" into one of these
   directories and "ls", you can examine the output generated by FLASH and by
   the program which ran a test on this output, if any exists.

7. Examine the output in a browser

   If the computer on which you are running FlashTest also hosts a web-server,
   you can immediately use FlashTestView to examine the results in a straight-
   forward and logical way.

   If your local computer does not host a web-server, but you have access to
   a computer that does, it is easy to have your FlashTest results sent to
   this computer so that you can use the FlashTestView tool. This will require
   just one change to the "config" file of FlashTest and one change to the
   "config" file of FlashTestView on the remote computer.

   For now, please ask me (Noel Taylor) for assistance if you wish to do this.


                                    THE END

Nomenclature:

The following terms are used throughout this manual:

An "invocation" represents a single call to the flashTest.py script, which
generates a single directory under output/[hostname]. This directory will have
the form of a date, possibly followed by a suffix. A single invocation can run
FlashTest against an arbitrary number of "test.info" files. Each of these is
called a "build".

A "build" represents the execution of instructions found in a single "test.info"
file, and usually corresponds to the compilation of a single FLASH executable.
Each build will get its own directory under output/[hostname]/[invocation-dir],
and each build can be run against an arbitray number of FLASH parfiles.

A "run" represents the execution of a FLASH executable against a single parfile.
Each run will generate its own directory under:

  output/[hostname]/[invocation-dir]/[build-dir]


FlashTest is run by invoking the script followed by options to FlashTest,
followed by a test-path with optional override-options.
Consider the following:

This runs the test 'SomeTest' with no options:

  ./flashTest SomeTest

This runs the test 'SomeTest' with some options:

  ./flashTest SomeTest setupOptions='-auto -2d'

This runs the test 'SomeTest' with options read from a "test.info" file

  ./flashTest SomeTest/

This runs the test 'SomeTest' with options read from a "test.info" file,
some of which may be overridden by same-named command-line options:

  ./flashTest SomeTest/ setupOptions='-auto -2d'

*************************************
** general options to flashTest.py **
*************************************
-c <file>: use <file> as "config" instead of default
-i <file>: use <file> as source of "test.info" data
-f <file>: read <file> (a "job-file") for list of test-paths
-o <dir> : direct output to <dir>
-s <name>: <name> is the name of this site (host)
-t       : test run (no archiving)
-u       : update FLASH before run
-v       : verbose output (print logfile as it is written)
-z <dir> : FLASH source rooted at <dir>

Each option must be entered with its own preceding dash. For example

  $ ./flashTest.py -t -v -f path/to/job/file

will work, but

  $ ./flashTest.py -tvf path/to/job/file

will not.

-i, -o, and -z options will override 'pathToInfo', 'pathToOutdir', and
'pathToFlash' in the "config" file.

Variables that are set in the "config" file or in a "test.info" file
may be overridden on the command line, or in a job-file, by so-called
"override options" e.g.:

  $ ./flashTest.py Comparison/Sod key1=val1 key2=val2

If an option passed in this way, for example, 'key1', requires no value
the syntax looks like:

  $ ./flashTest.py Comparison/Sod key1= key2=val2
