# Numerical Tests

This folder contains a collection of tests that perform black-box testing on different parts
of the library and later on validate their outputs from a numerical point of view.

In most cases, these tests use predefined, well-known, datasets, and compare the outputs to
the equivalent ones generated in other technologies, such as R or Matlab.

## Types of tests

At the moment these three types of test exist:

* `fit`: Fit bivariate copulas on some data and validate the estimated parameters.
* `cdf`: Compute the CDFs for a set of fixed inputs using a particular theta and validate the
  output values.
* `pdf`: Compute the PDFs for a set of fixed inputs using a particular theta and validate the
  output values.

## Folder Structure

The tests are organized by folders, one for each type of test.

Within each folder, the following elements can be found:

* `test_cases`: A folder containing JSON files that define all the details of each test case.
* `inputs`: A folder containing CSV files that will be used as test inputs. In some cases these
  CSV files have been crafted by hand. In other cases they have been generated using some script.
* `outputs` (optional): A folder containing the expected outputs from the test. This folder only
  exists if the output of the test is large, but is skipped in cases where the output is a single
  parameter.
* `scripts`: A folder containing the scripts used to generate the expected outputs in other
  technologies, such as R or Matlab. These scripts are not executed during the tests and exist
  here only for reproducibility.
* `test_xyz.py`: A script that parses the test case JSON files and executes them. Each test type
  folder contains its own script, which is different from type to type.

## Test Case Format

Each test case is defined in a JSON file that must contain the following entries:

* `metadata`: A sub-document containing information about the test, such as a human-readable
  description, the type of test and pointers to the scripts used to generate the expected
  outputs in other languages.
* `test`: Name of the class that is being tested and specification of the keyword arguments
  that need to be used when creating an instance.
* `test_case_inputs`: A sub-document containing the test inputs. Test inputs can either be
  float values or filenames. Test inputs differ from test type to test type, and knowing
  how to interpret each entry is responsibility of the test script. File paths are always relative
  to the `inputs` folder.
* `expected_output`: A sub-document containing the test outputs that were generated by
  each of the reference languages. Like inputs, the outputs can either be float values or
  paths to CSV files, always relative to the `outputs` folder.
* `settings`: configuration about the test, including how much tolerance needs to be used
  when comparing the outputs to the expected ones.

This is an example of a fit test for the Frank Copula:

```
{
    "metadata": {
        "test_type": "bivariate_fit",
        "dataset_type": "handcrafted",
        "scripts_to_generate": {
            "output": {
                "R": "bivariate_fit_output.R",
                "Matlab": "bivariate_fit_output.m"
            }
        },
        "description": "Test Frank on a highly correlated dataset."
    },
    "test": {
        "class": "copulas.bivariate.frank.Frank",
        "kwargs": {}
    },
    "test_case_inputs": {
        "points": "bivariate_fit_test_case_1_input.csv"
    },
    "expected_output": {
        "R": {
            "theta": 17.0227133037689,
            "tau": 0.787726361443376
        },
        "Matlab": {
            "theta": 17.0227133037689,
            "tau": 0.787726361443376
        }
    },
    "settings": {
        "rtol": 1e-5
    }
}
```
