AdvancedAnalytics
===================

A collection of python modules, classes and methods for simplifying the use of machine learning solutions.  **AdvancedAnalytics** provides easy access to advanced tools in **Sci-Learn**, **NLTK** and other machine learning packages.  **AdvancedAnalytics** was developed to simplify learning python from the book *The Art and Science of Data Analytics*.

Description
===========

From a high level view, building machine learning applications typically proceeds through three stages:

    1. Data Preprocessing
    2. Modeling or Analytics
    3. Postprocessing

The classes and methods in **AdvancedAnalytics** primarily support the first and last stages of machine learning applications. 

Data scientists report they spend 80% of their total effort in first and last stages. The first stage, *data preprocessing*, is concerned with preparing the data for analysis.  This includes:

    1. identifying and correcting outliers, 
    2. imputing missing values, and 
    3. encoding data. 

The last stage, *solution postprocessing*, involves developing graphic summaries of the solution, and metrics for evaluating the quality of the solution.

Documentation and Examples
============================

The API and documentation for all classes and examples are available at https://github.com/tandonneur/AdvancedAnalytics/. 

Usage
=====

Currently the most popular usage is for supporting solutions developed using these advanced machine learning packages:

    * Sci-Learn
    * StatsModels
    * NLTK

The intention is to expand this list to other packages.  This is a simple example for linear regression that uses the data map structure to preprocess data:

.. code-block:: python

    from AdvancedAnalytics.ReplaceImputeEncode import DT
    from AdvancedAnalytics.ReplaceImputeEncode import ReplaceImputeEncode
    from AdvancedAnalytics.Tree import tree_regressor
    from sklearn.tree import DecisionTreeRegressor, export_graphviz 
    # Data Map Using DT, Data Types
    data_map = {
        "Salary":         [DT.Interval, (20000.0, 2000000.0)],
        "Department":     [DT.Nominal, ("HR", "Sales", "Marketing")] 
        "Classification": [DT.Nominal, (1, 2, 3, 4, 5)]
        "Years":          [DT.Interval, (18, 60)] }
    # Preprocess data from data frame df
    rie = ReplaceImputeEncode(data_map=data_map, interval_scaling=None,
                              nominal_encoding= "SAS", drop=True)
    encoded_df = rie.fit_transform(df)
    y = encoded_df["Salary"]
    X = encoded_df.drop("Salary", axis=1)
    dt = DecisionTreeRegressor(criterion= "gini", max_depth=4,
                                min_samples_split=5, min_samples_leaf=5)
    dt = dt.fit(X,y)
    tree_regressor.display_importance(dt, encoded_df.columns)
    tree_regressor.display_metrics(dt, X, y)

Current Modules and Classes
=============================

ReplaceImputeEncode
    Classes for Data Preprocessing
        * DT defines new data types used in the data dictionary
        * ReplaceImputeEncode a class for data preprocessing

Regression
    Classes for Linear and Logistic Regression
        * linreg support for linear regressino
        * logreg support for logistic regression
        * stepwise a variable selection class

Tree
    Classes for Decision Tree Solutions
        * tree_regressor support for regressor decision trees
        * tree_classifier support for classification decision trees

Forest
    Classes for Random Forests
        * forest_regressor support for regressor random forests
        * forest_classifier support for classification random forests

NeuralNetwork
    Classes for Neural Networks
        * nn_regressor support for regressor neural networks
        * nn_classifier support for classification neural networks

Text
    Classes for Text Analytics
        * text_analysis support for topic analysis
        * text_plot for word clouds
        * sentiment_analysis support for sentiment analysis

Internet
    Classes for Internet Applications
        * scrape support for web scrapping
        * metrics a class for solution metrics

Installation and Dependencies
=============================

**AdvancedAnalytics** is designed to work on any operating system running python 3.  It can be installed using **pip** or **conda**.

.. code-block:: python

    pip install AdvancedAnalytics
    # or
    conda install -c dr.jones AdvancedAnalytics

General Dependencies
    There are dependencies.  Most classes import one or more modules from    
    **Sci-Learn**, referenced as *sklearn* in module imports, and 
    **StatsModels**.  These are both installed with the current version
    of **anaconda**.

Installed with AdvancedAnalytics
    Most packages used by **AdvancedAnalytics** are automatically 
    installed with its installation.  These consist of the following 
    packages.

        * statsmodels
        * scikit-learn
        * scikit-image
        * nltk
        * pydotplus

Other Dependencies
    The *Tree* and *Forest* modules plot decision trees and importance
    metrics using **pydotplus** and the **graphviz** packages.  These
    should also be automatically installed with **AdvancedAnalytics**.

    However, the **graphviz** install is sometimes not fully complete 
    with the conda install.  It may require an additional pip install.

    .. code-block:: python

        pip install graphviz

Text Analytics Dependencies
    The *TextAnalytics* module uses the **NLTK**, **Sci-Learn**, and 
    **wordcloud** packages.  Usually these are also automatically 
    installed automatically with **AdvancedAnalytics**.  You can verify 
    they are installed using the following commands.

    .. code-block:: python

        conda list nltk
        conda list sci-learn
        conda list wordcloud

    However, when the **NLTK** package is installed, it does not 
    install the data used by the package.  In order to load the
    **NLTK** data run the following code once before using the 
    *TextAnalytics* module.

    .. code-block:: python

        #The following NLTK commands should be run once
        nltk.download("punkt")
        nltk.download("averaged_preceptron_tagger")
        nltk.download("stopwords")
        nltk.download("wordnet")

    The **wordcloud** package also uses a little know package
    **tinysegmenter** version 0.3.  Run the following code to ensure
    it is installed.

    .. code-block:: python

        conda install -c conda-forge tinysegmenter==0.3
        # or
        pip install tinysegmenter==0.3

Internet Dependencies
    The *Internet* module contains a class *scrape* which has some   
    functions for scraping newsfeeds. Some of these use the 
    **newspaper3k** package.  It should be automatically installed with 
    **AdvancedAnalytics**.

    However, it also uses the package **newsapi-python**, which is not 
    automatically installed.  If you intended to use this news scraping
    scraping tool, it is necessary to install the package using the 
    following code:

    .. code-block:: python

        conda install -c conda-forge newsapi
        # or
        pip install newsapi

    In addition, the newsapi service is sponsored by a commercial company
    www.newsapi.com.  You will need to register with them to obtain an 
    *API* key required to access this service.  This is free of charge 
    for developers, but there is a fee if *newsapi* is used to broadcast 
    news with an application or at a website.

Code of Conduct
---------------

Everyone interacting in the AdvancedAnalytics project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct: https://www.pypa.io/en/latest/code-of-conduct/ .


