# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['pyemits',
 'pyemits.common',
 'pyemits.common.io',
 'pyemits.common.typing',
 'pyemits.common.utils',
 'pyemits.core',
 'pyemits.core.evaluation',
 'pyemits.core.ml',
 'pyemits.core.ml.anomaly_detection',
 'pyemits.core.ml.regression',
 'pyemits.core.preprocessing',
 'pyemits.services']

package_data = \
{'': ['*']}

install_requires = \
['SQLAlchemy>=1.4.29,<2.0.0',
 'arrow>=1.2.0,<2.0.0',
 'combo>=0.1.2,<0.2.0',
 'dask>=2021.9.1,<2022.0.0',
 'filterpy>=1.4.5,<2.0.0',
 'jax>=0.2.24,<0.3.0',
 'jaxlib>=0.1.73,<0.2.0',
 'joblib>=1.1.0,<2.0.0',
 'keras>=2.6.0,<3.0.0',
 'lightgbm>=3.3.0,<4.0.0',
 'modin>=0.11.1,<0.12.0',
 'pyarrow>=5.0.0,<6.0.0',
 'pydantic>=1.8.2,<2.0.0',
 'pykalman>=0.9.5,<0.10.0',
 'pyod>=0.9.4,<0.10.0',
 'pytorch-lightning>=1.4.9,<2.0.0',
 'ray>=1.7.0,<2.0.0',
 'scikit-learn>=1.0,<2.0',
 'sqlmodel>=0.0.6,<0.0.7',
 'statsmodels>=0.13.0,<0.14.0',
 'suod>=0.0.8,<0.0.9',
 'tensorflow>=2.6.0,<3.0.0',
 'xgboost>=1.5.0,<2.0.0']

setup_kwargs = {
    'name': 'pyemits',
    'version': '0.1.2.0',
    'description': 'python package for easy manipulation on time series data for quick insight',
    'long_description': '![Project Icon](./assets/icon.png)\n\n# What is Pyemits\n\nPyEmits, a python package for easy manipulation in time-series data.\n\nThe ultimate goal:\n\n> Keep it simple and stupid\n\n> Make everything configurable\n\n> Uniform API for machine learning and deep learning\n\n# Why need Pyemits?\n\nTime-series data is very common in real life.\n\n- Engineering\n- FSI industry (Financial Services Industry)\n- FMCG (Fast Moving Consumer Good)\n\nData scientist\'s work consists of:\n\n- forecasting\n- prediction/simulation\n- data preparation\n- cleansing\n- anomaly detection\n- descriptive data analysis/exploratory data analysis/data profile\n- data processing and ETL pipeline scripts\n\neach new business unit shall build the following wheels again and again\n\n## if you are facing these problems, then Pyemits is fit for you\n\n1. data processing and ETL pipeline\n    1. extraction\n    2. transformation\n        1. cleansing\n        2. feature engineering\n        3. remove outliers\n        4. AI landing for prediction, forecasting\n    3. write it back to database\n2. ml framework\n    1. multiple model training\n    2. multiple model prediction\n    3. kfold validation\n    4. anomaly detection\n    5. forecasting\n    6. develop deep learning model (regression)\n    7. ensemble modelling\n3. exploratory data analysis\n    1. descriptive data analysis\n    2. data profile\n    3. data set comparison\n\ndata scientist need to write different code to develop their model is there a package integrate all ml lib with single\nsimple api? That\'s why I create this project.\n\nThis project is under active development, free to use (Apache 2.0)\nI am happy to see anyone can contribute for more advancement on features\n\n# New feature:\n\n[data processing pipeline](#data-processing-pipeline)\n\n[db connection and manipulation](#io)\n\n# Development Progress\n\n<table>\n    <tr>\n        <td>Features</td>\n        <td>Progress</td>\n        <td>Available at version</td>\n        <td>Notes</td>\n    </tr>\n    <tr>\n        <td>PyOD integration</td>\n        <td>80%</td>\n        <td>0.1.2</td>\n        <td>model parameters config are not yet finished</td>\n    </tr>\n    <tr>\n        <td>XGBoost integration</td>\n        <td>80%</td>\n        <td>0.1.2</td>\n        <td>model parameters config are not yet finished</td>\n    </tr>\n    <tr>\n        <td>LightGBM integration</td>\n        <td>80%</td>\n        <td>0.1.2</td>\n        <td>model parameters config are not yet finished</td>\n    </tr>\n    <tr>\n        <td>Sklearn model integration</td>\n        <td>80%</td>\n        <td>0.1.2</td>\n        <td>model parameters config are not yet finished</td>\n    </tr>\n    <tr>\n        <td>Keras integration</td>\n        <td>100%</td>\n        <td>0.1.2</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>Pytorch_lightning integration</td>\n        <td>100%</td>\n        <td>0.1.2</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>MXnet integration</td>\n        <td>0%</td>\n        <td>tbc</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>DB connection</td>\n        <td>0%</td>\n        <td>tbc</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>aggregation</td>\n        <td>0%</td>\n        <td>0.1.3</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>cleansing</td>\n        <td>0%</td>\n        <td>0.1.3</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>dimensional reduction</td>\n        <td>0%</td>\n        <td>0.1.3</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>Kalman filtering</td>\n        <td>0%</td>\n        <td>0.1.3 or later</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>model evaluation and visualization</td>\n        <td>0%</td>\n        <td>0.1.3 or later</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>data profile for exploration</td>\n        <td>20%</td>\n        <td>0.1.3 or later</td>\n        <td>finished data statistics only</td>\n    </tr>\n    <tr>\n        <td>forecast at scale</td>\n        <td>100%</td>\n        <td>0.1.2</td>\n        <td>see preprocessing.scaling.py</td>\n    </tr>\n\n</table>\n\n# Release Update\n\n<table>\n    <tr>\n        <td>Version</td>\n        <td>Features</td>\n        <td>Notes</td>\n    </tr>\n    <tr>\n        <td>0.1</td>\n        <td>initialization of project</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>0.1.1</td>\n        <td>RegTrainer/ParallelTrainer/KFoldCV</td>\n        <td></td>\n    </tr>\n    <tr>\n        <td>0.1.2</td>\n        <td>PyOD/Keras/Pytorch_lightning/scaling/splitting</td>\n        <td></td>\n    </tr>\n\n</table>\n\n# Install\n\n```shell\npip install pyemits\n```\n\n# Features highlight\n\n> scikit-learn API style\n>\n> inherit the design concept of pyecharts, "everything is configurable"\n>\n> highly flexible configuration items, can easily integrate with existing model\n>\n> easily integrate to SaaS product for product proof of concept\n\n# Easy training\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegTrainer, RegressionDataModel\n\nX = np.random.randint(1, 100, size=(1000, 10))\ny = np.random.randint(1, 100, size=(1000, 1))\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = RegTrainer([\'XGBoost\'], [None], raw_data_model)\ntrainer.fit()\n\n```\n\n# Accept neural network as model\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegTrainer, RegressionDataModel\nfrom pyemits.core.ml.regression.nn import KerasWrapper\n\nX = np.random.randint(1, 100, size=(1000, 10, 10))\ny = np.random.randint(1, 100, size=(1000, 4))\n\nkeras_lstm_model = KerasWrapper.from_simple_lstm_model((10, 10), 4)\nraw_data_model = RegressionDataModel(X, y)\ntrainer = RegTrainer([keras_lstm_model], [None], raw_data_model)\ntrainer.fit()\n```\n\nalso keep flexibility on customized model\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegTrainer, RegressionDataModel\nfrom pyemits.core.ml.regression.nn import KerasWrapper\n\nX = np.random.randint(1, 100, size=(1000, 10, 10))\ny = np.random.randint(1, 100, size=(1000, 4))\n\nfrom keras.layers import Dense, Dropout, LSTM\nfrom keras import Sequential\n\nmodel = Sequential()\nmodel.add(LSTM(128,\n               activation=\'softmax\',\n               input_shape=(10, 10),\n               ))\nmodel.add(Dropout(0.1))\nmodel.add(Dense(4))\nmodel.compile(loss=\'mse\', optimizer=\'adam\', metrics=[\'mse\'])\n\nkeras_lstm_model = KerasWrapper(model, nickname=\'LSTM\')\nraw_data_model = RegressionDataModel(X, y)\ntrainer = RegTrainer([keras_lstm_model], [None], raw_data_model)\ntrainer.fit()\n```\n\nor attach it in algo config\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegTrainer, RegressionDataModel\nfrom pyemits.core.ml.regression.nn import KerasWrapper\nfrom pyemits.common.config_model import KerasSequentialConfig\n\nX = np.random.randint(1, 100, size=(1000, 10, 10))\ny = np.random.randint(1, 100, size=(1000, 4))\n\nfrom keras.layers import Dense, Dropout, LSTM\nfrom keras import Sequential\n\nkeras_lstm_model = KerasWrapper(nickname=\'LSTM\')\nconfig = KerasSequentialConfig(layer=[LSTM(128,\n                                           activation=\'softmax\',\n                                           input_shape=(10, 10),\n                                           ),\n                                      Dropout(0.1),\n                                      Dense(4)],\n                               compile=dict(loss=\'mse\', optimizer=\'adam\', metrics=[\'mse\']))\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = RegTrainer([keras_lstm_model],\n                     [config],\n                     raw_data_model,\n                     {\'fit_config\': [dict(epochs=10, batch_size=32)]})\ntrainer.fit()\n```\n\nPyTorch, MXNet under development you can leave me a message if you want to contribute\n\n# MultiOutput training\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegressionDataModel, MultiOutputRegTrainer\nfrom pyemits.core.preprocessing.splitting import SlidingWindowSplitter\n\nX = np.random.randint(1, 100, size=(10000, 1))\ny = np.random.randint(1, 100, size=(10000, 1))\n\n# when use auto-regressive like MultiOutput, pls set ravel = True\n# ravel = False, when you are using LSTM which support multiple dimension\nsplitter = SlidingWindowSplitter(24, 24, ravel=True)\nX, y = splitter.split(X, y)\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = MultiOutputRegTrainer([\'XGBoost\'], [None], raw_data_model)\ntrainer.fit()\n```\n\n# Parallel training\n\n- provide fast training using parallel job\n- use RegTrainer as base, but add Parallel running\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegressionDataModel, ParallelRegTrainer\n\nX = np.random.randint(1, 100, size=(10000, 1))\ny = np.random.randint(1, 100, size=(10000, 1))\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = ParallelRegTrainer([\'XGBoost\', \'LightGBM\'], [None, None], raw_data_model)\ntrainer.fit()\n```\n\nor you can use RegTrainer for multiple model, but it is not in Parallel job\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegressionDataModel, RegTrainer\n\nX = np.random.randint(1, 100, size=(10000, 1))\ny = np.random.randint(1, 100, size=(10000, 1))\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = RegTrainer([\'XGBoost\', \'LightGBM\'], [None, None], raw_data_model)\ntrainer.fit()\n```\n\n# KFold training\n\n- KFoldConfig is global config, will apply to all\n\n```python\nimport numpy as np\n\nfrom pyemits.core.ml.regression.trainer import RegressionDataModel, KFoldCVTrainer\nfrom pyemits.common.config_model import KFoldConfig\n\nX = np.random.randint(1, 100, size=(10000, 1))\ny = np.random.randint(1, 100, size=(10000, 1))\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = KFoldCVTrainer([\'XGBoost\', \'LightGBM\'], [None, None], raw_data_model,\n                         {\'kfold_config\': KFoldConfig(n_splits=10)})\ntrainer.fit()\n```\n\n# Easy prediction\n\n```python\nimport numpy as np\nfrom pyemits.core.ml.regression.trainer import RegressionDataModel, RegTrainer\nfrom pyemits.core.ml.regression.predictor import RegPredictor\n\nX = np.random.randint(1, 100, size=(10000, 1))\ny = np.random.randint(1, 100, size=(10000, 1))\n\nraw_data_model = RegressionDataModel(X, y)\ntrainer = RegTrainer([\'XGBoost\', \'LightGBM\'], [None, None], raw_data_model)\ntrainer.fit()\n\npredictor = RegPredictor(trainer.clf_models, \'RegTrainer\')\npredictor.predict(RegressionDataModel(X))\n\n```\n\n# Forecast at scale\n\n- see examples: [forecast at scale.ipynb](./examples/forecast%20at%20scale.ipynb)\n\n# Data Model\n\n```python\nfrom pyemits.common.data_model import RegressionDataModel\nimport numpy as np\n\nX = np.random.randint(1, 100, size=(1000, 10, 10))\ny = np.random.randint(1, 100, size=(1000, 1))\n\ndata_model = RegressionDataModel(X, y)\n\n```\n\ndirectly write an attribute to the data model\n\n```python\ndata_model._update_attributes(\'X_shape\', (1000, 10, 10))\ndata_model.X_shape\n>> > (1000, 10, 10)\n```\n\nwrite something to the meta data\n\n```python\ndata_model.add_meta_data(\'dimension\', (1000, 10, 10))\ndata_model.meta_data\n>> > {\'dimension\': (1000, 10, 10)}\n```\n\n# Anomaly detection (partial finished)\n\n- see: [anomaly detection](./examples/anomaly%20detector.ipynb)\n- root cause analyzer (under development)\n- Kalman filter (under development)\n\n```python\nfrom pyemits.core.ml.anomaly_detection.predictor import AnomalyPredictor\nfrom pyemits.core.ml.anomaly_detection.trainer import AnomalyTrainer, PyodWrapper\nfrom pyemits.common.data_model import AnomalyDataModel\nfrom pyemits.common.config_model import PyodIforestConfig\nfrom pyod.models.iforest import IForest\nfrom pyod.utils import generate_data\n\ncontamination = 0.1  # percentage of outliers\nn_train = 1000  # number of training points\nn_test = 200  # number of testing points\n\nX_train, y_train, X_test, y_test = generate_data(\n    n_train=n_train, n_test=n_test, contamination=contamination)\n\n# highly flexible model config, accept str, PyodWrapper with/without initialized model\n# either one\ntrainer = AnomalyTrainer([\'IForest\', PyodWrapper(IForest()), PyodWrapper(IForest), \'IForest\', \'IForest\', \'IForest\'],\n                         None, AnomalyDataModel(X_train))\ntrainer = AnomalyTrainer([PyodWrapper(IForest(contamination=0.05)), PyodWrapper(IForest)],\n                         [None, PyodIforestConfig(contamination=0.05)], AnomalyDataModel(X_train))\ntrainer.fit()\n\n# option 1\npredictor = AnomalyPredictor(trainer.clf_models)\n\n# option 2\npredictor = AnomalyPredictor(trainer.clf_models,\n                             other_config={\'standard_scaler\': predictor.misc_container[\'standard_scaler\']})\n\n# option 3\npredictor = AnomalyPredictor(trainer.clf_models,\n                             other_config={\'standard_scaler\': predictor.misc_container[\'standard_scaler\'],\n                                           \'combination_config\': {\'n_buckets\': 5}}, combination_method=\'moa\')\n\npredictor.predict(AnomalyDataModel(X_test))\n\n```\n\n# Data processing pipeline\n\nit features in the following:\n\n- easy configuration\n    - register steps, tasks in data processing pipeline\n- log data result in each tasks, each steps\n- record the flow of pipeline, from steps to work (from marco to micro)\n\nyou can embed other function features in the task, but parameter: "data" is required to be passed in\n\ne.g. add email notification, add log, upload to database etc...\n\n```python\n\nfrom pyemits.core.preprocessing.pipeline import DataNode, NumpyDataNode, PandasDataFrameDataNode, PandasSeriesDataNode,\n\nPipeline, Step, Task\nimport pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame(np.random.random(size=(20, 20)))\n\ndn = PandasDataFrameDataNode.from_pandas(df)\n\n\ndef sum_each_col(data, a=1, b=2):\n    return data.sum()\n\n\ndef sum_series(data):\n    return np.array([data.sum()])\n```\n\ntask registration and arguments registration\n\n```python\ntask_a = Task(sum_each_col)\ntask_a.register_args(a=10, b=10)\ntask_b = Task(sum_series)\n```\n\npipeline register step and execute\n\n```python\npipeline = Pipeline()\n\nstep_a = Step(\'step_a\', [task_a], \'\')\nstep_b = Step(\'step_b\', [task_b], \'\')\n\npipeline.register_step(step_a)\npipeline.register_step(step_b)\npipeline.execute(dn)\n```\n\nknow the steps and its tasks from start to end\n\n```python\npipeline.get_step_task_mapping()\n>> > {0: (\'test\', [\'sum_each_col\']), 1: (\'test1\', [\'sum_series\'])}\n```\n\nknow the snapshot result in each steps, each tasks, friendly to data scientist for debugging\n\n```markdown\npipeline.get_pipeline_snapshot_res(step_id=1,tasks_id=0)\n> > > array([197.70351007])\n```\n\n# Evaluation (under development)\n\n- see module: [evaluation](pyemits/core/evaluation)\n- backtesting\n- model evaluation\n\n# Ensemble (under development)\n\n- blending\n- stacking\n- voting\n- by combo package\n    - moa\n    - aom\n    - average\n    - median\n    - maximization\n\n# IO\n\n- db connection and manipulation\n\n```python\nfrom pyemits.common.io.db import DBConnectionBase\n\ndb = DBConnectionBase.from_full_db_path(\'sqlite:///test.db\')\n\ndb.execute(\'CREATE TABLE abcc(c1 int, c2 int, c3 int)\')\n\ndb.execute(\'INSERT INTO abcc(c1, c2, c3) VALUES (10, 10, 10)\', change_commit=True)\n\ndb.execute(\'SELECT * FROM abcc\', fetch=10)\ndb.execute(\'SELECT * FROM abcc\', fetch=\'all\')\n\nschemas = db.get_schemas()\n\nschemas[\'main\'][\'abcc\']\n>> [{\'name\': \'c1\',\n     \'type\': INTEGER(),\n     \'nullable\': True,\n     \'default\': None,\n     \'autoincrement\': \'auto\',\n     \'primary_key\': 0},\n    {\'name\': \'c2\',\n     \'type\': INTEGER(),\n     \'nullable\': True,\n     \'default\': None,\n     \'autoincrement\': \'auto\',\n     \'primary_key\': 0},\n    {\'name\': \'c3\',\n     \'type\': INTEGER(),\n     \'nullable\': True,\n     \'default\': None,\n     \'autoincrement\': \'auto\',\n     \'primary_key\': 0}]\n```\n\n- local\n\n# dashboard ???\n\n# other miscellaneous feature\n\n- continuous evaluation\n- aggregation\n- dimensional reduction\n- data profile (intensive data overview)\n\n# to be confirmed\n\n....\n\n# References\n\nthe following libraries gave me some idea/insight\n\n1. greykit\n    1. changepoint detection\n    2. model summary\n    3. seaonality\n2. pytorch-forecasting\n3. darts\n4. pyaf\n5. orbit\n6. kats/prophets by facebook\n7. sktime\n8. gluon ts\n9. tslearn\n10. pyts\n11. luminaries\n12. tods\n13. autots\n14. pyodds\n15. scikit-hts\n\n\n',
    'author': 'thompson0012',
    'author_email': '51963680+thompson0012@users.noreply.github.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/thompson0012/PyEmits',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.8.3,<3.10',
}


setup(**setup_kwargs)
