Metadata-Version: 2.1
Name: elastic-tools
Version: 1.0.2
Summary: UNKNOWN
Home-page: https://github.com/petricore-tech/elastictools.git
License: UNKNOWN
Platform: UNKNOWN
Description-Content-Type: text/markdown
Requires-Dist: aiofiles (==0.6.0)
Requires-Dist: aiohttp (==3.7.4.post0)
Requires-Dist: astroid (==2.5.6)
Requires-Dist: async-timeout (==3.0.1)
Requires-Dist: attrs (==20.3.0)
Requires-Dist: bleach (==3.3.0)
Requires-Dist: build (==0.4.0)
Requires-Dist: certifi (==2020.12.5)
Requires-Dist: chardet (==4.0.0)
Requires-Dist: colorama (==0.4.4)
Requires-Dist: docutils (==0.17.1)
Requires-Dist: elastic-tools (==1.0.1)
Requires-Dist: elasticsearch (==7.12.1)
Requires-Dist: idna (==3.1)
Requires-Dist: importlib-metadata (==4.0.1)
Requires-Dist: isort (==5.8.0)
Requires-Dist: keyring (==23.0.1)
Requires-Dist: lazy-object-proxy (==1.6.0)
Requires-Dist: mccabe (==0.6.1)
Requires-Dist: motor (==2.4.0)
Requires-Dist: multidict (==5.1.0)
Requires-Dist: mypy-extensions (==0.4.3)
Requires-Dist: packaging (==20.9)
Requires-Dist: pep517 (==0.10.0)
Requires-Dist: pkginfo (==1.7.0)
Requires-Dist: Pygments (==2.9.0)
Requires-Dist: pylint (==2.8.2)
Requires-Dist: pymongo (==3.11.4)
Requires-Dist: pyparsing (==2.4.7)
Requires-Dist: readme-renderer (==29.0)
Requires-Dist: requests (==2.25.1)
Requires-Dist: requests-toolbelt (==0.9.1)
Requires-Dist: rfc3986 (==1.5.0)
Requires-Dist: six (==1.16.0)
Requires-Dist: toml (==0.10.2)
Requires-Dist: tqdm (==4.60.0)
Requires-Dist: twine (==3.4.1)
Requires-Dist: typed-ast (==1.4.3)
Requires-Dist: typing-extensions (==3.10.0.0)
Requires-Dist: ujson (==4.0.2)
Requires-Dist: urllib3 (==1.26.4)
Requires-Dist: webencodings (==0.5.1)
Requires-Dist: websockets (==9.0.2)
Requires-Dist: wrapt (==1.12.1)
Requires-Dist: yarl (==1.6.3)
Requires-Dist: zipp (==3.4.1)

# ElasticTools - high-speed utilities for ElasticSearch data manipulations

`ElasticTools` is a python package developed as a handy utility for cases when it comes to **streaming**, **manipulation** or **processing** of any sort of data stored (or prone to be stored) within ElasticSearch.

The key features are:

- **Highly performant**. Written purely in python with a dense usage of concurrency thanks to AsyncIO.

- **Intuitive**. Though almost every commandline argument has a detailed explanation of *what* is it and *why* we need it, one can intuitively answer those questions.

## Installation

### Create and activate a virtual environment

```bash
python3 -m venv venv
source venv/bin/activate
```

### Install the package using **pip**

```bash
pip install elastic-tools
```

## Usage

### **1. Dump Elasticsearch index**

```bash
elasticdump --elastic_address <ES_ADDRESS> --index <INDEX_NAME> --output_dir <TARGET_DIRECTORY> --chunk_size <CHUNK_SIZE>
```

*Example call:*

```bash
elasticdump --elastic_address localhost:9200 --index products --output_dir ./dump/ --chunk_size 500
```

### **2. Restore Elasticsearch index**

```bash
elasticload --elastic_address <ES_ADDRESS> --index <INDEX_NAME> --input_dir <DIR_WITH_DUMPED_DATA> --chunk_size <CHUNK_SIZE> --connection_pool_size <PARALLEL_CONNECTIONS_COUNT> --mode <MODE>
```

*Example call:*

```bash
elasticload --elastic_address localhost:9200 --index products --input_dir ./dump/ --chunk_size 1000
```

Based on the situation you may want to load *only data* or *both data and settings*. For this purposes `mode` argument is very helpful. Run `mongo2elastic -h` for more info.

### **3. Stream data from MongoDB to Elasticsearch**

```bash
mongo2elastic --mongo_address <MONGO_ADDRESS> --mongo_db <MONGO_DATABASE_NAME> --mongo_collection <MONGO_COLLECTION_NAME> --elastic_address <ES_ADDRESS> --elastic_index <ES_INDEX_NAME> --batch_size <BATCH_SIZE> --connection_pool_size <CONNECTION_POOL_SIZE> --mode <MODE>
```

*Example call:*

```bash
mongo2elastic --mongo_address localhost:27017 --mongo_db store --mongo_collection products --elastic_address localhost:9200 --elastic_index products --batch_size 500 --connection_pool_size 5 --mode default
```

Based on the situation you may want to stream *only data*, *only updates* or *both*. For this purposes `mode` argument is very helpful. Run `mongo2elastic -h` for more info.


