Metadata-Version: 2.2
Name: shub-workflow
Version: 1.14.20.7
Summary: Workflow manager for Zyte ScrapyCloud tasks.
Home-page: https://github.com/scrapinghub/shub-workflow
Maintainer: Scrapinghub
License: BSD
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=3.12
Requires-Dist: scrapinghub[msgpack]>=2.3.1
Requires-Dist: jinja2>=2.7.3
Requires-Dist: sqlitedict==2.1.0
Requires-Dist: boto3>=1.9.92
Requires-Dist: bloom-filter2==2.0.0
Requires-Dist: collection-scanner
Requires-Dist: tenacity
Requires-Dist: typing-extensions
Requires-Dist: scrapy
Requires-Dist: prettytable
Provides-Extra: with-s3-tools
Requires-Dist: s3fs>=0.4.0; extra == "with-s3-tools"
Provides-Extra: with-gcs-tools
Requires-Dist: google-cloud-storage>=1.38.0; extra == "with-gcs-tools"
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: maintainer
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary

A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.

# Installation

```
pip install shub-workflow
```

If you want to support s3 tools:

```
pip install shub-workflow[with-s3-tools]
```

For google cloud storage tools support:

```
pip install shub-workflow[with-gcs-tools]
```

# Usage

Check [Project Wiki](https://github.com/scrapinghub/shub-workflow/wiki) for documentation. You can also see code tests for lots of examples of usage.

# Note

The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used
for setting up a development environment for shub-workflow library development and testing.


# For developers

For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:

```
> pipenv install --dev
> cp pre-commit .git/hooks/
```

for installing the environment, and:

```
> pipenv shell
```

for initiating it.

There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do `git commit` (provided
you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.

```
> ./lint.sh
```
