Metadata-Version: 2.1
Name: satip
Version: 2.11.24
Summary: Satip provides the functionality necessary for
Author: Jacob Bieker, Ayrton Bourn, Jack Kelly
Author-email: info@openclimatefix.org
License: MIT
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: affine==2.4.0
Requires-Dist: aiobotocore==2.8.0
Requires-Dist: aiohttp==3.9.1
Requires-Dist: aioitertools==0.11.0
Requires-Dist: aiosignal==1.3.1
Requires-Dist: alembic==1.13.0
Requires-Dist: appdirs==1.4.4
Requires-Dist: asciitree==0.3.3
Requires-Dist: attrs==23.1.0
Requires-Dist: blosc2==2.3.2
Requires-Dist: bokeh==3.3.2
Requires-Dist: botocore==1.33.1
Requires-Dist: Brotli==1.1.0
Requires-Dist: cached-property==1.5.2
Requires-Dist: Cartopy==0.22.0
Requires-Dist: certifi==2023.11.17
Requires-Dist: cftime==1.6.3
Requires-Dist: charset-normalizer==3.3.2
Requires-Dist: click==8.1.7
Requires-Dist: click-plugins==1.1.1
Requires-Dist: cligj==0.7.2
Requires-Dist: cloudpickle==3.0.0
Requires-Dist: colorama==0.4.6
Requires-Dist: configobj==5.0.8
Requires-Dist: contourpy==1.2.0
Requires-Dist: cycler==0.12.1
Requires-Dist: cytoolz==0.12.2
Requires-Dist: dask==2023.12.0
Requires-Dist: distributed==2023.12.0
Requires-Dist: docutils==0.20.1
Requires-Dist: donfig==0.8.1.post0
Requires-Dist: eumdac==2.1.0
Requires-Dist: fasteners==0.17.3
Requires-Dist: fonttools==4.46.0
Requires-Dist: freezegun==1.3.1
Requires-Dist: frozenlist==1.4.0
Requires-Dist: fsspec==2023.12.1
Requires-Dist: greenlet==3.0.1
Requires-Dist: h5netcdf==1.3.0
Requires-Dist: h5py==3.10.0
Requires-Dist: idna==3.6
Requires-Dist: imagecodecs==2023.9.18
Requires-Dist: importlib-metadata==7.0.0
Requires-Dist: iniconfig==2.0.0
Requires-Dist: Jinja2==3.1.2
Requires-Dist: jmespath==1.0.1
Requires-Dist: kiwisolver==1.4.5
Requires-Dist: locket==1.0.0
Requires-Dist: lz4==4.3.2
Requires-Dist: Mako==1.3.0
Requires-Dist: MarkupSafe==2.1.3
Requires-Dist: matplotlib==3.8.2
Requires-Dist: msgpack==1.0.7
Requires-Dist: multidict==6.0.4
Requires-Dist: munkres==1.1.4
Requires-Dist: ndindex==1.7
Requires-Dist: netCDF4==1.6.5
Requires-Dist: nowcasting-datamodel==1.5.26
Requires-Dist: numcodecs==0.12.1
Requires-Dist: numpy==1.23.2
Requires-Dist: ocf-blosc2==0.0.4
Requires-Dist: packaging==23.2
Requires-Dist: pandas==2.1.3
Requires-Dist: partd==1.4.1
Requires-Dist: Pillow==10.1.0
Requires-Dist: pip>=23.3.1
Requires-Dist: platformdirs==4.1.0
Requires-Dist: pluggy==1.3.0
Requires-Dist: pooch==1.8.0
Requires-Dist: psutil>=5.9.5
Requires-Dist: psycopg2-binary==2.9.9
Requires-Dist: py-cpuinfo==9.0.0
Requires-Dist: pyarrow==14.0.1
Requires-Dist: pyarrow-hotfix==0.6
Requires-Dist: pydantic==1.10.10
Requires-Dist: pykdtree==1.3.10
Requires-Dist: pyorbital==1.8.0
Requires-Dist: pyparsing==3.1.1
Requires-Dist: pyproj==3.6.1
Requires-Dist: pyresample==1.27.1
Requires-Dist: pyshp==2.3.1
Requires-Dist: PySocks==1.7.1
Requires-Dist: pyspectral==0.13.0
Requires-Dist: pytest==7.4.3
Requires-Dist: python-dateutil==2.8.2
Requires-Dist: python-geotiepoints==1.7.1
Requires-Dist: pytz==2023.3.post1
Requires-Dist: PyYAML==6.0.1
Requires-Dist: rasterio==1.3.9
Requires-Dist: requests>=2.31.0
Requires-Dist: s3fs==2023.12.1
Requires-Dist: satpy==0.38.0
Requires-Dist: SciPy==1.11.4
Requires-Dist: setuptools==68.2.2
Requires-Dist: shapely==2.0.2
Requires-Dist: six==1.16.0
Requires-Dist: snuggs==1.4.7
Requires-Dist: sortedcontainers==2.4.0
Requires-Dist: SQLAlchemy==2.0.23
Requires-Dist: structlog==23.2.0
Requires-Dist: tblib==2.0.0
Requires-Dist: toolz==0.12.0
Requires-Dist: tornado==6.3.3
Requires-Dist: tqdm==4.66.1
Requires-Dist: trollimage==1.22.2
Requires-Dist: trollsift==0.5.1
Requires-Dist: typing_extensions==4.8.0
Requires-Dist: tzdata==2023.3
Requires-Dist: urllib3>=1.26.18
Requires-Dist: wheel==0.42.0
Requires-Dist: wrapt==1.16.0
Requires-Dist: xarray==2023.11.0
Requires-Dist: xyzservices==2023.10.1
Requires-Dist: yarl==1.9.3
Requires-Dist: zarr==2.16.1
Requires-Dist: zict==3.0.0
Requires-Dist: zipp==3.17.0

# Satip
<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
[![All Contributors](https://img.shields.io/badge/all_contributors-6-orange.svg?style=flat-square)](#contributors-)
<!-- ALL-CONTRIBUTORS-BADGE:END -->

[![PyPI version](https://badge.fury.io/py/satip.svg)](https://badge.fury.io/py/satip)

[![codecov](https://codecov.io/gh/openclimatefix/Satip/branch/main/graph/badge.svg?token=GTQDR2ZZ2S)](https://codecov.io/gh/openclimatefix/Satip)

> Satip is a library for <b>sat</b>ellite <b>i</b>mage <b>p</b>rocessing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data

<br>


### Installation

To install the `satip` library please run:

```bash
pip install satip
```

Or if you're working in the development environment you can run the following from the directory root:

```bash
pip install -e .
```

#### Conda

Or, if you want to use `conda` from a cloned Satip repository:

```bash
conda env create -f environment.yml
conda activate satip
pip install -e .
```

If you plan to work on the development of Satip then also consider installing these development tools:

```bash
conda install pytest flake8 jedi mypy black pre-commit
pre-commit install
```

### Development Environment

In order to contribute:
- it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.
- At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.

## Operation

### Getting your own API key

In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:

1. Go to https://eoportal.eumetsat.int and register an account.
2. You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
3. Please make sure that you added the key and secret to your user's environment variables.

### Downloading EUMETSAT Data

The following command will download the last 2 hours of RSS imagery into NetCDF files at the specified location

```bash
python satip/app.py --api-key=<EUMETSAT API Key> --api-secret=<EUMETSAT API Secret> --save-dir="/path/to/saving/files/" --history="2 hours"
```

To download more historical data, the command below will download the native files, compress with bz2, and save into a subdirectory.

```bash
python satip/get_raw_eumetsat_data.py --user-key=<EUMETSAT API Key> --user-secret=<EUMETSAT API Secret>
```

### Converting Native files to Zarr
`scripts/convert_native_to_zarr.py` converts EUMETSAT `.nat` files to Zarr datasets, using very mild lossy [JPEG-XL](https://en.wikipedia.org/wiki/JPEG_XL) compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent `bz2` compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range `[0, 1]`. NaNs are encoded as the value `0.025`. All "real" values are in the range `[0.075, 1]`. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see [this comment for more details](https://github.com/openclimatefix/Satip/issues/67#issuecomment-1036456502)). Use `satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs` to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range `[0, 1]`.


## Running in Production

The live service uses `app.py` as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.

`--api-key` or `API_KEY` is the EUMETSAT API key

`--api-secret` or `API_SECRET` is the EUMETSAT API secret

`--save-dir` or `SAVE_DIR` is the top level directory to save the output files, a `latest` subfolder will be added to that directory to contain the latest data

`--history` or `HISTORY` is the amount of history timesteps to use in the `latest.zarr` files

`--db-url` or `DB_URL` is the URL to the database to save to when a run has finished

`--use-rescaler` or `USE_RESCALER` tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.

## Testing

To run tests, simply run ```pytest .``` from the root of the repository. To generate the test plots, run ```python scripts/generate_test_plots.py```.

### Environmental Variables
Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:
 - `EUMETSAT_USER_KEY`: the EUMETSAT API key
 - `EUMETSAT_USER_SECRET`: the EUMETSAT API secret

 These can be added using the `export` command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<table>
  <tr>
    <td align="center"><a href="https://www.jacobbieker.com"><img src="https://avatars.githubusercontent.com/u/7170359?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Jacob Bieker</b></sub></a><br /><a href="https://github.com/openclimatefix/Satip/commits?author=jacobbieker" title="Code">💻</a></td>
    <td align="center"><a href="http://jack-kelly.com"><img src="https://avatars.githubusercontent.com/u/460756?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Jack Kelly</b></sub></a><br /><a href="https://github.com/openclimatefix/Satip/commits?author=JackKelly" title="Code">💻</a></td>
    <td align="center"><a href="https://github.com/AyrtonB"><img src="https://avatars.githubusercontent.com/u/29051639?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Ayrton Bourn</b></sub></a><br /><a href="https://github.com/openclimatefix/Satip/commits?author=AyrtonB" title="Code">💻</a></td>
    <td align="center"><a href="http://laurencewatson.com"><img src="https://avatars.githubusercontent.com/u/1125376?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Laurence Watson</b></sub></a><br /><a href="https://github.com/openclimatefix/Satip/commits?author=Rabscuttler" title="Code">💻</a></td>
    <td align="center"><a href="https://github.com/notger"><img src="https://avatars.githubusercontent.com/u/1180540?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Notger Heinz</b></sub></a><br /><a href="https://github.com/openclimatefix/Satip/commits?author=notger" title="Documentation">📖</a></td>
    <td align="center"><a href="https://github.com/peterdudfield"><img src="https://avatars.githubusercontent.com/u/34686298?v=4?s=100" width="100px;" alt=""/><br /><sub><b>Peter Dudfield</b></sub></a><br /><a href="https://github.com/openclimatefix/Satip/commits?author=peterdudfield" title="Documentation">📖</a></td>
  </tr>
</table>

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->

<!-- ALL-CONTRIBUTORS-LIST:END -->

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
