kedro.contrib.io.azure.JSONBlobDataSet¶

class kedro.contrib.io.azure.JSONBlobDataSet(filepath, container_name, credentials, encoding='utf-8', blob_from_bytes_args=None, blob_to_bytes_args=None, load_args=None, save_args=None)[source]¶

Bases: kedro.io.core.AbstractDataSet

JSONBlobDataSet loads and saves json(line-delimited) files in Microsoft’s Azure blob storage. It uses Azure storage SDK to read and write in Azure and pandas to handle the json(l) file locally.

Example:

import pandas as pd

data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
                     'col3': [5, 6]})

data_set = JSONBlobDataSet(filepath="test.jsonl", credentials={"sas_token":"1234"},
                           load_args={"lines":True}, container_name="test",
                           save_args={"orient":"records", "lines":True})
data_set.save(data)
reloaded = data_set.load()

assert data.equals(reloaded)

Methods

`JSONBlobDataSet.__init__`(filepath, …[, …])	Creates a new instance of `JSONBlobDataSet` pointing to a concrete json(l) file on Azure blob storage.
`JSONBlobDataSet.exists`()	Checks whether a data set’s output already exists by calling the provided _exists() method.
`JSONBlobDataSet.from_config`(name, config[, …])	Create a data set instance using the configuration provided.
`JSONBlobDataSet.get_last_load_version`()	Versioned datasets should override this property to return last loaded version
`JSONBlobDataSet.get_last_save_version`()	Versioned datasets should override this property to return last saved version.
`JSONBlobDataSet.load`()	Loads data by delegation to the provided load method.
`JSONBlobDataSet.release`()	Release any cached data.
`JSONBlobDataSet.save`(data)	Saves data by delegation to the provided save method.

__init__(filepath, container_name, credentials, encoding='utf-8', blob_from_bytes_args=None, blob_to_bytes_args=None, load_args=None, save_args=None)[source]¶

Creates a new instance of JSONBlobDataSet pointing to a concrete json(l) file on Azure blob storage.

Parameters:

filepath (str) – path to a azure blob of a json(l) file.
container_name (str) – Azure container name.
credentials (Dict[str, Any]) – Credentials (account_name and account_key or sas_token) to access the Azure blob storage
encoding (str) – Default utf-8. Defines encoding of json files downloaded as binary streams.
blob_to_bytes_args (Optional[Dict[str, Any]]) – Any additional arguments to pass to Azure’s get_blob_to_bytes method: https://docs.microsoft.com/en-us/python/api/azure.storage.blob.baseblobservice.baseblobservice?view=azure-python#get-blob-to-bytes
blob_from_bytes_args (Optional[Dict[str, Any]]) – Any additional arguments to pass to Azure’s create_blob_from_bytes method: https://docs.microsoft.com/en-us/python/api/azure.storage.blob.blockblobservice.blockblobservice?view=azure-python#create-blob-from-bytes
load_args (Optional[Dict[str, Any]]) – Pandas options for loading json(l) files. Here you can find all available arguments: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html All defaults are preserved.
save_args (Optional[Dict[str, Any]]) – Pandas options for saving json(l) files. Here you can find all available arguments: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_json.html All defaults are preserved, but “index”, which is set to False.

Return type:

None

exists()¶

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:	`bool`
Returns:	Flag indicating whether the output already exists.
Raises:	`DataSetError` – when underlying exists method raises error.

classmethod from_config(name, config, load_version=None, save_version=None)¶

Create a data set instance using the configuration provided.

Parameters:	name (`str`) – Data set name. config (`Dict`[`str`, `Any`]) – Data set config dictionary. load_version (`Optional`[`str`]) – Version string to be used for `load` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled. save_version (`Optional`[`str`]) – Version string to be used for `save` operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
Return type:	`AbstractDataSet`
Returns:	An instance of an `AbstractDataSet` subclass.
Raises:	`DataSetError` – When the function fails to create the data set from its config.

get_last_load_version()¶

Versioned datasets should override this property to return last loaded version

Return type:	`Optional`[`str`]

get_last_save_version()¶

Versioned datasets should override this property to return last saved version.

Return type:	`Optional`[`str`]

load()¶

Loads data by delegation to the provided load method.

Return type:	`Any`
Returns:	Data returned by the provided load method.
Raises:	`DataSetError` – When underlying load method raises error.

release()¶

Release any cached data.

Raises:	`DataSetError` – when underlying exists method raises error.
Return type:	`None`

save(data)¶

Saves data by delegation to the provided save method.

Parameters:	data (`Any`) – the value to be saved by provided save method.
Raises:	`DataSetError` – when underlying save method raises error.
Return type:	`None`