Downloadfroms3

flowtask.components.DownloadFromS3

DownloadFromS3

DownloadFromS3(loop=None, job=None, stat=None, **kwargs)

Bases: Boto3Client, DownloadFromBase

DownloadFromS3.

Overview

Download a file from an Amazon S3 bucket using the functionality from DownloadFrom.

Properties

.. table:: Properties :widths: auto

+--------------------+----------+-----------+-----------------------------------------------------------------+
| Name               | Required | Summary                                                                     |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| credentials        |   Yes    | Credentials to establish connection with S3 service (username and password) |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| bucket             |   Yes    | The name of the S3 bucket to download files from.                           |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| source_dir         |   No     | The directory path within the S3 bucket to download files from.             |
|                    |          | Defaults to the root directory (`/`).                                       |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| source             |   No     | A dictionary specifying the filename to download.                           |
|                    |          | If provided, takes precedence over `source_dir` and `_srcfiles`.            |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| _srcfiles          |   No     | A list of filenames to download from the S3 bucket.                         |
|                    |          | Used in conjunction with `source_dir`.                                      |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| rename             |   No     | A new filename to use for the downloaded file.                              |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| directory          |   Yes    | The local directory path to save the downloaded files.                      |
+--------------------+----------+-----------+-----------------------------------------------------------------+
| create_destination |   No     | A boolean flag indicating whether to create the destination directory       |
|                    |          | if it doesn't exist. Defaults to `True`.                                    |
+--------------------+----------+-----------+-----------------------------------------------------------------+

save the file on the new destination.

Methods

• start()
• close()
• run()
• s3_list(s3_client, suffix="")
• save_attachment(self, filepath, content)

• download_file(self, filename, obj)

  Example:

  DownloadFromS3:
    credentials:
      use_credentials: false
      region_name: us-east-2
    bucket: placer-navigator-data
    source_dir: placer-analytics/bulk-export/monthly-weekly/2025-03-06/metadata/
    destination:
      directory: /nfs/symbits/placerai/2025-03-06/metadata/
    create_destination: true