Copytopg¶
Bases: CopyTo
CopyToPg
This component allows copy data into a Postgres table,
Copy into main postgres using copy_to_table functionality.
TODO: Design an Upsert feature with Copy to Pg.
.. table:: Properties :widths: auto
+----------------+----------+----------------------------------------------------------------------------------+
| Name | Required | Summary |
+----------------+----------+----------------------------------------------------------------------------------+
| schema | Yes | Name of the schema where the table resides. |
+----------------+----------+----------------------------------------------------------------------------------+
| tablename | Yes | Name of the table to insert data into. |
+----------------+----------+----------------------------------------------------------------------------------+
| truncate | No | Boolean flag indicating whether to truncate the table before inserting. |
| | | Defaults to False. |
+----------------+----------+----------------------------------------------------------------------------------+
| use_chunks | No | Boolean flag indicating whether to insert data in chunks (for large datasets). |
| | | Defaults to False. |
| | | Requires specifying a chunksize property for chunk size determination. |
+----------------+----------+----------------------------------------------------------------------------------+
| chunksize | No | Integer value specifying the size of each data chunk when use_chunks is True. |
| | | Defaults to None (chunk size will be calculated based on CPU cores). |
+----------------+----------+----------------------------------------------------------------------------------+
| use_buffer | No | Boolean flag indicating whether to use a buffer for data insertion (optional). |
| | | Defaults to False. |
| | | Using a buffer can improve performance for large datasets. |
+----------------+----------+----------------------------------------------------------------------------------+
| array_columns | No | List of column names containing JSON arrays. These columns will be formatted |
| | | appropriately before insertion. |
| | | Requires use_buffer to be True. |
+----------------+----------+----------------------------------------------------------------------------------+
| use_quoting | No | Boolean flag indicating whether to use quoting for CSV data insertion (optional).|
| | | Defaults to False. |
| | | Using quoting can be helpful for data containing special characters. |
+----------------+----------+----------------------------------------------------------------------------------+
| datasource | No | Using a Datasource instead manual credentials |
+----------------+----------+----------------------------------------------------------------------------------+
| credentials | No | Supporting manual postgresql credentials |
+----------------+----------+----------------------------------------------------------------------------------+
Returns a dictionary containing metrics about the copy operation:
* ROWS_SAVED (int): The number of rows successfully inserted into the target table.
* NUM_ROWS (int): The total number of rows processed from the input data.
* NUM_COLUMNS (int): The number of columns found in the input data.
* (optional): Other metrics specific to the implementation.