Metadata-Version: 2.4
Name: cometx
Version: 3.3.0
Summary: Python tools for Comet
Home-page: https://github.com/comet-ml/cometx/
Author: cometx development team
License: MIT License
Keywords: ai,artificial intelligence,python,machine learning
Platform: Linux
Platform: Mac OS X
Platform: Windows
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Framework :: Jupyter
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pillow>=11.1.0
Requires-Dist: comet_ml>=3.49.1
Requires-Dist: reportlab
Requires-Dist: rich>=13.0.0
Requires-Dist: six
Requires-Dist: requests
Requires-Dist: tqdm
Requires-Dist: matplotlib
Requires-Dist: streamlit
Provides-Extra: all
Requires-Dist: opik; extra == "all"
Requires-Dist: comet_mpm; extra == "all"
Requires-Dist: numpy; extra == "all"
Requires-Dist: scikit-learn; extra == "all"
Requires-Dist: matplotlib; extra == "all"
Requires-Dist: scipy; extra == "all"
Requires-Dist: ipywidgets; extra == "all"
Requires-Dist: markdown; extra == "all"
Requires-Dist: IPython; extra == "all"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: platform
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# cometx

```
   _________  ____ ___  ___  / /__  __
  / ___/ __ \/ __ `__ \/ _ \/ __/ |/_/
 / /__/ /_/ / / / / / /  __/ /__>  <
 \___/\____/_/ /_/ /_/\___/\__/_/|_|
```

Open source extensions for the [Comet](https://www.comet.com/site/?utm_source=cometx&utm_medium=referral&utm_campaign=cometx_2022&utm_content=github) SDK.

These extensions are created and supported by the community and are
not an official project of Comet ML. We welcome contributions!

## Installation

```
pip install cometx --upgrade
```

To use these command-line functions, you can set your Comet API key and URL override using command-line flags:

```
cometx --api-key="YOUR-COMET-API-KEY" COMMAND
```

If you are a Comet on-prem user, and your installation does not use smart-keys, you'll also need to set the URL override:

```
cometx --api-key="YOUR-COMET-API-KEY" --url-override="https://your-companys-comet.com/clientlib/" COMMAND
```

## Usage

`cometx` is composed of a series of commands that are useful
independently, and can be used together to create sophisticated tools
for ML management.

### Commands

* [cometx admin](#cometx-admin)
* [cometx config](#cometx-config)
* [cometx copy](#cometx-copy)
* [cometx count](#cometx-count)
* [cometx delete-assets](#cometx-delete-assets)
* [cometx download](#cometx-download)
* [cometx list](#cometx-list)
* [cometx log](#cometx-log)
* [cometx reproduce](#cometx-reproduce)
* [cometx smoke-test](#cometx-smoke-test)
* [cometx update](#cometx-update)

For all commands, use the `--help` flag to get additional information.

## Global Options

These flags are availble before a command:

* `--api-key API_KEY` - Set the COMET_API_KEY
* `--url-override URL_OVERRIDE` - Set the COMET_URL_OVERRIDE

This command shows the cometx version:

* `--version` - Display comet_ml version

This command can be used globally, or for individual commands:

* `-h, --help` - Show help message

## cometx list

This command is used to:

* get a list of all workspaces that you are a member of
* get a list of all projects in a workspace
* get a list of all experiments (by name or key) in a project

cometx list examples:

```
cometx list WORKSPACE/PROJECT/EXPERIMENT-KEY-OR-NAME
cometx list WORKSPACE/PROJECT
cometx list WORKSPACE
cometx list
```

### Flags

* `-u, --use-name` - Use experiment names for experiment folders and listings
* `--query QUERY` - Only list experiments that match this Comet query string
* `--debug` - Provide debug info

For more information, `cometx list --help`

## cometx count

This command is used to:

* count the number of workspaces you are a member of
* count the number of projects in workspaces
* count the number of experiments in projects
* count the number of artifacts in workspaces

cometx count examples:

```
cometx count
cometx count --workspaces-only
cometx count --with-projects
cometx count --with-experiments
cometx count --count-all
cometx count --limit 10
```

### Flags

* `--workspaces-only` - Count only workspaces (fastest)
* `--with-projects` - Count workspaces and projects (default)
* `--with-experiments` - Count workspaces, projects, and experiments (slowest, most detailed)
* `--count-all` - Count everything: workspaces, projects, artifacts, and experiments (most comprehensive)
* `--limit LIMIT` - Process only the first N workspaces (useful for testing)
* `--debug` - Provide debug info

For more information, `cometx count --help`

## cometx download

This command is used to:

* download all workspaces, projects, and experiments of workspaces that you are a member of
* download all projects, and experiments of a given workspace
* download all experiments of a given workspace/project
* download artifacts and models from the registry
* download panels

> **Note**: For detailed information on copying experiments from one Comet installation to another, see [MIGRATIONS.md](https://github.com/comet-ml/cometx/blob/main/MIGRATIONS.md).

cometx download examples:

```
cometx download WORKSPACE/PROJECT/EXPERIMENT-KEY-OR-NAME [RESOURCE ...] [FLAGS ...]
cometx download WORKSPACE/PROJECT [RESOURCE ...] [FLAGS ...]
cometx download WORKSPACE [RESOURCE ...] [FLAGS ...]
cometx download [RESOURCE ...] [FLAGS ...]
```

Where [RESOURCE ...] is zero or more of the following names:

* run - alias for: code, git, output, graph, and requirements
* system
* others
* parameters
* metadata
* metrics
* assets
* html
* project - alias for: project_notes, project_metadata

If no RESOURCE is given it will download all of them.

### Flags

* `--from from` - Source of data to copy. Should be: comet, wandb, or neptune
* `-i IGNORE [IGNORE ...], --ignore IGNORE [IGNORE ...]` - Resource(s) (or 'experiments') to ignore
* `-j PARALLEL, --parallel PARALLEL` - The number of threads to use for parallel downloading; default (None) is based on CPUs
* `-o OUTPUT, --output OUTPUT` - Output directory for downloads
* `-u, --use-name` - Use experiment names for experiment folders and listings
* `-l, --list` - List the items at this level (workspace, project, experiment, artifacts, or model-registry) rather than download
* `--flat` - Download the files without subfolders
* `-f, --ask` - Queries the user; if flag not included system will answer `yes` for all queries
* `--filename FILENAME` - Only get resources ending with this
* `--query QUERY` - Only download experiments that match this Comet query string
* `--asset-type ASSET_TYPE` - Only get assets with this type
* `--sync SYNC` - What level to sync at: all, experiment, project, or workspace
* `--debug` - Provide debug info

To download artifacts:

```
cometx download WORKSPACE/artifacts/NAME [FLAGS ...]
cometx download WORKSPACE/artifacts/NAME/VERSION-OR-ALIAS [FLAGS ...]
```

To download models from the model registry:

```
cometx download WORKSPACE/model-registry/NAME [FLAGS ...]
cometx download WORKSPACE/model-registry/NAME/VERSION-OR-STAGE [FLAGS ...]
```

To download panels:

```
cometx download WORKSPACE/panels/NAME-OR-ID [FLAGS ...]
cometx download WORKSPACE/panels [FLAGS ...]
```

For more information, `cometx download --help`

## cometx copy

This command is used to:

* copy downloaded data to a new experiment
* create a symlink from one project to existing experiments
* copy panels

> **Note**: For detailed information on copying experiments from one Comet installation to another, see [MIGRATIONS.md](https://github.com/comet-ml/cometx/blob/main/MIGRATIONS.md).

cometx copy examples:

```
cometx copy SOURCE DESTINATION
cometx copy --symlink SOURCE DESTINATION
cometx copy --path /base/path SOURCE DESTINATION
cometx copy --path ~/Downloads SOURCE DESTINATION
```

where SOURCE is:

* if not `--symlink`, "WORKSPACE/PROJECT/EXPERIMENT", "WORKSPACE/PROJECT", or "WORKSPACE" folder
* if `--symlink`, then it is a Comet path to workspace or workspace/project
* "WORKSPACE/panels" or "WORKSPACE/panels/PANEL-ZIP-FILENAME" to copy panels

where DESTINATION is:

* WORKSPACE
* WORKSPACE/PROJECT

Not all combinations are possible:

| Destination → <br/>Source ↓ | WORKSPACE            | WORKSPACE/PROJECT      |
|--------------------|----------------------|------------------------|
| `WORKSPACE/*/*`      | Copies all projects  | N/A                    |
| `WORKSPACE/PROJ/*`   | N/A                  | Copies all experiments |
| `WORKSPACE/PROJ/EXP` | N/A                  | Copies experiment      |

### Asset Types

* 3d-image
* 3d-points - deprecated
* audio
* confusion-matrix - may contain assets
* curve
* dataframe
* dataframe-profile
* datagrid
* embeddings - may reference image asset
* histogram2d - not used
* histogram3d - internal only, single histogram, partial logging
* histogram_combined_3d
* image
* llm_data
* model-element
* notebook
* source_code
* tensorflow-model-graph-text - not used
* text-sample
* video

### Flags

* `-i IGNORE [IGNORE ...], --ignore IGNORE [IGNORE ...]` - Resource(s) (or 'experiments') to ignore
* `-j PARALLEL, --parallel PARALLEL` - The number of threads to use for parallel uploading; default (None) is based on CPUs
* `--debug` - If given, allow debugging
* `--quiet` - If given, don't display update info
* `--symlink` - Instead of copying, create a link to an experiment in a project
* `--sync` - Check to see if experiment name has been created first; if so, skip
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)

### Using --path

The `--path` option allows you to specify a base directory where your workspace folders are located. This is useful when your downloaded experiments are stored in a specific directory structure.

Examples:
```bash
# Copy from experiments in /data/experiments/workspace
cometx copy --path /data/experiments workspace dest-workspace

# Copy from experiments in your home directory
cometx copy --path ~ workspace dest-workspace

# Copy from experiments in Downloads folder
cometx copy --path ~/Downloads workspace dest-workspace
```

For more information, `cometx copy --help`

## cometx log

This command is used to log a resource (metrics, parameters, asset,
etc) file to a specific experiment or experiments.

cometx log examples:

```
cometx log WORKSPACE/PROJECT/EXPERIMENT-KEY FILENAME ... --type=TYPE
cometx log WORKSPACE PANEL-ZIP-FILENAME ... --type=panel
cometx log WORKSPACE PANEL.py ... --type=panel
cometx log WORKSPACE PANEL-URL ... --type=panel
cometx log WORKSPACE/PROJECT --type=other --set "key:value"
cometx log WORKSPACE --type=other --set "key:value"
```

Where TYPE is one of the following names:

* all
* asset
* audio
* code
* image
* metrics
* notebook
* panel
* tensorflow-file
* text-sample
* video
* other
* tensorboard-folder-assets

### Flags

* `--type TYPE` - The type of item to log
* `--set SET` - The key:value to log
* `--query QUERY` - A Comet Query string, see https://www.comet.com/docs/v2/api-and-sdk/python-sdk/reference/API/#apiquery
* `--debug` - If given, allow debugging
* `--use-base-name` - If given, using the basename for logging assets

For more information, `cometx log --help`

## cometx delete-assets

To delete experiments assets:

```
cometx delete-assets WORKSPACE/PROJECT --type=image
cometx delete-assets WORKSPACE/PROJECT/EXPERIMENT --type=all
```

Type can be valid asset type, including:

* all
* asset
* audio
* code
* image
* notebook
* text-sample
* video

### Flags

* `--type TYPE` - The type of item to log
* `--debug` - If given, allow debugging
* `--query QUERY` - Only delete experiments that match this Comet query string

For more information, `cometx delete-assets --help`

## cometx config

To enable auto-logging of your notebooks in Jupyter environments:

```python
cometx config --auto-log-notebook yes
```

To turn auto-logging of notebooks off, use:

```python
cometx config --auto-log-notebook no
```

If you keep the generated experiment URLs in the notebook, but later edit the notebook, the notebooks will be updated in all of the experiments created in the notebook.

### Flags

* `--debug` - If given, allow debugging
* `--auto-log-notebook AUTO_LOG_NOTEBOOK` - Takes a 1/yes/true, or 0/no/false

For more information, `cometx config --help`

## cometx reproduce

```
cometx reproduce [-h] [--run] [--executable EXECUTABLE] COMET_PATH OUTPUT_DIR
```

This command is used to reproduce experiments by copying files to a specified output directory.

### Flags

* `--run` - Run the reproducible script
* `--executable EXECUTABLE` - Run the reproducible script with specified executable

For more information, `cometx reproduce --help`

## cometx update

```
cometx update [-h] [--debug] COMET_SOURCE COMET_DESTINATION
```

To update existing experiments.

cometx update SOURCE DESTINATION

where SOURCE is a folder:

* "WORKSPACE/PROJECT/EXPERIMENT"
* "WORKSPACE/PROJECT"
* "WORKSPACE"

where DESTINATION is a Comet:

* WORKSPACE
* WORKSPACE/PROJECT

### Flags

* `--debug` - If given, allow debugging

For more information, `cometx update --help`

## cometx admin

```
cometx admin [-h] [--host HOST] [--debug] ACTION [ARGUMENTS ...]
```

To perform admin functions

### Actions

#### chargeback-report

Generate a chargeback report from the Comet server.

```
cometx admin chargeback-report [YEAR-MONTH]
```

**Arguments:**
* `YEAR-MONTH` (optional, deprecated) - The YEAR-MONTH to run report for, eg 2024-09. If not provided, generates a report for all available periods.

**Output:**
* Saves a JSON file: `comet-chargeback-report.json` (or `comet-chargeback-report-{YEAR-MONTH}.json`)

**Examples:**
```
cometx admin chargeback-report
<<<<<<< Updated upstream
cometx admin usage-report
=======
cometx admin chargeback-report 2024-09
```
>>>>>>> Stashed changes

#### usage-report

Generate a usage report with experiment counts and statistics for one or more workspaces/projects.

```
cometx admin usage-report WORKSPACE [WORKSPACE ...]
cometx admin usage-report WORKSPACE/PROJECT [WORKSPACE/PROJECT ...]
cometx admin usage-report --app
```

**Arguments:**
* `WORKSPACE_PROJECT` (required, one or more, unless using `--app`) - One or more `WORKSPACE` or `WORKSPACE/PROJECT` to run usage report for. If `WORKSPACE` is provided without a project, all projects in that workspace will be included.

**Options:**
* `--units {month,week,day,hour}` - Time unit for grouping experiments (default: month)
  * `month`: Group by month (YYYY-MM format)
  * `week`: Group by ISO week (YYYY-WW format)
  * `day`: Group by day (YYYY-MM-DD format)
  * `hour`: Group by hour (YYYY-MM-DD-HH format)
* `--max-experiments-per-chart N` - Maximum number of workspaces/projects per chart (default: 5). If more workspaces/projects are provided, multiple charts will be generated.
* `--no-open` - Don't automatically open the generated PDF file after generation.
* `--app` - Launch interactive Streamlit web app instead of generating PDF. When using this option, you don't need to specify workspace/project arguments.

**Output:**
* **PDF Report** (default): Generates a PDF report containing:
  * Summary statistics (total experiments, users, run times, GPU utilization)
  * Experiment count charts by time unit
  * GPU utilization charts (if GPU data is available)
  * GPU memory utilization charts (if GPU data is available)
* **Interactive Web App** (with `--app`): Launches a Streamlit web interface where you can:
  * Select workspace and project from dropdowns
  * View statistics and charts interactively
  * Change time units and regenerate reports
  * View "All Projects" from a workspace

**Examples:**
```
# Generate report for a single workspace
cometx admin usage-report my-workspace

# Generate report for multiple projects
cometx admin usage-report my-workspace/project1 my-workspace/project2

# Generate report with weekly grouping
cometx admin usage-report workspace1 workspace2 --units week

# Generate report with daily grouping, don't auto-open
cometx admin usage-report workspace --units day --no-open

# Launch interactive web app
cometx admin usage-report --app
```

#### gpu-report

Generate a GPU usage report for one or more workspaces/projects with detailed GPU metrics analysis.

```
cometx admin gpu-report WORKSPACE [WORKSPACE ...] --start-date DATE
cometx admin gpu-report WORKSPACE/PROJECT [WORKSPACE/PROJECT ...] --start-date DATE
```

**Arguments:**
* `WORKSPACE_PROJECT` (required, one or more) - One or more `WORKSPACE` or `WORKSPACE/PROJECT` to run GPU report for. If `WORKSPACE` is provided without a project, all projects in that workspace will be included.

**Options:**
* `--start-date DATE` (required) - Start date for the report in YYYY-MM-DD format (e.g., `2024-01-01`)
* `--end-date DATE` (optional) - End date for the report in YYYY-MM-DD format (e.g., `2024-12-31`). If not provided, reports from start-date onwards.
* `--metrics METRIC [METRIC ...]` (optional) - List of metrics to track. If not provided, uses default GPU metrics:
  * `sys.gpu.0.gpu_utilization` - GPU utilization percentage
  * `sys.gpu.0.memory_utilization` - GPU memory utilization percentage
  * `sys.gpu.0.used_memory` - GPU memory used in GB
  * `sys.gpu.0.power_usage` - GPU power usage in watts
  * `sys.gpu.0.temperature` - GPU temperature in Celsius
* `--open` - Automatically open the generated PDF file after generation.

**Output:**
* Generates a PDF report containing:
  * Summary statistics (total experiments, workspaces, metrics tracked)
  * Breakdown by workspace (if multiple workspaces)
  * Average metrics by workspace charts (bar charts)
  * Maximum metrics by month charts (time series line charts with workspace legend)
* Generates individual PNG chart files for each metric:
  * `gpu_report_avg_{metric}_by_workspace.png` - Average metric value per workspace
  * `gpu_report_max_{metric}_by_month.png` - Maximum metric value per month over time

**Examples:**
```
# Generate report for a single workspace from a start date
cometx admin gpu-report my-workspace --start-date 2024-01-01

# Generate report with date range
cometx admin gpu-report my-workspace --start-date 2024-01-01 --end-date 2024-12-31

# Generate report for multiple projects
cometx admin gpu-report workspace1/project1 workspace2 --start-date 2024-01-01

# Generate report with custom metrics
cometx admin gpu-report my-workspace --start-date 2024-01-01 --metrics sys.gpu.0.gpu_utilization sys.gpu.0.memory_utilization

# Generate report and automatically open PDF
cometx admin gpu-report my-workspace --start-date 2024-01-01 --open
```

### Global Flags

* `--host HOST` - Override the HOST URL
* `--debug` - If given, allow debugging
* `--api-key API_KEY` - Set the COMET_API_KEY
* `--url-override URL_OVERRIDE` - Set the COMET_URL_OVERRIDE

For more information, `cometx admin --help`

> **Note**: For detailed information on admin commands, see [README-ADMIN.md](https://github.com/comet-ml/cometx/blob/main/README-ADMIN.md).

## cometx smoke-test

```
cometx smoke-test [-h] [--exclude [EXCLUDE ...]] [--debug DEBUG] COMET_PATH [include ...]
```

Perform a smoke test on a Comet installation. Logs results to WORKSPACE/smoke-tests or WORKSPACE/PROJECT.

Examples:

Run all tests:
```
cometx smoke-test WORKSPACE    # project defaults to smoke-tests
cometx smoke-test WORKSPACE/PROJECT
```

Run everything except mpm tests:
```
cometx smoke-test WORKSPACE/PROJECT --exclude mpm
```

Run just optimizer tests:
```
cometx smoke-test WORKSPACE/PROJECT optimizer
```

Run just metric tests:
```
cometx smoke-test WORKSPACE/PROJECT metric
```

Items to include or exclude:

* optimizer
* mpm
* panel
* opik
* experiment
  * metric
  * image
  * asset
  * dataset-info
  * confusion-matrix
  * embedding

### Flags

* `--exclude [EXCLUDE ...]` - Items to exclude; any of: asset, confusion-matrix, dataset-info, embedding, experiment, image, metric, mpm, opik, optimizer, panel
* `--debug DEBUG` - Show debugging information

For more information, `cometx smoke-test --help`

## Copy/Download Use Cases

In this section we'll explore some common scenarios.

1. Copy a specific project from one Comet installation to another
2. Copy all projects in workspace to a new workspace
3. Copy specific experiments in a project to new experiments

### 1. Copy a specific project from one comet installation to another

A useful idiom is to set your Comet environment variables on the line
of a command. In this manner, you can set the `COMET_URL_OVERRIDE`
and `COMET_API_KEY` for different installations.

Of course, you don't have to set the environment variables if you are
copying experiments on the same Comet installation.

Here is how you one could download the experiments in
WORKSPACE/PROJECT from http://comet.a.com:

```shell
cometx --api-key=A-KEY download WORKSPACE/PROJECT
```

The `cometx download` command downloads all of the Comet experiment
data into local files. Note that WORKSPACE/PROJECT refers to a
workspace and project on http://comet.a.com.

One could then copy the downloaded experiment data with a similar command:

```shell
cometx --api-key=B-KEY copy WORKSPACE/PROJECT NEW-WORKSPACE/NEW-PROJECT
```

Note that WORKSPACE/PROJECT now refers to a directory, and
NEW-WORKSPACE/NEW-PROJECT refers to a workspace and project on
http://comet.b.com.

### 2. Copy all projects in workspace to a new workspace

Similarly, one can copy all of the projects by first downloading them:

```shell
cometx --api-key=A-KEY download WORKSPACE
```

and then copying them:

```shell
cometx --api-key=B-KEY copy WORKSPACE NEW-WORKSPACE
```

### 3. Copy specific experiments in a project to new experiments

Similarly, one can copy a single experiment first downloading it:

```shell
cometx --api-key=A-KEY download WORKSPACE/PROJECT/EXPERIMENT-NAME-OR-ID
```

and then copying it:

```shell
cometx --api-key=B-KEY copy WORKSPACE/PROJECT/EXPERIMENT-NAME-OR-ID NEW-WORKSPACE/NEW-PROJECT
```

## Running Tests

WARNING: Running the tests will create experiments, models, assets, etc.
in your default workspace if not set otherwise.

To run the tests, you can either export all of these items in the
environment:

```shell
$ export COMET_USER="<USERNAME>"
$ export COMET_WORKSPACE="<WORKSPACE>"
$ export COMET_API_KEY="<API-KEY>"
$ pytest tests
```

Or, define `workspace` and `api_key` in your ~/.comet.config file:

```shell
$ export COMET_USER="<USERNAME>"
$ pytest tests
```
