Metadata-Version: 2.4
Name: np_codeocean
Version: 0.3.17
Summary: Tools for uploading and interacting with Mindscope Neuropixels experiments on Code Ocean
Author-email: Ben Hardcastle <ben.hardcastle@alleninstitute.org>, Chris Mochizuki <chrism@alleninstitute.org>, Arjun Sridhar <arjun.sridhar@alleninstitute.org>
License: MIT
Project-URL: Source, https://github.com/AllenInstitute/np_codeocean
Project-URL: Issues, https://github.com/AllenInstitute/np_codeocean/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: np_session>=0.6.44
Requires-Dist: np-tools>=0.1.23
Requires-Dist: np-config>=0.4.33
Requires-Dist: requests>=2.31.0
Requires-Dist: npc-session>=0.1.34
Requires-Dist: polars>=0.20.16
Requires-Dist: npc-lims>=0.1.168
Requires-Dist: npc-ephys>=0.1.32
Requires-Dist: wavpack-numcodecs<0.2
Requires-Dist: cryptography<43.0
Requires-Dist: aind-data-transfer-service>=1.15.0
Requires-Dist: aind-slurm-rest-v2==0.0.3
Requires-Dist: aind-codeocean-pipeline-monitor>=0.5.2
Provides-Extra: dynamicrouting
Requires-Dist: npc-lims>=0.1.154; extra == "dynamicrouting"
Requires-Dist: npc-sessions[metadata]>=0.0.275; extra == "dynamicrouting"
Requires-Dist: aind-codeocean-pipeline-monitor[full]>=0.5.0; extra == "dynamicrouting"
Provides-Extra: openscope
Requires-Dist: aind-metadata-mapper; extra == "openscope"

# np_codeocean
Tools for uploading Mindscope Neuropixels experiments to S3 (for Code Ocean).

Requires running as admin on Windows in order to create remote-to-remote symlinks
on the Isilon.

## Development workflow
- clone the repo, or pull with rebase
- sync the development environment using `uv` (see below)
- push changes to main (always commit changes to `uv.lock`)
- github action formats and publishes a new version
- pull the bump commit

## Install 
Setup/sync the development environment for working on a specific project:
```shell
uv sync --extra <dynamicrouting|openscope>
```
This ensures all developers are using the same package versions.

## Add dependencies
For shared utilities:
```shell
uv add <package-name>
```

For project-specific utilities (added to optional dependency groups):
```shell
uv add <package-name> --optional <dynamicrouting|openscope>
```

## Update dependencies
All:
```shell
uv lock --upgrade
```

Single package:
```shell
uv lock --upgrade-package  <package-name>
```

## Usage 
- `upload` CLI tool is provided, which uses the
  [`np_session`](https://github.com/AllenInstitute/np_session) interface to find
  and upload
  raw data for one ecephys session:

    ```
    pip install np_codeocean
    upload <session-id>
    ```
 
    where session-id is any valid input to `np_session.Session()`, e.g.: 
    - a lims ID (`1333741475`) 
    - a workgroups foldername (`DRPilot_366122_20230101`) 
    - a path to a session folder (    `\\allen\programs\mindscope\workgroups\np-exp\1333741475_719667_20240227`)
    
- a folder of symlinks pointing to the raw data is created, with a new structure suitable for the KS2.5 sorting pipeline on Code Ocean
- the symlink folder, plus metadata, are entered into a csv file, which is
  submitted to [`http://aind-data-transfer-service`](http://aind-data-transfer-service), which in turn runs the
  [`aind-data-transfer`](https://github.com/AllenNeuralDynamics/aind-data-transfer)
  tool on the HPC, which follows the symlinks to the original data,
  median-subtracts/scales/compresses ephys data, then uploads with the AWS CLI tool
- all compression/zipping acts on copies in temporary folders: the original raw data is not altered in anyway 
