Metadata-Version: 2.4
Name: lr-shuttle
Version: 1.0.0
Summary: CLI and Python client for host-side of json based serial communication with embedded device bridge.
Author-email: Jonas Estberger <jonas.estberger@lumenradio.com>
License: MIT
Keywords: CLI
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: rich>=13.7
Requires-Dist: pydantic>=2.8
Requires-Dist: typer>=0.12
Requires-Dist: pyserial>=3.5
Requires-Dist: esptool>=4.7
Provides-Extra: dev
Requires-Dist: build>=1.2.1; extra == "dev"
Requires-Dist: twine>=5.1.1; extra == "dev"
Requires-Dist: wheel; extra == "dev"
Requires-Dist: pytest>=8.4.2; extra == "dev"
Requires-Dist: black>=25.9.0; extra == "dev"
Requires-Dist: pytest-html; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"

# Shuttle

`shuttle` is a Typer-based command-line interface & python library for interacting with the ESP32-C5 devboard over its NDJSON serial protocol. The tool is packaged as `lr-shuttle` for PyPI distribution and exposes high-level helpers for common workflows such as probing firmware info, querying protocol metadata, and issuing TimoTwo SPI sequences.


## Installation

```bash
python3 -m pip install lr-shuttle
```

The package supports Python 3.11 and later. When working from this repository you can install it in editable mode:

```bash
make -C host dev
```

## Connecting to the Devboard

- The CLI talks to the board over a serial device supplied via `--port` or the `SHUTTLE_PORT` environment variable (e.g., `/dev/ttyUSB0`).
- Default baud rate is `921600` with a `2s` read timeout. Both can be overridden using `--baud` and `--timeout`.
- Use `--log SERIAL.log` to capture raw RX/TX NDJSON lines with UTC timestamps for later inspection.

## Core Commands

| Command | Description |
| --- | --- |
| `shuttle ping` | Sends a `ping` command to fetch firmware/protocol metadata. |
| `shuttle get-info` | Calls `get.info` and pretty-prints the returned capability snapshot. |
| `shuttle spi-cfg [options]` | Queries or updates the devboard SPI defaults (wraps the `spi.cfg` protocol command). |
| `shuttle uart-cfg [options]` | Queries or updates the devboard UART defaults (wraps the `uart.cfg` protocol command). |
| `shuttle uart-tx [payload]` | Transmits bytes over the devboard UART (wraps the `uart.tx` protocol command). |
| `shuttle flash --port /dev/ttyUSB0` | Programs the ESP32-C5 devboard using the bundled firmware artifacts. |


### SPI Configuration Workflow

- Run `shuttle spi-cfg --port /dev/ttyUSB0` with no extra flags to fetch the current board-level SPI defaults (the response mirrors the firmware’s `spi` object).
- Provide overrides such as `--hz 1500000 --clock-phase trailing` to persist new defaults in the device’s NVS store. String arguments are case-insensitive; the CLI normalizes them to the lowercase values expected by the firmware.
- If you need to push a raw JSON document (e.g., the sample in [`src/spi.cfg`](../src/spi.cfg)), pipe it through a future send-file helper or `screen`/`cat` directly; `spi-cfg` itself focuses on structured flag input.


### UART Configuration Workflow

- Use `shuttle uart-cfg --port /dev/ttyUSB0` with no overrides to dump the persisted UART defaults (`baudrate`, `stopbits`, `parity`).
- Supply `--baudrate`, `--stopbits`, or `--parity` (accepts `n/none`, `e/even`, `o/odd`) to persist new values. Arguments are validated client-side to match the firmware’s accepted ranges (baudrate 1.2 k–4 M, stopbits 1–2).
- Like the SPI helper, UART updates are persisted to the device’s NVS region, so you only need to run the command when changing settings.

### UART Transmission

- `shuttle uart-tx [HEX] --port /dev/ttyUSB0` forwards a hex-encoded payload to the devboard using the `uart.tx` protocol command. The CLI trims whitespace/underscores and validates the string before sending.
- To avoid manual hex encoding, pass `--text "Hello"` (optionally with `--newline`) to send UTF-8 text, `--file payload.bin` to transmit the raw bytes of a file, or provide `-` as the argument to read from stdin. Exactly one payload source can be used per invocation.
- Use `--uart-port` if a future firmware exposes multiple UART instances; otherwise the option can be omitted and the default device UART is used.
- Responses echo the number of bytes accepted by the firmware, matching the `n` field returned by `uart.tx`.

### Flashing Bundled Firmware

- `shuttle flash --port /dev/ttyUSB0` invokes `esptool.py` under the hood and writes the bundled ESP32-C5 firmware (bootloader, partitions, and application images) to the selected device. Pass `--erase-first` to issue a chip erase before programming.
- Firmware bundles live under `shuttle/firmware/<board>` inside the Python package. Run `make -f Makefile.arduino arduino-python` from the repo root after compiling with Arduino CLI to refresh the packaged binaries and `manifest.json` for distribution builds. The helper also copies `boot_app0.bin` from the ESP-IDF core (needed for the USB CDC-on-boot option) so the CLI uses the same flashing layout as `arduino-cli upload`.
- Use `--board <name>` if additional bundles are added; the command enumerates available bundles automatically and validates the provided identifier.


### Transport Reliability (Protocol v2)

Shuttle uses a bidirectional ACK/retransmit layer built into the NDJSON message stream to handle the USB CDC serial link's occasional dropped or truncated messages.

- After sending a command the host waits up to **10 ms** for a device ACK (`{"type":"ack","id":…}`). If no ACK arrives the command is retransmitted with the same `id`, up to 5 times before raising a timeout. The device deduplicates repeated commands so a retransmitted command is never executed twice.
- The device caches every response and event it sends and retransmits them every 10 ms until the host acknowledges receipt with `{"type":"ack","seq":…}`.
- On connect the host sends `get.info` and asserts `protocol_version == 2`. If the device returns an older version the host raises a clear error: **re-flash the firmware** using `shuttle flash --port <tty>`.

### Sequence Integrity Checks

Every device message carries a monotonically increasing `seq` counter. Shuttle uses this for gap detection on top of the ACK/retransmit layer:

- During a command, any unexpected gap in response/event sequence numbers raises a `ShuttleSerialError`.
- Pass `--seq-meta /path/to/seq.meta` to persist the last observed sequence number. Subsequent Shuttle runs expect the very next `seq` value; if a gap is detected (for example because the device dropped messages while Shuttle was offline), the CLI exits with an error detailing the missing value.
- The metadata file stores a single integer. Delete it (or point `--seq-meta` to another location) if the device was power-cycled and its counter reset.


### Logging and Diagnostics

- `--log FILE` appends every raw NDJSON line (RX and TX) along with an ISO-8601 timestamp. This is useful for post-run audits or attaching transcripts to bug reports.
- Combine `--log` with `--seq-meta` to maintain both a byte-perfect trace and an audit trail of sequence continuity.
- Rich panels highlight non-`ok` responses and include firmware error codes returned by the device, making it straightforward to spot invalid arguments or transport failures.

### Environment Tips

- Export `SHUTTLE_PORT` in your shell profile to avoid typing `--port` for each command.
- For scripted flows, prefer `shuttle timotwo read-reg` and `shuttle timotwo nop` helpers instead of manually streaming raw JSON—they take care of command IDs, transfer framing, and error presentation.
- Use `make -C host test` to run the CLI unit tests and verify local changes before publishing to PyPI.



## TimoTwo SPI Commands

Commands implementing the SPI protocol as described at [docs.lumenradio.io/timotwo/spi-interface](https://docs.lumenrad.io/timotwo/spi-interface/).

| Command | Description |
| --- | --- |
| `shuttle timotwo nop` | Issues a single-frame TimoTwo NOP SPI transfer through the devboard. |
| `shuttle timotwo read-reg --addr 0x05 --length 2` | Performs the two-phase TimoTwo register read sequence and decodes the resulting payload/IRQ flags. |
| `shuttle timotwo write-reg --addr 0x05 --data cafebabe` | Performs the two-phase TimoTwo register write sequence to write bytes to a register. |
| `shuttle timotwo read-dmx --length 12` | Reads the latest received DMX values from the TimoTwo device using a two-phase SPI sequence. |
| `shuttle timotwo version [--format text\|json]` | Reads the VERSION register (0x10) and reports firmware version (dotted string, e.g. `1.2.0.0`) and hardware version (integer). |
| `shuttle timotwo update-fw TIMO.cci --port /dev/ttyUSB0` | Streams a TimoTwo `.cci` firmware image via FW_BLOCK commands (requires SPI ≤ 2 MHz and ≥ 255-byte transfers). |

All commands respect the global options declared on the root CLI (`--log`, `--seq-meta`, `--port`, etc.). Rich tables are used to render human-friendly summaries of responses and decoded payloads.


### TimoTwo Register Read Example

To read bytes from a TimoTwo register, use the `read-reg` command. I.e. to read the device name:

```bash
$ shuttle timotwo read-reg --addr 0x36 --length 12
                    TimoTwo read-reg
 Status     OK
 Command    spi.xfer (payload phase)
 RX         00 48 65 6c 6c 6f 20 57 6f 72 6c 64 00
 IRQ level  {'level': 'low'}
                      TimoTwo read-reg
 Address        0x36
 Length         12
 Data           48 65 6c 6c 6f 20 57 6f 72 6c 64 00
 IRQ (command)  0x00
 IRQ (payload)  0x00
 Command RX     00
 Payload RX     00 48 65 6c 6c 6f 20 57 6f 72 6c 64 00
```


### TimoTwo Register Write Example

To write bytes to a TimoTwo register, use the `write-reg` command. I.e. to set the device name to `Hello World`:

```bash
shuttle timotwo write-reg --addr 0x36 --data 48656c6c6f20576f726c6400 --port /dev/ttyUSB0
```

- `--addr` specifies the register address (decimal or 0x-prefixed, 0-63)
- `--data` is a hex string of bytes to write (1-32 bytes)
- `--port` is your serial device

The command will print a summary table with the address, data written, and IRQ flags for each phase. If bit 7 of the IRQ flags is set, the sequence should be retried per the TimoTwo protocol.


### TimoTwo DMX Read Example

Read the latest received DMX values from the window set up by the DMX_WINDOW register:

```bash
shuttle timotwo read-dmx --length 12 --port /dev/ttyUSB0
```

This will print a summary table with the length, data bytes (hex), and IRQ flags for each phase. If bit 7 of the IRQ flags is set, the sequence should be retried per the TimoTwo protocol.

- `--length` specifies the number of DMX bytes to read (1 - max_transfer_bytes)
- `--port` is your serial device


### TimoTwo Firmware Update

Use `shuttle timotwo update-fw` to push official `.cci` images (for example `timotwo-fx-b50f26ad.cci`; the companion `.hex` is provided for reference only) through the Shuttle bridge without touching an external programmer:

```bash
shuttle timotwo update-fw timotwo-fx-b50f26ad.cci --port /dev/ttyUSB0
```

- The command first checks `spi_caps.max_transfer_bytes` and the current SPI clock. Firmware updates require at least 255 bytes per `spi.xfer` call and a clock ≤ 2 MHz. Run `shuttle spi-cfg --hz 2000000` (or lower) if the persisted setting is faster.
- Shuttle enables SPI, sets TimoTwo into UPDATE_MODE by writing `0x40` to CONFIG, waits for the IRQ reboot window (0.6 s), and verifies bit 7 of STATUS before streaming data.
- `.cci` files contain a 4-byte header followed by 272-byte chunks. Because the TimoTwo FW loader accepts at most 255 contiguous bytes, each chunk is split into one `FW_BLOCK_CMD_1` transfer (0x8E + 254 bytes) and one `FW_BLOCK_CMD_2` transfer (0x8F + 18 bytes).
- The first chunk after the header carries metadata. After every 16 data chunks the device writes flash internally, so the CLI pauses for `--flush-wait-ms` (defaults to 500 ms) before continuing. When the whole image has been sent it waits `--final-wait-ms` (defaults to 1000 ms) to let TimoTwo finalize the update.
- Once STATUS clears UPDATE_MODE the command reads the VERSION register, prints FW/HW revisions, and confirms completion. If any step fails (IRQ bit 7, transport error, malformed `.cci`) the CLI aborts with a helpful message.

Tip: combine `--flush-wait-ms 0` and `--final-wait-ms 0` with a lab DUT when replaying the same firmware repeatedly, but keep the defaults when programming production hardware to honour the vendor timing guidelines.


### Using the Library from Python

Use the transport helpers for HIL tests with explicit request→response pairing:

```python
from shuttle.serial_client import NDJSONSerialClient
from shuttle import timotwo

with NDJSONSerialClient("/dev/ttyUSB0") as client:
    # Fire a TimoTwo read-reg using the async API
    commands = timotwo.read_reg_sequence(address=0x05, length=2)
    responses = [client.send_command("spi.xfer", cmd).result(timeout=1.0) for cmd in commands]
    print("Command RX:", responses[0]["rx"])
    print("Payload RX:", responses[1]["rx"])
```

Legacy helpers (`spi_xfer`, `ping`, etc.) remain for simple sequential calls; prefer `send_command` when you need explicit request→response control.

#### Parsing registers with `REGISTER_MAP`

`REGISTER_MAP` in `shuttle.timotwo` (and `shuttle.timo`) documents the bit layout of device registers. Example: read the `VERSION` register (0x10) and decode firmware/hardware versions using the `parse_version` helper.

```python
from shuttle.serial_client import NDJSONSerialClient
from shuttle import timotwo

def read_version(serial_port):
    with NDJSONSerialClient(serial_port) as client:
        seq = timotwo.version_sequence()
        responses = [client.send_command("spi.xfer", cmd).result(timeout=1.0) for cmd in seq]
        rx_payload = bytes.fromhex(responses[1]["rx"])
        version_data = rx_payload[-timotwo.REGISTER_MAP["VERSION"]["length"]:]
        return timotwo.parse_version(version_data)

result = read_version("/dev/ttyUSB0")
print(f"FW version: {result.fw_version}")   # e.g. "1.2.0.0"
print(f"HW version: {result.hw_version}")   # e.g. 15
```

Or using the CLI directly:

```bash
$ shuttle timotwo version --port /dev/ttyUSB0
 TimoTwo VERSION
 FW Version  1.2.0.0
 HW Version  15

$ shuttle timo version --port /dev/ttyUSB0 --format json
{"command": "version", "status": "ok", "fw_version": "1.2.0.0", "hw_version": 15}
```

Use the field metadata in `timotwo.REGISTER_MAP` to interpret other registers (e.g., check `REGISTER_MAP[0x01]["fields"]` for status flags).

More examples can be found in the [examples directory](examples/).

### Async-style Command and Event Handling

`NDJSONSerialClient` now dispatches in a background reader thread and exposes futures so you can fan out work without changing the client for new ops:

- `send_command(op, params)` returns a `Future` that resolves to the matching response or raises on timeout/sequence gap. You can issue multiple commands back-to-back and wait later.
- `register_event_listener("ev.name")` returns a subscription whose `.next(timeout=…)` yields each event payload; multiple listeners can subscribe to the same event (e.g., IRQ and DMX streams).

Example HIL sketch:

```python
client = NDJSONSerialClient(port, baudrate=DEFAULT_BAUD, timeout=DEFAULT_TIMEOUT)
irq_sub = client.register_event_listener("spi.irq")
cmd_future = client.send_command("timotwo.read-reg", {"address": 0x05, "length": 1})

# Wait for either side as your test requires
reg_resp = cmd_future.result(timeout=1)
irq_event = irq_sub.next(timeout=1)
```

Events continue to emit until you close the subscription or client, so you can assert on multiple DMX frames or IRQ edges without recreating listeners.


## Production Test SPI Commands

| Command | Description |
| --- | --- |
| `shuttle prodtest reset` | Reset GPIO pins, IRQ pin and Radio |
| `shuttle prodtest ping` | Send '+' and expect '-' to verify SPI link |
| `shuttle prodtest io-self-test` | Perform GPIO self-test on pins given as argument |
| `shuttle prodtest antenna` | Select antenna |
| `shuttle prodtest continuous-tx` | Continuous transmitter test |
| `shuttle prodtest hw-device-id` | Read the 8-byte HW Device ID |
| `shuttle prodtest serial-number [--value HEX]` | Read or write the 8-byte serial number |
| `shuttle prodtest config [--value HEX]` | Read or write the 5-byte config payload |
| `shuttle prodtest erase-nvmc HW_ID` | Erase NVMC if the provided 8-byte HW ID matches |
