{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c4e7a505",
   "metadata": {},
   "source": [
    "# Data analysis\n",
    "\n",
    "This feature able the user to develop real-time data analysis, consist of the complete Python-powered environment, with a set of custom methods for agile development.\n",
    "\n",
    "\n",
    "    Extensions > New Extension > Data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224d6ecf",
   "metadata": {},
   "source": [
    "## Bare minimum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21b41ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bci_framework.extensions.data_analysis import DataAnalysis\n",
    "\n",
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    Analysis()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206e6ee3",
   "metadata": {},
   "source": [
    "## Data stream access\n",
    "\n",
    "The data stream is accessed asynchronously with the `loop_consumer` decorator from `bci_framework.extensions.data_analysis`, this decorator requires the Kafka topics to access.\n",
    "\n",
    "There is 2 topics availables for `loop_consumer`: `eeg` and `marker`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f593de0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bci_framework.extensions.data_analysis import DataAnalysis, loop_consumer\n",
    "\n",
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.stream()\n",
    "\n",
    "    @loop_consumer('eeg', 'marker')\n",
    "    def stream(self):\n",
    "        print('Incoming data...')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    Analysis()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db9d6e43",
   "metadata": {},
   "source": [
    "The decorated method receives 3 optional arguments: `data`, `topic` and `frame`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b61fc6e3",
   "metadata": {},
   "source": [
    "**data:** `(eeg, aux)` if topic is `eeg`, `marker_value` if topic is `marker`  \n",
    "**kafka_stream:** The `stream` object from Kafka.  \n",
    "**topic:** The topic of the Kafka stream, this object is available too in the object `data.topic`  \n",
    "**frame:** Incremental flag with the counter of streamed data.  \n",
    "**latency:** The time bewteen acquisition and read. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f26d427",
   "metadata": {},
   "outputs": [],
   "source": [
    "@loop_consumer('eeg')\n",
    "def stream(self, data, topic, frame, latency):\n",
    "    eeg, aux = data\n",
    "    \n",
    "    print(f'Incoming data #{frame}')\n",
    "    print(f'EEG{eeg.shape}')\n",
    "    print(f'AUX{aux.shape}')\n",
    "    print(f'Topic: {topic}')\n",
    "    print(f'Latency: {latency}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5e917a2",
   "metadata": {},
   "source": [
    "The above code will execute every data stream input, and the below code will execute only when a marker is streamed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e88f7a5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "@loop_consumer('marker')\n",
    "def stream(self, data, topic, frame):\n",
    "    marker_value = data\n",
    "    \n",
    "    print(f'Incoming marker: {marker_value}')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a67fe56",
   "metadata": {},
   "source": [
    "Is **not possible**, use the decorator `loop_consumer` in more than one place, so the argument `topic` could be used to create a flow control."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2461604a",
   "metadata": {},
   "outputs": [],
   "source": [
    "@loop_consumer('eeg', 'marker')\n",
    "def stream(self, data, topic, frame):\n",
    "    \n",
    "    if topic == 'eeg':\n",
    "        eeg, aux = data\n",
    "        print(\"EEG data incomming..\")\n",
    "    \n",
    "    elif topic == 'marker':\n",
    "        marker_value = data\n",
    "        print(\"Marker incomming..\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c99a0ab2",
   "metadata": {},
   "source": [
    "## Simulate data stream\n",
    "\n",
    "Using `fake_loop_consumer` instead of `loop_consumer` is possible to create a fake data stream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1174e2bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bci_framework.extensions.data_analysis import DataAnalysis, fake_loop_consumer\n",
    "import logging\n",
    "\n",
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.stream()\n",
    "\n",
    "    @fake_loop_consumer('eeg')\n",
    "    def stream(self):\n",
    "        logging.debug('Incoming data...')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    Analysis()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "733faeff",
   "metadata": {},
   "source": [
    "## Built in methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75665ecb",
   "metadata": {},
   "source": [
    "### Buffer / Sliding window\n",
    "\n",
    "We can use `self.create_buffer` to implement an automatic buffer with a fixed time view, for example, a buffer of 30 seconds:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e460b19b",
   "metadata": {},
   "outputs": [],
   "source": [
    "self.create_buffer(seconds=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a0d4bf",
   "metadata": {},
   "source": [
    "The data can be accesed with `self.buffer_eeg` and `self.buffer_aux`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bed46f61",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.create_buffer(seconds=30)\n",
    "        self.stream()\n",
    "\n",
    "    @loop_consumer('eeg')\n",
    "    def stream(self):\n",
    "         eeg = self.buffer_eeg\n",
    "         aux = self.buffer_aux"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60a0ab4a",
   "metadata": {},
   "source": [
    "The `self.create_buffer` method receives other arguments like `aux_shape`, `fill` and `samples`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "426ac90a",
   "metadata": {},
   "outputs": [],
   "source": [
    "self.create_buffer(seconds=30, aux_shape=3, fill=0, resampling=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ef6c164",
   "metadata": {},
   "source": [
    "**aux_shape:** The dimension of the auxiliary data, 3 by default.  \n",
    "**fill:** Initialize buffet with this value, 0 by default.  \n",
    "**resampling:** This value is used to resampling the data.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29cf99fa",
   "metadata": {},
   "source": [
    "### Resampling\n",
    "\n",
    "The resampling is defined when the buffer is created, with the argument `resampling` this value is not strictly used, instead a near and optimal value is calculated based on the sampling rate and the buffer size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcfe5ec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.create_buffer(seconds=30, resampling=1000)\n",
    "        self.stream()\n",
    "\n",
    "    @loop_consumer('eeg')\n",
    "    def stream(self):\n",
    "         eeg = self.buffer_eeg_resampled\n",
    "         aux = self.buffer_aux_resampled\n",
    "         \n",
    "         print(f'EEG{eeg.shape}')\n",
    "         print(f'AUX{aux.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035ea85b",
   "metadata": {},
   "source": [
    "The resampling will not affect the buffer, the both data are accessible all the time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e639b7a9",
   "metadata": {},
   "source": [
    "### Data slicing referenced by markers\n",
    "\n",
    "Consist of a method that crops the available data with a marker reference. The  decorator `@marker_slicing` do the trick. Receives the `markers`, a `t0` that indicate how many seconds to crop before the marker and the `duration` of the slice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c23cd8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bci_framework.extensions.data_analysis import DataAnalysis, marker_slicing\n",
    "\n",
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        \n",
    "        # Needs to be greater than the duration of the slice.\n",
    "        self.create_buffer(seconds=30, aux_shape=3)\n",
    "        self.slicing()\n",
    "\n",
    "    @marker_slicing(['Right', 'Left'], t0=-2, duration=6)\n",
    "    def slicing(self, eeg, aux, timestamp, marker):\n",
    "        \n",
    "        print(eeg.shape)\n",
    "        print(aux.shape)\n",
    "        print(timestamp.shape)\n",
    "        print(marker)\n",
    "        print()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    Analysis()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44886b59",
   "metadata": {},
   "source": [
    "The above code will be executed each time that and slice is available. The `buffer` must be greater than the duration of the desired slice.\n",
    "\n",
    "**eeg:** The EEG data croped (`channels, time`).  \n",
    "**aux:** The AUX data croped (`aux, time`).  \n",
    "**timestamp:** The `timestamp` vector.  \n",
    "**marker:** The `marker` that trigger the crop.  \n",
    "**latency:** The time bewteen acquisition and read. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab51c89e",
   "metadata": {},
   "source": [
    "## Receive markers\n",
    "\n",
    "The markers can be accessed specifying the topic `marker` in the `loop_consumer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caa025f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "@loop_consumer('marker')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebdb00d0",
   "metadata": {},
   "source": [
    "## Send commands, annotations and feedbacks\n",
    "\n",
    "The commands are used to communicate outputs into the real world, or other systems, they can also be read in the **Stimuli delivery** to create neurofeedback applications. To activate this feature just add the `enable_produser` argument as `True` into the `DataAnalysis` subclass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31427004",
   "metadata": {},
   "outputs": [],
   "source": [
    "if __name__ == '__main__':\n",
    "    Analysis(enable_produser=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8283028b",
   "metadata": {},
   "source": [
    "Once activate the producer, the methods `self.send_command`, `self.send_feedback` and `self.end_annotation`are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da032b5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "@loop_consumer('eeg')\n",
    "def stream(self):\n",
    "     eeg = self.buffer_eeg_resampled\n",
    "     aux = self.buffer_aux_resampled\n",
    "\n",
    "    [...]  # amazing data analysis\n",
    "    \n",
    "    self.send_command('MyCommand', value=45)        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f854504b",
   "metadata": {},
   "source": [
    "The `self.send_annotation` also receive the optional argument `duration`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0057097b",
   "metadata": {},
   "outputs": [],
   "source": [
    "self.send_annotation('Data record start')\n",
    "self.send_annotation('The subject yawn', duration=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dbde28c",
   "metadata": {},
   "source": [
    "The self.send_feedback receive any kind of Python data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0838b830",
   "metadata": {},
   "outputs": [],
   "source": [
    "feed = {'var1': 0,\n",
    "        'var2': True,\n",
    "        'var3': 'Right',\n",
    "       }\n",
    "self.send_feedback(**feed)\n",
    "\n",
    "self.send_feedback(a=0, b=2.3, c='Left')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "676e5313",
   "metadata": {},
   "source": [
    "A generic producer also is available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "827af423",
   "metadata": {},
   "outputs": [],
   "source": [
    "self.generic_produser(topic, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e509b4b",
   "metadata": {},
   "source": [
    "## Communication between analysis process\n",
    "\n",
    "Let's build a script that will acts like **Kafka transformer**, this script reads the raw EEG data, calculate their EEG spectrum using Fourier and inject back again into the stream. This can be other advanced processing tasks, like classifications using neural networks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eeaabee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bci_framework.extensions.data_analysis import DataAnalysis, loop_consumer\n",
    "from bci_framework.extensions import properties as prop\n",
    "from gcpds.utils.processing import fourier\n",
    "\n",
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.create_buffer(seconds=30, resampling=1000)\n",
    "        self.stream()\n",
    "\n",
    "    @loop_consumer('eeg')\n",
    "    def stream(self):\n",
    "         W, EEG = fourier(self.buffer_eeg, fs=prop.SAMPLE_RATE, axis=1)\n",
    "         data = {'amplitude': EEG,\n",
    "                 'frequency': W}\n",
    "         self.generic_produser('spectrum', data)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    Analysis(enable_produser=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "643bb5cb",
   "metadata": {},
   "source": [
    "Now, in another script, we will write a **Kafka consumer** this script will consume from the previously created stream."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45569cfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bci_framework.extensions.data_analysis import DataAnalysis, loop_consumer\n",
    "from bci_framework.extensions import properties as prop\n",
    "from gcpds.utils.processing import fourier\n",
    "\n",
    "class Analysis(DataAnalysis):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.stream()\n",
    "\n",
    "    @loop_consumer('spectrum')\n",
    "    def stream(self, data):\n",
    "        data = data.value['data']\n",
    "        \n",
    "        EEG = data['amplitude']\n",
    "        W = data['frequency']\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    Analysis()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a301ff2",
   "metadata": {},
   "source": [
    "This examples are available by default in the framework extensions explorer.\n",
    "\n",
    "The `spectrum` topic must be created before to use the topic:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1628b591",
   "metadata": {},
   "outputs": [],
   "source": [
    "kafka-topics.sh --create --bootstrap-server localhost:2181 --replication-factor 1 --partitions 1 --topic feedback"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f05e4f7",
   "metadata": {},
   "source": [
    "## Framework integration\n",
    "\n",
    "BCI-Framework can execute any number of scripts as an independent process, the system will handle the interruption and show information about the CPU and memory usage.\n",
    "\n",
    "    Data analysis > Data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78283930",
   "metadata": {},
   "source": [
    "<img src='images/analysis_process.png'></img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
