{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plotting massive data sets\n",
    "\n",
    "This notebook plots about half a million LIDAR points around Toronto from the KITTI data set. ([Source](http://www.cvlibs.net/datasets/kitti/raw_data.php)) The data is meant to be played over time. With pydeck, we can render these points and interact with them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning the data\n",
    "\n",
    "First we need to import the data. Each row of data represents one x/y/z coordinate for a point in space at a point in time, with each frame representing about 115,000 points.\n",
    "\n",
    "We also need to scale the points to plot closely on a map. These point coordinates are not given in latitude and longitude, so as a workaround we'll plot them very close to (0, 0) on the earth.\n",
    "\n",
    "In future versions of pydeck other viewports, like a flat plane, will be supported out-of-the-box. For now, we'll make do with scaling the points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "all_lidar = pd.concat([\n",
    "    pd.read_csv('https://raw.githubusercontent.com/ajduberstein/kitti_subset/master/kitti_1.csv'),\n",
    "    pd.read_csv('https://raw.githubusercontent.com/ajduberstein/kitti_subset/master/kitti_2.csv'),\n",
    "    pd.read_csv('https://raw.githubusercontent.com/ajduberstein/kitti_subset/master/kitti_3.csv'),\n",
    "    pd.read_csv('https://raw.githubusercontent.com/ajduberstein/kitti_subset/master/kitti_4.csv'),\n",
    "])\n",
    "\n",
    "# Filter to one frame of data\n",
    "lidar = all_lidar[all_lidar['source'] == 136]\n",
    "lidar.loc[: , ['x', 'y']] = lidar[['x', 'y']] / 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting the data\n",
    "\n",
    "We'll define a single `PointCloudLayer` and plot it.\n",
    "\n",
    "Pydeck by default expects the input of `get_position` to be a string name indicating a single position value. For convenience, you can pass in a string indicating the X/Y/Z coordinate, here `get_position='[x, y, z]'`. You also have access to a small expression parser--in our `get_position` function here, we increase the size of the z coordinate times 10.\n",
    "\n",
    "Using `pydeck.data_utils.compute_view`, we'll zoom to the approximate center of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pydeck as pdk\n",
    "\n",
    "\n",
    "point_cloud = pdk.Layer(\n",
    "    'PointCloudLayer',\n",
    "    lidar[['x', 'y', 'z']],\n",
    "    get_position=['x', 'y', 'z * 10'],\n",
    "    get_normal=[0, 0, 1],\n",
    "    get_color=[255, 0, 100, 200],\n",
    "    pickable=True,  \n",
    "    auto_highlight=True,\n",
    "    point_size=1)\n",
    "\n",
    "\n",
    "view_state = pdk.data_utils.compute_view(lidar[['x', 'y']], 0.9)\n",
    "view_state.max_pitch = 360\n",
    "view_state.pitch = 80\n",
    "view_state.bearing = 120\n",
    "\n",
    "r = pdk.Deck(\n",
    "    point_cloud,\n",
    "    initial_view_state=view_state,\n",
    "    map_provider=None,\n",
    ")\n",
    "r.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from collections import deque\n",
    "\n",
    "# Choose a handful of frames to loop through\n",
    "frame_buffer = deque([42, 56, 81, 95])\n",
    "print('Press the stop icon to exit')\n",
    "while True:\n",
    "    current_frame = frame_buffer[0]\n",
    "    lidar = all_lidar[all_lidar['source'] == current_frame]\n",
    "    r.layers[0].get_position = '@@=[x / 10000, y / 10000, z * 10]'\n",
    "    r.layers[0].data = lidar.to_dict(orient='records')\n",
    "    frame_buffer.rotate()\n",
    "    r.update()\n",
    "    time.sleep(0.5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
