{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing and clustering scATAC PBMCs\n",
    "\n",
    "The data consist in ~3000 cells of human PBMCs. Cells were FAC sorted thus providing a ground truth for cell type identification. The data was produces by [Buenrostro et al. 2018](https://pubmed.ncbi.nlm.nih.gov/29706549/)\n",
    "The original count matrix (using bulk peaks as features) from Buenrostro et al. is available here: [GSE96772](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE96772)\n",
    "\n",
    "On a unix system, you can uncomment and run the following to download the count matri in its anndata format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !mkdir data\n",
    "# !wget https://www.dropbox.com/s/cwlaaxl70t27tb2/data_tutorial_buenrostro.tar.gz?dl=0 -O data/data_tutorial_buenrostro.tar.gz\n",
    "# !cd data; tar -xzf data_tutorial_buenrostro.tar.gz\n",
    "# !mkdir write"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If episcanpy isn't installed already. Or if some of the functions aren't working. Install the following version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install git+https://github.com/colomemaria/epiScanpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/annadanese/opt/anaconda3/lib/python3.7/site-packages/scanpy/api/__init__.py:7: FutureWarning: \n",
      "\n",
      "In a future version of Scanpy, `scanpy.api` will be removed.\n",
      "Simply use `import scanpy as sc` and `import scanpy.external as sce` instead.\n",
      "\n",
      "  FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "import scanpy as sc\n",
    "import anndata as ad\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import episcanpy.api as epi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.settings.set_figure_params(dpi=80, color_map='gist_earth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_file = 'write/buenrostro_pbmc.h5ad'  # the file that will store the analysis results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the count matrix into an [AnnData](https://anndata.readthedocs.io/en/latest/anndata.AnnData.html) object, which holds many slots for annotations and different representations of the data. It also comes with its own HDF5 file format: .h5ad.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 491436\n",
       "    obs: 'batch', 'cell_name'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata = ad.read('data/data_tutorial_buenrostro/all_buenrostro_bulk_peaks.h5ad')\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>batch</th>\n",
       "      <th>cell_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>BM1077-MPP-Frozen-160105-1.dedup.st.bam</th>\n",
       "      <td>0</td>\n",
       "      <td>BM1077-MPP-Frozen-160105-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-20160726-scATAC-BM1137-GMP3high-HYC-88.dedup.st.bam</th>\n",
       "      <td>0</td>\n",
       "      <td>singles-20160726-scATAC-BM1137-GMP3high-HYC-88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-160808-scATAC-BM1137-GMP2mid-LS-65.dedup.st.bam</th>\n",
       "      <td>0</td>\n",
       "      <td>singles-160808-scATAC-BM1137-GMP2mid-LS-65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-BM0828-LMPP-frozen-151105-20.dedup.st.bam</th>\n",
       "      <td>0</td>\n",
       "      <td>singles-BM0828-LMPP-frozen-151105-20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-160819-BM1137-CMP-LS-95.dedup.st.bam</th>\n",
       "      <td>0</td>\n",
       "      <td>singles-160819-BM1137-CMP-LS-95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BM1077-LMPP-Frozen-160107-40.dedup.st.bam</th>\n",
       "      <td>1</td>\n",
       "      <td>BM1077-LMPP-Frozen-160107-40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BM1077-MPP-Frozen-160105-74.dedup.st.bam</th>\n",
       "      <td>1</td>\n",
       "      <td>BM1077-MPP-Frozen-160105-74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-BM1214-GMP-160421-9.dedup.st.bam</th>\n",
       "      <td>1</td>\n",
       "      <td>singles-BM1214-GMP-160421-9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-BM0828-LMPP-frozen-151105-62.dedup.st.bam</th>\n",
       "      <td>1</td>\n",
       "      <td>singles-BM0828-LMPP-frozen-151105-62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>singles-BM0828-MEP-160420-43.dedup.st.bam</th>\n",
       "      <td>1</td>\n",
       "      <td>singles-BM0828-MEP-160420-43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2034 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   batch  \\\n",
       "BM1077-MPP-Frozen-160105-1.dedup.st.bam                0   \n",
       "singles-20160726-scATAC-BM1137-GMP3high-HYC-88....     0   \n",
       "singles-160808-scATAC-BM1137-GMP2mid-LS-65.dedu...     0   \n",
       "singles-BM0828-LMPP-frozen-151105-20.dedup.st.bam      0   \n",
       "singles-160819-BM1137-CMP-LS-95.dedup.st.bam           0   \n",
       "...                                                  ...   \n",
       "BM1077-LMPP-Frozen-160107-40.dedup.st.bam              1   \n",
       "BM1077-MPP-Frozen-160105-74.dedup.st.bam               1   \n",
       "singles-BM1214-GMP-160421-9.dedup.st.bam               1   \n",
       "singles-BM0828-LMPP-frozen-151105-62.dedup.st.bam      1   \n",
       "singles-BM0828-MEP-160420-43.dedup.st.bam              1   \n",
       "\n",
       "                                                                                         cell_name  \n",
       "BM1077-MPP-Frozen-160105-1.dedup.st.bam                                 BM1077-MPP-Frozen-160105-1  \n",
       "singles-20160726-scATAC-BM1137-GMP3high-HYC-88....  singles-20160726-scATAC-BM1137-GMP3high-HYC-88  \n",
       "singles-160808-scATAC-BM1137-GMP2mid-LS-65.dedu...      singles-160808-scATAC-BM1137-GMP2mid-LS-65  \n",
       "singles-BM0828-LMPP-frozen-151105-20.dedup.st.bam             singles-BM0828-LMPP-frozen-151105-20  \n",
       "singles-160819-BM1137-CMP-LS-95.dedup.st.bam                       singles-160819-BM1137-CMP-LS-95  \n",
       "...                                                                                            ...  \n",
       "BM1077-LMPP-Frozen-160107-40.dedup.st.bam                             BM1077-LMPP-Frozen-160107-40  \n",
       "BM1077-MPP-Frozen-160105-74.dedup.st.bam                               BM1077-MPP-Frozen-160105-74  \n",
       "singles-BM1214-GMP-160421-9.dedup.st.bam                               singles-BM1214-GMP-160421-9  \n",
       "singles-BM0828-LMPP-frozen-151105-62.dedup.st.bam             singles-BM0828-LMPP-frozen-151105-62  \n",
       "singles-BM0828-MEP-160420-43.dedup.st.bam                             singles-BM0828-MEP-160420-43  \n",
       "\n",
       "[2034 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display information stored in adata.obs\n",
    "adata.obs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['chr1_10279_10779', 'chr1_13252_13752', 'chr1_16019_16519',\n",
      "       'chr1_29026_29526', 'chr1_96364_96864', 'chr1_115440_115940',\n",
      "       'chr1_237535_238035', 'chr1_240811_241311', 'chr1_540469_540969',\n",
      "       'chr1_713909_714409',\n",
      "       ...\n",
      "       'chrX_154822578_154823078', 'chrX_154840821_154841321',\n",
      "       'chrX_154841449_154841949', 'chrX_154841956_154842456',\n",
      "       'chrX_154842517_154843017', 'chrX_154862057_154862557',\n",
      "       'chrX_154870909_154871409', 'chrX_154880741_154881241',\n",
      "       'chrX_154891824_154892324', 'chrX_154896342_154896842'],\n",
      "      dtype='object', name='index', length=491436)\n"
     ]
    }
   ],
   "source": [
    "# checking the variable names (bulk peaks coordinates)\n",
    "print(adata.var_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the FACs information from the file names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 491436\n",
       "    obs: 'batch', 'cell_name', 'facs_label'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata.obs['facs_label'] = ['MEP' if 'MEP' in line else line.split('.bam')[0].lstrip('singles-').split('BM')[-1].split('-')[1] for line in adata.obs['cell_name'].tolist()]\n",
    "adata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the additional metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label\tcell_type\r\n",
      "BM1077-CLP-Frozen-160106-13\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-14\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-2\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-21\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-27\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-3\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-36\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-42\tCLP\r\n",
      "BM1077-CLP-Frozen-160106-44\tCLP\r\n"
     ]
    }
   ],
   "source": [
    "!head 'data/data_tutorial_buenrostro/metadata.tsv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['BM1077-MPP-Frozen-160105-1',\n",
       "       'singles-20160726-scATAC-BM1137-GMP3high-HYC-88',\n",
       "       'singles-160808-scATAC-BM1137-GMP2mid-LS-65',\n",
       "       'singles-BM0828-LMPP-frozen-151105-20',\n",
       "       'singles-160819-BM1137-CMP-LS-95', 'BM1077-MPP-Frozen-160105-36',\n",
       "       'singles-20160726-scATAC-BM1214-CMP-LS-84',\n",
       "       'BM1077-CMP-Frozen-160106-21', 'singles-BM0106-HSC-SIM-160219-36',\n",
       "       'singles-BM1214-GMP-160421-60',\n",
       "       ...\n",
       "       'singles-160822-BM1137-CMP-LS-91',\n",
       "       'singles-BM0828-CMP-frozen-151118-69',\n",
       "       'singles-20160617-scATAC-BM1214-CMP-LS-40',\n",
       "       'singles-BM0828-GMP-151027-2',\n",
       "       'singles-20160726-scATAC-BM1214-CMP-LS-19',\n",
       "       'BM1077-LMPP-Frozen-160107-40', 'BM1077-MPP-Frozen-160105-74',\n",
       "       'singles-BM1214-GMP-160421-9', 'singles-BM0828-LMPP-frozen-151105-62',\n",
       "       'singles-BM0828-MEP-160420-43'],\n",
       "      dtype='object', length=2034)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# format the cell names to match the metadata file\n",
    "adata.obs_names = [x.split('/')[-1].split('.dedup.st.bam')[0] for x in adata.obs_names.tolist()]\n",
    "adata.obs_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 491436\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type\\n'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "epi.pp.load_metadata(adata,\n",
    "                     'data/data_tutorial_buenrostro/metadata.tsv',\n",
    "                     separator='\\t')\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata.obs['cell_type'] = [x.rstrip('\\n') for x in adata.obs['cell_type\\n']]\n",
    "del adata.obs['cell_type\\n'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 491436\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>facs_label</th>\n",
       "      <th>CLP</th>\n",
       "      <th>CMP</th>\n",
       "      <th>GMP</th>\n",
       "      <th>GMP1low</th>\n",
       "      <th>GMP2mid</th>\n",
       "      <th>GMP3high</th>\n",
       "      <th>HSC</th>\n",
       "      <th>LMPP</th>\n",
       "      <th>MCP</th>\n",
       "      <th>MEP</th>\n",
       "      <th>MPP</th>\n",
       "      <th>UNK</th>\n",
       "      <th>mono</th>\n",
       "      <th>pDC</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cell_type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CLP</th>\n",
       "      <td>78</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CMP</th>\n",
       "      <td>0</td>\n",
       "      <td>502</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GMP</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>216</td>\n",
       "      <td>68</td>\n",
       "      <td>44</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HSC</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>347</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LMPP</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>160</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MEP</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>138</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MPP</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UNK</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mono</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pDC</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>73</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "facs_label  CLP  CMP  GMP  GMP1low  GMP2mid  GMP3high  HSC  LMPP  MCP  MEP  \\\n",
       "cell_type                                                                    \n",
       "CLP          78    0    0        0        0         0    0     0    0    0   \n",
       "CMP           0  502    0        0        0         0    0     0    0    0   \n",
       "GMP           0    0  216       68       44        74    0     0    0    0   \n",
       "HSC           0    0    0        0        0         0  347     0    0    0   \n",
       "LMPP          0    0    0        0        0         0    0   160    0    0   \n",
       "MEP           0    0    0        0        0         0    0     0    0  138   \n",
       "MPP           0    0    0        0        0         0    0     0    0    0   \n",
       "UNK           0    0    0        0        0         0    0     0    0    0   \n",
       "mono          0    0    0        0        0         0    0     0    0    0   \n",
       "pDC           0    0    0        0        0         0    0     0   73    0   \n",
       "\n",
       "facs_label  MPP  UNK  mono  pDC  \n",
       "cell_type                        \n",
       "CLP           0    0     0    0  \n",
       "CMP           0    0     0    0  \n",
       "GMP           0    0     0    0  \n",
       "HSC           0    0     0    0  \n",
       "LMPP          0    0     0    0  \n",
       "MEP           0    0     0    0  \n",
       "MPP         142    0     0    0  \n",
       "UNK           0   60     0    0  \n",
       "mono          0    0    64    0  \n",
       "pDC           0    0     0   68  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the 2 different annotation obtained\n",
    "pd.crosstab(adata.obs['cell_type'], adata.obs['facs_label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load gene/transcript annotation\n",
    "\n",
    "Download annotation file (the data are aligned on hg19)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !wget ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_19/gencode.v19.annotation.gtf.gz -O data/data_tutorial_buenrostro/gencode.v19.annotation.gtf.gz\n",
    "# !cd data/data_tutorial_buenrostro; gunzip gencode.v19.annotation.gtf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.tl.find_genes(adata,\n",
    "           gtf_file='data/data_tutorial_buenrostro/gencode.v19.annotation.gtf',\n",
    "           key_added='transcript_annotation',\n",
    "           upstream=2000,\n",
    "           feature_type='transcript',\n",
    "           annotation='HAVANA',\n",
    "           raw=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 491436\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type'\n",
       "    var: 'transcript_annotation'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check if the data matrix is binary - if not, binarize the data matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(np.max(adata.X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here, the data is already binary. If not, run the following commands\n",
    "# epi.pp.binarize(adata)\n",
    "# print(np.max(adata.X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Proceed to basic filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 491436\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type'\n",
       "    var: 'transcript_annotation'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove any potential empty features or barcodes\n",
    "epi.pp.filter_cells(adata, min_features=1)\n",
    "epi.pp.filter_features(adata, min_cells=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 467226\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filtered out 24210 peaks\n",
    "adata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quality controls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 2034 × 467226\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata.obs['log_nb_features'] = [np.log10(x) for x in adata.obs['nb_features']]\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "... storing 'facs_label' as categorical\n",
      "... storing 'cell_type' as categorical\n",
      "... storing 'transcript_annotation' as categorical\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 73528 characters]\n",
      "text/plain": [
       "<Figure size 372.24x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 281,
       "width": 310
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 101152 characters]\n",
      "text/plain": [
       "<Figure size 372.24x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 281,
       "width": 289
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.violin(adata, ['nb_features'])\n",
    "epi.pl.violin(adata, ['log_nb_features'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34312 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 318
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 45236 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 331
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# set a minimum number of cells to keep \n",
    "min_features = 1000\n",
    "\n",
    "epi.pp.coverage_cells(adata, binary=True, log=False, bins=50,\n",
    "               threshold=min_features, save='Buenrostro_bulk_peaks_coverage_cells.png')\n",
    "epi.pp.coverage_cells(adata, binary=True, log=10, bins=50,\n",
    "               threshold=min_features, save='Buenrostro_bulk_peaks_coverage_cells_log10.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 47644 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 344
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 52480 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 350
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# minimum number of cells sharing a feature\n",
    "min_cells = 5\n",
    "epi.pp.coverage_features(adata, binary=True, log=False, \n",
    "                        threshold=min_cells, save='Buenrostro_bulk_peaks_coverage_peaks.png')\n",
    "epi.pp.coverage_features(adata, binary=True, log=True, \n",
    "                        threshold=min_cells, save='Buenrostro_bulk_peaks_coverage_peaks_log10.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Actually proceed to filter the cells and peaks based on the QC plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 1945 × 467226\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_features = 1000\n",
    "epi.pp.filter_cells(adata, min_features=min_features)\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 1945 × 311035\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min_cells = 5\n",
    "epi.pp.filter_features(adata, min_cells=min_cells)\n",
    "adata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looking at the QC plots after filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 45336 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 331
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pp.coverage_cells(adata, binary=True, log='log10', bins=50, threshold=min_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54588 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 350
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pp.coverage_features(adata, binary=True, log='log10', bins=50, threshold=min_cells)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identifying the most variable features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We aim to select a cuttof after the elbow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 45268 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 2 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 317
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pp.cal_var(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 50224 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 320
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 50648 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 300,
       "width": 314
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "min_score_value = 0.515\n",
    "nb_feature_selected = 120000\n",
    "epi.pl.variability_features(adata,log=None,\n",
    "                     min_score=min_score_value, nb_features=nb_feature_selected,\n",
    "                     save='variability_features_plot_bonemarrow_peakmatrix.png')\n",
    "\n",
    "epi.pl.variability_features(adata,log='log10',\n",
    "                     min_score=min_score_value, nb_features=nb_feature_selected,\n",
    "                     save='variability_features_plot_bonemarrow_peakmatrix_log10.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the current matrix in the raw layer\n",
    "adata.raw = adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new AnnData containing only the most variable features\n",
    "adata = epi.pp.select_var_feature(adata,\n",
    "                                  nb_features=nb_feature_selected,\n",
    "                                  show=False,\n",
    "                                  copy=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "View of AnnData object with n_obs × n_vars = 1945 × 122511\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 77808 characters]\n",
      "text/plain": [
       "<Figure size 372.24x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 281,
       "width": 310
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 115344 characters]\n",
      "text/plain": [
       "<Figure size 372.24x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 281,
       "width": 289
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.violin(adata, ['nb_features'])\n",
    "epi.pl.violin(adata, ['log_nb_features'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "View of AnnData object with n_obs × n_vars = 1945 × 122511\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Trying to set attribute `.obs` of view, copying.\n"
     ]
    }
   ],
   "source": [
    "epi.pp.filter_cells(adata, min_features=2000)\n",
    "epi.pp.filter_cells(adata, max_features=25000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 1722 × 122511\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 110752 characters]\n",
      "text/plain": [
       "<Figure size 372.24x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 281,
       "width": 310
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 119728 characters]\n",
      "text/plain": [
       "<Figure size 372.24x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 281,
       "width": 289
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.violin(adata, ['nb_features'])\n",
    "epi.pl.violin(adata, ['log_nb_features'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quick visualisation before normalisation\n",
    "\n",
    "The lazy function compute PCA, neighbor graph umap and tsne embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " NumbaWarning:/Users/annadanese/opt/anaconda3/lib/python3.7/site-packages/numba/np/ufunc/parallel.py:355: \u001b[1mThe TBB threading layer requires TBB version 2019.5 or later i.e., TBB_INTERFACE_VERSION >= 11005. Found TBB_INTERFACE_VERSION = 11000. The TBB threading layer is disabled.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Consider installing the package MulticoreTSNE (https://github.com/DmitryUlyanov/Multicore-TSNE). Even for n_jobs=1 this speeds up the computation considerably and might yield better converged results.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 426148 characters]\n",
      "text/plain": [
       "<Figure size 832x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 299,
       "width": 740
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pp.lazy(adata)\n",
    "sc.pl.umap(adata, color=['nb_features', 'cell_type'], wspace=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the current version of the matrix (binary, not normalised) in a layer of the Anndata.\n",
    "adata.layers['binary'] = adata.X.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Library size correction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.pp.normalize_total(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the current version of the matrix (normalised) in a layer of the Anndata.\n",
    "adata.layers['normalised'] = adata.X.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 1722 × 122511\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'\n",
       "    uns: 'pca', 'neighbors', 'umap', 'cell_type_colors'\n",
       "    obsm: 'X_pca', 'X_tsne', 'X_umap'\n",
       "    varm: 'PCs'\n",
       "    layers: 'binary', 'normalised'\n",
       "    obsp: 'distances', 'connectivities'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 287720 characters]\n",
      "text/plain": [
       "<Figure size 772.8x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 299,
       "width": 694
      }
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " RuntimeWarning:/Users/annadanese/opt/anaconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py:214: Glyph 8942 missing from current font.\n",
      " RuntimeWarning:/Users/annadanese/opt/anaconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py:183: Glyph 8942 missing from current font.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 501400 characters]\n",
      "text/plain": [
       "<Figure size 960x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 386,
       "width": 836
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 43836 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 321,
       "width": 308
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sc.settings.set_figure_params(dpi=80, color_map='gist_earth')\n",
    "epi.pl.pca_overview(adata, color=['nb_features', 'cell_type'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Consider installing the package MulticoreTSNE (https://github.com/DmitryUlyanov/Multicore-TSNE). Even for n_jobs=1 this speeds up the computation considerably and might yield better converged results.\n"
     ]
    }
   ],
   "source": [
    "epi.pp.lazy(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 409276 characters]\n",
      "text/plain": [
       "<Figure size 772.8x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 299,
       "width": 694
      }
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " RuntimeWarning:/Users/annadanese/opt/anaconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py:214: Glyph 8942 missing from current font.\n",
      " RuntimeWarning:/Users/annadanese/opt/anaconda3/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py:183: Glyph 8942 missing from current font.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 506596 characters]\n",
      "text/plain": [
       "<Figure size 960x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 390,
       "width": 827
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 52148 characters]\n",
      "text/plain": [
       "<Figure size 320x320 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 321,
       "width": 316
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.pca_overview(adata, color=['nb_features', 'cell_type'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 569160 characters]\n",
      "text/plain": [
       "<Figure size 1248x320 with 5 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 299,
       "width": 1074
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['nb_features', 'log_nb_features', 'cell_type'], wspace=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Log transform. \n",
    "This isn't a mandatory step. It really depends of the dataset. Here, the number of features recovered per cell vary greatly. Thus, it benefits from a log transformation after normalisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.pp.log1p(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Consider installing the package MulticoreTSNE (https://github.com/DmitryUlyanov/Multicore-TSNE). Even for n_jobs=1 this speeds up the computation considerably and might yield better converged results.\n"
     ]
    }
   ],
   "source": [
    "epi.pp.lazy(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 457992 characters]\n",
      "text/plain": [
       "<Figure size 832x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 299,
       "width": 740
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['nb_features', 'cell_type'], wspace=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Louvain clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 1722 × 122511\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'\n",
       "    uns: 'pca', 'neighbors', 'umap', 'cell_type_colors', 'log1p'\n",
       "    obsm: 'X_pca', 'X_tsne', 'X_umap'\n",
       "    varm: 'PCs'\n",
       "    layers: 'binary', 'normalised'\n",
       "    obsp: 'distances', 'connectivities'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.tl.louvain(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 578160 characters]\n",
      "text/plain": [
       "<Figure size 1344x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1179
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['louvain', 'cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0\n",
      "got 17 at resolution 1.5\n",
      "step 1\n",
      "got 11 at resolution 0.75\n",
      "step 2\n",
      "got 12 at resolution 1.125\n",
      "step 3\n",
      "got 15 at resolution 1.3125\n",
      "step 4\n",
      "got 14 at resolution 1.21875\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1.21875,\n",
       " AnnData object with n_obs × n_vars = 1722 × 122511\n",
       "     obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features', 'louvain'\n",
       "     var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'\n",
       "     uns: 'pca', 'neighbors', 'umap', 'cell_type_colors', 'log1p', 'louvain', 'louvain_colors', 'facs_label_colors'\n",
       "     obsm: 'X_pca', 'X_tsne', 'X_umap'\n",
       "     varm: 'PCs'\n",
       "     layers: 'binary', 'normalised'\n",
       "     obsp: 'distances', 'connectivities')"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "epi.tl.getNClusters(adata, n_cluster=14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 577764 characters]\n",
      "text/plain": [
       "<Figure size 1344x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1179
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['louvain', 'cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. kmeans clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.tl.kmeans(adata, num_clusters=14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 586028 characters]\n",
      "text/plain": [
       "<Figure size 1344x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1179
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['kmeans', 'cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. hierarchical clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.tl.hc(adata, num_clusters=14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 582852 characters]\n",
      "text/plain": [
       "<Figure size 1344x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1179
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['hc', 'cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. leiden clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.tl.leiden(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 579212 characters]\n",
      "text/plain": [
       "<Figure size 1344x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1179
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['leiden', 'cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0\n",
      "got 20 at resolution 1.5\n",
      "step 1\n",
      "got 13 at resolution 0.75\n",
      "step 2\n",
      "got 16 at resolution 1.125\n",
      "step 3\n",
      "got 16 at resolution 0.9375\n",
      "step 4\n",
      "got 14 at resolution 0.84375\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.84375,\n",
       " AnnData object with n_obs × n_vars = 1722 × 122511\n",
       "     obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features', 'louvain', 'kmeans', 'hc', 'leiden'\n",
       "     var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'\n",
       "     uns: 'pca', 'neighbors', 'umap', 'cell_type_colors', 'log1p', 'louvain', 'louvain_colors', 'facs_label_colors', 'kmeans_colors', 'hc_colors', 'leiden', 'leiden_colors'\n",
       "     obsm: 'X_pca', 'X_tsne', 'X_umap'\n",
       "     varm: 'PCs'\n",
       "     layers: 'binary', 'normalised'\n",
       "     obsp: 'distances', 'connectivities')"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "epi.tl.getNClusters(adata, n_cluster=14, method='leiden')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 573460 characters]\n",
      "text/plain": [
       "<Figure size 1344x320 with 3 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1179
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.umap(adata, color=['leiden', 'cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Comparison and Adjusted Rand Index (ARI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 415812 characters]\n",
      "text/plain": [
       "<Figure size 896x320 with 2 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 819
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# True labels\n",
    "epi.pl.umap(adata, color=['cell_type', 'facs_label'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 676168 characters]\n",
      "text/plain": [
       "<Figure size 1792x320 with 4 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 316,
       "width": 1486
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Clusters\n",
    "epi.pl.umap(adata, color=['louvain', 'kmeans', 'hc', 'leiden'], wspace=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A few metrics to compare the quality of the different clusterings. \n",
    "For all these methods. The best value possible is 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1)** Compute the **Adjusted Rand Index** for the different clustering to determine which one perform best.\n",
    "It computes a similarity measure between two clusterings (predicted and true labels)by counting cells that are assigned in the same or different clusters between the two clusterings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "louvain:\t 0.48143459493055457\n",
      "kmeans:\t 0.34473337260007614\n",
      "hc:\t 0.4869084669795262\n",
      "leiden:\t 0.5215454468198376\n"
     ]
    }
   ],
   "source": [
    "print('louvain:\\t', epi.tl.ARI(adata, 'louvain', 'cell_type'))\n",
    "print('kmeans:\\t', epi.tl.ARI(adata, 'kmeans', 'cell_type'))\n",
    "print('hc:\\t', epi.tl.ARI(adata, 'hc', 'cell_type'))\n",
    "print('leiden:\\t', epi.tl.ARI(adata, 'leiden', 'cell_type'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2)** Compute the **Homogeneity score**. The score is higher when the different clusters contain only cells with the same ground truth label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "louvain:\t 0.6027317462511458\n",
      "kmeans:\t 0.5218117148571952\n",
      "hc:\t 0.597155399046304\n",
      "leiden:\t 0.6462131688120222\n"
     ]
    }
   ],
   "source": [
    "print('louvain:\\t', epi.tl.homogeneity(adata, 'louvain', 'cell_type'))\n",
    "print('kmeans:\\t', epi.tl.homogeneity(adata, 'kmeans', 'cell_type'))\n",
    "print('hc:\\t', epi.tl.homogeneity(adata, 'hc', 'cell_type'))\n",
    "print('leiden:\\t', epi.tl.homogeneity(adata, 'leiden', 'cell_type'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3)** Compute the **Adjusted Mutual Information**, it is measure of the similarity between two labels of the same data, while accounting for chance (the Mutual information is generally higher for two set of labels with a large number of clusters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "louvain:\t 0.6489850295660261\n",
      "kmeans:\t 0.5151669485843612\n",
      "hc:\t 0.607839434804083\n",
      "leiden:\t 0.6905252709103069\n"
     ]
    }
   ],
   "source": [
    "print('louvain:\\t', epi.tl.AMI(adata, 'louvain', 'cell_type'))\n",
    "print('kmeans:\\t', epi.tl.AMI(adata, 'kmeans', 'cell_type'))\n",
    "print('hc:\\t', epi.tl.AMI(adata, 'hc', 'cell_type'))\n",
    "print('leiden:\\t', epi.tl.AMI(adata, 'leiden', 'cell_type'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the processed Anndata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata.write(results_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differential chromatin analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 1722 × 122511\n",
       "    obs: 'batch', 'cell_name', 'facs_label', 'label', 'cell_type', 'nb_features', 'log_nb_features', 'louvain', 'kmeans', 'hc', 'leiden'\n",
       "    var: 'transcript_annotation', 'n_cells', 'commonness', 'prop_shared_cells', 'variability_score'\n",
       "    uns: 'cell_type_colors', 'facs_label_colors', 'hc_colors', 'kmeans_colors', 'leiden', 'leiden_colors', 'louvain', 'louvain_colors', 'neighbors', 'pca', 'umap'\n",
       "    obsm: 'X_pca', 'X_tsne', 'X_umap'\n",
       "    varm: 'PCs'\n",
       "    layers: 'binary', 'normalised'\n",
       "    obsp: 'connectivities', 'distances'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata = ad.read(results_file)\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "epi.tl.rank_features(adata, 'louvain', omic='ATAC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1776152 characters]\n",
      "text/plain": [
       "<Figure size 1280x1280 with 14 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 1095,
       "width": 1071
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.rank_feat_groups(adata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 787824 characters]\n",
      "text/plain": [
       "<Figure size 1280x1280 with 14 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 1048,
       "width": 1071
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.rank_feat_groups(adata, feature_symbols='transcript_annotation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: dendrogram data not found (using key=dendrogram_louvain). Running `sc.tl.dendrogram` with default parameters. For fine tuning it is recommended to run `sc.tl.dendrogram` independently.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 801352 characters]\n",
      "text/plain": [
       "<Figure size 3664x344 with 4 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 478,
       "width": 2944
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epi.pl.rank_feat_groups_matrixplot(adata)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
