{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# To download 5k Peripheral blood mononuclear cells (PBMCs) from a healthy donor (Next GEM v1.1)\n",
    "\n",
    "data available on the 10x Genomics website here: https://support.10xgenomics.com/single-cell-atac/datasets/1.2.0/atac_pbmc_5k_nextgem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# downloading the fragment file --> atac_pbmc_5k_nextgem_fragments.tsv.gz\n",
    "#!wget https://cf.10xgenomics.com/samples/cell-atac/1.2.0/atac_pbmc_5k_nextgem/atac_pbmc_5k_nextgem_fragments.tsv.gz\n",
    "\n",
    "# downloading the index of the fragment file --> atac_pbmc_5k_nextgem_fragments.tsv.gz.tbi\n",
    "#!wget https://cf.10xgenomics.com/samples/cell-atac/1.2.0/atac_pbmc_5k_nextgem/atac_pbmc_5k_nextgem_fragments.tsv.gz.tbi\n",
    "\n",
    "# downloading per barcode metrics --> atac_pbmc_5k_nextgem_singlecell.csv\n",
    "#!wget https://cf.10xgenomics.com/samples/cell-atac/1.2.0/atac_pbmc_5k_nextgem/atac_pbmc_5k_nextgem_singlecell.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to install the latest development version of episcanpy\n",
    "#!pip install git+https://github.com/colomemaria/epiScanpy@update_load_features\n",
    "\n",
    "# to install the latest stable version of episcanpy\n",
    "#!pip install episcanpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import episcanpy as epi\n",
    "import anndata as ad "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# To select a feature space\n",
    "\n",
    "To build a count matrix using scATAC-seq data, it is important to choose a feature space on which to build the count matrix. \n",
    "\n",
    "epiScanpy offers to load different set of custom features (bed, gtf and gff input files accepted), or to load peaks called using macs2 (see below on how to call peaks). Alternatively, epiScanpy can also generate windows of a given size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading annotation files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_bed_file = \"tmp_file_merged_peaks.bed\"   \n",
    "features = epi.ct.load_features(test_bed_file, path=\"\", input_file_format=None, sort=True)\n",
    "\n",
    "test_gff_file = \"HGAP3_Tb427v10.gff\"\n",
    "features = epi.ct.load_features_gff(test_gff_file,\n",
    "                             filter_per_source=[\"AUGUSTUS\", \"Pfam\"],\n",
    "                             filter_per_feature_type='gene')\n",
    "\n",
    "test_gtf_file = \"gencode.vM17.basic.annotation.gtf\"\n",
    "features = epi.ct.load_features_gtf(test_gtf_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling peaks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO  @ Wed, 24 Mar 2021 11:13:57: \n",
      "# Command line: callpeak --nomodel --keep-dup all --extsize 200 --shift -100 -t ./atac_pbmc_5k_nextgem_fragments.tsv.gz\n",
      "# ARGUMENTS LIST:\n",
      "# name = NA\n",
      "# format = AUTO\n",
      "# ChIP-seq file = ['./atac_pbmc_5k_nextgem_fragments.tsv.gz']\n",
      "# control file = None\n",
      "# effective genome size = 2.70e+09\n",
      "# band width = 300\n",
      "# model fold = [5, 50]\n",
      "# qvalue cutoff = 5.00e-02\n",
      "# The maximum gap between significant sites is assigned as the read length/tag size.\n",
      "# The minimum length of peaks is assigned as the predicted fragment length \"d\".\n",
      "# Larger dataset will be scaled towards smaller dataset.\n",
      "# Range for calculating regional lambda is: 10000 bps\n",
      "# Broad region calling is off\n",
      "# Paired-End mode is off\n",
      " \n",
      "INFO  @ Wed, 24 Mar 2021 11:13:57: #1 read tag files... \n",
      "INFO  @ Wed, 24 Mar 2021 11:13:57: #1 read treatment tags... \n",
      "INFO  @ Wed, 24 Mar 2021 11:13:57: Detected format is: BED \n",
      "INFO  @ Wed, 24 Mar 2021 11:13:57: * Input file is gzipped. \n",
      "INFO  @ Wed, 24 Mar 2021 11:13:59:  1000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:02:  2000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:03:  3000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:06:  4000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:08:  5000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:10:  6000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:11:  7000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:13:  8000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:15:  9000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:17:  10000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:20:  11000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:22:  12000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:25:  13000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:27:  14000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:28:  15000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:30:  16000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:32:  17000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:34:  18000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:36:  19000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:38:  20000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:40:  21000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:41:  22000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:43:  23000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:45:  24000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:46:  25000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:48:  26000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:50:  27000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:52:  28000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:54:  29000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:56:  30000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:14:58:  31000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:00:  32000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:02:  33000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:04:  34000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:05:  35000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:07:  36000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:11:  37000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:13:  38000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:15:  39000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:18:  40000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:20:  41000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:22:  42000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:24:  43000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:25:  44000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:27:  45000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:29:  46000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:30:  47000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:32:  48000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:34:  49000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:35:  50000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:37:  51000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:39:  52000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:41:  53000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:42:  54000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:44:  55000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:46:  56000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:48:  57000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:49:  58000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:51:  59000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:52:  60000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:54:  61000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:56:  62000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:15:58:  63000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:00:  64000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:02:  65000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:04:  66000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:05:  67000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:07:  68000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:09:  69000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:11:  70000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:13:  71000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:14:  72000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:16:  73000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:17:  74000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:20:  75000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:22:  76000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:23:  77000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:26:  78000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:27:  79000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:29:  80000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:31:  81000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:34:  82000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:36:  83000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:39:  84000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:41:  85000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:42:  86000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:44:  87000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:46:  88000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:48:  89000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:51:  90000000 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #1 tag size is determined as 150 bps \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #1 tag size = 150.0 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #1  total tags in treatment: 90498340 \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #1 finished! \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #2 Build Peak Model... \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #2 Skipped... \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #2 Use 200 as fragment length \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #2 Sequencing ends will be shifted towards 5' by 100 bp(s) \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #3 Call peaks... \n",
      "INFO  @ Wed, 24 Mar 2021 11:16:53: #3 Pre-compute pvalue-qvalue table... \n",
      "INFO  @ Wed, 24 Mar 2021 11:19:53: #3 Call peaks for each chromosome... \n",
      "INFO  @ Wed, 24 Mar 2021 11:21:22: #4 Write output xls file... NA_peaks.xls \n",
      "INFO  @ Wed, 24 Mar 2021 11:21:22: #4 Write peak in narrowPeak format file... NA_peaks.narrowPeak \n",
      "INFO  @ Wed, 24 Mar 2021 11:21:23: #4 Write summits bed file... NA_summits.bed \n",
      "INFO  @ Wed, 24 Mar 2021 11:21:23: Done! \n"
     ]
    }
   ],
   "source": [
    "# calling peaks --> it will create file in the local directory\n",
    "!macs2 callpeak --nomodel --keep-dup all --extsize 200 --shift -100 -t ./atac_pbmc_5k_nextgem_fragments.tsv.gz\n",
    "\n",
    "\n",
    "# load the peaks and normalise their size to 500 (250*2)\n",
    "peaks = epi.ct.load_peaks(\"NA_peaks.narrowPeak\")\n",
    "epi.ct.norm_peaks(peaks, extension=250)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chr1\t237619\t237838\tNA_peak_1\t230\t.\t6.36068\t25.1441\t23.0551\t44\r\n",
      "chr1\t565193\t565410\tNA_peak_2\t388\t.\t8.56744\t41.0653\t38.8709\t132\r\n",
      "chr1\t569269\t569499\tNA_peak_3\t1794\t.\t23.2359\t182.034\t179.409\t61\r\n",
      "chr1\t713651\t714402\tNA_peak_4\t13223\t.\t25.0938\t1326.22\t1322.39\t403\r\n",
      "chr1\t752254\t752832\tNA_peak_5\t2531\t.\t21.7928\t255.88\t253.125\t392\r\n",
      "chr1\t762357\t763179\tNA_peak_6\t10143\t.\t25.9794\t1017.91\t1014.39\t515\r\n",
      "chr1\t779439\t780172\tNA_peak_7\t378\t.\t8.43763\t40.0713\t37.8825\t586\r\n",
      "chr1\t793385\t793640\tNA_peak_8\t180\t.\t5.58182\t20.0898\t18.0455\t81\r\n",
      "chr1\t804943\t805512\tNA_peak_9\t7684\t.\t30.203\t771.717\t768.419\t318\r\n",
      "chr1\t832576\t832964\tNA_peak_10\t172\t.\t5.45201\t19.2808\t17.2444\t197\r\n"
     ]
    }
   ],
   "source": [
    "# quick look at the file\n",
    "!head NA_peaks.narrowPeak"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the peaks and normalise their size to 500 (250*2)\n",
    "peaks = epi.ct.load_peaks(\"NA_peaks.narrowPeak\")\n",
    "epi.ct.norm_peaks(peaks, extension=250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### generating windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "windows = epi.ct.make_windows(5000, chromosomes='human') # generate 5000bp windows across the huamn genome. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now, to build the count matrix\n",
    "In this example we are using the peak features and only 2 threads. For a faster result, you should consider using more threads.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chromosome 1: 684.0747020244598 sec\n",
      "Chromosome 2: 503.49666714668274 sec\n",
      "Chromosome 3: 420.7269206047058 sec\n",
      "Chromosome 4: 267.5270185470581 sec\n",
      "Chromosome 5: 335.92078971862793 sec\n",
      "Chromosome 6: 483.4750199317932 sec\n",
      "Chromosome 7: 381.748735666275 sec\n",
      "Chromosome 8: 412.0700845718384 sec\n",
      "Chromosome 9: 281.22125816345215 sec\n",
      "Chromosome 10: 415.1694209575653 sec\n",
      "Chromosome 11: 479.1664354801178 sec\n",
      "Chromosome 12: 608.7532119750977 sec\n",
      "Chromosome 13: 169.2280580997467 sec\n",
      "Chromosome 14: 213.66552996635437 sec\n",
      "Chromosome 15: 276.6346945762634 sec\n",
      "Chromosome 16: 424.7526617050171 sec\n",
      "Chromosome 17: 550.9005522727966 sec\n",
      "Chromosome 18: 147.62225031852722 sec\n",
      "Chromosome 19: 571.224552154541 sec\n",
      "Chromosome 20: 233.46129488945007 sec\n",
      "Chromosome 21: 92.89729404449463 sec\n",
      "Chromosome 22: 178.4060080051422 sec\n",
      "Chromosome X: 196.7464315891266 sec\n",
      "Chromosome Y: 66.34237909317017 sec\n",
      "All data contains:\n",
      "AnnData object with n_obs × n_vars = 400798 × 107529\n",
      "Total time is 8447.611531734467 sec\n"
     ]
    }
   ],
   "source": [
    "adata = epi.ct.bld_mtx_bed(fragment_file=\"atac_pbmc_5k_nextgem_fragments.tsv.gz\",\n",
    "                          feature_region=peaks, \n",
    "                          thread=2, \n",
    "                          save='atac_pbmc_5k_nextgem_fragments_macs2_peaks.h5ad')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's now build the geneactivity matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 400798 × 107529"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adata = ad.read(\"atac_pbmc_5k_nextgem_fragments_macs2_peaks.h5ad\")\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnnData object with n_obs × n_vars = 5876 × 107529\n",
       "    obs: 'nb_features'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### filtering low quality barcodes\n",
    "epi.pp.filter_cells(adata, min_features=100)\n",
    "adata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# building the gene activity matrix\n",
    "activity = epi.tl.geneactivity(adata,\n",
    "                               gtf_file=\"gencode.v28.annotation.gtf\",\n",
    "                              annotation='HAVANA')\n",
    "activity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activity.write(\"atac_pbmc_5k_nextgem_fragments_gene_activity.h5ad\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
