pipeline_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing & plotting data pipeline example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates the main components of the data processing and plotting pipeline for parsing simulation files (`.json`) into pickled numpy arrays (`.pkl`) and subsequently selecting, plotting, and analyzing various subsets or all of the data. This notebook shows the logical order in which data is processed to obtain the results in the paper."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Realistic co-culture `dish`, `tissue`, and vasculature graph imaging\n",
    "2. Heuristic data plotting\n",
    "3. Simulated data processing & plotting\n",
    "    - 3.1 Parse sample co-culture `dish` data\n",
    "    - 3.2 Analyze parsed sample co-culture `dish` data\n",
    "    - 3.3 Subset sample co-culture `dish` data\n",
    "    - 3.4 Plot full subsetted co-culture `dish` and `tissue` data\n",
    "    - 3.5 Multi-feature & outcome analysis of full co-culture `dish` and `tissue` data\n",
    "4. Experimental literature data plotting\n",
    "6. Ranked data plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Realistic co-culture `dish`, `tissue`, and vasculature graph imaging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Images of the simulations in each context are generated to highlight the differences in cancer and healthy cell spatial distributions over time.\n",
    "\n",
    "The main function `image` takes in a `.json` file output from the simulation and produces an image of either the populations, cell states, volume density, or graphs. For this analysis, only the population and graph figures were used. \n",
    "\n",
    "The population images were generated for the following files:\n",
    "    \n",
    "    VITRO_DISH_TREAT_CH_0_NA_NA_1000_100_00.json\n",
    "    VIVO_TISSUE_TREAT_CH_0_NA_NA_1000_100_00.json\n",
    "\n",
    "The graph images were generated for the following file:\n",
    "\n",
    "    VIVO_TISSUE_TREAT_CH_0_NA_NA_1000_100_00.GRAPH.json\n",
    "    \n",
    "These represent the untreated realistic co-culture `dish` and `tissue` simulations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "+ `DATA_PATH` variables are the path to subsetted data files (`.json` files generated from simulation output)\n",
    "+ `...TIMES` variables indicate which time points to make images at\n",
    "+ `SIZE` variable indicates the size to make the image\n",
    "+ `POPS_TO_IGNORE` indicate which cell population numbers to ignore (where CAR T-cell populations are listed, but are not present in these untreated simulations)\n",
    "+ `BGCOL` indicates what color to make the background of the image\n",
    "+ `RADIUS` indicates the simulation radius out which to draw to (cells stop at radius 36, but the graph exists within the margins and out to the full radius of 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Untreated co-culture dish images\n",
    "DATA_PATH_IMAGE_UNTREATED_COCULTURE_CELLS = 'examples/files/full/coculture/jsons/'\n",
    "DATA_PATH_IMAGE_UNTREATED_COCULTURE_SVG = 'examples/figures/coculture/images/'\n",
    "DISH_TIMES = '0,4,7'\n",
    "\n",
    "# Untreated tissue images\n",
    "DATA_PATH_IMAGE_UNTREATED_TISSUE_CELLS = 'examples/files/full/tissue/jsons/cells/'\n",
    "DATA_PATH_IMAGE_UNTREATED_TISSUE_SVG = 'examples/figures/tissue/images/'\n",
    "TISSUE_TIMES = '1,16,31'\n",
    "\n",
    "# Population image specifications\n",
    "SIZE = '5'\n",
    "POPS_TO_IGNORE = '2,3'\n",
    "BGCOL = '#FFFFFF'\n",
    "RADIUS = '40'\n",
    "\n",
    "# Tissue graph image specifications\n",
    "DATA_PATH_IMAGE_UNTREATED_TISSUE_GRAPH = 'examples/files/full/tissue/jsons/graph/'\n",
    "DATA_PATH_IMAGE_UNTREATED_TISSUE_SVG = 'examples/figures/tissue/images/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Image full untreated realistic co-culture `dish` and `tissue` data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.image.image import image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "image(DATA_PATH_IMAGE_UNTREATED_COCULTURE_CELLS, DATA_PATH_IMAGE_UNTREATED_COCULTURE_SVG, size=SIZE,\n",
    "      time=DISH_TIMES, ignore=POPS_TO_IGNORE, radius=RADIUS, bgcol=BGCOL, pops=True)\n",
    "image(DATA_PATH_IMAGE_UNTREATED_TISSUE_CELLS, DATA_PATH_IMAGE_UNTREATED_TISSUE_SVG, size=SIZE, \n",
    "      time=TISSUE_TIMES, ignore=POPS_TO_IGNORE, radius=RADIUS, bgcol=BGCOL, pops=True)\n",
    "image(DATA_PATH_IMAGE_UNTREATED_TISSUE_GRAPH, DATA_PATH_IMAGE_UNTREATED_TISSUE_SVG, size=SIZE, \n",
    "      time=TISSUE_TIMES, radius=RADIUS, graph=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Heuristic data plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main function (`plot_heuristics_data`) will make a plot the probability of binding and/or killing based on the CAR-antigen and PD1-PDL1 binding heuristics used in the paper across various values of ligand/receptor and/or binding affinity. Each time this function is run the same output will be produced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "+ `RESULTS_PATH_HEURISTICS` variable indicates where to save the heuristic plots (`.svg` files as a result of plotting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "RESULTS_PATH_HEURISTICS = 'examples/figures/heuristics/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot heuristic data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.plot.plot_data import plot_heuristics_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_heuristics_data(RESULTS_PATH_HEURISTICS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Simulated data processing & plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data processing and plotting pipeline is used for both `dish` and `tissue` data. A small set of co-culture `dish` data, which is not used in the paper and is a shorter time scale than paper simulations for ease of running sections 3.1-3.3 of this notebook in a reasonable time scale, is provided for the purpose of exploring this data processing pipeline. For the plotting, full sets of the co-culture `dish` and `tissue` data used in the paper are provided to see figure outputs. Only a subset of `tissue` simulations were generated in the paper and used in this example. This set are those that showed effective treatment in the realistic co-culture `dish` context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Parse sample co-culture `dish` data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main parsing function (`parse`) iterates through each file in the data path and parses each simulation instance, extracting fields from the simulation setup, cells, and environment.\n",
    "\n",
    "The parsed arrays are organized as:\n",
    "\n",
    "`{\n",
    "    \"setup\": {\n",
    "        \"radius\": R,\n",
    "        \"height\": H,\n",
    "        \"time\": [],\n",
    "        \"pops\": [],\n",
    "        \"types\": [],\n",
    "        \"coords\": []\n",
    "    },\n",
    "    \"agents\": (N seeds) x (T timepoints) x (H height) x (C coordinates) x (P positions),\n",
    "    \"environments\": {\n",
    "        \"glucose\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "        \"oxygen\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "        \"tgfa\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "        \"IL-2\": (N seeds) x (T timepoints) x (H height) x (R radius)\n",
    "    }\n",
    "}\n",
    "`\n",
    "\n",
    "where each entry in the agents array is a structured entry of the shape:\n",
    "\n",
    "`\n",
    "\"pop\"       int8    population code\n",
    "\"type\"      int8    cell type code\n",
    "\"volume\"    int16   cell volume (rounded)\n",
    "\"cycle\"     int16   average cell cycle length (rounded)\n",
    "`\n",
    "The `parse.py` file contains general parsing functions.\n",
    "\n",
    "Parsing can take some time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "Set up workspace variables for parsing simulations. \n",
    "\n",
    "+ `DATA_PATH` variables are the path to simulation output data files (`.tar.xz` files of compressed simulation outputs)\n",
    "+ `RESULTS_PATH` variables are the path for result files (`.pkl` files generated by parsing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Toy data workspace variables\n",
    "DATA_PATH_TOY_COCULTURE_TAR = 'examples/files/toy/coculture/tars/'\n",
    "RESULTS_PATH_TOY_COCULTURE_PARSED = 'examples/files/toy/coculture/parsed/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parse sample co-culture `dish` simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.parse.parse import parse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parse(DATA_PATH_TOY_COCULTURE_TAR, RESULTS_PATH_TOY_COCULTURE_PARSED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Analyze parsed sample co-culture `dish` data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each main analyzing function (`analyze_cells`, `analyze_env`, `analyze_spatial`, and `analyze_lysis`) iterate through each parsed file (`.pkl`) in the data path and analyzes each simulation instance, extracting fields from the simulation setup, cells, and environment depending on the function.\n",
    "\n",
    "`analyze_cells` collects cell counts for each population and state over time (files produced will end with `ANALYZED`).\n",
    "\n",
    "When the `sharedLocs` flag is set to `True`, only the data for CAR T-cells that share a location with at least one cancer cell is collected. When this flag is used, files produced will end with `SHAREDLOCS`. This was used to further analyze the spatial differences between the `dish` and `tissue` contexts by providing an additional analysis on the effective treatments is compared to show the cell state dynamics over time for only those CAR T-cells that share locations with at least one cancer cell.\n",
    "\n",
    "`analyze_env` collects information on environmental species concentrations over time (files produced will end with `ENVIRONMENT`).\n",
    "\n",
    "`analyze_spatial` collects cell counts for each population across simulation radii over time (files produced will end with `SPATIAL`).\n",
    "\n",
    "`analyze_lysis` collects lysed cell information over time (files produced will end with `LYSED`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "Set up workspace variables for analyzing simulations.\n",
    "\n",
    "+ `DATA_PATH...PARSED` variables are the path to parsed data files (`.pkl` files generated by parsing), where for the `co-culture dish` data, one may need to manually put the effective treatment files into a folder separate from the rest of the data.\n",
    "+ `DATA_PATH...LYSIS` variables are the path to `LYSIS` data files (`.LYSIS.json` files generated directly from the simulation outputs)\n",
    "+ `RESULTS_PATH` variables are the path for result files (`.pkl` files generated by analyzing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Toy data workspace variables\n",
    "DATA_PATH_TOY_COCULTURE_PARSED = 'examples/files/toy/coculture/parsed/'\n",
    "DATA_PATH_TOY_COCULTURE_LYSIS = 'examples/files/toy/coculture/lysis/'\n",
    "RESULTS_PATH_TOY_COCULTURE_CELLS = 'examples/files/toy/coculture/analyzed/cells/'\n",
    "RESULTS_PATH_TOY_COCULTURE_ENVIRONMENT = 'examples/files/toy/coculture/analyzed/environment/'\n",
    "RESULTS_PATH_TOY_COCULTURE_SPATIAL = 'examples/files/toy/coculture/analyzed/spatial/'\n",
    "RESULTS_PATH_TOY_COCULTURE_LYSED = 'examples/files/toy/coculture/analyzed/lysed/'\n",
    "RESULTS_PATH_TOY_COCULTURE_SHAREDLOCS = 'examples/files/toy/coculture/analyzed/sharedlocs/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Analyze parsed sample co-culture `dish` simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.analyze.analyze_cells import analyze_cells\n",
    "from scripts.analyze.analyze_env import analyze_env\n",
    "from scripts.analyze.analyze_spatial import analyze_spatial\n",
    "from scripts.analyze.analyze_lysis import analyze_lysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Analyze co-culture dish data\n",
    "analyze_cells(DATA_PATH_TOY_COCULTURE_PARSED, RESULTS_PATH_TOY_COCULTURE_CELLS)\n",
    "analyze_env(DATA_PATH_TOY_COCULTURE_PARSED, RESULTS_PATH_TOY_COCULTURE_ENVIRONMENT)\n",
    "analyze_spatial(DATA_PATH_TOY_COCULTURE_PARSED, RESULTS_PATH_TOY_COCULTURE_SPATIAL)\n",
    "analyze_lysis(DATA_PATH_TOY_COCULTURE_LYSIS, RESULTS_PATH_TOY_COCULTURE_LYSED)\n",
    "analyze_cells(DATA_PATH_TOY_COCULTURE_PARSED, RESULTS_PATH_TOY_COCULTURE_SHAREDLOCS, sharedLocs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Subset analyzed sample co-culture `dish` data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main subsetting function (`subset_data`) takes in a given desired subset of data and iterates through each analyzed file in the data path (`.pkl`) and adds simulations matching the subset requirements to a single data file. Each subset will also automatically include the untreated control if it is in the file directory where the data is being pulled from.\n",
    "\n",
    "Since we are working with a limited toy example for this exercise, we will select a few small subsets of data, but they will not be exhaustive collections of the full data set as in the paper.\n",
    "\n",
    "For this example, we will select all simulations where indicated features meet the following requirements:\n",
    "\n",
    "+ `CAR AFFINITY` : 1e-7\n",
    "\n",
    "This means all data within this subset will have the specified values of the `CAR AFFINITY` listed above, but all features values of `DOSE`, `TREAT RATIO`, `ANTIGENS CANCER`, and if applicable `ANTIGENS HEALTHY`, will be included. All subsets will be saved in the following format:\n",
    "\n",
    "`XML_NAME` + `DOSE` + `TREAT RATIO` + `CAR AFFINITY` + `ANTIGENS CANCER` + `ANTIGENS HEALTHY`\n",
    "\n",
    "where the gap between values is separated by a `_`, values specified in the subset are replaced with the desired value, and values not specified in the subset are replaced by an `X` to indicate that all values of that feature are present.\n",
    "\n",
    "Thus, the above example will produce the following name for the toy data:\n",
    "    \n",
    "    VITRO_DISH_TREAT_CH_2D_X_X_1e-07_X_X_DATATYPE.pkl\n",
    "\n",
    "\n",
    "Where `DATATYPE` is either `ANALYZED` for cells, `ENVIRONMENT` for environment, `SPATIAL` for spatial, or `LYSED` for lysed analyses.\n",
    "\n",
    "We will also make a subset with the following requirements:\n",
    "\n",
    "+ `CAR AFFINITY` : 1e-6\n",
    "+ `ANTIGENS CANCER` : 1000\n",
    "\n",
    "Which will produce the following toy dataset subset:\n",
    "\n",
    "    VITRO_DISH_TREAT_CH_2D_X_X_1e-06_1000_X_DATATYPE.pkl\n",
    "    \n",
    "And finally, we will collect all of the cell data into a single file by not selecting a subset, producing data with the following name:\n",
    "\n",
    "    VITRO_DISH_TREAT_CH_2D_X_X_X_X_X_DATATYPE.pkl\n",
    "\n",
    "We will additionally do this with and without the `states` flag, such that we are collecting only the cell population and state counts data over time, but will exclude the volume and cell cycle distribution data. The file with the `states` information only will be listed as:\n",
    "\n",
    "\n",
    "    VITRO_DISH_TREAT_CH_2D_X_X_X_X_X_STATES_ANALYZED.pkl\n",
    "\n",
    "For the `tissue`, only the subset of all data is required. Additionally, for the sharedLocs data, only the subset of all data is required for both effective treatments in realistic co-culture `dish` and `tissue` contexts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "Set up workspace variables for subsetting simulations.\n",
    "\n",
    "+ `DATA_PATH` variables are the path to analyzed data files (`.pkl` files generated by analyzing data)\n",
    "+ `RESULTS_PATH` variables are the path for result files (`.pkl` files generated by subsetting)\n",
    "+ `...XML_NAME` variables are strings (that vary based on data setup) that will precede the file name extension (that vary based on subset selected)\n",
    "+ `...SUBSET...` variables are the requested data subsets to make (`;` separated lists with tuples containing keys to specific feature values that simulations in subset must include)\n",
    "+ `...ALL` variables indicate to collect all data without subsetting\n",
    "+ `DATA_TYPE` variables indicate which type of analyzed data is being fed into the subsetting function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Co-culture dish workspace variables\n",
    "DATA_PATH_TOY_COCULTURE_CELLS = 'examples/files/toy/coculture/analyzed/cells/'\n",
    "DATA_PATH_TOY_COCULTURE_ENVIRONMENT = 'examples/files/toy/coculture/analyzed/environment/'\n",
    "DATA_PATH_TOY_COCULTURE_SPATIAL = 'examples/files/toy/coculture/analyzed/spatial/'\n",
    "DATA_PATH_TOY_COCULTURE_LYSED = 'examples/files/toy/coculture/analyzed/lysed/'\n",
    "DATA_PATH_TOY_COCULTURE_SHAREDLOCS = 'examples/files/toy/coculture/analyzed/sharedlocs/'\n",
    "\n",
    "TOY_COCULTURE_XML_NAME = 'VITRO_DISH_TREAT_CH_2D'\n",
    "\n",
    "RESULTS_PATH_TOY_COCULTURE_SUBSET_CELLS = 'examples/files/toy/coculture/subset/cells/'\n",
    "RESULTS_PATH_TOY_COCULTURE_SUBSET_ENVIRONMENT = 'examples/files/toy/coculture/subset/environment/'\n",
    "RESULTS_PATH_TOY_COCULTURE_SUBSET_SPATIAL = 'examples/files/toy/coculture/subset/spatial/'\n",
    "RESULTS_PATH_TOY_COCULTURE_SUBSET_LYSED = 'examples/files/toy/coculture/subset/lysed/'\n",
    "RESULTS_PATH_TOY_COCULTURE_SUBSET_SHAREDLOCS = 'examples/files/toy/coculture/subset/sharedlocs/'\n",
    "\n",
    "TOY_COCULTURE_SUBSETS = '[(CAR AFFINITY:1e-7)];[(CAR AFFINITY:1e-6),(ANTIGENS CANCER:10000)]'\n",
    "TOY_COCULTURE_ALL = ''\n",
    "\n",
    "# Types of analyses\n",
    "DATA_TYPE_CELLS = 'ANALYZED'\n",
    "DATA_TYPE_ENVIRONMENT = 'ENVIRONMENT'\n",
    "DATA_TYPE_SPATIAL = 'SPATIAL'\n",
    "DATA_TYPE_LYSED = 'LYSED'\n",
    "DATA_TYPE_SHAREDLOCS = 'SHAREDLOCS'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subset sample co-culture `dish` simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.subset.subset import subset_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Subset co-culture dish data\n",
    "subset_data(DATA_PATH_TOY_COCULTURE_CELLS, TOY_COCULTURE_XML_NAME, DATA_TYPE_CELLS, \n",
    "            RESULTS_PATH_TOY_COCULTURE_SUBSET_CELLS, TOY_COCULTURE_SUBSETS)\n",
    "subset_data(DATA_PATH_TOY_COCULTURE_CELLS, TOY_COCULTURE_XML_NAME, DATA_TYPE_CELLS, \n",
    "            RESULTS_PATH_TOY_COCULTURE_SUBSET_CELLS, TOY_COCULTURE_ALL, states=True)\n",
    "subset_data(DATA_PATH_TOY_COCULTURE_ENVIRONMENT, TOY_COCULTURE_XML_NAME, DATA_TYPE_ENVIRONMENT, \n",
    "            RESULTS_PATH_TOY_COCULTURE_SUBSET_ENVIRONMENT, TOY_COCULTURE_SUBSETS)\n",
    "subset_data(DATA_PATH_TOY_COCULTURE_SPATIAL, TOY_COCULTURE_XML_NAME, DATA_TYPE_SPATIAL, \n",
    "            RESULTS_PATH_TOY_COCULTURE_SUBSET_SPATIAL, TOY_COCULTURE_SUBSETS)\n",
    "subset_data(DATA_PATH_TOY_COCULTURE_LYSED, TOY_COCULTURE_XML_NAME, DATA_TYPE_LYSED, \n",
    "            RESULTS_PATH_TOY_COCULTURE_SUBSET_LYSED, TOY_COCULTURE_SUBSETS)\n",
    "subset_data(DATA_PATH_TOY_COCULTURE_SHAREDLOCS, TOY_COCULTURE_XML_NAME, DATA_TYPE_SHAREDLOCS, \n",
    "            RESULTS_PATH_TOY_COCULTURE_SUBSET_SHAREDLOCS, subsetsRequested=TOY_COCULTURE_ALL, states=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Plot full subsetted co-culture `dish` and `tissue` data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main plotting function (`plot_data`) iterates through each subsetted file (`.pkl`) in the data path and plots relevant data for each subset instance.\n",
    "\n",
    "The function enables choosing which feature to color the data by. Choosing the color to be `X` will enable the function to automatically color the data based on whichever features are not held constant in the subset. \n",
    "\n",
    "For the `tissue` data only: since not all possible combinations of the `tissue` data were collected, each possible feature will need to be specified (and stored in different locations as the feature color is not stored in the file name). For this example, we use `ANTIGENS CANCER` as the sample variable and put all resulting figures in the `antigens_cancer` subfolders. The same set of code can be run for the `DOSE`, `CAR AFFINITY`, and `TREAT RATIO` variables, which would go in the `dose`, `car_affinity`, and `treat_ratio` subfolders, respectively. \n",
    "\n",
    "For the sharedLocs data, since not all possible combinations of the `tissue` data was collected and only the effective treatment realistic co-culture `dish` are desired, each desired feature to color by will need to be specified (and stored in different locations as the feature color is not stored in the file name). In this analysis, only `ANTIGENS CANCER` was used.\n",
    "\n",
    "For the purpose of this example, instead of using the toy data, we have provided a few full, pre-computed subsets of all the entire co-culture `dish` and `tissue` datasets used in the paper for exploring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "Set up workspace variables for plotting simulations.\n",
    "\n",
    "+ `DATA_PATH` variables are the path to subsetted data files (`.pkl` files generated by subsetting data)\n",
    "+ `RESULTS_PATH` variables are the path for result files (`.svg` files generated by plotting)\n",
    "+ `...COLOR` variables indicate which feature to color the variables by (choosing the color to be `X` will enable the function to automatically color the data based on whichever features are not held constant in the subset)\n",
    "+ `SHAREDLOCS_COLOR` variables indicate which feature to color the variables by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Co-culture dish workspace variables\n",
    "DATA_PATH_DISH_COCULTURE_SUBSET_CELLS = 'examples/files/full/coculture/subset/cells/'\n",
    "DATA_PATH_DISH_COCULTURE_SUBSET_ENVIRONMENT = 'examples/files/full/coculture/subset/environment/'\n",
    "DATA_PATH_DISH_COCULTURE_SUBSET_SPATIAL = 'examples/files/full/coculture/subset/spatial/'\n",
    "DATA_PATH_DISH_COCULTURE_SUBSET_LYSED = 'examples/files/full/coculture/subset/lysed/'\n",
    "DATA_PATH_DISH_COCULTURE_EFFECTIVE_TREATMENTS_SUBSET_SHAREDLOCS = 'examples/files/full/coculture/subset/sharedlocs/'\n",
    "\n",
    "DISH_COCULTURE_COLOR = 'X'\n",
    "\n",
    "RESULTS_PATH_DISH_COCULTURE_FIGURES_CELLS = 'examples/figures/coculture/cells/'\n",
    "RESULTS_PATH_DISH_COCULTURE_FIGURES_ENVIRONMENT = 'examples/figures/coculture/environment/'\n",
    "RESULTS_PATH_DISH_COCULTURE_FIGURES_SPATIAL = 'examples/figures/coculture/spatial/'\n",
    "RESULTS_PATH_DISH_COCULTURE_FIGURES_LYSED = 'examples/figures/coculture/lysed/'\n",
    "RESULTS_PATH_DISH_COCULTURE_EFFECTIVE_TREATMENTS_FIGURES_SHAREDLOCS = 'examples/figures/coculture/sharedlocs/'\n",
    "\n",
    "# Tissue workspace variables\n",
    "DATA_PATH_TISSUE_SUBSET_CELLS = 'examples/files/full/tissue/subset/cells/'\n",
    "DATA_PATH_TISSUE_SUBSET_ENVIRONMENT = 'examples/files/full/tissue/subset/environment/'\n",
    "DATA_PATH_TISSUE_SUBSET_SPATIAL = 'examples/files/full/tissue/subset/spatial/'\n",
    "DATA_PATH_TISSUE_SUBSET_LYSED = 'examples/files/full/tissue/subset/lysed/'\n",
    "DATA_PATH_TISSUE_SUBSET_SHAREDLOCS = 'examples/files/full/tissue/subset/sharedlocs/'\n",
    "\n",
    "\n",
    "TISSUE_COLOR_ANTIGENS_CANCER = 'ANTIGENS CANCER'\n",
    "\n",
    "RESULTS_PATH_TISSUE_FIGURES_CELLS_ANTIGENS_CANCER = 'examples/figures/tissue/cells/antigens_cancer/'\n",
    "RESULTS_PATH_TISSUE_FIGURES_ENVIRONMENT_ANTIGENS_CANCER = 'examples/figures/tissue/environment/antigens_cancer/'\n",
    "RESULTS_PATH_TISSUE_FIGURES_SPATIAL_ANTIGENS_CANCER = 'examples/figures/tissue/spatial/antigens_cancer/'\n",
    "RESULTS_PATH_TISSUE_FIGURES_LYSED_ANTIGENS_CANCER = 'examples/figures/tissue/lysed/antigens_cancer/'\n",
    "RESULTS_PATH_TISSUE_FIGURES_SHAREDLOCS = 'examples/figures/tissue/sharedlocs/'\n",
    "\n",
    "# Shared locations workspace variables\n",
    "SHAREDLOCS_COLOR = 'ANTIGENS CANCER'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot full co-culture `dish` and `tissue` simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.plot.plot_data import plot_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Plot co-culture dish data\n",
    "plot_data(DATA_PATH_DISH_COCULTURE_SUBSET_CELLS, DISH_COCULTURE_COLOR, \n",
    "          RESULTS_PATH_DISH_COCULTURE_FIGURES_CELLS)\n",
    "plot_data(DATA_PATH_DISH_COCULTURE_SUBSET_ENVIRONMENT, DISH_COCULTURE_COLOR, \n",
    "          RESULTS_PATH_DISH_COCULTURE_FIGURES_ENVIRONMENT)\n",
    "plot_data(DATA_PATH_DISH_COCULTURE_SUBSET_SPATIAL, DISH_COCULTURE_COLOR, \n",
    "          RESULTS_PATH_DISH_COCULTURE_FIGURES_SPATIAL)\n",
    "plot_data(DATA_PATH_DISH_COCULTURE_SUBSET_LYSED, DISH_COCULTURE_COLOR, \n",
    "          RESULTS_PATH_DISH_COCULTURE_FIGURES_LYSED)\n",
    "plot_data(DATA_PATH_DISH_COCULTURE_EFFECTIVE_TREATMENTS_SUBSET_SHAREDLOCS, SHAREDLOCS_COLOR, \n",
    "          RESULTS_PATH_DISH_COCULTURE_EFFECTIVE_TREATMENTS_FIGURES_SHAREDLOCS, partial=True)\n",
    "\n",
    "# Plot tissue data\n",
    "plot_data(DATA_PATH_TISSUE_SUBSET_CELLS, TISSUE_COLOR_ANTIGENS_CANCER, \n",
    "          RESULTS_PATH_TISSUE_FIGURES_CELLS_ANTIGENS_CANCER, partial=True)\n",
    "plot_data(DATA_PATH_TISSUE_SUBSET_ENVIRONMENT, TISSUE_COLOR_ANTIGENS_CANCER, \n",
    "          RESULTS_PATH_TISSUE_FIGURES_ENVIRONMENT_ANTIGENS_CANCER, partial=True)\n",
    "plot_data(DATA_PATH_TISSUE_SUBSET_SPATIAL, TISSUE_COLOR_ANTIGENS_CANCER, \n",
    "          RESULTS_PATH_TISSUE_FIGURES_SPATIAL_ANTIGENS_CANCER, partial=True)\n",
    "plot_data(DATA_PATH_TISSUE_SUBSET_LYSED, TISSUE_COLOR_ANTIGENS_CANCER, \n",
    "          RESULTS_PATH_TISSUE_FIGURES_LYSED_ANTIGENS_CANCER, partial=True)\n",
    "plot_data(DATA_PATH_TISSUE_SUBSET_SHAREDLOCS, SHAREDLOCS_COLOR, \n",
    "          RESULTS_PATH_TISSUE_FIGURES_SHAREDLOCS, partial=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Multi-feature & outcome analysis of full co-culture `dish` and `tissue` data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main outcome analysis function (`stats`) iterates through each subsetted file (`.pkl`) in the data path and plots and analyzes relevant data for each subset instance. When the `average` flag is set to `True`, data will be averaged across replicates for analysis.\n",
    "\n",
    "For these analyses, only full data (analyzed by cell counts) subsets were used. We provided the following real, full co-culture `dish` subset for this analysis:\n",
    "\n",
    "    VITRO_DISH_TREAT_CH_2D_X_X_X_X_X_STATES_ANALYZED.pkl\n",
    "    \n",
    "and the following real, full `tissue` subset:\n",
    "\n",
    "    VIVO_TISSUE_TREAT_C_2D_X_X_X_X_X_STATES_ANALYZED.pkl\n",
    "\n",
    "We will analyze this data both in full and averaged across replicates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "Set up workspace variables for plotting simulations.\n",
    "\n",
    "+ `DATA_PATH` variables are the path to subsetted data files (`.pkl` files generated by subsetting data)\n",
    "+ `RESULTS_PATH` variables are the path for result files (`.svg` or `.pdf` or `.csv` files generated by analyzing the data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Co-culture dish workspace variables\n",
    "DATA_PATH_DISH_COCULTURE_FULL = 'examples/files/full/coculture/subset/cells/VITRO_DISH_TREAT_CH_2D_X_X_X_X_X_STATES_ANALYZED.pkl'\n",
    "\n",
    "RESULTS_PATH_DISH_COCULTURE_STATS = 'examples/figures/coculture/stats/'\n",
    "RESULTS_PATH_DISH_COCULTURE_STATS_AVERAGE = 'examples/figures/coculture/stats/average/'\n",
    "\n",
    "# Tissue workspace variables\n",
    "DATA_PATH_TISSUE_SUBSET_CELLS = 'examples/files/full/tissue/subset/cells/'\n",
    "\n",
    "RESULTS_PATH_TISSUE_STATS = 'examples/figures/tissue/stats/'\n",
    "RESULTS_PATH_TISSUE_STATS_AVERAGE = 'examples/figures/tissue/stats/average/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Analyze full co-culture `dish` and `tissue` simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.stats.stats import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Co-culture dish outcome analysis\n",
    "stats(DATA_PATH_DISH_COCULTURE_FULL, RESULTS_PATH_DISH_COCULTURE_STATS, average=False)\n",
    "stats(DATA_PATH_DISH_COCULTURE_FULL, RESULTS_PATH_DISH_COCULTURE_STATS_AVERAGE, average=True)\n",
    "\n",
    "# Tissue outcome analysis\n",
    "stats(DATA_PATH_TISSUE_SUBSET_CELLS, RESULTS_PATH_TISSUE_STATS, average=False)\n",
    "stats(DATA_PATH_TISSUE_SUBSET_CELLS, RESULTS_PATH_TISSUE_STATS_AVERAGE, average=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Experimental literature data plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main function (`plot_kill_curve_exp_data`) will plot experimental data kill curves using data extracted from a variety of reference papers. Each time this function is run the same output will be produced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "+ `RESULTS_PATH_EXPERIMENTAL_LITERATURE_KILL_CURVES` variable indicates where to save plots based on extracted data from literature (`.svg` files as a result of plotting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "RESULTS_PATH_EXPERIMENTAL_LITERATURE_KILL_CURVES = 'examples/figures/kill_curves/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot experimental literature data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.plot.plot_data import plot_kill_curve_exp_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_kill_curve_exp_data(RESULTS_PATH_EXPERIMENTAL_LITERATURE_KILL_CURVES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Ranked data plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After finding the effective treatments from the realistic co-culture `dish` context and analyzing them in `tissue`, a score and rank for each simulation (averaged across replicates) were provided in both contexts and combined into a single `.csv` file for analysis.\n",
    "\n",
    "The main function `plot_dish_tissue_compare_data` takes this `.csv` file as an input and generates parity and ladder plots for the rank and score of these simulations in the `dish` compared to the the `tissue` context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Workspace variables\n",
    "\n",
    "+ `DATA_PATH` variables are the path to excel data file (`.csv` file generated for this analysis)\n",
    "+ `RESULTS_PATH` variables are the path for result files (`.svg` files generated by plotting the data)\n",
    "+ `COMPARE_COLOR` variables indicate which feature to color the variables by (choosing the color to be `X` will enable the function to automatically color the data based on each feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "DATA_PATH_COMPARE_DISH_TISSUE_CSV = 'examples/files/full/ranked/RANK_SCORE_COMPARE_DISH_TISSUE.csv'\n",
    "RESULTS_PATH_COMPARE_DISH_TISSUE_FIGURES = 'examples/figures/ranked/'\n",
    "\n",
    "COMPARE_COLOR = 'X'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compare full effective treatment realistic co-culture `dish` and `tissue` simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scripts.plot.plot_data import plot_dish_tissue_compare_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_dish_tissue_compare_data(DATA_PATH_COMPARE_DISH_TISSUE_CSV, COMPARE_COLOR, RESULTS_PATH_COMPARE_DISH_TISSUE_FIGURES)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}