generate_figure_inputs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"toc\"></a>\n",
    "# GENERATE FIGURE INPUTS\n",
    "\n",
    "This notebook provides functions and scripts for generating input files for visualization with D3.\n",
    "\n",
    "---\n",
    "- [WORKSPACE VARIABLES](#workspace-variables)\n",
    "- **[SYSTEM REPRESENTATION](#system-representation)**\n",
    "    - [colony borders](#system-representation-colony-borders)\n",
    "    - [simulation outlines](#system-representation-simulation-outlines)\n",
    "    - [simulation states](#system-representation-simulation-states)\n",
    "    - [simulation metrics](#system-representation-simulation-metrics)\n",
    "- **[CELL VARIABILITY](#cell-variability)**\n",
    "    - [feature quantiles](#cell-variability-feature-quantiles)\n",
    "    - [simulation outlines](#cell-variability-simulation-outlines)\n",
    "    - [simulation states](#cell-variability-simulation-states)\n",
    "    - [simulation metrics](#cell-variability-simulation-metrics)\n",
    "- **[NUTRIENT DYNAMICS](#nutrient-dynamics)**\n",
    "    - [concentration profiles](#nutrient-dynamics-concentration-profiles)\n",
    "    - [simulation outlines](#nutrient-dynamics-simulation-outlines)\n",
    "    - [simulation states](#nutrient-dynamics-simulation-states)\n",
    "    - [simulation metrics](#nutrient-dynamics-simulation-metrics)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the simulation sets has a corresponding script of the same name, which contains a class (of the same name) with relevant condition variables and methods to iterate through these condition.\n",
    "The `loop` method will be used the most often to iterate through all conditions, extract relevant information, and then compile the information into a single output file.\n",
    "The `loop` method works with both individual files for each condition (`.csv` or `.json`) or a `.tar.xz` compressed archive of the files for each condition.\n",
    "For a given figure, not all conditions may be used.\n",
    "\n",
    "All generated figure inputs are provided in the `analysis` directory.\n",
    "As much as possible, files are provided in compressed archive form, using the provided `compress.sh` script, taking three arguments: `[NAME OF SIMULATION SET] [EXTENSION NAME] [json OR csv]`.\n",
    "The corresponding arguments for each of the following figures is provided in the description.\n",
    "\n",
    "The `generate.py` file contains functions to generate figure inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.generate import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"workspace-variables\"></a>\n",
    "\n",
    "### WORKSPACE VARIABLES\n",
    "\n",
    "Set up workspace variables for analyzing simulations.\n",
    "\n",
    "- **`DATA_PATH`** is the path to data files (`.tar.xz` files of compressed simulation outputs)\n",
    "- **`RESULTS_PATH`** is the path to result files (`.csv` files generated by parsing)\n",
    "- **`ANALYSIS_PATH`** is the path for analysis files (`.json` and `.csv` files, `.tar.xz` compressed archives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH = \"/path/to/data/files/\"\n",
    "RESULTS_PATH = \"/path/to/result/files/\"\n",
    "ANALYSIS_PATH = \"/path/to/analysis/files/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"system-representation\"></a>\n",
    "\n",
    "### SYSTEM REPRESENTATION\n",
    "\n",
    "Generate `SYSTEM_REPRESENTATION` figure inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.SYSTEM_REPRESENTATION import SYSTEM_REPRESENTATION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"system-representation-colony-borders\"></a>\n",
    "\n",
    "**colony borders**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/colony_borders.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The colony borders figure shows overlay of borders for 2D and different 3D simulation conditions across context and geometries for seed 0 at t = 15 days.\n",
    "\n",
    "The function `get_colony_borders` extracts colony borders for the given timepoint and seed.\n",
    "Files created by the function can be compressed with `compress.sh` using `SYSTEM_REPRESENTATION .BORDERS csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SYSTEM_REPRESENTATION.run(ANALYSIS_PATH, RESULTS_PATH, get_colony_borders, timepoints=15.0, seeds=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"system-representation-simulation-outlines\"></a>\n",
    "\n",
    "**simulation outlines**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_outlines.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation outlines figure shows colony outlines for each seed and simulation condition.\n",
    "\n",
    "The `.OUTLINES` files produced in the basic analysis step are used as input.\n",
    "Make sure to EXTRACT THE ARCHIVE FILES from `SYSTEM_REPRESENTATION.OUTLINES.tar.xz`.\n",
    "Only the conditions with extension `.150.csv` are needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"system-representation-simulation-states\"></a>\n",
    "\n",
    "**simulation states**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_states.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation states figure shows states of each cell for seed 0 for each seed and simulation condition.\n",
    "\n",
    "The `.POSITIONS` files produced in the basic analysis step are used as input.\n",
    "Make sure to EXTRACT THE ARCHIVE FILES from `SYSTEM_REPRESENTATION.POSITIONS.tar.xz`.\n",
    "Only the conditions with extension `.150.00.csv` are needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"system-representation-simulation-metrics\"></a>\n",
    "\n",
    "**simulation metrics**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_metrics.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation metrics figures shows emergent metrics over time for populations grouped by selected conditions and context.\n",
    "\n",
    "The `.METRICS` files produced in the basic analysis step are merged into a single file per metric (`SIMULATION_REPRESENTATION.METRICS.*.json`) which are then used as inputs for D3.\n",
    "\n",
    "For simulations in 3D, we also extract metrics from only the center layer (z = 0).\n",
    "Files created by the function `get_center_layers` can be compressed with `compress.sh` using `SYSTEM_REPRESENTATION .LAYERS json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dimensions = [\"3D\"]\n",
    "SYSTEM_REPRESENTATION.run(ANALYSIS_PATH, RESULTS_PATH, get_center_layers_metrics, \\\n",
    "                          dimensions=dimensions)\n",
    "SYSTEM_REPRESENTATION.run(ANALYSIS_PATH, RESULTS_PATH, get_center_layer_seeds, \\\n",
    "                          timepoints=[15.0], dimensions=dimensions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "METRICS = [\"GROWTH\", \"SYMMETRY\", \"CYCLES\"]\n",
    "for metric in METRICS:\n",
    "    SYSTEM_REPRESENTATION.loop(ANALYSIS_PATH, merge_metrics, save_metrics, f\".METRICS.{metric}\", timepoints=[None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cell-variability\"></a>\n",
    "\n",
    "### CELL VARIABILITY\n",
    "\n",
    "Generate `CELL_VARIABILITY` figure inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.CELL_VARIABILITY import CELL_VARIABILITY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cell-variability-feature-quantiles\"></a>\n",
    "\n",
    "**feature quantiles**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/feature_quantiles.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The feature quantiles figure shows distributions of age and volume for selected contexts and conditions.\n",
    "\n",
    "The function `get_feature_quantiles` calculates quantiles across all seeds and timepoints.\n",
    "These files are then merged into a single file `CELL_VARIABILITY.QUANTILES.json` that is used as input for D3.\n",
    "Files created by `get_feature_quantiles` can be compressed with `compress.sh` using `CELL_VARIABILITY .QUANTILES csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timepoints = [1, 2, 3, 4, 8, 15]\n",
    "CELL_VARIABILITY.load(ANALYSIS_PATH, DATA_PATH, get_feature_quantiles, timepoints=timepoints)\n",
    "CELL_VARIABILITY.loop(ANALYSIS_PATH, merge_feature_quantiles, save_feature_quantiles, \\\n",
    "                        f\".QUANTILES\", timepoints=[None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cell-variability-simulation-outlines\"></a>\n",
    "\n",
    "**simulation outlines**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_outlines.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation outlines figure shows colony outlines for each seed and simulation condition.\n",
    "\n",
    "The `.OUTLINES` files produced in the basic analysis step are used as input.\n",
    "Make sure to EXTRACT THE ARCHIVE FILES from `CELL_VARIABILITY.OUTLINES.tar.xz`.\n",
    "Only the conditions with extension `.150.csv` are needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cell-variability-simulation-states\"></a>\n",
    "\n",
    "**simulation states**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_states.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation states figure shows states of each cell for seed 0 for each seed and simulation condition.\n",
    "\n",
    "The `.POSITIONS` files produced in the basic analysis step are used as input.\n",
    "Make sure to EXTRACT THE ARCHIVE FILES from `CELL_VARIABILITY.POSITIONS.tar.xz`.\n",
    "Only the conditions with extension `.150.00.csv` are needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"cell-variability-simulation-metrics\"></a>\n",
    "\n",
    "**simulation metrics**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_metrics.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation metrics figures shows emergent metrics over time for populations grouped by selected conditions and context.\n",
    "\n",
    "The `.METRICS` files produced in the basic analysis step are merged into a single file per metric (`CELL_VARIABILITY.METRICS.*.json`) which are then used as inputs for D3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "METRICS = [\"GROWTH\", \"SYMMETRY\", \"CYCLES\"]\n",
    "for metric in METRICS:\n",
    "    CELL_VARIABILITY.loop(ANALYSIS_PATH, merge_metrics, save_metrics, f\".METRICS.{metric}\", timepoints=[None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"nutrient-dynamics\"></a>\n",
    "\n",
    "### NUTRIENT DYNAMICS\n",
    "\n",
    "Generate `NUTRIENT_DYNAMICS` figure inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.NUTRIENT_DYNAMICS import NUTRIENT_DYNAMICS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"nutrient-dynamics-concentration-profiles\"></a>\n",
    "\n",
    "**concentration profiles**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/concentration_profiles.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The concentration profiles figure shows timecourse of average glucose concentration at the center of the simulation across seeds for all conditions.\n",
    "\n",
    "The function `make_concentration_profiles` extracts center concentrations for the every timepoint.\n",
    "Files created by the function can be compressed with `compress.sh` using `NUTRIENT_DYNAMICS .PROFILES csv`.\n",
    "The `.PROFILES` files produced are merged into a single file (`NUTRIENT_DYNAMICS.PROFILES.csv`) which is used as input for D3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUTRIENT_DYNAMICS.load(ANALYSIS_PATH, DATA_PATH, make_concentration_profiles, timepoints=None)\n",
    "NUTRIENT_DYNAMICS.loop(ANALYSIS_PATH, merge_concentration_profiles, save_concentration_profiles, \\\n",
    "                       f\".PROFILES\", timepoints=[None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"nutrient-dynamics-simulation-outlines\"></a>\n",
    "\n",
    "**simulation outlines**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_outlines.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation outlines figure shows colony outlines for each seed and simulation condition.\n",
    "\n",
    "The `.OUTLINES` files produced in the basic analysis step are used as input.\n",
    "Make sure to EXTRACT THE ARCHIVE FILES from `NUTRIENT_DYNAMICS.OUTLINES.tar.xz`.\n",
    "Only the conditions with extension `.150.csv` are needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"nutrient-dynamics-simulation-states\"></a>\n",
    "\n",
    "**simulation states**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_states.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation states figure shows states of each cell for seed 0 for each seed and simulation condition.\n",
    "\n",
    "The `.POSITIONS` files produced in the basic analysis step are used as input.\n",
    "Make sure to EXTRACT THE ARCHIVE FILES from `NUTRIENT_DYNAMICS.POSITIONS.tar.xz`.\n",
    "Only the conditions with extension `.150.00.csv` are needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"nutrient-dynamics-simulation-metrics\"></a>\n",
    "\n",
    "**simulation metrics**\n",
    "<span style=\"float:right;\">\n",
    "    [go to figure](http://0.0.0.0:8000/figures/simulation_metrics.html) &#x2022;\n",
    "    [back to top](#toc)\n",
    "</span>\n",
    "\n",
    "The simulation metrics figures shows emergent metrics over time for populations grouped by selected conditions and context.\n",
    "\n",
    "The `.METRICS` files produced in the basic analysis step are merged into a single file per metric (`NUTRIENT_DYNAMICS.METRICS.*.json`) which are then used as inputs for D3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "METRICS = [\"GROWTH\", \"SYMMETRY\", \"CYCLES\"]\n",
    "for metric in METRICS:\n",
    "    NUTRIENT_DYNAMICS.loop(ANALYSIS_PATH, merge_metrics, save_metrics, f\".METRICS.{metric}\", timepoints=[None])"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}