Lab1 (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\\rightarrow$Run All).\n",
    "\n",
    "Make sure you fill in any place that says `YOUR CODE HERE` or \"YOUR ANSWER HERE\", as well as your name and collaborators below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"Sreya Dhar\"\n",
    "COLLABORATORS = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f6c72f56e392728627cf96c31d0c35ae",
     "grade": false,
     "grade_id": "cell-d7130d89d1ed3eb3",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Introduction\n",
    "\n",
    "This is the first of three labs that you will do in this course. In this lab, you will parse a variant call format file (https://en.wikipedia.org/wiki/Variant_Call_Format). This file has one thousand variants in it. Each line after the header lines is a variant. Open the variant file and inspect its contents. You will parse each line and transform it into a dictionary. The final output will be a list of dictionaries. A dictionary will look something like this (I have truncated the output):\n",
    "\n",
    "```python\n",
    "{\n",
    "        \"ALT\": \"G\",\n",
    "        \"CHROM\": \"4\",\n",
    "        \"FILTER\": \"PASS\",\n",
    "        \"ID\": \".\",\n",
    "        \"INFO\": {\n",
    "\n",
    "            \"Gene.ensGene\": \"ENSG00000109471,ENSG00000138684\",\n",
    "            \"Gene.refGene\": \"IL2,IL21\",\n",
    "            \"GeneDetail.ensGene\": \"dist=38306,dist=117597\",\n",
    "            \"GeneDetail.refGene\": \"dist=38536,dist=117597\"\n",
    "        },\n",
    "        \"POS\": 123416186,\n",
    "        \"QUAL\" :23.25,\n",
    "        \"REF\": \"A\",\n",
    "        \"SAMPLE\": {\n",
    "            \"XG102\": {\n",
    "                \"AD\": \"51,8\",\n",
    "                \"DP\": \"59\",\n",
    "                \"GQ\": \"32\",\n",
    "                \"GT\": \"0/1\",\n",
    "                \"PL\": \"32,0,1388\"\n",
    "            }\n",
    "    }\n",
    "```\n",
    "\n",
    "The lab will guide you through writing small functions that will be used to generate the output above. Note that the method outlined in this lab is not necessarily optimal. It is just one way of parsing the file and teaching you how to break a large problem into small parts.\n",
    "\n",
    "Note that the header that corresponds to the columns starts with one hash/pound (#) symbol. Skip all headers that start with two hashes/pounds (##)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "65282f9954e4babc991d0dda89a7c240",
     "grade": false,
     "grade_id": "cell-89c9383b6911e6e0",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 1 (5 points)\n",
    "Write a function that determines the data type of its input string. The data types can be a float, int, or string. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "72f9aa18325877c42ca6fe9883f3b9f0",
     "grade": false,
     "grade_id": "cell-b051456fee8432e2",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def determine_data_type(value):\n",
    "    \"\"\"\n",
    "    The function takes a string input and determines its data type to be either a float, int, or string. \n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "        # YOUR CODE HERE\n",
    "    while True:\n",
    "        try:\n",
    "            val = int(value)\n",
    "            #print(val) \n",
    "            return type(val)\n",
    "            break;\n",
    "        except ValueError:\n",
    "            try:\n",
    "                val=float(value)\n",
    "                #print(val)\n",
    "                return type(val)\n",
    "                \n",
    "                break;\n",
    "            except ValueError:\n",
    "            \n",
    "                return type(value)\n",
    "                break;\n",
    "\n",
    "                    \n",
    "                    \n",
    "# determine_data_type('1.2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b68a0fe63fe9992293622eb0c020dbe6",
     "grade": true,
     "grade_id": "cell-d5bac7f216838077",
     "locked": true,
     "points": 5,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert determine_data_type('1.2') == float\n",
    "assert determine_data_type('4') == int\n",
    "assert determine_data_type('EAS503') == str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fe9d30ef77c57f8928c9aa8de74ac529",
     "grade": false,
     "grade_id": "cell-d331435fc6a5e116",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 2 (5 points)\n",
    "\n",
    "Write a function whose input is a list of strings. This function determines the correct data type of all the elements in the list. For example, ['1', '2', '3'] is int, ['1.1', '2.2', '3.3'] is float, ['1.1', '2', '3.3'] is also float, and ['1.1', '234String', '3.3'] is str. The purpose of this function to figure out what to cast an array of strings to. Some lists might be all ints, in which case the data type is int. Some might be a mixture of ints and floats, in which case the data type will be a float. Some lists might be a mixture of ints, floats, and strings, in which case the data type of the list will be a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2377d20ab018529a9a7bdff58a3ce07f",
     "grade": false,
     "grade_id": "cell-163887217764f9d8",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def determine_data_type_of_list(values):\n",
    "    \"\"\"\n",
    "    Function takes a list of strings and determines their data type. \n",
    "\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    string_ = None\n",
    "    float_ = None\n",
    "    \n",
    "    for value in values: \n",
    "        if determine_data_type(value)== str:\n",
    "            string_=True\n",
    "            break;\n",
    "            \n",
    "        elif determine_data_type(value)== float:\n",
    "            float_=True\n",
    "            \n",
    "    if string_:\n",
    "        return str\n",
    "    elif float_:\n",
    "        return float\n",
    "    else:\n",
    "        return int\n",
    "\n",
    "                    \n",
    "# determine_data_type_of_list(['1.1', '234String', '3.3'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d4239274647dd805b81cbe9e6427af52",
     "grade": true,
     "grade_id": "cell-994c2dcfd49a7e14",
     "locked": true,
     "points": 5,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert determine_data_type_of_list(['1', '2', '3']) == int\n",
    "assert determine_data_type_of_list(['1.1', '2.2', '3.3']) == float\n",
    "assert determine_data_type_of_list(['1.1', '2', '3.3']) == float\n",
    "assert determine_data_type_of_list(['1.1', '234String', '3.3']) == str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1c1d9e36b4aa9d2a053a401d3f226317",
     "grade": false,
     "grade_id": "cell-f169f51164c98ed8",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 3 (10 points)\n",
    "\n",
    "Write a function whose inputs are a format field and sample field. The format field looks like format_field = 'GT:AD:DP:GQ:PGT:PID:PL' and the sample field looks like\n",
    "\n",
    "```python\n",
    "sample_field = {'XG102': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG103': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG104': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG202': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG203': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG204': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG302': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG303': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG304': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG402': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG403': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG404': '0/1:34,38:72:99:.:.:938,0,796'}\n",
    "```\n",
    "\n",
    "Transform the inputs such that the output looks like this:\n",
    "\n",
    "\n",
    "```python\n",
    "output = {'XG102': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG103': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG104': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'},\n",
    " 'XG202': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG203': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG204': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'},\n",
    " 'XG302': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG303': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG304': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'},\n",
    " 'XG402': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG403': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG404': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'}}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c065afb1772d05574834f151083796a4",
     "grade": false,
     "grade_id": "cell-b8b5e6df500af89b",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "format_field = \"GT:AD:DP:GQ:PGT:PID:PL\"\n",
    "sample_field = {'XG102': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG103': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG104': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG202': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG203': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG204': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG302': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG303': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG304': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG402': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG403': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG404': '0/1:34,38:72:99:.:.:938,0,796'}\n",
    "\n",
    "\n",
    "def format_sample_fields(format_field, sample_field):\n",
    "    \"\"\"\n",
    "    Format the sample fields given the description above. Data is already provided.\n",
    "    \"\"\"\n",
    "    \n",
    "    form_field = format_field.split(':')\n",
    "    len_field = len(form_field)\n",
    "    #print(len_field)\n",
    "    for key, value in sample_field.items():\n",
    "            sample_dict = {}\n",
    "            values = value.split(':')\n",
    "            for i in range(len_field):\n",
    "                sample_dict[form_field[i]] = values[i]\n",
    "            sample_field[key]= sample_dict\n",
    "    return sample_field\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fdda11a3dddf014fbbd1180db33dc5aa",
     "grade": true,
     "grade_id": "cell-a8d94bc8f1701072",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "solution = {'XG102': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG103': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG104': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'},\n",
    " 'XG202': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG203': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG204': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'},\n",
    " 'XG302': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG303': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG304': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'},\n",
    " 'XG402': {'AD': '0,76',\n",
    "           'DP': '76',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '1|1',\n",
    "           'PID': '48306945_C_G',\n",
    "           'PL': '3353,229,0'},\n",
    " 'XG403': {'AD': '0,52',\n",
    "           'DP': '52',\n",
    "           'GQ': '99',\n",
    "           'GT': '1/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '1517,156,0'},\n",
    " 'XG404': {'AD': '34,38',\n",
    "           'DP': '72',\n",
    "           'GQ': '99',\n",
    "           'GT': '0/1',\n",
    "           'PGT': '.',\n",
    "           'PID': '.',\n",
    "           'PL': '938,0,796'}}\n",
    "\n",
    "format_field = \"GT:AD:DP:GQ:PGT:PID:PL\"\n",
    "sample_field = {'XG102': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG103': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG104': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG202': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG203': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG204': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG302': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG303': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG304': '0/1:34,38:72:99:.:.:938,0,796',\n",
    "             'XG402': '1/1:0,76:76:99:1|1:48306945_C_G:3353,229,0',\n",
    "             'XG403': '1/1:0,52:52:99:.:.:1517,156,0',\n",
    "             'XG404': '0/1:34,38:72:99:.:.:938,0,796'}\n",
    "\n",
    "format_output = format_sample_fields(format_field, sample_field)\n",
    "\n",
    "assert format_output['XG102'] == solution ['XG102']\n",
    "assert format_output['XG302'] == solution ['XG302']\n",
    "assert format_output['XG404'] == solution ['XG404']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d0d641abf9414b46a2c06b4959373383",
     "grade": false,
     "grade_id": "cell-9e67a97fc5f7697b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 4 (10 points)\n",
    "\n",
    "Write a function whose inputs are a list containing the vcf header and a variant line. The function should return a dictionary using the header as keys and the variant line as values. The function should use the format_sample_fields you wrote previously to format the sample fields. The output of the first line looks like this:\n",
    "\n",
    "```python\n",
    "{'ALT': 'G',\n",
    " 'CHROM': '4',\n",
    " 'FILTER': 'PASS',\n",
    " 'ID': '.',\n",
    " 'INFO': 'AC=1;AF=0.167;AN=6;BaseQRankSum=-2.542;ClippingRankSum=0;DP=180;ExcessHet=3.0103;FS=0;MLEAC=1;MLEAF=0.167;MQ=52.77;MQRankSum=-4.631;QD=0.39;ReadPosRankSum=1.45;SOR=0.758;VQSLOD=-8.209;culprit=MQ;ANNOVAR_DATE=2018-04-16;Func.refGene=intergenic;Gene.refGene=IL2,IL21;GeneDetail.refGene=dist=38536,dist=117597;ExonicFunc.refGene=.;AAChange.refGene=.;Func.ensGene=intergenic;Gene.ensGene=ENSG00000109471,ENSG00000138684;GeneDetail.ensGene=dist=38306,dist=117597;ExonicFunc.ensGene=.;AAChange.ensGene=.;cytoBand=4q27;gwasCatalog=.;tfbsConsSites=.;wgRna=.;targetScanS=.;Gene_symbol=.;OXPHOS_Complex=.;Ensembl_Gene_ID=.;Ensembl_Protein_ID=.;Uniprot_Name=.;Uniprot_ID=.;NCBI_Gene_ID=.;NCBI_Protein_ID=.;Gene_pos=.;AA_pos=.;AA_sub=.;Codon_sub=.;dbSNP_ID=.;PhyloP_46V=.;PhastCons_46V=.;PhyloP_100V=.;PhastCons_100V=.;SiteVar=.;PolyPhen2_prediction=.;PolyPhen2_score=.;SIFT_prediction=.;SIFT_score=.;FatHmm_prediction=.;FatHmm_score=.;PROVEAN_prediction=.;PROVEAN_score=.;MutAss_prediction=.;MutAss_score=.;EFIN_Swiss_Prot_Score=.;EFIN_Swiss_Prot_Prediction=.;EFIN_HumDiv_Score=.;EFIN_HumDiv_Prediction=.;CADD_score=.;CADD_Phred_score=.;CADD_prediction=.;Carol_prediction=.;Carol_score=.;Condel_score=.;Condel_pred=.;COVEC_WMV=.;COVEC_WMV_prediction=.;PolyPhen2_score_transf=.;PolyPhen2_pred_transf=.;SIFT_score_transf=.;SIFT_pred_transf=.;MutAss_score_transf=.;MutAss_pred_transf=.;Perc_coevo_Sites=.;Mean_MI_score=.;COSMIC_ID=.;Tumor_site=.;Examined_samples=.;Mutation_frequency=.;US=.;Status=.;Associated_disease=.;Presence_in_TD=.;Class_predicted=.;Prob_N=.;Prob_P=.;SIFT_score=.;SIFT_converted_rankscore=.;SIFT_pred=.;Polyphen2_HDIV_score=.;Polyphen2_HDIV_rankscore=.;Polyphen2_HDIV_pred=.;Polyphen2_HVAR_score=.;Polyphen2_HVAR_rankscore=.;Polyphen2_HVAR_pred=.;LRT_score=.;LRT_converted_rankscore=.;LRT_pred=.;MutationTaster_score=.;MutationTaster_converted_rankscore=.;MutationTaster_pred=.;MutationAssessor_score=.;MutationAssessor_score_rankscore=.;MutationAssessor_pred=.;FATHMM_score=.;FATHMM_converted_rankscore=.;FATHMM_pred=.;PROVEAN_score=.;PROVEAN_converted_rankscore=.;PROVEAN_pred=.;VEST3_score=.;VEST3_rankscore=.;MetaSVM_score=.;MetaSVM_rankscore=.;MetaSVM_pred=.;MetaLR_score=.;MetaLR_rankscore=.;MetaLR_pred=.;M-CAP_score=.;M-CAP_rankscore=.;M-CAP_pred=.;CADD_raw=.;CADD_raw_rankscore=.;CADD_phred=.;DANN_score=.;DANN_rankscore=.;fathmm-MKL_coding_score=.;fathmm-MKL_coding_rankscore=.;fathmm-MKL_coding_pred=.;Eigen_coding_or_noncoding=.;Eigen-raw=.;Eigen-PC-raw=.;GenoCanyon_score=.;GenoCanyon_score_rankscore=.;integrated_fitCons_score=.;integrated_fitCons_score_rankscore=.;integrated_confidence_value=.;GERP++_RS=.;GERP++_RS_rankscore=.;phyloP100way_vertebrate=.;phyloP100way_vertebrate_rankscore=.;phyloP20way_mammalian=.;phyloP20way_mammalian_rankscore=.;phastCons100way_vertebrate=.;phastCons100way_vertebrate_rankscore=.;phastCons20way_mammalian=.;phastCons20way_mammalian_rankscore=.;SiPhy_29way_logOdds=.;SiPhy_29way_logOdds_rankscore=.;Interpro_domain=.;GTEx_V6_gene=.;GTEx_V6_tissue=.;esp6500siv2_all=.;esp6500siv2_aa=.;esp6500siv2_ea=.;ExAC_ALL=.;ExAC_AFR=.;ExAC_AMR=.;ExAC_EAS=.;ExAC_FIN=.;ExAC_NFE=.;ExAC_OTH=.;ExAC_SAS=.;ExAC_nontcga_ALL=.;ExAC_nontcga_AFR=.;ExAC_nontcga_AMR=.;ExAC_nontcga_EAS=.;ExAC_nontcga_FIN=.;ExAC_nontcga_NFE=.;ExAC_nontcga_OTH=.;ExAC_nontcga_SAS=.;ExAC_nonpsych_ALL=.;ExAC_nonpsych_AFR=.;ExAC_nonpsych_AMR=.;ExAC_nonpsych_EAS=.;ExAC_nonpsych_FIN=.;ExAC_nonpsych_NFE=.;ExAC_nonpsych_OTH=.;ExAC_nonpsych_SAS=.;1000g2015aug_all=.;1000g2015aug_afr=.;1000g2015aug_amr=.;1000g2015aug_eur=.;1000g2015aug_sas=.;CLNALLELEID=.;CLNDN=.;CLNDISDB=.;CLNREVSTAT=.;CLNSIG=.;dbscSNV_ADA_SCORE=.;dbscSNV_RF_SCORE=.;snp138NonFlagged=.;avsnp150=.;CADD13_RawScore=0.015973;CADD13_PHRED=2.741;Eigen=-0.3239;REVEL=.;MCAP=.;Interpro_domain=.;ICGC_Id=.;ICGC_Occurrence=.;gnomAD_genome_ALL=0.0003;gnomAD_genome_AFR=0.0001;gnomAD_genome_AMR=0;gnomAD_genome_ASJ=0;gnomAD_genome_EAS=0.0007;gnomAD_genome_FIN=0.0009;gnomAD_genome_NFE=0.0002;gnomAD_genome_OTH=0.0011;gerp++gt2=.;cosmic70=.;InterVar_automated=.;PVS1=.;PS1=.;PS2=.;PS3=.;PS4=.;PM1=.;PM2=.;PM3=.;PM4=.;PM5=.;PM6=.;PP1=.;PP2=.;PP3=.;PP4=.;PP5=.;BA1=.;BS1=.;BS2=.;BS3=.;BS4=.;BP1=.;BP2=.;BP3=.;BP4=.;BP5=.;BP6=.;BP7=.;Kaviar_AF=.;Kaviar_AC=.;Kaviar_AN=.;ALLELE_END',\n",
    " 'POS': '123416186',\n",
    " 'QUAL': '23.25',\n",
    " 'REF': 'A',\n",
    " 'SAMPLE': {'XG102': {'AD': '51,8',\n",
    "                      'DP': '59',\n",
    "                      'GQ': '32',\n",
    "                      'GT': '0/1',\n",
    "                      'PL': '32,0,1388'},\n",
    "            'XG103': {'AD': '47,0',\n",
    "                      'DP': '47',\n",
    "                      'GQ': '99',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,114,1353'},\n",
    "            'XG104': {'AD': '74,0',\n",
    "                      'DP': '74',\n",
    "                      'GQ': '51',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,51,1827'},\n",
    "            'XG202': {'AD': '51,8',\n",
    "                      'DP': '59',\n",
    "                      'GQ': '32',\n",
    "                      'GT': '0/1',\n",
    "                      'PL': '32,0,1388'},\n",
    "            'XG203': {'AD': '47,0',\n",
    "                      'DP': '47',\n",
    "                      'GQ': '99',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,114,1353'},\n",
    "            'XG204': {'AD': '74,0',\n",
    "                      'DP': '74',\n",
    "                      'GQ': '51',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,51,1827'},\n",
    "            'XG302': {'AD': '51,8',\n",
    "                      'DP': '59',\n",
    "                      'GQ': '32',\n",
    "                      'GT': '0/1',\n",
    "                      'PL': '32,0,1388'},\n",
    "            'XG303': {'AD': '47,0',\n",
    "                      'DP': '47',\n",
    "                      'GQ': '99',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,114,1353'},\n",
    "            'XG304': {'AD': '74,0',\n",
    "                      'DP': '74',\n",
    "                      'GQ': '51',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,51,1827'},\n",
    "            'XG402': {'AD': '51,8',\n",
    "                      'DP': '59',\n",
    "                      'GQ': '32',\n",
    "                      'GT': '0/1',\n",
    "                      'PL': '32,0,1388'},\n",
    "            'XG403': {'AD': '47,0',\n",
    "                      'DP': '47',\n",
    "                      'GQ': '99',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,114,1353'},\n",
    "            'XG404': {'AD': '74,0',\n",
    "                      'DP': '74',\n",
    "                      'GQ': '51',\n",
    "                      'GT': '0/0',\n",
    "                      'PL': '0,51,1827'}}}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "378fd6c6b20ef3f5c8b44b34840a3add",
     "grade": false,
     "grade_id": "cell-797a895b1e430349",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "header = \"CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tXG102\tXG103\tXG104\tXG202\tXG203\tXG204\tXG302\tXG303\tXG304\tXG402\tXG403\tXG404\".split('\\t')\n",
    "line = '4\t123416186\t.\tA\tG\t23.25\tPASS\tAC=1;AF=0.167;AN=6;BaseQRankSum=-2.542;ClippingRankSum=0;DP=180;ExcessHet=3.0103;FS=0;MLEAC=1;MLEAF=0.167;MQ=52.77;MQRankSum=-4.631;QD=0.39;ReadPosRankSum=1.45;SOR=0.758;VQSLOD=-8.209;culprit=MQ;ANNOVAR_DATE=2018-04-16;Func.refGene=intergenic;Gene.refGene=IL2,IL21;GeneDetail.refGene=dist=38536,dist=117597;ExonicFunc.refGene=.;AAChange.refGene=.;Func.ensGene=intergenic;Gene.ensGene=ENSG00000109471,ENSG00000138684;GeneDetail.ensGene=dist=38306,dist=117597;ExonicFunc.ensGene=.;AAChange.ensGene=.;cytoBand=4q27;gwasCatalog=.;tfbsConsSites=.;wgRna=.;targetScanS=.;Gene_symbol=.;OXPHOS_Complex=.;Ensembl_Gene_ID=.;Ensembl_Protein_ID=.;Uniprot_Name=.;Uniprot_ID=.;NCBI_Gene_ID=.;NCBI_Protein_ID=.;Gene_pos=.;AA_pos=.;AA_sub=.;Codon_sub=.;dbSNP_ID=.;PhyloP_46V=.;PhastCons_46V=.;PhyloP_100V=.;PhastCons_100V=.;SiteVar=.;PolyPhen2_prediction=.;PolyPhen2_score=.;SIFT_prediction=.;SIFT_score=.;FatHmm_prediction=.;FatHmm_score=.;PROVEAN_prediction=.;PROVEAN_score=.;MutAss_prediction=.;MutAss_score=.;EFIN_Swiss_Prot_Score=.;EFIN_Swiss_Prot_Prediction=.;EFIN_HumDiv_Score=.;EFIN_HumDiv_Prediction=.;CADD_score=.;CADD_Phred_score=.;CADD_prediction=.;Carol_prediction=.;Carol_score=.;Condel_score=.;Condel_pred=.;COVEC_WMV=.;COVEC_WMV_prediction=.;PolyPhen2_score_transf=.;PolyPhen2_pred_transf=.;SIFT_score_transf=.;SIFT_pred_transf=.;MutAss_score_transf=.;MutAss_pred_transf=.;Perc_coevo_Sites=.;Mean_MI_score=.;COSMIC_ID=.;Tumor_site=.;Examined_samples=.;Mutation_frequency=.;US=.;Status=.;Associated_disease=.;Presence_in_TD=.;Class_predicted=.;Prob_N=.;Prob_P=.;SIFT_score=.;SIFT_converted_rankscore=.;SIFT_pred=.;Polyphen2_HDIV_score=.;Polyphen2_HDIV_rankscore=.;Polyphen2_HDIV_pred=.;Polyphen2_HVAR_score=.;Polyphen2_HVAR_rankscore=.;Polyphen2_HVAR_pred=.;LRT_score=.;LRT_converted_rankscore=.;LRT_pred=.;MutationTaster_score=.;MutationTaster_converted_rankscore=.;MutationTaster_pred=.;MutationAssessor_score=.;MutationAssessor_score_rankscore=.;MutationAssessor_pred=.;FATHMM_score=.;FATHMM_converted_rankscore=.;FATHMM_pred=.;PROVEAN_score=.;PROVEAN_converted_rankscore=.;PROVEAN_pred=.;VEST3_score=.;VEST3_rankscore=.;MetaSVM_score=.;MetaSVM_rankscore=.;MetaSVM_pred=.;MetaLR_score=.;MetaLR_rankscore=.;MetaLR_pred=.;M-CAP_score=.;M-CAP_rankscore=.;M-CAP_pred=.;CADD_raw=.;CADD_raw_rankscore=.;CADD_phred=.;DANN_score=.;DANN_rankscore=.;fathmm-MKL_coding_score=.;fathmm-MKL_coding_rankscore=.;fathmm-MKL_coding_pred=.;Eigen_coding_or_noncoding=.;Eigen-raw=.;Eigen-PC-raw=.;GenoCanyon_score=.;GenoCanyon_score_rankscore=.;integrated_fitCons_score=.;integrated_fitCons_score_rankscore=.;integrated_confidence_value=.;GERP++_RS=.;GERP++_RS_rankscore=.;phyloP100way_vertebrate=.;phyloP100way_vertebrate_rankscore=.;phyloP20way_mammalian=.;phyloP20way_mammalian_rankscore=.;phastCons100way_vertebrate=.;phastCons100way_vertebrate_rankscore=.;phastCons20way_mammalian=.;phastCons20way_mammalian_rankscore=.;SiPhy_29way_logOdds=.;SiPhy_29way_logOdds_rankscore=.;Interpro_domain=.;GTEx_V6_gene=.;GTEx_V6_tissue=.;esp6500siv2_all=.;esp6500siv2_aa=.;esp6500siv2_ea=.;ExAC_ALL=.;ExAC_AFR=.;ExAC_AMR=.;ExAC_EAS=.;ExAC_FIN=.;ExAC_NFE=.;ExAC_OTH=.;ExAC_SAS=.;ExAC_nontcga_ALL=.;ExAC_nontcga_AFR=.;ExAC_nontcga_AMR=.;ExAC_nontcga_EAS=.;ExAC_nontcga_FIN=.;ExAC_nontcga_NFE=.;ExAC_nontcga_OTH=.;ExAC_nontcga_SAS=.;ExAC_nonpsych_ALL=.;ExAC_nonpsych_AFR=.;ExAC_nonpsych_AMR=.;ExAC_nonpsych_EAS=.;ExAC_nonpsych_FIN=.;ExAC_nonpsych_NFE=.;ExAC_nonpsych_OTH=.;ExAC_nonpsych_SAS=.;1000g2015aug_all=.;1000g2015aug_afr=.;1000g2015aug_amr=.;1000g2015aug_eur=.;1000g2015aug_sas=.;CLNALLELEID=.;CLNDN=.;CLNDISDB=.;CLNREVSTAT=.;CLNSIG=.;dbscSNV_ADA_SCORE=.;dbscSNV_RF_SCORE=.;snp138NonFlagged=.;avsnp150=.;CADD13_RawScore=0.015973;CADD13_PHRED=2.741;Eigen=-0.3239;REVEL=.;MCAP=.;Interpro_domain=.;ICGC_Id=.;ICGC_Occurrence=.;gnomAD_genome_ALL=0.0003;gnomAD_genome_AFR=0.0001;gnomAD_genome_AMR=0;gnomAD_genome_ASJ=0;gnomAD_genome_EAS=0.0007;gnomAD_genome_FIN=0.0009;gnomAD_genome_NFE=0.0002;gnomAD_genome_OTH=0.0011;gerp++gt2=.;cosmic70=.;InterVar_automated=.;PVS1=.;PS1=.;PS2=.;PS3=.;PS4=.;PM1=.;PM2=.;PM3=.;PM4=.;PM5=.;PM6=.;PP1=.;PP2=.;PP3=.;PP4=.;PP5=.;BA1=.;BS1=.;BS2=.;BS3=.;BS4=.;BP1=.;BP2=.;BP3=.;BP4=.;BP5=.;BP6=.;BP7=.;Kaviar_AF=.;Kaviar_AC=.;Kaviar_AN=.;ALLELE_END\tGT:AD:DP:GQ:PL\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827'\n",
    "def create_dict_from_line(header, line):\n",
    "    \"\"\"\n",
    "    Given the header and a single line, transform them into dictionary as described above. \n",
    "    Header and line input are provided in this cell. \n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    my_dict = {}\n",
    "    sample_dict = {}\n",
    "    lines = line.split('\\t')\n",
    "    format_field = ''\n",
    "    \n",
    "    #print((len(header), len(lines))\n",
    "    \n",
    "    for x in range(len(header)):\n",
    "        if header[x].upper()=='FORMAT':\n",
    "            format_field = lines[x]\n",
    "        elif header[x].startswith('XG'):\n",
    "            sample_dict[header[x]]= lines[x]\n",
    "        else:\n",
    "            my_dict[header[x]] = lines[x]\n",
    "            \n",
    "    my_dict['SAMPLE'] = format_sample_fields(format_field, sample_dict)\n",
    "    return my_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "24c12ba29aab52ec25b18eedc353108e",
     "grade": true,
     "grade_id": "cell-1ff8dfa70dbf2f1e",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "solution = {'CHROM': '4', 'POS': '123416186', 'ID': '.', 'REF': 'A', 'ALT': 'G', 'QUAL': '23.25', 'FILTER': 'PASS', 'INFO': 'AC=1;AF=0.167;AN=6;BaseQRankSum=-2.542;ClippingRankSum=0;DP=180;ExcessHet=3.0103;FS=0;MLEAC=1;MLEAF=0.167;MQ=52.77;MQRankSum=-4.631;QD=0.39;ReadPosRankSum=1.45;SOR=0.758;VQSLOD=-8.209;culprit=MQ;ANNOVAR_DATE=2018-04-16;Func.refGene=intergenic;Gene.refGene=IL2,IL21;GeneDetail.refGene=dist=38536,dist=117597;ExonicFunc.refGene=.;AAChange.refGene=.;Func.ensGene=intergenic;Gene.ensGene=ENSG00000109471,ENSG00000138684;GeneDetail.ensGene=dist=38306,dist=117597;ExonicFunc.ensGene=.;AAChange.ensGene=.;cytoBand=4q27;gwasCatalog=.;tfbsConsSites=.;wgRna=.;targetScanS=.;Gene_symbol=.;OXPHOS_Complex=.;Ensembl_Gene_ID=.;Ensembl_Protein_ID=.;Uniprot_Name=.;Uniprot_ID=.;NCBI_Gene_ID=.;NCBI_Protein_ID=.;Gene_pos=.;AA_pos=.;AA_sub=.;Codon_sub=.;dbSNP_ID=.;PhyloP_46V=.;PhastCons_46V=.;PhyloP_100V=.;PhastCons_100V=.;SiteVar=.;PolyPhen2_prediction=.;PolyPhen2_score=.;SIFT_prediction=.;SIFT_score=.;FatHmm_prediction=.;FatHmm_score=.;PROVEAN_prediction=.;PROVEAN_score=.;MutAss_prediction=.;MutAss_score=.;EFIN_Swiss_Prot_Score=.;EFIN_Swiss_Prot_Prediction=.;EFIN_HumDiv_Score=.;EFIN_HumDiv_Prediction=.;CADD_score=.;CADD_Phred_score=.;CADD_prediction=.;Carol_prediction=.;Carol_score=.;Condel_score=.;Condel_pred=.;COVEC_WMV=.;COVEC_WMV_prediction=.;PolyPhen2_score_transf=.;PolyPhen2_pred_transf=.;SIFT_score_transf=.;SIFT_pred_transf=.;MutAss_score_transf=.;MutAss_pred_transf=.;Perc_coevo_Sites=.;Mean_MI_score=.;COSMIC_ID=.;Tumor_site=.;Examined_samples=.;Mutation_frequency=.;US=.;Status=.;Associated_disease=.;Presence_in_TD=.;Class_predicted=.;Prob_N=.;Prob_P=.;SIFT_score=.;SIFT_converted_rankscore=.;SIFT_pred=.;Polyphen2_HDIV_score=.;Polyphen2_HDIV_rankscore=.;Polyphen2_HDIV_pred=.;Polyphen2_HVAR_score=.;Polyphen2_HVAR_rankscore=.;Polyphen2_HVAR_pred=.;LRT_score=.;LRT_converted_rankscore=.;LRT_pred=.;MutationTaster_score=.;MutationTaster_converted_rankscore=.;MutationTaster_pred=.;MutationAssessor_score=.;MutationAssessor_score_rankscore=.;MutationAssessor_pred=.;FATHMM_score=.;FATHMM_converted_rankscore=.;FATHMM_pred=.;PROVEAN_score=.;PROVEAN_converted_rankscore=.;PROVEAN_pred=.;VEST3_score=.;VEST3_rankscore=.;MetaSVM_score=.;MetaSVM_rankscore=.;MetaSVM_pred=.;MetaLR_score=.;MetaLR_rankscore=.;MetaLR_pred=.;M-CAP_score=.;M-CAP_rankscore=.;M-CAP_pred=.;CADD_raw=.;CADD_raw_rankscore=.;CADD_phred=.;DANN_score=.;DANN_rankscore=.;fathmm-MKL_coding_score=.;fathmm-MKL_coding_rankscore=.;fathmm-MKL_coding_pred=.;Eigen_coding_or_noncoding=.;Eigen-raw=.;Eigen-PC-raw=.;GenoCanyon_score=.;GenoCanyon_score_rankscore=.;integrated_fitCons_score=.;integrated_fitCons_score_rankscore=.;integrated_confidence_value=.;GERP++_RS=.;GERP++_RS_rankscore=.;phyloP100way_vertebrate=.;phyloP100way_vertebrate_rankscore=.;phyloP20way_mammalian=.;phyloP20way_mammalian_rankscore=.;phastCons100way_vertebrate=.;phastCons100way_vertebrate_rankscore=.;phastCons20way_mammalian=.;phastCons20way_mammalian_rankscore=.;SiPhy_29way_logOdds=.;SiPhy_29way_logOdds_rankscore=.;Interpro_domain=.;GTEx_V6_gene=.;GTEx_V6_tissue=.;esp6500siv2_all=.;esp6500siv2_aa=.;esp6500siv2_ea=.;ExAC_ALL=.;ExAC_AFR=.;ExAC_AMR=.;ExAC_EAS=.;ExAC_FIN=.;ExAC_NFE=.;ExAC_OTH=.;ExAC_SAS=.;ExAC_nontcga_ALL=.;ExAC_nontcga_AFR=.;ExAC_nontcga_AMR=.;ExAC_nontcga_EAS=.;ExAC_nontcga_FIN=.;ExAC_nontcga_NFE=.;ExAC_nontcga_OTH=.;ExAC_nontcga_SAS=.;ExAC_nonpsych_ALL=.;ExAC_nonpsych_AFR=.;ExAC_nonpsych_AMR=.;ExAC_nonpsych_EAS=.;ExAC_nonpsych_FIN=.;ExAC_nonpsych_NFE=.;ExAC_nonpsych_OTH=.;ExAC_nonpsych_SAS=.;1000g2015aug_all=.;1000g2015aug_afr=.;1000g2015aug_amr=.;1000g2015aug_eur=.;1000g2015aug_sas=.;CLNALLELEID=.;CLNDN=.;CLNDISDB=.;CLNREVSTAT=.;CLNSIG=.;dbscSNV_ADA_SCORE=.;dbscSNV_RF_SCORE=.;snp138NonFlagged=.;avsnp150=.;CADD13_RawScore=0.015973;CADD13_PHRED=2.741;Eigen=-0.3239;REVEL=.;MCAP=.;Interpro_domain=.;ICGC_Id=.;ICGC_Occurrence=.;gnomAD_genome_ALL=0.0003;gnomAD_genome_AFR=0.0001;gnomAD_genome_AMR=0;gnomAD_genome_ASJ=0;gnomAD_genome_EAS=0.0007;gnomAD_genome_FIN=0.0009;gnomAD_genome_NFE=0.0002;gnomAD_genome_OTH=0.0011;gerp++gt2=.;cosmic70=.;InterVar_automated=.;PVS1=.;PS1=.;PS2=.;PS3=.;PS4=.;PM1=.;PM2=.;PM3=.;PM4=.;PM5=.;PM6=.;PP1=.;PP2=.;PP3=.;PP4=.;PP5=.;BA1=.;BS1=.;BS2=.;BS3=.;BS4=.;BP1=.;BP2=.;BP3=.;BP4=.;BP5=.;BP6=.;BP7=.;Kaviar_AF=.;Kaviar_AC=.;Kaviar_AN=.;ALLELE_END', 'SAMPLE': {'XG102': {'GT': '0/1', 'AD': '51,8', 'DP': '59', 'GQ': '32', 'PL': '32,0,1388'}, 'XG103': {'GT': '0/0', 'AD': '47,0', 'DP': '47', 'GQ': '99', 'PL': '0,114,1353'}, 'XG104': {'GT': '0/0', 'AD': '74,0', 'DP': '74', 'GQ': '51', 'PL': '0,51,1827'}, 'XG202': {'GT': '0/1', 'AD': '51,8', 'DP': '59', 'GQ': '32', 'PL': '32,0,1388'}, 'XG203': {'GT': '0/0', 'AD': '47,0', 'DP': '47', 'GQ': '99', 'PL': '0,114,1353'}, 'XG204': {'GT': '0/0', 'AD': '74,0', 'DP': '74', 'GQ': '51', 'PL': '0,51,1827'}, 'XG302': {'GT': '0/1', 'AD': '51,8', 'DP': '59', 'GQ': '32', 'PL': '32,0,1388'}, 'XG303': {'GT': '0/0', 'AD': '47,0', 'DP': '47', 'GQ': '99', 'PL': '0,114,1353'}, 'XG304': {'GT': '0/0', 'AD': '74,0', 'DP': '74', 'GQ': '51', 'PL': '0,51,1827'}, 'XG402': {'GT': '0/1', 'AD': '51,8', 'DP': '59', 'GQ': '32', 'PL': '32,0,1388'}, 'XG403': {'GT': '0/0', 'AD': '47,0', 'DP': '47', 'GQ': '99', 'PL': '0,114,1353'}, 'XG404': {'GT': '0/0', 'AD': '74,0', 'DP': '74', 'GQ': '51', 'PL': '0,51,1827'}}}\n",
    "\n",
    "header = \"CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tXG102\tXG103\tXG104\tXG202\tXG203\tXG204\tXG302\tXG303\tXG304\tXG402\tXG403\tXG404\".split('\\t')\n",
    "line = '4\t123416186\t.\tA\tG\t23.25\tPASS\tAC=1;AF=0.167;AN=6;BaseQRankSum=-2.542;ClippingRankSum=0;DP=180;ExcessHet=3.0103;FS=0;MLEAC=1;MLEAF=0.167;MQ=52.77;MQRankSum=-4.631;QD=0.39;ReadPosRankSum=1.45;SOR=0.758;VQSLOD=-8.209;culprit=MQ;ANNOVAR_DATE=2018-04-16;Func.refGene=intergenic;Gene.refGene=IL2,IL21;GeneDetail.refGene=dist=38536,dist=117597;ExonicFunc.refGene=.;AAChange.refGene=.;Func.ensGene=intergenic;Gene.ensGene=ENSG00000109471,ENSG00000138684;GeneDetail.ensGene=dist=38306,dist=117597;ExonicFunc.ensGene=.;AAChange.ensGene=.;cytoBand=4q27;gwasCatalog=.;tfbsConsSites=.;wgRna=.;targetScanS=.;Gene_symbol=.;OXPHOS_Complex=.;Ensembl_Gene_ID=.;Ensembl_Protein_ID=.;Uniprot_Name=.;Uniprot_ID=.;NCBI_Gene_ID=.;NCBI_Protein_ID=.;Gene_pos=.;AA_pos=.;AA_sub=.;Codon_sub=.;dbSNP_ID=.;PhyloP_46V=.;PhastCons_46V=.;PhyloP_100V=.;PhastCons_100V=.;SiteVar=.;PolyPhen2_prediction=.;PolyPhen2_score=.;SIFT_prediction=.;SIFT_score=.;FatHmm_prediction=.;FatHmm_score=.;PROVEAN_prediction=.;PROVEAN_score=.;MutAss_prediction=.;MutAss_score=.;EFIN_Swiss_Prot_Score=.;EFIN_Swiss_Prot_Prediction=.;EFIN_HumDiv_Score=.;EFIN_HumDiv_Prediction=.;CADD_score=.;CADD_Phred_score=.;CADD_prediction=.;Carol_prediction=.;Carol_score=.;Condel_score=.;Condel_pred=.;COVEC_WMV=.;COVEC_WMV_prediction=.;PolyPhen2_score_transf=.;PolyPhen2_pred_transf=.;SIFT_score_transf=.;SIFT_pred_transf=.;MutAss_score_transf=.;MutAss_pred_transf=.;Perc_coevo_Sites=.;Mean_MI_score=.;COSMIC_ID=.;Tumor_site=.;Examined_samples=.;Mutation_frequency=.;US=.;Status=.;Associated_disease=.;Presence_in_TD=.;Class_predicted=.;Prob_N=.;Prob_P=.;SIFT_score=.;SIFT_converted_rankscore=.;SIFT_pred=.;Polyphen2_HDIV_score=.;Polyphen2_HDIV_rankscore=.;Polyphen2_HDIV_pred=.;Polyphen2_HVAR_score=.;Polyphen2_HVAR_rankscore=.;Polyphen2_HVAR_pred=.;LRT_score=.;LRT_converted_rankscore=.;LRT_pred=.;MutationTaster_score=.;MutationTaster_converted_rankscore=.;MutationTaster_pred=.;MutationAssessor_score=.;MutationAssessor_score_rankscore=.;MutationAssessor_pred=.;FATHMM_score=.;FATHMM_converted_rankscore=.;FATHMM_pred=.;PROVEAN_score=.;PROVEAN_converted_rankscore=.;PROVEAN_pred=.;VEST3_score=.;VEST3_rankscore=.;MetaSVM_score=.;MetaSVM_rankscore=.;MetaSVM_pred=.;MetaLR_score=.;MetaLR_rankscore=.;MetaLR_pred=.;M-CAP_score=.;M-CAP_rankscore=.;M-CAP_pred=.;CADD_raw=.;CADD_raw_rankscore=.;CADD_phred=.;DANN_score=.;DANN_rankscore=.;fathmm-MKL_coding_score=.;fathmm-MKL_coding_rankscore=.;fathmm-MKL_coding_pred=.;Eigen_coding_or_noncoding=.;Eigen-raw=.;Eigen-PC-raw=.;GenoCanyon_score=.;GenoCanyon_score_rankscore=.;integrated_fitCons_score=.;integrated_fitCons_score_rankscore=.;integrated_confidence_value=.;GERP++_RS=.;GERP++_RS_rankscore=.;phyloP100way_vertebrate=.;phyloP100way_vertebrate_rankscore=.;phyloP20way_mammalian=.;phyloP20way_mammalian_rankscore=.;phastCons100way_vertebrate=.;phastCons100way_vertebrate_rankscore=.;phastCons20way_mammalian=.;phastCons20way_mammalian_rankscore=.;SiPhy_29way_logOdds=.;SiPhy_29way_logOdds_rankscore=.;Interpro_domain=.;GTEx_V6_gene=.;GTEx_V6_tissue=.;esp6500siv2_all=.;esp6500siv2_aa=.;esp6500siv2_ea=.;ExAC_ALL=.;ExAC_AFR=.;ExAC_AMR=.;ExAC_EAS=.;ExAC_FIN=.;ExAC_NFE=.;ExAC_OTH=.;ExAC_SAS=.;ExAC_nontcga_ALL=.;ExAC_nontcga_AFR=.;ExAC_nontcga_AMR=.;ExAC_nontcga_EAS=.;ExAC_nontcga_FIN=.;ExAC_nontcga_NFE=.;ExAC_nontcga_OTH=.;ExAC_nontcga_SAS=.;ExAC_nonpsych_ALL=.;ExAC_nonpsych_AFR=.;ExAC_nonpsych_AMR=.;ExAC_nonpsych_EAS=.;ExAC_nonpsych_FIN=.;ExAC_nonpsych_NFE=.;ExAC_nonpsych_OTH=.;ExAC_nonpsych_SAS=.;1000g2015aug_all=.;1000g2015aug_afr=.;1000g2015aug_amr=.;1000g2015aug_eur=.;1000g2015aug_sas=.;CLNALLELEID=.;CLNDN=.;CLNDISDB=.;CLNREVSTAT=.;CLNSIG=.;dbscSNV_ADA_SCORE=.;dbscSNV_RF_SCORE=.;snp138NonFlagged=.;avsnp150=.;CADD13_RawScore=0.015973;CADD13_PHRED=2.741;Eigen=-0.3239;REVEL=.;MCAP=.;Interpro_domain=.;ICGC_Id=.;ICGC_Occurrence=.;gnomAD_genome_ALL=0.0003;gnomAD_genome_AFR=0.0001;gnomAD_genome_AMR=0;gnomAD_genome_ASJ=0;gnomAD_genome_EAS=0.0007;gnomAD_genome_FIN=0.0009;gnomAD_genome_NFE=0.0002;gnomAD_genome_OTH=0.0011;gerp++gt2=.;cosmic70=.;InterVar_automated=.;PVS1=.;PS1=.;PS2=.;PS3=.;PS4=.;PM1=.;PM2=.;PM3=.;PM4=.;PM5=.;PM6=.;PP1=.;PP2=.;PP3=.;PP4=.;PP5=.;BA1=.;BS1=.;BS2=.;BS3=.;BS4=.;BP1=.;BP2=.;BP3=.;BP4=.;BP5=.;BP6=.;BP7=.;Kaviar_AF=.;Kaviar_AC=.;Kaviar_AN=.;ALLELE_END\tGT:AD:DP:GQ:PL\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827\t0/1:51,8:59:32:32,0,1388\t0/0:47,0:47:99:0,114,1353\t0/0:74,0:74:51:0,51,1827'\n",
    "\n",
    "assert create_dict_from_line(header, line) == solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "36fb690b7ddc223135d11fa3fbb381d8",
     "grade": false,
     "grade_id": "cell-6154b4499449fb82",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 5 (10 points)\n",
    "\n",
    "Write a function whose input is a filename for a vcf. The function reads the vcf file one variant at a time and transforms it into a dictionary using the create_dict_from_line function. It returns a list containing all the variant dictionaries. NOTE: The file used for the automated test only has one line. Your function should be able to handle multiple lines.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "793670fcc309dadc4f8249475f9962ba",
     "grade": false,
     "grade_id": "cell-eabb7e5d7d13d274",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def read_vcf_file(filename):\n",
    "    \"\"\"\n",
    "    See description above\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    vcf_1=[]\n",
    "    vcf_1 = [i.strip('\\n') \n",
    "             for i in open(filename).readlines() \n",
    "                 if not ('##') in i]\n",
    "    header = vcf_1[0][1:].split('\\t')\n",
    "    lines = vcf_1[1:]\n",
    "    outdict=[]\n",
    "    for word in lines:\n",
    "        outdict.append(create_dict_from_line(header, word))\n",
    "    return outdict\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9fe3f4cae91800408ba1d30af59de18b",
     "grade": true,
     "grade_id": "cell-56b74f86f9822086",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "expected_output = [{'CHROM': '4',\n",
    "  'POS': '123416186',\n",
    "  'ID': '.',\n",
    "  'REF': 'A',\n",
    "  'ALT': 'G',\n",
    "  'QUAL': '23.25',\n",
    "  'FILTER': 'PASS',\n",
    "  'INFO': 'AC=1;AF=0.167;AN=6;BaseQRankSum=-2.542;ClippingRankSum=0;DP=180;ExcessHet=3.0103;FS=0;MLEAC=1;MLEAF=0.167;MQ=52.77;MQRankSum=-4.631;QD=0.39;ReadPosRankSum=1.45;SOR=0.758;VQSLOD=-8.209;culprit=MQ;ANNOVAR_DATE=2018-04-16;Func.refGene=intergenic;Gene.refGene=IL2,IL21;GeneDetail.refGene=dist=38536,dist=117597;ExonicFunc.refGene=.;AAChange.refGene=.;Func.ensGene=intergenic;Gene.ensGene=ENSG00000109471,ENSG00000138684;GeneDetail.ensGene=dist=38306,dist=117597;ExonicFunc.ensGene=.;AAChange.ensGene=.;cytoBand=4q27;gwasCatalog=.;tfbsConsSites=.;wgRna=.;targetScanS=.;Gene_symbol=.;OXPHOS_Complex=.;Ensembl_Gene_ID=.;Ensembl_Protein_ID=.;Uniprot_Name=.;Uniprot_ID=.;NCBI_Gene_ID=.;NCBI_Protein_ID=.;Gene_pos=.;AA_pos=.;AA_sub=.;Codon_sub=.;dbSNP_ID=.;PhyloP_46V=.;PhastCons_46V=.;PhyloP_100V=.;PhastCons_100V=.;SiteVar=.;PolyPhen2_prediction=.;PolyPhen2_score=.;SIFT_prediction=.;SIFT_score=.;FatHmm_prediction=.;FatHmm_score=.;PROVEAN_prediction=.;PROVEAN_score=.;MutAss_prediction=.;MutAss_score=.;EFIN_Swiss_Prot_Score=.;EFIN_Swiss_Prot_Prediction=.;EFIN_HumDiv_Score=.;EFIN_HumDiv_Prediction=.;CADD_score=.;CADD_Phred_score=.;CADD_prediction=.;Carol_prediction=.;Carol_score=.;Condel_score=.;Condel_pred=.;COVEC_WMV=.;COVEC_WMV_prediction=.;PolyPhen2_score_transf=.;PolyPhen2_pred_transf=.;SIFT_score_transf=.;SIFT_pred_transf=.;MutAss_score_transf=.;MutAss_pred_transf=.;Perc_coevo_Sites=.;Mean_MI_score=.;COSMIC_ID=.;Tumor_site=.;Examined_samples=.;Mutation_frequency=.;US=.;Status=.;Associated_disease=.;Presence_in_TD=.;Class_predicted=.;Prob_N=.;Prob_P=.;SIFT_score=.;SIFT_converted_rankscore=.;SIFT_pred=.;Polyphen2_HDIV_score=.;Polyphen2_HDIV_rankscore=.;Polyphen2_HDIV_pred=.;Polyphen2_HVAR_score=.;Polyphen2_HVAR_rankscore=.;Polyphen2_HVAR_pred=.;LRT_score=.;LRT_converted_rankscore=.;LRT_pred=.;MutationTaster_score=.;MutationTaster_converted_rankscore=.;MutationTaster_pred=.;MutationAssessor_score=.;MutationAssessor_score_rankscore=.;MutationAssessor_pred=.;FATHMM_score=.;FATHMM_converted_rankscore=.;FATHMM_pred=.;PROVEAN_score=.;PROVEAN_converted_rankscore=.;PROVEAN_pred=.;VEST3_score=.;VEST3_rankscore=.;MetaSVM_score=.;MetaSVM_rankscore=.;MetaSVM_pred=.;MetaLR_score=.;MetaLR_rankscore=.;MetaLR_pred=.;M-CAP_score=.;M-CAP_rankscore=.;M-CAP_pred=.;CADD_raw=.;CADD_raw_rankscore=.;CADD_phred=.;DANN_score=.;DANN_rankscore=.;fathmm-MKL_coding_score=.;fathmm-MKL_coding_rankscore=.;fathmm-MKL_coding_pred=.;Eigen_coding_or_noncoding=.;Eigen-raw=.;Eigen-PC-raw=.;GenoCanyon_score=.;GenoCanyon_score_rankscore=.;integrated_fitCons_score=.;integrated_fitCons_score_rankscore=.;integrated_confidence_value=.;GERP++_RS=.;GERP++_RS_rankscore=.;phyloP100way_vertebrate=.;phyloP100way_vertebrate_rankscore=.;phyloP20way_mammalian=.;phyloP20way_mammalian_rankscore=.;phastCons100way_vertebrate=.;phastCons100way_vertebrate_rankscore=.;phastCons20way_mammalian=.;phastCons20way_mammalian_rankscore=.;SiPhy_29way_logOdds=.;SiPhy_29way_logOdds_rankscore=.;Interpro_domain=.;GTEx_V6_gene=.;GTEx_V6_tissue=.;esp6500siv2_all=.;esp6500siv2_aa=.;esp6500siv2_ea=.;ExAC_ALL=.;ExAC_AFR=.;ExAC_AMR=.;ExAC_EAS=.;ExAC_FIN=.;ExAC_NFE=.;ExAC_OTH=.;ExAC_SAS=.;ExAC_nontcga_ALL=.;ExAC_nontcga_AFR=.;ExAC_nontcga_AMR=.;ExAC_nontcga_EAS=.;ExAC_nontcga_FIN=.;ExAC_nontcga_NFE=.;ExAC_nontcga_OTH=.;ExAC_nontcga_SAS=.;ExAC_nonpsych_ALL=.;ExAC_nonpsych_AFR=.;ExAC_nonpsych_AMR=.;ExAC_nonpsych_EAS=.;ExAC_nonpsych_FIN=.;ExAC_nonpsych_NFE=.;ExAC_nonpsych_OTH=.;ExAC_nonpsych_SAS=.;1000g2015aug_all=.;1000g2015aug_afr=.;1000g2015aug_amr=.;1000g2015aug_eur=.;1000g2015aug_sas=.;CLNALLELEID=.;CLNDN=.;CLNDISDB=.;CLNREVSTAT=.;CLNSIG=.;dbscSNV_ADA_SCORE=.;dbscSNV_RF_SCORE=.;snp138NonFlagged=.;avsnp150=.;CADD13_RawScore=0.015973;CADD13_PHRED=2.741;Eigen=-0.3239;REVEL=.;MCAP=.;Interpro_domain=.;ICGC_Id=.;ICGC_Occurrence=.;gnomAD_genome_ALL=0.0003;gnomAD_genome_AFR=0.0001;gnomAD_genome_AMR=0;gnomAD_genome_ASJ=0;gnomAD_genome_EAS=0.0007;gnomAD_genome_FIN=0.0009;gnomAD_genome_NFE=0.0002;gnomAD_genome_OTH=0.0011;gerp++gt2=.;cosmic70=.;InterVar_automated=.;PVS1=.;PS1=.;PS2=.;PS3=.;PS4=.;PM1=.;PM2=.;PM3=.;PM4=.;PM5=.;PM6=.;PP1=.;PP2=.;PP3=.;PP4=.;PP5=.;BA1=.;BS1=.;BS2=.;BS3=.;BS4=.;BP1=.;BP2=.;BP3=.;BP4=.;BP5=.;BP6=.;BP7=.;Kaviar_AF=.;Kaviar_AC=.;Kaviar_AN=.;ALLELE_END',\n",
    "  'SAMPLE': {'XG102': {'GT': '0/1',\n",
    "    'AD': '51,8',\n",
    "    'DP': '59',\n",
    "    'GQ': '32',\n",
    "    'PL': '32,0,1388'},\n",
    "   'XG103': {'GT': '0/0',\n",
    "    'AD': '47,0',\n",
    "    'DP': '47',\n",
    "    'GQ': '99',\n",
    "    'PL': '0,114,1353'},\n",
    "   'XG104': {'GT': '0/0',\n",
    "    'AD': '74,0',\n",
    "    'DP': '74',\n",
    "    'GQ': '51',\n",
    "    'PL': '0,51,1827'},\n",
    "   'XG202': {'GT': '0/1',\n",
    "    'AD': '51,8',\n",
    "    'DP': '59',\n",
    "    'GQ': '32',\n",
    "    'PL': '32,0,1388'},\n",
    "   'XG203': {'GT': '0/0',\n",
    "    'AD': '47,0',\n",
    "    'DP': '47',\n",
    "    'GQ': '99',\n",
    "    'PL': '0,114,1353'},\n",
    "   'XG204': {'GT': '0/0',\n",
    "    'AD': '74,0',\n",
    "    'DP': '74',\n",
    "    'GQ': '51',\n",
    "    'PL': '0,51,1827'},\n",
    "   'XG302': {'GT': '0/1',\n",
    "    'AD': '51,8',\n",
    "    'DP': '59',\n",
    "    'GQ': '32',\n",
    "    'PL': '32,0,1388'},\n",
    "   'XG303': {'GT': '0/0',\n",
    "    'AD': '47,0',\n",
    "    'DP': '47',\n",
    "    'GQ': '99',\n",
    "    'PL': '0,114,1353'},\n",
    "   'XG304': {'GT': '0/0',\n",
    "    'AD': '74,0',\n",
    "    'DP': '74',\n",
    "    'GQ': '51',\n",
    "    'PL': '0,51,1827'},\n",
    "   'XG402': {'GT': '0/1',\n",
    "    'AD': '51,8',\n",
    "    'DP': '59',\n",
    "    'GQ': '32',\n",
    "    'PL': '32,0,1388'},\n",
    "   'XG403': {'GT': '0/0',\n",
    "    'AD': '47,0',\n",
    "    'DP': '47',\n",
    "    'GQ': '99',\n",
    "    'PL': '0,114,1353'},\n",
    "   'XG404': {'GT': '0/0',\n",
    "    'AD': '74,0',\n",
    "    'DP': '74',\n",
    "    'GQ': '51',\n",
    "    'PL': '0,51,1827'}}},\n",
    " {'CHROM': '12',\n",
    "  'POS': '81444551',\n",
    "  'ID': 'rs10746177',\n",
    "  'REF': 'T',\n",
    "  'ALT': 'C',\n",
    "  'QUAL': '5022.69',\n",
    "  'FILTER': 'PASS',\n",
    "  'INFO': 'AC=6;AF=1;AN=6;DP=185;ExcessHet=3.0103;FS=0;MLEAC=6;MLEAF=1;MQ=59.95;QD=27.3;SOR=1.042;VQSLOD=4.51;culprit=QD;DB;POSITIVE_TRAIN_SITE;ANNOVAR_DATE=2018-04-16;Func.refGene=intergenic;Gene.refGene=LIN7A,ACSS3;GeneDetail.refGene=dist=112857,dist=27258;ExonicFunc.refGene=.;AAChange.refGene=.;Func.ensGene=intronic;Gene.ensGene=ENSG00000111058;GeneDetail.ensGene=.;ExonicFunc.ensGene=.;AAChange.ensGene=.;cytoBand=12q21.31;gwasCatalog=.;tfbsConsSites=.;wgRna=.;targetScanS=.;Gene_symbol=.;OXPHOS_Complex=.;Ensembl_Gene_ID=.;Ensembl_Protein_ID=.;Uniprot_Name=.;Uniprot_ID=.;NCBI_Gene_ID=.;NCBI_Protein_ID=.;Gene_pos=.;AA_pos=.;AA_sub=.;Codon_sub=.;dbSNP_ID=.;PhyloP_46V=.;PhastCons_46V=.;PhyloP_100V=.;PhastCons_100V=.;SiteVar=.;PolyPhen2_prediction=.;PolyPhen2_score=.;SIFT_prediction=.;SIFT_score=.;FatHmm_prediction=.;FatHmm_score=.;PROVEAN_prediction=.;PROVEAN_score=.;MutAss_prediction=.;MutAss_score=.;EFIN_Swiss_Prot_Score=.;EFIN_Swiss_Prot_Prediction=.;EFIN_HumDiv_Score=.;EFIN_HumDiv_Prediction=.;CADD_score=.;CADD_Phred_score=.;CADD_prediction=.;Carol_prediction=.;Carol_score=.;Condel_score=.;Condel_pred=.;COVEC_WMV=.;COVEC_WMV_prediction=.;PolyPhen2_score_transf=.;PolyPhen2_pred_transf=.;SIFT_score_transf=.;SIFT_pred_transf=.;MutAss_score_transf=.;MutAss_pred_transf=.;Perc_coevo_Sites=.;Mean_MI_score=.;COSMIC_ID=.;Tumor_site=.;Examined_samples=.;Mutation_frequency=.;US=.;Status=.;Associated_disease=.;Presence_in_TD=.;Class_predicted=.;Prob_N=.;Prob_P=.;SIFT_score=.;SIFT_converted_rankscore=.;SIFT_pred=.;Polyphen2_HDIV_score=.;Polyphen2_HDIV_rankscore=.;Polyphen2_HDIV_pred=.;Polyphen2_HVAR_score=.;Polyphen2_HVAR_rankscore=.;Polyphen2_HVAR_pred=.;LRT_score=.;LRT_converted_rankscore=.;LRT_pred=.;MutationTaster_score=.;MutationTaster_converted_rankscore=.;MutationTaster_pred=.;MutationAssessor_score=.;MutationAssessor_score_rankscore=.;MutationAssessor_pred=.;FATHMM_score=.;FATHMM_converted_rankscore=.;FATHMM_pred=.;PROVEAN_score=.;PROVEAN_converted_rankscore=.;PROVEAN_pred=.;VEST3_score=.;VEST3_rankscore=.;MetaSVM_score=.;MetaSVM_rankscore=.;MetaSVM_pred=.;MetaLR_score=.;MetaLR_rankscore=.;MetaLR_pred=.;M-CAP_score=.;M-CAP_rankscore=.;M-CAP_pred=.;CADD_raw=.;CADD_raw_rankscore=.;CADD_phred=.;DANN_score=.;DANN_rankscore=.;fathmm-MKL_coding_score=.;fathmm-MKL_coding_rankscore=.;fathmm-MKL_coding_pred=.;Eigen_coding_or_noncoding=.;Eigen-raw=.;Eigen-PC-raw=.;GenoCanyon_score=.;GenoCanyon_score_rankscore=.;integrated_fitCons_score=.;integrated_fitCons_score_rankscore=.;integrated_confidence_value=.;GERP++_RS=.;GERP++_RS_rankscore=.;phyloP100way_vertebrate=.;phyloP100way_vertebrate_rankscore=.;phyloP20way_mammalian=.;phyloP20way_mammalian_rankscore=.;phastCons100way_vertebrate=.;phastCons100way_vertebrate_rankscore=.;phastCons20way_mammalian=.;phastCons20way_mammalian_rankscore=.;SiPhy_29way_logOdds=.;SiPhy_29way_logOdds_rankscore=.;Interpro_domain=.;GTEx_V6_gene=.;GTEx_V6_tissue=.;esp6500siv2_all=.;esp6500siv2_aa=.;esp6500siv2_ea=.;ExAC_ALL=.;ExAC_AFR=.;ExAC_AMR=.;ExAC_EAS=.;ExAC_FIN=.;ExAC_NFE=.;ExAC_OTH=.;ExAC_SAS=.;ExAC_nontcga_ALL=.;ExAC_nontcga_AFR=.;ExAC_nontcga_AMR=.;ExAC_nontcga_EAS=.;ExAC_nontcga_FIN=.;ExAC_nontcga_NFE=.;ExAC_nontcga_OTH=.;ExAC_nontcga_SAS=.;ExAC_nonpsych_ALL=.;ExAC_nonpsych_AFR=.;ExAC_nonpsych_AMR=.;ExAC_nonpsych_EAS=.;ExAC_nonpsych_FIN=.;ExAC_nonpsych_NFE=.;ExAC_nonpsych_OTH=.;ExAC_nonpsych_SAS=.;1000g2015aug_all=0.9998;1000g2015aug_afr=0.9992;1000g2015aug_amr=1;1000g2015aug_eur=1;1000g2015aug_sas=1;CLNALLELEID=.;CLNDN=.;CLNDISDB=.;CLNREVSTAT=.;CLNSIG=.;dbscSNV_ADA_SCORE=.;dbscSNV_RF_SCORE=.;snp138NonFlagged=rs10746177;avsnp150=rs10746177;CADD13_RawScore=-0.101662;CADD13_PHRED=1.718;Eigen=-0.4730;REVEL=.;MCAP=.;Interpro_domain=.;ICGC_Id=.;ICGC_Occurrence=.;gnomAD_genome_ALL=0.9988;gnomAD_genome_AFR=0.9961;gnomAD_genome_AMR=1;gnomAD_genome_ASJ=1;gnomAD_genome_EAS=1;gnomAD_genome_FIN=1;gnomAD_genome_NFE=0.9998;gnomAD_genome_OTH=1;gerp++gt2=.;cosmic70=.;InterVar_automated=.;PVS1=.;PS1=.;PS2=.;PS3=.;PS4=.;PM1=.;PM2=.;PM3=.;PM4=.;PM5=.;PM6=.;PP1=.;PP2=.;PP3=.;PP4=.;PP5=.;BA1=.;BS1=.;BS2=.;BS3=.;BS4=.;BP1=.;BP2=.;BP3=.;BP4=.;BP5=.;BP6=.;BP7=.;Kaviar_AF=0.953358;Kaviar_AC=24814;Kaviar_AN=26028;ALLELE_END',\n",
    "  'SAMPLE': {'XG102': {'GT': '1/1',\n",
    "    'AD': '0,72',\n",
    "    'DP': '72',\n",
    "    'GQ': '99',\n",
    "    'PL': '1905,214,0'},\n",
    "   'XG103': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1568,168,0'},\n",
    "   'XG104': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1563,168,0'},\n",
    "   'XG202': {'GT': '1/1',\n",
    "    'AD': '0,72',\n",
    "    'DP': '72',\n",
    "    'GQ': '99',\n",
    "    'PL': '1905,214,0'},\n",
    "   'XG203': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1568,168,0'},\n",
    "   'XG204': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1563,168,0'},\n",
    "   'XG302': {'GT': '1/1',\n",
    "    'AD': '0,72',\n",
    "    'DP': '72',\n",
    "    'GQ': '99',\n",
    "    'PL': '1905,214,0'},\n",
    "   'XG303': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1568,168,0'},\n",
    "   'XG304': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1563,168,0'},\n",
    "   'XG402': {'GT': '1/1',\n",
    "    'AD': '0,72',\n",
    "    'DP': '72',\n",
    "    'GQ': '99',\n",
    "    'PL': '1905,214,0'},\n",
    "   'XG403': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1568,168,0'},\n",
    "   'XG404': {'GT': '1/1',\n",
    "    'AD': '0,56',\n",
    "    'DP': '56',\n",
    "    'GQ': '99',\n",
    "    'PL': '1563,168,0'}}}]\n",
    "result = read_vcf_file('two_variants.vcf')\n",
    "assert expected_output == result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f5f76399c4a590e128eb1b2810d9d8bb",
     "grade": false,
     "grade_id": "cell-a899884d841f13ac",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 6 (5 points)\n",
    "\n",
    "Write a function that extracts the info field from the data dictionary that was created in the previous part. The function should return all the info field dictionaries as list. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3a1d1988fc40a1790b86296a6ed87ba0",
     "grade": false,
     "grade_id": "cell-8a653a060e0f679c",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def extract_info_field(data):\n",
    "    \"\"\"\n",
    "    See description in part 6\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    outlist = []\n",
    "    for word in data:\n",
    "        outlist.append(word['INFO'])\n",
    "    return outlist\n",
    "\n",
    "#extract_info_field(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4cee01cf3a50ba49b6b18b90683c7edb",
     "grade": true,
     "grade_id": "cell-59c0cf835a495bdc",
     "locked": true,
     "points": 5,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "expected_results = open('info_field.vcf', 'r').readline().strip()\n",
    "info_field = extract_info_field(result)\n",
    "\n",
    "assert expected_results == info_field[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e6871ce2f3251a4388335ddcf689e675",
     "grade": false,
     "grade_id": "cell-b518bc208514b5fa",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 7 (10 points)\n",
    "\n",
    "You now need to figure out that data types for each of the info fields. Begin by writing a function that first takes the info fields and turns them into a dictionary. Make sure to skip any fields that do not have a value or are missing a value. Also replace `\\x3b` with a comma and `\\x3d` with an equal sign.\n",
    "\n",
    "Note: only return keys that have values! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ddf5f127cadad9653453509b00b55af9",
     "grade": false,
     "grade_id": "cell-288839e06fb1e24e",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def create_dictionary_of_info_field_values(data):\n",
    "    \"\"\"\n",
    "    See description of part 7\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "    outdict = {}\n",
    "    for word in data:\n",
    "        dict_replace = word.replace('\\x3d', '=').split(';')\n",
    "        for items_ in dict_replace:\n",
    "            items_ = items_.split('=')\n",
    "            \n",
    "            if len(items_) <= 1:\n",
    "                continue\n",
    "                \n",
    "            key = items_[0]\n",
    "            value = '='.join(items_[1:])\n",
    "            \n",
    "            if not value or value == '.':\n",
    "                continue\n",
    "            else:\n",
    "                if  outdict.get(key)==[value] or outdict.get(key) == None:\n",
    "                    outdict[key] = [value]\n",
    "                else:\n",
    "                    outdict[key].append(value)\n",
    "                        \n",
    "    return outdict\n",
    "            \n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "12d0ffd9113349bd75b9f8dafe2a2bbd",
     "grade": true,
     "grade_id": "cell-952e3bdcb65b9a03",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "data = read_vcf_file('two_variants.vcf')\n",
    "info_field = extract_info_field(data)\n",
    "expected_results = {'AC': ['1', '6'], 'AF': ['0.167', '1'], 'AN': ['6'], 'BaseQRankSum': ['-2.542'], 'ClippingRankSum': ['0'], 'DP': ['180', '185'], 'ExcessHet': ['3.0103'], 'FS': ['0'], 'MLEAC': ['1', '6'], 'MLEAF': ['0.167', '1'], 'MQ': ['52.77', '59.95'], 'MQRankSum': ['-4.631'], 'QD': ['0.39', '27.3'], 'ReadPosRankSum': ['1.45'], 'SOR': ['0.758', '1.042'], 'VQSLOD': ['-8.209', '4.51'], 'culprit': ['MQ', 'QD'], 'ANNOVAR_DATE': ['2018-04-16'], 'Func.refGene': ['intergenic'], 'Gene.refGene': ['IL2,IL21', 'LIN7A,ACSS3'], 'GeneDetail.refGene': ['dist=38536,dist=117597', 'dist=112857,dist=27258'], 'Func.ensGene': ['intergenic', 'intronic'], 'Gene.ensGene': ['ENSG00000109471,ENSG00000138684', 'ENSG00000111058'], 'GeneDetail.ensGene': ['dist=38306,dist=117597'], 'cytoBand': ['4q27', '12q21.31'], 'CADD13_RawScore': ['0.015973', '-0.101662'], 'CADD13_PHRED': ['2.741', '1.718'], 'Eigen': ['-0.3239', '-0.4730'], 'gnomAD_genome_ALL': ['0.0003', '0.9988'], 'gnomAD_genome_AFR': ['0.0001', '0.9961'], 'gnomAD_genome_AMR': ['0', '1'], 'gnomAD_genome_ASJ': ['0', '1'], 'gnomAD_genome_EAS': ['0.0007', '1'], 'gnomAD_genome_FIN': ['0.0009', '1'], 'gnomAD_genome_NFE': ['0.0002', '0.9998'], 'gnomAD_genome_OTH': ['0.0011', '1'], '1000g2015aug_all': ['0.9998'], '1000g2015aug_afr': ['0.9992'], '1000g2015aug_amr': ['1'], '1000g2015aug_eur': ['1'], '1000g2015aug_sas': ['1'], 'snp138NonFlagged': ['rs10746177'], 'avsnp150': ['rs10746177'], 'Kaviar_AF': ['0.953358'], 'Kaviar_AC': ['24814'], 'Kaviar_AN': ['26028']}\n",
    "\n",
    "\n",
    "\n",
    "results = create_dictionary_of_info_field_values(info_field)\n",
    "assert expected_results == results\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2c0126eff825572ef5b0b92d87ef5642",
     "grade": false,
     "grade_id": "cell-62985edcd7fba77a",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 8 (5 points)\n",
    "\n",
    "Write a function whose input is the output from `create_dictionary_of_info_field_values` and uses the previously written function `determine_data_type_of_list` to determine the data type of each of the info fields. The output is a dictionary whose keys are the name of the info fields and values are the data type.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9799cbbd231f27bd0893093fded9358f",
     "grade": false,
     "grade_id": "cell-e307233a51135767",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def determine_data_type_of_info_fields(data):\n",
    "    \"\"\"\n",
    "    See desription in part 8\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "    return {value:determine_data_type_of_list(data[value]) for value in data}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "de5f50348a3de73ff8d2b30706eeb960",
     "grade": true,
     "grade_id": "cell-5327bd6c0ba8ab32",
     "locked": true,
     "points": 5,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "data = {'key1': ['1', '2', '3'], 'key2':['1.1', '2.2', '3.3'], 'key3': ['1.1', '2', '3.3'], 'key4': ['1.1', '234String', '3.3']}\n",
    "expected_results = {'key1': int, 'key2': float, 'key3': float, 'key4': str}\n",
    "\n",
    "assert determine_data_type_of_info_fields(data) == expected_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "98af1728515ce6700e4714d46617b40a",
     "grade": false,
     "grade_id": "cell-ff1699590f0852bd",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 9  (No points)\n",
    "\n",
    "Write a function whose first input is the data from  `read_vcf_file` and the second input is the output from `determine_data_type_of_info_fields`. The function converts the info field into a dictionary and uses the data types that you determined to cast each field into the correct data type. Make sure to convert the `POS` to int and `QUAL` to float. The output will look something like this (I have removed most of the fields):\n",
    "\n",
    "The output will look something like this \n",
    "\n",
    "```python\n",
    "{\n",
    "        \"ALT\": \"G\",\n",
    "        \"CHROM\": \"4\",\n",
    "        \"FILTER\": \"PASS\",\n",
    "        \"ID\": \".\",\n",
    "        \"INFO\": {\n",
    "           \n",
    "            \"Gene.ensGene\": \"ENSG00000109471,ENSG00000138684\",\n",
    "            \"Gene.refGene\": \"IL2,IL21\",\n",
    "            \"GeneDetail.ensGene\": \"dist=38306,dist=117597\",\n",
    "            \"GeneDetail.refGene\": \"dist=38536,dist=117597\"\n",
    "        },\n",
    "        \"POS\": 123416186,\n",
    "        \"QUAL\" :23.25,\n",
    "        \"REF\": \"A\",\n",
    "        \"SAMPLE\": {\n",
    "            \"XG102\": {\n",
    "                \"AD\": \"51,8\",\n",
    "                \"DP\": \"59\",\n",
    "                \"GQ\": \"32\",\n",
    "                \"GT\": \"0/1\",\n",
    "                \"PL\": \"32,0,1388\"\n",
    "            }\n",
    "    }\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ee76702fd1a43afe2cc3df04062dc35b",
     "grade": false,
     "grade_id": "cell-9dd8c86420d606c3",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "data= read_vcf_file(\"two_variants.vcf\")\n",
    "info_field= extract_info_field(data)\n",
    "info_list = create_dictionary_of_info_field_values(info_field)\n",
    "info_field_data_type = determine_data_type_of_info_fields(info_list)\n",
    "\n",
    "\n",
    "def format_data(data, info_field_data_type):\n",
    "    # YOUR CODE HERE\n",
    "    for ele in data:\n",
    "        ele['QUAL']=float(ele['QUAL'])\n",
    "        ele['POS'] = int(ele['POS'])\n",
    "        ele['INFO'] = create_dictionary_of_info_field_values([ele['INFO']])\n",
    "        for key, value in  ele['INFO'].items():\n",
    "            ele['INFO'][key] = list(map(info_field_data_type[key], value))[0]\n",
    "\n",
    "    return data\n",
    "    \n",
    "\n",
    "# format_data(data, info_field_data_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e6b3e62cd0fbff8dd5c06e9662946e29",
     "grade": false,
     "grade_id": "cell-644bedf01e77becd",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 10 (No Points)\n",
    "\n",
    "Write a function whose inputs are a Python dictionary and filename. The function will saves the dictionary as a json file using the filename given. Use the json library. Use these options to correctly format your JSON -- `sort_keys=True, indent=4, separators=(',', ': '), ensure_ascii=False`. \n",
    "Use this function to save your parsed data as a json file. \n",
    "\n",
    "USE THE FOLLOWING FILENAME TO SAVE YOUR FILE: `lab1.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8df429edc18d06a9eab29bef48abc282",
     "grade": false,
     "grade_id": "cell-e350d04329de6ab7",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def save_data_as_json(data, filename):\n",
    "    # YOUR CODE HERE\n",
    "    \n",
    "    import json\n",
    "    \n",
    "    with open(filename, \"w\") as file: \n",
    "        json.dump(data, file, sort_keys=True, indent=4, separators= (',',': '), ensure_ascii= False)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c0e408b3c720a560bc889cbc8134418a",
     "grade": false,
     "grade_id": "cell-35ab52204d553858",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 11 (10 points)\n",
    "\n",
    "Write a function whose input is a filename for a json file. The function should use the filename to read the JSON file in which you saved your final parsed data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ab6bf7a67dc1548060b9efa228c82728",
     "grade": false,
     "grade_id": "cell-e4b0e898f94ee719",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def load_data_from_json(filename):\n",
    "    # YOUR CODE HERE\n",
    "    import json\n",
    "    with open(filename, 'r') as f:\n",
    "        load_data=json.load(f)\n",
    "    return load_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4e00356929605884510397fd1e1ee6fb",
     "grade": true,
     "grade_id": "cell-47a7658311edecb1",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_save_data = {'key1': ['1', '2', '3'], 'key2':['1.1', '2.2', '3.3'], 'key3': ['1.1', '2', '3.3'], 'key4': ['1.1', '234String', '3.3']}\n",
    "save_data_as_json(test_save_data, 'test_save.json')\n",
    "results = load_data_from_json('test_save.json')\n",
    "\n",
    "assert results == test_save_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b850526abce04cdc04d692e3b75c2960",
     "grade": false,
     "grade_id": "cell-c8e64adf70d98a4c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Part 12 (30 points)\n",
    "\n",
    "Write a function whose inputs are `CHROM`, `REF`, `ALT`, `POS`, and `filename`.  Using these inputs, the function should load a JSON file using the given filename and return a list of variants that match \n",
    "the given CHROM, REF, ALT, and POS. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2cde46ae76ec238ae65dc537c0356892",
     "grade": false,
     "grade_id": "cell-f3ca58c8b73040f4",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "def find_variant(CHROM, REF, ALT, POS, filename):\n",
    "    # YOUR CODE HERE\n",
    "    load_data=load_data_from_json(filename)\n",
    "    list_var = [i for i in load_data if (i['CHROM'] == CHROM and i['REF'] == REF and i['ALT']==ALT and i['POS']==POS)]\n",
    "    return list_var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5f63b995745e7d00c993cc850a01cd34",
     "grade": true,
     "grade_id": "cell-d86c013b6117c7c7",
     "locked": true,
     "points": 30,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "filename = 'lab1_data.vcf' \n",
    "data = read_vcf_file(filename) # read vcf file \n",
    "info_field_data = extract_info_field(data) # extract all the info fields\n",
    "info_field_list = create_dictionary_of_info_field_values(info_field_data) # create dictionary from info fields\n",
    "info_field_data_type = determine_data_type_of_info_fields(info_field_list) # Determine data type of each info field\n",
    "data = format_data(data, info_field_data_type) # format the data variable -- from data = read_vcf_file(filename)\n",
    "save_data_as_json(data, 'lab1.json') # save the formatted data\n",
    "data_loaded = load_data_from_json('lab1.json') # load saved data \n",
    "\n",
    "# Now I will fetch three variants using your find_variant function and check that they match my results\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'ALT': 'G',\n",
       "  'CHROM': '13',\n",
       "  'FILTER': 'PASS',\n",
       "  'ID': 'rs4421887',\n",
       "  'INFO': {'1000g2015aug_afr': 0.8313,\n",
       "   '1000g2015aug_all': 0.952676,\n",
       "   '1000g2015aug_amr': 0.9841,\n",
       "   '1000g2015aug_eur': 1.0,\n",
       "   '1000g2015aug_sas': 0.9969,\n",
       "   'AC': 6,\n",
       "   'AF': 1.0,\n",
       "   'AN': 6,\n",
       "   'ANNOVAR_DATE': '2018-04-16',\n",
       "   'BaseQRankSum': 1.77,\n",
       "   'CADD13_PHRED': 3.712,\n",
       "   'CADD13_RawScore': 0.109684,\n",
       "   'ClippingRankSum': 0,\n",
       "   'DP': 176,\n",
       "   'Eigen': -0.7945,\n",
       "   'ExcessHet': 3.0103,\n",
       "   'FS': 0.0,\n",
       "   'Func.ensGene': 'intergenic',\n",
       "   'Func.refGene': 'intergenic',\n",
       "   'Gene.ensGene': 'ENSG00000264387,ENSG00000228611',\n",
       "   'Gene.refGene': 'MIR5007,PRR20D',\n",
       "   'GeneDetail.ensGene': 'dist=543620,dist=281031',\n",
       "   'GeneDetail.refGene': 'dist=543620,dist=1422749',\n",
       "   'Kaviar_AC': 3,\n",
       "   'Kaviar_AF': 0.0001153,\n",
       "   'Kaviar_AN': 26028,\n",
       "   'MLEAC': 6,\n",
       "   'MLEAF': 1.0,\n",
       "   'MQ': 60.0,\n",
       "   'MQRankSum': 0.0,\n",
       "   'QD': 28.95,\n",
       "   'ReadPosRankSum': -0.338,\n",
       "   'SOR': 0.257,\n",
       "   'VQSLOD': 22.53,\n",
       "   'avsnp150': 'rs4421887',\n",
       "   'culprit': 'MQ',\n",
       "   'cytoBand': '13q21.1',\n",
       "   'gnomAD_genome_AFR': 0.8475,\n",
       "   'gnomAD_genome_ALL': 0.9546,\n",
       "   'gnomAD_genome_AMR': 0.9843,\n",
       "   'gnomAD_genome_ASJ': 0.9834,\n",
       "   'gnomAD_genome_EAS': 1.0,\n",
       "   'gnomAD_genome_FIN': 0.9951,\n",
       "   'gnomAD_genome_NFE': 0.9987,\n",
       "   'gnomAD_genome_OTH': 0.9887,\n",
       "   'snp138NonFlagged': 'rs4421887'},\n",
       "  'POS': 56292303,\n",
       "  'QUAL': 5037.69,\n",
       "  'REF': 'T',\n",
       "  'SAMPLE': {'XG102': {'AD': '0,62',\n",
       "    'DP': '62',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1796,185,0'},\n",
       "   'XG103': {'AD': '0,41',\n",
       "    'DP': '41',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1218,122,0'},\n",
       "   'XG104': {'AD': '1,70',\n",
       "    'DP': '71',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '2037,202,0'},\n",
       "   'XG202': {'AD': '0,62',\n",
       "    'DP': '62',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1796,185,0'},\n",
       "   'XG203': {'AD': '0,41',\n",
       "    'DP': '41',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1218,122,0'},\n",
       "   'XG204': {'AD': '1,70',\n",
       "    'DP': '71',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '2037,202,0'},\n",
       "   'XG302': {'AD': '0,62',\n",
       "    'DP': '62',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1796,185,0'},\n",
       "   'XG303': {'AD': '0,41',\n",
       "    'DP': '41',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1218,122,0'},\n",
       "   'XG304': {'AD': '1,70',\n",
       "    'DP': '71',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '2037,202,0'},\n",
       "   'XG402': {'AD': '0,62',\n",
       "    'DP': '62',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1796,185,0'},\n",
       "   'XG403': {'AD': '0,41',\n",
       "    'DP': '41',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '1218,122,0'},\n",
       "   'XG404': {'AD': '1,70',\n",
       "    'DP': '71',\n",
       "    'GQ': '99',\n",
       "    'GT': '1/1',\n",
       "    'PL': '2037,202,0'}}}]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "find_variant(\"13\", \"T\", \"G\", 56292303, 'lab1.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}