consolidation

Scraper/Notes/errors.md

@@ -0,0 +1,31 @@
+Crashing randomly - catch & restart?
+---------
+####### SNPedia import for rs25531
+Traceback (most recent call last):
+  File "SNPedia/DataScraper.py", line 265, in <module>
+    dfCrawl = SNPCrawl(rsids=rsid)
+  File "SNPedia/DataScraper.py", line 45, in __init__
+    self.initcrawl(rsids)
+  File "SNPedia/DataScraper.py", line 54, in initcrawl
+    self.grabTable(rsid) # imports
+  File "SNPedia/DataScraper.py", line 81, in grabTable
+    response = urllib.request.urlopen(url)
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
+    return opener.open(url, data, timeout)
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
+    response = self._open(req, data)
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
+    '_open', req)
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
+    result = func(*args)
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
+    context=self._context, check_hostname=self._check_hostname)
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1320, in do_open
+    r = h.getresponse()
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1321, in getresponse
+    response.begin()
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 296, in begin
+    version, status, reason = self._read_status()
+  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 265, in _read_status
+    raise RemoteDisconnected("Remote end closed connection without"
+http.client.RemoteDisconnected: Remote end closed connection without response

Scraper/Notes/notes.md

@@ -0,0 +1 @@
+rs7412 is listed as pathogenic - however You need both to have APOE4. If your rs429358 is T then you are APOE3 (normal). And when Promethease finds APOE4 it highlights it for you right up front.

Scraper/README.md

@@ -0,0 +1,61 @@
+# Phenotype.dev
+
+![Example of output](https://github.com/glasgowm148/Phenotype/blob/master/images/phenotype.png)
+## Description
+Phenotype is an open source web application that allows users to gather the information they need to make sense of their own genome without needing to rely on outside services with unknown privacy policies. OS Genome's goal is to crawl various sources and give meaning to an individual's genome. It creates a Responsive Grid of the user's specific genome. This allows for everything from filtering to excel exporting. All of which using Flask, Kendo, and Python programming.
+
+## Disclaimer
+Raw Data coming from Genetic tests done by Direct To Consumer companies such as 23andMe and Ancestry.com were found to have a false positive rate of 40% for genes with clinical significance in a March 2018 study [*False-positive results released by direct-to-consumer genetic tests highlight the importance of clinical confirmation testing for appropriate patient care*](https://www.nature.com/articles/gim201838). For this reason, it's important to confirm any at risk clinical SNPs with your doctor who can provide genetic tests and send them to a clinical laboratory.
+
+With genome analysis, sometimes sites will report on the negative strand - meaning that if a rare gene pops up which is pathogenic for some thing you've never heard of - this is likely the case. If you had such a gene it's likely 23andme would've notified you - This tool is more aimed at created custom phenotypes which combine several low-risk, benign or carrier genes - to demonstrate phenotypes where diseases could be manifesting as the cause of a combination of several genes - rather than one SNP.
+
+
+# Log
+* Done
+    * Enabled cross-platform support
+    * snps_of_interest.txt can be loaded with SNPs to specifically query against your dataset
+    * one_thousand_and_you.txt is ~1000 rsids related to health, drug metabolism, hormones, autoimmune, eds, asd/adhd, pots/mcas, etc..
+    * dbSNP lookup functionality added
+    * CSV Export working
+    * Crawl dbSNP - added to import risk allele / freq
+    * Crawl dbSNP - import clinical significance
+    * Highlighting based on Risk Allele (Still some bugs)
+    * Support for AncestryDNA
+    * GeneticGenie + NutriHacker SNPs loaded
+    * Export to PDF
+* In Progress
+  * Tidying up HTML/CSS/Tabular 
+  * Filter by mutations only
+* To Do
+  * +/- strand orientation check
+  * 23andme i -> rsid
+  * User Login with upload
+    * Custom reports / phenotypes
+    * 'Save to ->' 
+    * Email notifications when new studies are published with specified rsids
+  * Other sources to utilise
+    * Reload DB Data
+    * Genomix source
+    * SNPEdia_Scraper
+    * genome_report
+
+## Installation:
+
+
+0.
+```
+## Ensure you are using Python3
+pip install -r requirements.txt
+
+## Starts scraping (This will take several hours but exports periodically)
+python3 SNPedia/DataScraper.py -f [Absolute path of your downloaded raw DNA data]
+
+## Once it's exported 5 results - In a new terminal window
+python3 SNPedia/SnpApi.py
+
+## Access the Local Server
+http://127.0.0.1:5000
+```
+
+This Git is based on [OSGenome](osgenome/SNPedia at master · mentatpsi/OSGenome), [SNPApi](https://github.com/leaena/snp-api), and [Snappy](https://github.com/zhaofengli/snappy)
+