0_Sklearn-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "68c18f9f-115f-490e-b81f-1b3794c428d0"
    }
   },
   "source": [
    "# Intro to Machine Learning with scikit-learn\n",
    "\n",
    "## An oft-quoted definition\n",
    "> A computer program is said to learn from experience E with respect to some task T and some performance measure P if its performance on T, as measured by P, improves with experience E. \n",
    "\n",
    "> ( Mitchell 1997)\n",
    "\n",
    "Example Experiences: Supervised and Unsupervised learning\n",
    "\n",
    "Example Tasks: Classification, Regression, Clustering\n",
    "\n",
    "Example Performance: Accuracy, F1-Score, RMSE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Things you can do with scikit-learn\n",
    "[![ml-map](src/img/ml_map.png)](http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html)\n",
    "\n",
    "For a full list, check out the [User Guide](http://scikit-learn.org/stable/user_guide.html).\n",
    "\n",
    "### Example algorithms:\n",
    "* Linear regression (trained with gradient descent), logistic regression\n",
    "* KNN\n",
    "* SVMs\n",
    "* Ensemble decision tree methods: Random Forests, Gradient boosted decision trees\n",
    "    * boosting vs bagging\n",
    "    * see the docs: http://scikit-learn.org/stable/modules/ensemble.html\n",
    "* Naive Bayes (Gaussian, Multinomial)\n",
    "\n",
    "## Further motivation\n",
    "![algo-comp](src/img/Model_comparison.jpg)\n",
    "_Olson 2017 https://arxiv.org/abs/1708.05070_\n",
    "\n",
    "### In Neuroscience\n",
    "#### From Konrad Kording's group\n",
    "* Encoding: [Modern machine learning outperforms GLMs at predicting spikes](https://www.biorxiv.org/content/early/2017/10/04/111450), with [code](https://github.com/KordingLab/spykesML)\n",
    "![Fig4](src/img/ML_GLM_Fig4.png)\n",
    "* Decoding: [Machine learning for neural decoding](https://arxiv.org/abs/1708.00909), with [code](https://github.com/KordingLab/Neural_Decoding)\n",
    "\n",
    "#### Neural Networks\n",
    "* Also an emergence of RNN (LFADS, Sussillo) and CNN papers (Yamins, Ecker) to help explain neural responses\n",
    "\n",
    "### Examples of ensemble methods in biology\n",
    "* ensemble methods are not only useful for Kaggle competitions but also in biology\n",
    "* bagging, boosting, and stacking\n",
    "* \"Wisdom of crowds for robust gene network inference\"\n",
    "     > Reconstructing gene regulatory networks from high-throughput data is a long-standing challenge. Through the Dialogue on Reverse Engineering Assessment and Methods (DREAM) project, we performed a comprehensive blind assessment of over 30 network inference methods on Escherichia coli, Staphylococcus aureus, Saccharomyces cerevisiae and in silico microarray data. We characterize the performance, data requirements and inherent biases of different inference approaches, and we provide guidelines for algorithm application and development. We observed that no single inference method performs optimally across all data sets. In contrast, integration of predictions from multiple inference methods shows robust and high performance across diverse data sets. We thereby constructed high-confidence networks for E. coli and S. aureus, each comprising ∼1,700 transcriptional interactions at a precision of ∼50%. We experimentally tested 53 previously unobserved regulatory interactions in E. coli, of which 23 (43%) were supported. Our results establish community-based methods as a powerful and robust tool for the inference of transcriptional gene regulatory networks.\n",
    "\n",
    "    > from https://www.nature.com/articles/nmeth.2016\n",
    "* Other DREAM competitions, e.g. Keller et al. 2017 [Predicting human olfactory perception from chemical features of odor molecules](http://science.sciencemag.org/content/355/6327/820.long)\n",
    "\n",
    "### Another note\n",
    "#### Machine learning is not a substitute for proper experiment design\n",
    "* What do you think are the differences between these two experiments?\n",
    "<img src=\"src/img/carey-eyeblink-conditioning.png\" width=\"600\">\n",
    "<img src=\"src/img/carey-control-for-speed.png\" width=\"300\">\n",
    "> from _https://www.biorxiv.org/content/early/2017/01/11/099721_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The whole process\n",
    "\n",
    "1. Preprocess your data and feature transformation / engineering\n",
    "    * [sklearn.preprocessing](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing)\n",
    "    * remove outliers and impute NaNs\n",
    "    * unit mean and variance\n",
    "    * convert categorial variables to numerical (one-hot encondings to binary) \n",
    "    * log-transform\n",
    "    * expanded or reduced bases, see [sklearn.decomposition](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition)\n",
    "    * polynomial, interaction terms\n",
    "    * also see [sklearn.feature_selection](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_selection)\n",
    "3. Divide data into training, validation, and test sets\n",
    "    * splitter classes of [sklearn.model_selection](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection)\n",
    "    * KFold, leave one out\n",
    "4. Tune model and regularization parameters\n",
    "    * search over relevant models and hyperparameters\n",
    "    * hyper-parameter-optimizers in [sklearn.model_selection](http://scikit-learn.org/stable/modules/classes.html#hyper-parameter-optimizers)\n",
    "        * GridSearchCV and RandomizedSearchCV \n",
    "        * Randomly sampling parameters is generally better![rand](src/img/Random_opt.png)\n",
    "        * See Bergstra and Bengio 2012 [Random Search for Hyper-Parameter Optimization](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf)\n",
    "    * Fit model to training data, using validation set to assess model [score](http://scikit-learn.org/stable/modules/model_evaluation.html)\n",
    "    * `model.fit()`\n",
    "5. Evaluate on held-out test data\n",
    "    * [sklearn.metrics](http://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics)\n",
    "    * `scorer(ytrue, model.predict())`\n",
    "    * evaluate predictions: confusion matrix, residuals\n",
    "    * examine best model: coefficients, feature_importance\n",
    "6. Repeat (in the case of nested cross-validation)\n",
    "    * test new models, add or remove features\n",
    "\n",
    "## Common API\n",
    "* initialize, fit, predict, score:\n",
    "```\n",
    "# Class to extend Sklearn models\n",
    "class SklearnHelper(object):\n",
    "    def __init__(self, clf, scorer, seed=42, params={}):\n",
    "        params['random_state'] = seed\n",
    "        self.clf = clf(**params)\n",
    "        self.scorer = scorer\n",
    "\n",
    "    def train(self, x_train, y_train):\n",
    "        self.clf.fit(x_train, y_train)\n",
    "\n",
    "    def predict(self, x_test):\n",
    "        return self.clf.predict(x_test)\n",
    "    \n",
    "    def score(self, x_test, y_true):\n",
    "        return self.scorer(y_true, self.predict(x_test))\n",
    "# Example usage:\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "model = SklearnHelper(LogisticRegression, accuracy_score)\n",
    "model.train(X_train, Y_train)\n",
    "model.score(X_test, Y_true)\n",
    "```\n",
    "* For an overview of the API see [API design for machine learning software: experiences from the scikit-learn project](https://arxiv.org/pdf/1309.0238.pdf), or check the docs for the [full API](http://scikit-learn.org/stable/modules/classes.html#)\n",
    "\n",
    "* Integrates well with other packages, eg. scipy sparse matrics (CSR, CSC), pandas DataFrames, visualization with matplotlib and seaborn\n",
    "\n",
    "### Bias-Variance tradeoff\n",
    "![bias-var](src/img/bias-variance.png)\n",
    "_from http://www.brnt.eu/phd/node14.html_\n",
    "\n",
    "Also see the example chapter from Jake VanderPlas [here](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Other Resources\n",
    "\n",
    "## Machine learning (with scikit-learn)\n",
    "Besides checking out the tutorials and examples that are part of scikit-learn's documentation I'd recommend:\n",
    "* Jake VanderPlas's book, [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/index.html#5.-Machine-Learning). All of the notebooks are also available through [Binder](https://mybinder.org/v2/gh/jakevdp/PythonDataScienceHandbook/master?filepath=notebooks%2FIndex.ipynb)\n",
    "* Browsing Kaggle [kernels](https://www.kaggle.com/kernels)\n",
    "* Nature Methods Points of Significance columns: http://mkweb.bcgsc.ca/pointsofsignificance/\n",
    "\n",
    "## Books\n",
    "* Hastie, Elements of Statistical Learning\n",
    "* Bishop, Pattern Recognition and Machine Learning\n",
    "* Murphy, Machine Learning: a Probabilistic Perspective\n",
    "\n",
    "## Hyperparameter optimzation and AutoML\n",
    "* AutoML packages: [TPOT](https://github.com/rhiever/tpot), the [AutoML](https://github.com/automl) packages like [auto-sklearn](https://github.com/automl/auto-sklearn). These packages use genetic and bayesian optimization algorithms to evaluate the \"fitness\" or relationship between hyperparameter settings and model performance to search both across spaces where the relationship is uncertain as well as to focus in the subspaces that perform well. Can optimize not only the hyperparameters but also the type of model and preprocessing steps.\n",
    "* Bayesian optimization pacakges: [hyperopt](https://github.com/hyperopt/hyperopt), [Spearmint](https://github.com/HIPS/Spearmint), or [MOE](https://github.com/Yelp/MOE)\n",
    "\n",
    "## Other ML libraries in python\n",
    "* [XGBoost](https://github.com/dmlc/xgboost) or [LightGBM](https://github.com/Microsoft/LightGBM) for gradient boosting. This packages also have Scikit-Learn Wrappers so you can use them with GridSearch and pipelines with other sklearn algorithms.\n",
    "* MLlib for Spark\n",
    "* [scikit-learn-contrib](https://github.com/scikit-learn-contrib)\n",
    "* feature selection algorithms, such as [scikit-rebate](https://github.com/EpistasisLab/scikit-rebate)\n",
    "\n",
    "## Deep learning\n",
    "* TensorFlow, PyTorch, MXNet\n",
    "\n",
    "## Datasets\n",
    "* Besides the AllenSDK, check out [CRCNS](https://crcns.org)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}