Machine Learning

My machine learning code written by Python.

1. Environment Setup

• (1) Install Python 3.5 at Windows10.
• (2) Install IPython 4.0.3.
• (3) Install machine learning packages installer Anaconda.
• (4) Run IPython and access "http://127.0.0.1:8888" at browser.

> ipython notebook

2. ML libs/packages

2.1 numpy

• Matrix operations: add, subtraction, inverse
• linspace
• zeros
• meshgrid
• sin/cos/pi
• random
• np.where to find index
• batch replace array values
• np.split
• np.arange
• multi-index for 1d array
• polynomial fit

2.2 matplotlib

• draw line
• draw image
• draw 2 lists to bar
• draw bar with x
• draw image without axes
• figure resize
• plot 2 lines at one fiture
• plot point and line at one fiture
• 3D plot with pca
• plot 2 lines with different line width
• save to png image
• plot x/y/title label with font_size
• sub-plot
• set x/y axes limits
• add legend
• plot multi-lines with labels for each line
• plot confusion matrix

2.3 scipy

• leastsq
• load matlab mat file
• sparse matrix basic
• gaussian filter
• get data distribution probability
• cdist euclidean
• downsample/down-sample data
• optimize with constraints equal
• load matlab mat file then fft

2.4 pandas

• Series basic for pandas
• DataFrame basic for pandas
• create DataFrame from list of data
• read csv and iterator row by row
• read excel and iterator row by row
• replace/fill all nan
• parse csv with special seperator
• read csv without header
• remove missing values row/column
• null/missing values
• drop columns contains only zero
• data merge
• data sort by count
• data sort by mean
• modify string to int for each column data
• count special column values distribution
• write excel to multiple sheets
• merge two df: hstack
• merge two df: vstack

2.5 seaborn

• hotmap for confusion matrix
• seaborn clustermap
• catplot

3. ML algorithms

Samples should be opened by ipython.

3.1 Supervised Learning

3.1.1 Classification

Linear Model

• Linear Discriminant Analysis

Decision Tree

• sklearn DecisionTreeClassifier
• digit recognition by kaggle data(MNIST)
• shannon entropy calculation

Random Forest

• Random Forest result
• Random Forest predict probability

SVM

• SVM with diffirent kernels

Neural Network

• Basic Three Layers Network

Gradient Boosting for classification

• GradientBoostingClassifier

CNN (Deep Learning)

• CNN for kaggle MNIST
• Create diffirent CNN structure
• CNN Visualization

XGBoost

• Basic useage of XGB
• XGB for IRIS dataset
• XGB for kaggle digit recognition

DBN

• Install DBN lib at win
• DBN code example for python

RNN

• RNN for MNIST

DCNN

• Deconvolution network

3.1.2 Regression

Linear Regression

• sklearn LinearRegression and linear function parameters
• difference between np.linalg.lstsq and linear_model.LinearRegression
• regression by CNN

3.2 Un-Supervised Learning

3.2.1 HMM

• HMM basic
• HMM application
  • https://www.cnblogs.com/wangzming/p/7512607.html
  • https://zhuanlan.zhihu.com/p/20688517

3.2.1 Cluster

k-Nearest Neighbor

• self-written kNN
• kNN by lib sklearn
• kNN cluster example

DBSCAN

• dbscan with precomputed metric

3.2.2 PCA

• pca algorithm

3.3 Model evaluation

• KFold and StratifiedKFold
• accuracy score
• confusion matrix
• P, R, F1 value
• corss validation
• MSE
• Logloss
• classification report
• feature selection by feature importance
• homogeneity & completeness & v_measure for unsupervised learning cluster evaluation metric

3.4 Model selection

• GridSearchCV
• train & test split
• Learning Curves
• select k best feature

3.5 Tensorflow

3.5.1 TF Basic

• Install Tensorflow at windows
• "hello world" for basic calculation
• basic linear model by tf-core low level api
• basic linear model training by tf-core low level api
• basic linear model by tf.estimator high level api
• basic softmax regression at mnist dataset

3.5.2 tf.estimator

DNNClassifier set batchsize and epoch

3.5.3 TF models

• DNN-200N-100N-50N-10N
• basic CNN
• CNN-120C5-MP2-200C3-MP2-100N-10N
• CNN-120C5-200C3-MP2-100N-10N

3.6 Tensorboard

3.6.1 Tensorboard by Tensorflow

• TF linearmodel Tensorboard
• TF DNN Tensorboard

3.6.2 Tensorboard by Keras

• Tensorboard with keras
• Tensorboard with keras CNN
• EMBEDDING: mnist data
• EMBEDDING: your own data

3.7 keras

3.7.1 basic

• Install keras at Windows
• Modify keras backend between theano and tf
• Keras basic neural network
• int to one-hot & one-hot to int
• get log loss each epoch
• model dump, load
• plot loss,acc history
• early stop
• adam optimization
• validation
• param of model.summary
• keras fixed model random seed

3.7.2 models

• Keras 1D-CNN
• Keras 2D-CNN
• RNN by keras
• RESNET-50 by keras
• LSTM
• LSTM+Attention
• complex attention models
• BI-LSTM

3.7.3 complex

• get middle layer output

3.8 theano

• Install theano at win

3.9 Incremental learning

• Incremental learning by SGDClassifier partial_fit

3.10 outlier detection

• IsolationForest

3.11 sklearn

• model persistence(dump & load)
• euclidean distance
• boston house-prices dataset for regression
• decision tree ensembel with adaboost
• kmeans cluster
• mini batch kmeans cluster
• random forest grid search
• logloss
• GMM cluster
• TSNE/T-SNE plot
• F1-P-R calculation for multi-class output by sklearn
• Isomap plot

3.12 jupyter

• timeit/time performance
• jupyter reload module

3.13 mxnet

3.13.1 NDArray

• array create/reshape/index
• array concat
• array operation
• basic linear regression
• linear regression by gluon

3.13.2 Basic

• get dataset fashion mnist
• softmax regression by gloun
• plot activation relu/sigmoid with gradient
• mlp and model parameter access
• CNN LeNet
• dataset jaychou
• generate jay text by rnn
• compute gradient

4. Feature Engineering

4.1 Working With Text Data

• Extract 3 types of text feature: bag of words, TF, TF-IDF

4.2 String Hash

• Extract string hash value by FeatureHasher
• HashingVectorizer

4.3 Normalization

• MinMaxScaler
• StandardScaler

4.4 Feature selection

• feature selection by chi-square/chi2
• feature selection by recursive feature elimination/rfe

4.5 imbalance data process

• RandomOverSampler for Imbalance Data

4.6 missing values

• count null/missing_value item for each column
• drop the column if it contains one missing_value
• drop the column if the missing_value count > 0.5*full_count
• check the data with null data > 20%
• insert mean value for missing values

5. Image process

5.1 OpenCV

5.1.1 OpenCV Python

Installation

• Install opencv-python at Win-64 by conda py2 env
• Install opencv-python at Win-64 with conda(python3)
• Install opencv-python at Win by py2

Basic

• Image Read/Cut/Display
• Image Read/Cut/Display by Jupyter
• save image/imwrite
• cvtColor
• rgb to hsv
• hsv image

Preprocess

• Image smooth by blurring, Gaussian, median and bilateral filtering
• modify pixel color
• modify pixel color fast
• flattern rgb
• split rgb
• find connected component
• convert white color to green
• blur detection

Projects

• defect detection

5.1.2 OpenCV CPP

opencv 2.4.9 & windows-7

• Init opencv cpp project

5.1.3 Features & Matcher

• SIFT
• Images matching with average distance output
• Images matching with most-similar-average distance output
• Different size images match by ORB feature and BFMatcher
• Different direction images match by ORB feature and BFMatcher

5.1.4 Geometric Transformations

• resize
• Perspective Transformation
• Affine Transformation

5.2 Useful features

• Image smooth, shift, rotate, zoom by scipy.ndimage
• image enhancement for ocr
• Keep gray image pixel value range [0,255] when room/shift/rotate by setting order=1. commit-220ac520a0d008e74165fe3aace42b93844aedde
• template match

5.3 OCR

• tesseract basic feature
• image enhancement for tesseract

5.4 3D graph process

• stl file parse

5.5 face_recognition

• install face_recognition at ubuntu
• face location
• find facial features
• face recognition

6. Distributed ML

6.1 Spark

6.1.1 Spark Cluster Deployment

• Deployment cluster environment

6.1.2 Jupyter Integrate with Spark

• get sc at jupyter

6.1.3 Spark One Node

• Run at One Node

6.1.4 Spark Cluster

• Run at Cluster
• Distributed programming demo for feature extraction
• Get running slave current directory and IP address
• Submit multiple .py files which have dependency

6.1.5 Mlib

• recommandation system
• kmeans at cluster

6.1.6 spark at aws emr

• steps to create/run spark code at emr

6.2 Hadoop

6.2.1 Environment Setup

• Standalone Environment Setup
• Single Node Cluster Environment Setup

6.2.2 Run Hadoop self-example at Standalone mode

• hadoop example

6.2.3 HDFS

• HDFS basic operation at single node cluster

6.2.4 mrjob

• word count for mrjob map reduce basic

7. NLP

7.1 nltk

• nltk basic usage such as tokenize/tagging
• nltk tag meaning en & zh
• identify named entities basic
• nltk load and process built-in data/dataset/corpus
• nltk load and process external data/dataset/corpus
• normalizing text by stemmer
• normalizing text by lemmatization
• nltk.Text.similar
• more details about nltk.Text.similar
• sentiment analysis
• offline isntall nltk_data

7.2 word2vec

• word2vec basic
• gensim word2vec basic

7.3 Others

• n-gram
• preprocess for en text
• Bags of words feature
• TF, TF-IDF feature
• basic text clean

7.4 keyword & abstract extraction

• from Chinese text

7.5 gensim

• get dict for text
• dict filter
• doc2bow
• tfidf/tf-idf

7.6 AllenNLP

• install allennlp
• NER by biLSTM with CRF layer and ELMo embeddingstrained on the CoNLL-2003 NER dataset 2018
• fact predict by trained decomposable attention model 2017
• question answer from passage by BiDAF 2017

7.7 Spacy

• NER by spacy
• Tokenization with Part-of-speech (POS) Tagging

7.8 gensim

• Latent Dirichlet Allocation，LDA Model
• doc to BOW features

7.9 keras-bert

• Get text embeddings by pretrained BERT model

7.10 wordcloud

• plot wordcloud basic

7.11 wordnet

• wordnet basic and environment setup

7.12 NER

• BiLSTM-CRF-NER

7.13 LDA

• LDA of sklearn

8. Audio

8.1 pyAudioAnalysis

• basic intro
• frequency and data extraction from wav file
• audio feature extraction
• extract same length features for different audio

8.2 signal data augmentation

• add gaussian noise

9. GPU

• GPU environment setup
• GPU vs CPU basic
• check GPU hardware status
• check if keras/tensorflow run on GPU
• 矩池云GPU

10. Video

• read mp4 by opencv py3
• get video fps by opencv py3

11. recommandation system

11.1 surprise

• load local dataset by pandas
• load local dataset by surprise
• load buidin dataset by surprise
• ml-100k dataset movie name to id, or id to movie
• basic/official example
• basic algorithm and testing for local data
• predict rating for test dataset
• user based collaborative filtering and predict one item rating
• item based collaborative filtering and predict one item rating
• top-n recommandation for buildin data
• top-n recommandation for local data
• SVD collaborative filtering

12. other machine learning related algorithm

• correlation analysis
• association analysis
• jaccard similarity
• 2 strings/list similarity (unequal length)
• peak detect
• panel data analysis: fixed/random effects model by linearmodels
• panel data analysis: mixed effects model by statsmodels

13. Small project/features

• Human face image completion
• Human detection by opencv hug & svm
• gibberish text detection
• poker game AI
• learn deeper for pklearn
• Industrial Control System ICS Cyber Attack Dataset Classification
• red wine quelity classification
• chemistry dataset tox21 test by chainer-chemistry
• spam email/sms classification
• jaychou text generate by lstm
• SI/SIR model by scipy

14. related tools

14.1 conda

• create env, delete env
• export env to yml, create env from yml

15. front-end AI

15.1 JS access camera

• js open camera and take photo

15.2 face-api.js

• run face-api.js examples

16. D3.js

• tutorials and referencs
• js, select, svg basics
• mouse over multiple lines chart

17. LOFO

• get feature importance by LOFO and FastLOFO

18. vaderSentiment

• sentiment analysis by the lib

19. offline deployment

1. build sklearn model and py files to elf

20. keras-nlp

*keras-nlp dataset and classifier basic demo