SKADI is a Deep Learning framework developed to be simple, intuitive and friendly with beginners in deep learning.
The objective is that users can use the framework and consult to better understand the functioning of a neural network. Maybe jobs with deep learning models can be difficult to apply and understand, so this project was created to help the user to understand how a neural network works to create their own deep learning model.
The framework is ready to use, you can apply it in your projects. The features to be implemented are to improve and increase the capacity of the neural network, making it capable of solving the most varied types of problems in different ways, but them dont are impediments.
Python was choiced because it is a powerful language and the most used in Artificial Intelligence field. Is a easy tool to use, dynamic and user friendly.
Numpy it's an important and powerful math library that facilitates the implementation with your vectorization, statistics methods and another powerful tools.
- learning rate
- time-based learning rate decay
- exponential learning rate decay
- staircase learning rate decay
- cost functions
- mean squared error
- mean absolute error
- root mean squared error
- root mean squared logarithmic error
- binary cross-entropy
- sigmoid cross-entropy
- negative log-likelihood
- softmax negative log-likelihood
- hinge
- huber
- kullback-leibler
- momentum
- layers
- common
- dropout
- regularization
- l1
- l2
- batch normalization
- mini-batch gradient descent
- early stopping
- save/load
- freezing
- weights initialization
- glorot normal (xavier normal)
- glorot uniform
- random normal
- random uniform
- zeros
- ones
- gradient checking
To use this framework in your work, firstly you must install the python language, the git and numpy library in your computer, do the both last with this command lines
apt install git
pip install numpy
Now go to your project folder and there execute the follow command line
git clone https://github.com/Dellonath/SKADI.git
Copy the SKADI/skadi folder beside your file.py where you will import the SKADI, like below
Now, in your python file, you need to import the framework, you can do it with this lines:
from skadi.model import Model
from skadi.layers import Layer, DropoutIt is everything, now you are ready to work!!
For certain types of problems, whether regression or classification, it is necessary to adopt different approaches and structures for the neural network to perform well. For this, the table below defines some structures for the network to be able to return good results.
| Problem | Cost Function | Last Layer Activation | Number of Neurons in last layer |
|---|---|---|---|
| Linear Regression | MSE, MAE | Linear | Number of values to predict |
| Binary Classification 1 | Binary Cross-Entropy | Sigmoid | 1 |
| Binary Classification 2 | Softmax + Neg. Log-Likelihood | Linear | 2 |
| Multiclass Classification | Softmax + Neg. Log-Likelihood | Linear | Number of categories |
Below we have some examples of use and performance of the framework, resulting in a comparative graph between loss of training and loss of validation on the left and accuracy on the right.
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.datasets import make_classification
# creating a classification problem
X_class, Y_class = make_classification(n_samples=500, n_features=4, n_classes=4, n_clusters_per_class=1)
X_class = StandardScaler().fit_transform(X_class)
Y_class = OneHotEncoder(sparse=False).fit_transform(Y_class.reshape(-1, 1))
# creating the base of the model
nn = Model(
cost_function = 'softmax_neg_log_likelihood',
learning_rate = 0.01,
momentum = .9,
batch = ('sequential', 50),
lr_decay = ('staircase', .99, 5)
)
# adding different type layers with different hyperparameters
nn.add(Layer(4)) # input layer
nn.add(Layer(16, activation_function = 'relu', regularization = ('l1', 0.01)))
nn.add(Dropout(units = 16, activation_function = 'tanh', p = 0.3, regularization = ('l2', 0.1)))
nn.add(Layer(4, activation_function = 'linear')) # out layer
# fitting the model
nn.fit(X_class,
Y_class,
epochs = 3000,
verbose = 500,
validation_rate = 0.2
)
Out:
epoch: 0/3000 loss: 1.3509106021 val_acc: 0.0
epoch: 500/3000 loss: 0.4634072936 val_acc: 0.87
epoch: 1000/3000 loss: 0.4100287291 val_acc: 0.88
epoch: 1500/3000 loss: 0.4496435452 val_acc: 0.87
epoch: 2000/3000 loss: 0.4166560359 val_acc: 0.87
epoch: 2500/3000 loss: 0.5221857908 val_acc: 0.87
epoch: 3000/3000 loss: 0.4676211429 val_acc: 0.87
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
X_class, Y_class = make_classification(n_samples=500, n_features=4, n_classes=2, n_clusters_per_class=1)
X_class = StandardScaler().fit_transform(X_class)
Y_class = Y_class.reshape(-1, 1)
nn = Model(
cost_function = 'binary_cross_entropy',
learning_rate = 1e-2,
momentum = .99,
lr_decay = ('exponential', 0.9, None)
)
nn.add(Layer(4))
nn.add(Layer(10, 'tanh'))
nn.add(Layer(1, 'sigmoid'))
nn.fit(
X_class,
Y_class,
epochs = 1000,
verbose = 500,
validation_rate = 0.3,
loss_stop = (500, 1e-4)
)
Out:
epoch: 0/1000 loss: 0.7545087989 val_acc: 0.427
epoch: 500/1000 loss: 0.0923987324 val_acc: 0.940
loss_stop activated: 914/1000 loss: 0.0677572368 val_acc: 0.973
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_circles
X_class, Y_class = make_circles(n_samples = 500, noise = .1, factor = .4)
Y_class = Y_class.reshape(-1, 1)
X_class = StandardScaler().fit_transform(X_class)
nn = Model('sigmoid_cross_entropy', learning_rate=0.5)
nn.add(Layer(2))
nn.add(Layer(10, 'tanh'))
nn.add(Dropout(10, 'tanh', .1))
nn.add(Layer(1, 'linear'))
nn.fit(X_class, Y_class, epochs = 3000, verbose = 500, validation_rate = 0.2)
Out:
epoch: 0/3000 loss: 0.7131146386 val_acc: 0.700
epoch: 500/3000 loss: 0.0123973415 val_acc: 1.000
epoch: 1000/3000 loss: 0.0031569242 val_acc: 1.000
epoch: 1500/3000 loss: 0.0026429120 val_acc: 1.000
epoch: 2000/3000 loss: 0.0019530552 val_acc: 1.000
epoch: 2500/3000 loss: 0.0011332451 val_acc: 1.000
epoch: 3000/3000 loss: 0.0010627856 val_acc: 1.000
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.datasets import make_moons
X_class, Y_class = make_moons(n_samples = 100, noise = .1)
Y_class = OneHotEncoder(sparse=False).fit_transform(Y_class.reshape(-1, 1))
X_class = StandardScaler().fit_transform(X_class)
nn = Model('softmax_neg_log_likelihood', learning_rate=1e-1)
nn.add(Layer(2))
nn.add(Layer(12, 'tanh', regularization = ('l2', 0.001)))
nn.add(Layer(12, 'tanh', regularization = ('l2', 0.001)))
nn.add(Layer(2, 'linear'))
nn.fit(X_class, Y_class, epochs = 5000, verbose = 1000, validation_rate = 0.2)
Out:
epoch: 0/5000 loss: 1.1460166199 val_acc: 0.350
epoch: 1000/5000 loss: 0.0084950357 val_acc: 1.000
epoch: 2000/5000 loss: 0.0026057702 val_acc: 1.000
epoch: 3000/5000 loss: 0.0015145193 val_acc: 1.000
epoch: 4000/5000 loss: 0.0010876888 val_acc: 1.000
epoch: 5000/5000 loss: 0.0008667776 val_acc: 1.000
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression
X_linear, Y_linear = make_regression(n_samples = 100, n_features=1, n_targets = 1, shuffle = True, noise = 15)
Y_linear = Y_linear.reshape(len(X_linear), 1)
X_linear = StandardScaler().fit_transform(X_linear)
Y_linear = StandardScaler().fit_transform(Y_linear)
nn = Model('mae', learning_rate=1e-1)
nn.add(Layer(1))
nn.add(Layer(10))
nn.add(Layer(1))
nn.fit(X_linear, Y_linear, epochs = 300, verbose = 100, validation_rate = 0.3)
Out:
epoch: 0/300 loss: 0.8991597990 val_acc: 24.668
epoch: 100/300 loss: 0.2203735767 val_acc: 7.783
epoch: 200/300 loss: 0.2200636261 val_acc: 7.789
epoch: 300/300 loss: 0.2179173881 val_acc: 7.751
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler
X_linear, Y_linear = make_exp(n_samples=100, x_min = 0, x_max = 5, noise = 10)
X_linear = StandardScaler().fit_transform(X_linear)
Y_linear = StandardScaler().fit_transform(Y_linear)
nn = Model('mae', learning_rate=1e-2)
nn.add(Layer(1))
nn.add(Layer(10, 'tanh'))
nn.add(Layer(10, 'tanh'))
nn.add(Layer(1, 'linear'))
nn.fit(X_linear, Y_linear, epochs = 10000, verbose = 2500, validation_rate = .05)
Out:
epoch: 0/10000 loss: 1.0819791908 val_acc: 5.167
epoch: 2500/10000 loss: 0.1388328752 val_acc: 1.023
epoch: 5000/10000 loss: 0.1337075506 val_acc: 0.948
epoch: 7500/10000 loss: 0.1323456180 val_acc: 0.916
epoch: 10000/10000 loss: 0.1311388838 val_acc: 0.831
from skadi.model import Model # to import the framework base Model
from skadi.layers import Layer, Dropout # to import the framework Layers
from sklearn.preprocessing import StandardScaler
X_linear, Y_linear = make_cubic(n_samples=100, x_min=-4, x_max=4, a=1, b=0, c=-10, d=0, noise=3)
X_linear = StandardScaler().fit_transform(X_linear)
Y_linear = StandardScaler().fit_transform(Y_linear)
nn = Model('mae', learning_rate=1e-1)
nn.add(Layer(1))
nn.add(Layer(16, 'relu'))
nn.add(Layer(8, 'tanh'))
nn.add(Layer(1, 'linear'))
nn.fit(X_linear, Y_linear, epochs = 3000, verbose = 600, validation_rate = .02, loss_stop=(100, 1e-3))
Out:
epoch: 0/3000 loss: 1.3110190012 val_acc: 1.212
epoch: 600/3000 loss: 0.2925245185 val_acc: 0.383
epoch: 1200/3000 loss: 0.2856206497 val_acc: 0.312
epoch: 1800/3000 loss: 0.2997886400 val_acc: 0.318
epoch: 2400/3000 loss: 0.2594068682 val_acc: 0.325
epoch: 3000/3000 loss: 0.2519903201 val_acc: 0.303
I am a Computer Science student at the Universidade Federal de Itajubá, with a focus on statistics, data science, data analysis, machine learning, deep learning among others. I made this project as a personal challenge with the intention of applying several concepts that I learned in the excellent course Manual Prático do Deep Learning - Redes Neurais Profundas offered by Arnaldo Gualberto, in addition to the objective of incorporating this work into my portfolio.
After the implementation of this project, I had a significant improvement in the theoretical and technical part of Deep Learning, as in the functioning of a neural network, the feedforward and backpropagation algorithms, partial derivatives and how they are applied in the latter, which are activation functions and cost and how to implement them, what each hyperparameter means and how they affect the final result of the network, how to create an adequate structure for each type of problem and many other very important concepts for the development of a neural network. Another important learning was the fact of better understanding how I should prepare the data for a neural network, from its cleaning to the application of techniques such as Normalization, Standardization, One-Hot-Encoder and others.
