Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
+ ${display_result} +
+This section assumes familiarity with the MLJ model API
If one subtypes a new model type as either MLJFlux.MLJFluxProbabilistic or MLJFlux.MLJFluxDeterministic, then instead of defining new methods for MLJModelInterface.fit and MLJModelInterface.update one can make use of fallbacks by implementing the lower level methods shape, build, and fitresult. See the classifier source code for an example.
One still needs to implement a new predict method.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
An expanded version of this example, with early stopping and snapshots, is available here.
We define a builder that builds a chain with six alternating convolution and max-pool layers, and a final dense layer, which we apply to the MNIST image dataset.
First we define a generic builder (working for any image size, color or gray):
using MLJ
+using Flux
+using MLDatasets
+
+# helper function
+function flatten(x::AbstractArray)
+ return reshape(x, :, size(x)[end])
+end
+
+import MLJFlux
+mutable struct MyConvBuilder
+ filter_size::Int
+ channels1::Int
+ channels2::Int
+ channels3::Int
+end
+
+function MLJFlux.build(b::MyConvBuilder, rng, n_in, n_out, n_channels)
+
+ k, c1, c2, c3 = b.filter_size, b.channels1, b.channels2, b.channels3
+
+ mod(k, 2) == 1 || error("`filter_size` must be odd. ")
+
+ # padding to preserve image size on convolution:
+ p = div(k - 1, 2)
+
+ front = Chain(
+ Conv((k, k), n_channels => c1, pad=(p, p), relu),
+ MaxPool((2, 2)),
+ Conv((k, k), c1 => c2, pad=(p, p), relu),
+ MaxPool((2, 2)),
+ Conv((k, k), c2 => c3, pad=(p, p), relu),
+ MaxPool((2 ,2)),
+ flatten,
+ )
+ d = Flux.outputsize(front, (n_in..., n_channels, 1)) |> first
+ return Chain(front, Dense(d, n_out))
+endNext, we load some of the MNIST data and check scientific types conform to those is the table above:
N = 500
+Xraw, yraw = MNIST(split=:train)[:];
+Xraw = Xraw[:,:,1:N];
+yraw = yraw[1:N];
+
+scitype(Xraw)scitype(yraw)Inputs should have element scitype GrayImage:
X = coerce(Xraw, GrayImage);For classifiers, target must have element scitype <: Finite:
y = coerce(yraw, Multiclass);Instantiating an image classifier model:
ImageClassifier = @load ImageClassifier
+clf = ImageClassifier(
+ builder=MyConvBuilder(3, 16, 32, 32),
+ epochs=10,
+ loss=Flux.crossentropy,
+ )And evaluating the accuracy of the model on a 30% holdout set:
mach = machine(clf, X, y)
+
+evaluate!(
+ mach,
+ resampling=Holdout(rng=123, fraction_train=0.7),
+ measure=misclassification_rate,
+ )Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
A Julia package integrating deep learning Flux models with MLJ.
Provide a user-friendly and high-level interface to fundamental Flux deep learning models while still being extensible by supporting custom models written with Flux
Make building deep learning models more convenient to users already familiar with the MLJ workflow
Make it easier to apply machine learning techniques provided by MLJ, including: out-of-sample performance evaluation, hyper-parameter optimization, iteration control, and more, to deep learning models
MLJFlux support is focused on fundamental and widely used deep learning models. Sophisticated architectures or techniques such as online learning, reinforcement learning, and adversarial networks are currently beyond its scope.
Also note that MLJFlux is limited to training models only when all training data fits into memory, though it still supports automatic batching of data.
import Pkg
+Pkg.activate("my_environment", shared=true)
+Pkg.add(["MLJ", "MLJFlux", "Flux"])You only need Flux if you need to build a custom architecture or experiment with different optimizers, loss functions and activations.
using MLJ, Flux, MLJFlux
+import RDatasets
+
+# 1. Load Data
+iris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
+
+# 2. Load and instantiate model
+NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg="MLJFlux"
+clf = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Flux.ADAM(0.01),
+ batch_size=8,
+ epochs=100,
+ acceleration=CUDALibs() # For GPU support
+ )
+
+# 3. Wrap it in a machine
+mach = machine(clf, X, y)
+
+# 4. Evaluate the model
+cv=CV(nfolds=5)
+evaluate!(mach, resampling=cv, measure=accuracy) As you can see we were able to use MLJ functionality (i.e., cross validation) with a Flux deep learning model. All arguments provided also have defaults.
Notice that we were also able to define the neural network in a high-level fashion by only specifying the number of neurons in each hidden layer and the activation function. Meanwhile, MLJFlux was able to infer the input and output layer as well as use a suitable default for the loss function and output activation given the classification task. Notice as well that we did not need to implement a training or prediction loop as in Flux.
As in the example above, any MLJFlux model has a builder hyperparameter, an object encoding instructions for creating a neural network given the data that the model eventually sees (e.g., the number of classes in a classification problem). While each MLJ model has a simple default builder, users may need to define custom builders to get optimal results, and this will require familiarity with the Flux API for defining a neural network chain.
Flux is a deep learning framework in Julia that comes with everything you need to build deep learning models (i.e., GPU support, automatic differentiation, layers, activations, losses, optimizers, etc.). MLJFlux wraps models built with Flux which provides a more high-level interface for building and training such models. More importantly, it empowers Flux models by extending their support to many common machine learning workflows that are possible via MLJ such as:
Estimating performance of your model using a holdout set or other resampling strategy (e.g., cross-validation) as measured by one or more metrics (e.g., loss functions) that may not have been used in training
Optimizing hyper-parameters such as a regularization parameter (e.g., dropout) or a width/height/nchannnels of convolution layer
Compose with other models such as introducing data pre-processing steps (e.g., missing data imputation) into a pipeline. It might make sense to include non-deep learning models in this pipeline. Other kinds of model composition could include blending predictions of a deep learner with some other kind of model (as in “model stacking”). Models composed with MLJ can be also tuned as a single unit.
Controlling iteration by adding an early stopping criterion based on an out-of-sample estimate of the loss, dynamically changing the learning rate (eg, cyclic learning rates), periodically save snapshots of the model, generate live plots of sample weights to judge training progress (as in tensor board)
A comparable project, FastAI/FluxTraining, also provides a high-level interface for interacting with Flux models and supports a set of features that may overlap with (but not include all of) those supported by MLJFlux.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
MLJFlux.Linear — TypeLinear(; σ=Flux.relu, rng=Random.GLOBAL_RNG)MLJFlux builder that constructs a fully connected two layer network with activation function σ. The number of input and output nodes is determined from the data. The bias and coefficients are initialized using Flux.glorot_uniform(rng). If rng is an integer, it is instead used as the seed for a MersenneTwister.
MLJFlux.Short — TypeShort(; n_hidden=0, dropout=0.5, σ=Flux.sigmoid, rng=GLOBAL_RNG)MLJFlux builder that constructs a full-connected three-layer network using n_hidden nodes in the hidden layer and the specified dropout (defaulting to 0.5). An activation function σ is applied between the hidden and final layers. If n_hidden=0 (the default) then n_hidden is the geometric mean of the number of input and output nodes. The number of input and output nodes is determined from the data.
The each layer is initialized using Flux.glorot_uniform(rng). If rng is an integer, it is instead used as the seed for a MersenneTwister.
MLJFlux.MLP — TypeMLP(; hidden=(100,), σ=Flux.relu, rng=GLOBAL_RNG)MLJFlux builder that constructs a Multi-layer perceptron network. The ith element of hidden represents the number of neurons in the ith hidden layer. An activation function σ is applied between each layer.
The each layer is initialized using Flux.glorot_uniform(rng). If rng is an integer, it is instead used as the seed for a MersenneTwister.
MLJFlux.@builder — Macro@builder neural_netCreates a builder for neural_net. The variables rng, n_in, n_out and n_channels can be used to create builders for any random number generator rng, input and output sizes n_in and n_out and number of input channels n_channels.
Examples
julia> import MLJFlux: @builder;
+
+julia> nn = NeuralNetworkRegressor(builder = @builder(Chain(Dense(n_in, 64, relu),
+ Dense(64, 32, relu),
+ Dense(32, n_out))));
+
+julia> conv_builder = @builder begin
+ front = Chain(Conv((3, 3), n_channels => 16), Flux.flatten)
+ d = Flux.outputsize(front, (n_in..., n_channels, 1)) |> first
+ Chain(front, Dense(d, n_out));
+ end
+
+julia> conv_nn = NeuralNetworkRegressor(builder = conv_builder);Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
MLJFlux.NeuralNetworkClassifier — TypeNeuralNetworkClassifierA model type for constructing a neural network classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFluxDo model = NeuralNetworkClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkClassifier(builder=...).
NeuralNetworkClassifier is for training a data-dependent Flux.jl neural network for making probabilistic predictions of a Multiclass or OrderedFactor target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any AbstractVector whose element scitype is Multiclass or OrderedFactor; check the scitype with scitype(y)
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Short(): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux.jl documentation for examples of user-defined builders. See also finaliser below.
optimiser::Flux.Adam(): A Flux.Optimise optimiser. The optimiser performs the updating of the weights of the network. For further reference, see the Flux optimiser documentation. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.crossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.crossentropy: Standard multiclass classification loss, also known as the log loss.
Flux.logitcrossentopy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with softmax and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default softmax finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞).
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training.
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.softmax: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.softmax.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we build a classification model using the Iris dataset. This is a very basic example, using a default builder and no standardization. For a more advanced illustration, see NeuralNetworkRegressor or ImageClassifier, and examples in the MLJFlux.jl documentation.
using MLJ
+using Flux
+import RDatasetsFirst, we can load the data:
iris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), rng=123); # a vector and a table
+NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
+clf = NeuralNetworkClassifier()Next, we can train the model:
mach = machine(clf, X, y)
+fit!(mach)We can train the model in an incremental fashion, altering the learning rate as we go, provided optimizer_changes_trigger_retraining is false (the default). Here, we also change the number of (total) iterations:
clf.optimiser.eta = clf.optimiser.eta * 2
+clf.epochs = clf.epochs + 5
+
+fit!(mach, verbosity=2) # trains 5 more epochsWe can inspect the mean training loss using the cross_entropy function:
training_loss = cross_entropy(predict(mach, X), y) |> meanAnd we can access the Flux chain (model) using fitted_params:
chain = fitted_params(mach).chainFinally, we can see how the out-of-sample performance changes over time, using MLJ's learning_curve function:
r = range(clf, :epochs, lower=1, upper=200, scale=:log10)
+curve = learning_curve(clf, X, y,
+ range=r,
+ resampling=Holdout(fraction_train=0.7),
+ measure=cross_entropy)
+using Plots
+plot(curve.parameter_values,
+ curve.measurements,
+ xlab=curve.parameter_name,
+ xscale=curve.parameter_scale,
+ ylab = "Cross Entropy")
+See also ImageClassifier.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
Following is an example defining a new builder for creating a simple fully-connected neural network with two hidden layers, with n1 nodes in the first hidden layer, and n2 nodes in the second, for use in any of the first three models in Table 1. The definition includes one mutable struct and one method:
mutable struct MyBuilder <: MLJFlux.Builder
+ n1 :: Int
+ n2 :: Int
+end
+
+function MLJFlux.build(nn::MyBuilder, rng, n_in, n_out)
+ init = Flux.glorot_uniform(rng)
+ return Chain(
+ Dense(n_in, nn.n1, init=init),
+ Dense(nn.n1, nn.n2, init=init),
+ Dense(nn.n2, n_out, init=init),
+ )
+endNote here that n_in and n_out depend on the size of the data (see Table 1.
For a concrete image classification example, see the Image Classification Example.
More generally, defining a new builder means defining a new struct sub-typing MLJFlux.Builder and defining a new MLJFlux.build method with one of these signatures:
MLJFlux.build(builder::MyBuilder, rng, n_in, n_out)
+MLJFlux.build(builder::MyBuilder, rng, n_in, n_out, n_channels) # for use with `ImageClassifier`This method must return a Flux.Chain instance, chain, subject to the following conditions:
chain(x) must make sense:
x <: Array{<:AbstractFloat, 2} of size (n_in, batch_size) where batch_size is any integer (for use with one of the first three model types); orx <: Array{<:Float32, 4} of size (W, H, n_channels, batch_size), where (W, H) = n_in, n_channels is 1 or 3, and batch_size is any integer (for use with ImageClassifier)The object returned by chain(x) must be an AbstractFloat vector of length n_out.
Alternatively, use MLJFlux.@builder(neural_net) to automatically create a builder for any valid Flux chain expression neural_net, where the symbols n_in, n_out, n_channels and rng can appear literally, with the interpretations explained above. For example,
builder = MLJFlux.@builder Chain(Dense(n_in, 128), Dense(128, n_out, tanh))Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
MLJFlux.ImageClassifier — TypeImageClassifierA model type for constructing a image classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
ImageClassifier = @load ImageClassifier pkg=MLJFluxDo model = ImageClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in ImageClassifier(builder=...).
ImageClassifier classifies images using a neural network adapted to the type of images provided (color or gray scale). Predictions are probabilistic. Users provide a recipe for constructing the network, based on properties of the image encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is any AbstractVector of images with ColorImage or GrayImage scitype; check the scitype with scitype(X) and refer to ScientificTypes.jl documentation on coercing typical image formats into an appropriate type.
y is the target, which can be any AbstractVector whose element scitype is Multiclass; check the scitype with scitype(y).
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder: An MLJFlux builder that constructs the neural network. The fallback builds a depth-16 VGG architecture adapted to the image size and number of target classes, with no batch normalization; see the Metalhead.jl documentation for details. See the example below for a user-specified builder. A convenience macro @builder is also available. See also finaliser below.
optimiser::Flux.Adam(): A Flux.Optimise optimiser. The optimiser performs the updating of the weights of the network. For further reference, see the Flux optimiser documentation. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.crossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.crossentropy: Standard multiclass classification loss, also known as the log loss.
Flux.logitcrossentopy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with softmax and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default softmax finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞).
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training.
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.softmax: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.softmax.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we use MLJFlux and a custom builder to classify the MNIST image dataset.
using MLJ
+using Flux
+import MLJFlux
+import MLJIteration # for `skip` controlFirst we want to download the MNIST dataset, and unpack into images and labels:
import MLDatasets: MNIST
+data = MNIST(split=:train)
+images, labels = data.features, data.targetsIn MLJ, integers cannot be used for encoding categorical data, so we must coerce them into the Multiclass scitype:
labels = coerce(labels, Multiclass);Above images is a single array but MLJFlux requires the images to be a vector of individual image arrays:
images = coerce(images, GrayImage);
+images[1]We start by defining a suitable builder object. This is a recipe for building the neural network. Our builder will work for images of any (constant) size, whether they be color or black and white (ie, single or multi-channel). The architecture always consists of six alternating convolution and max-pool layers, and a final dense layer; the filter size and the number of channels after each convolution layer is customizable.
import MLJFlux
+
+struct MyConvBuilder
+ filter_size::Int
+ channels1::Int
+ channels2::Int
+ channels3::Int
+end
+
+make2d(x::AbstractArray) = reshape(x, :, size(x)[end])
+
+function MLJFlux.build(b::MyConvBuilder, rng, n_in, n_out, n_channels)
+ k, c1, c2, c3 = b.filter_size, b.channels1, b.channels2, b.channels3
+ mod(k, 2) == 1 || error("`filter_size` must be odd. ")
+ p = div(k - 1, 2) # padding to preserve image size
+ init = Flux.glorot_uniform(rng)
+ front = Chain(
+ Conv((k, k), n_channels => c1, pad=(p, p), relu, init=init),
+ MaxPool((2, 2)),
+ Conv((k, k), c1 => c2, pad=(p, p), relu, init=init),
+ MaxPool((2, 2)),
+ Conv((k, k), c2 => c3, pad=(p, p), relu, init=init),
+ MaxPool((2 ,2)),
+ make2d)
+ d = Flux.outputsize(front, (n_in..., n_channels, 1)) |> first
+ return Chain(front, Dense(d, n_out, init=init))
+endIt is important to note that in our build function, there is no final softmax. This is applied by default in all MLJFlux classifiers (override this using the finaliser hyperparameter).
Now that our builder is defined, we can instantiate the actual MLJFlux model. If you have a GPU, you can substitute in acceleration=CUDALibs() below to speed up training.
ImageClassifier = @load ImageClassifier pkg=MLJFlux
+clf = ImageClassifier(builder=MyConvBuilder(3, 16, 32, 32),
+ batch_size=50,
+ epochs=10,
+ rng=123)You can add Flux options such as optimiser and loss in the snippet above. Currently, loss must be a flux-compatible loss, and not an MLJ measure.
Next, we can bind the model with the data in a machine, and train using the first 500 images:
mach = machine(clf, images, labels);
+fit!(mach, rows=1:500, verbosity=2);
+report(mach)
+chain = fitted_params(mach)
+Flux.params(chain)[2]We can tack on 20 more epochs by modifying the epochs field, and iteratively fit some more:
clf.epochs = clf.epochs + 20
+fit!(mach, rows=1:500, verbosity=2);We can also make predictions and calculate an out-of-sample loss estimate, using any MLJ measure (loss/score):
predicted_labels = predict(mach, rows=501:1000);
+cross_entropy(predicted_labels, labels[501:1000]) |> meanThe preceding fit!/predict/evaluate workflow can be alternatively executed as follows:
evaluate!(mach,
+ resampling=Holdout(fraction_train=0.5),
+ measure=cross_entropy,
+ rows=1:1000,
+ verbosity=0)See also NeuralNetworkClassifier.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
MLJFlux.MultitargetNeuralNetworkRegressor — TypeMultitargetNeuralNetworkRegressorA model type for constructing a multitarget neural network regressor, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
MultitargetNeuralNetworkRegressor = @load MultitargetNeuralNetworkRegressor pkg=MLJFluxDo model = MultitargetNeuralNetworkRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in MultitargetNeuralNetworkRegressor(builder=...).
MultitargetNeuralNetworkRegressor is for training a data-dependent Flux.jl neural network to predict a multi-valued Continuous target, represented as a table, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any table or matrix of output targets whose element scitype is Continuous; check column scitypes with schema(y). If y is a Matrix, it is assumed to have columns corresponding to variables and rows corresponding to observations.
Hyper-parameters
builder=MLJFlux.Linear(σ=Flux.relu): An MLJFlux builder that constructs a neural network. Possible builders include: Linear, Short, and MLP. See MLJFlux documentation for more on builders, and the example below for using the @builder convenience macro.
optimiser::Flux.Adam(): A Flux.Optimise optimiser. The optimiser performs the updating of the weights of the network. For further reference, see the Flux optimiser documentation. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.mse: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a regression task, natural loss functions are:
Flux.mse
Flux.mae
Flux.msle
Flux.huber_loss
Currently MLJ measures are not supported as loss functions here.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞).
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training.
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew having the same scitype as X above. Predictions are deterministic.Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network.Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we apply a multi-target regression model to synthetic data:
using MLJ
+import MLJFlux
+using FluxFirst, we generate some synthetic data (needs MLJBase 0.20.16 or higher):
X, y = make_regression(100, 9; n_targets = 2) # both tables
+schema(y)
+schema(X)Splitting off a test set:
(X, Xtest), (y, ytest) = partition((X, y), 0.7, multi=true);Next, we can define a builder, making use of a convenience macro to do so. In the following @builder call, n_in is a proxy for the number input features and n_out the number of target variables (both known at fit! time), while rng is a proxy for a RNG (which will be passed from the rng field of model defined below).
builder = MLJFlux.@builder begin
+ init=Flux.glorot_uniform(rng)
+ Chain(
+ Dense(n_in, 64, relu, init=init),
+ Dense(64, 32, relu, init=init),
+ Dense(32, n_out, init=init),
+ )
+endInstantiating the regression model:
MultitargetNeuralNetworkRegressor = @load MultitargetNeuralNetworkRegressor
+model = MultitargetNeuralNetworkRegressor(builder=builder, rng=123, epochs=20)We will arrange for standardization of the the target by wrapping our model in TransformedTargetModel, and standardization of the features by inserting the wrapped model in a pipeline:
pipe = Standardizer |> TransformedTargetModel(model, target=Standardizer)If we fit with a high verbosity (>1), we will see the losses during training. We can also see the losses in the output of report(mach)
mach = machine(pipe, X, y)
+fit!(mach, verbosity=2)
+
+# first element initial loss, 2:end per epoch training losses
+report(mach).transformed_target_model_deterministic.model.training_lossesFor experimenting with learning rate, see the NeuralNetworkRegressor example.
pipe.transformed_target_model_deterministic.model.optimiser.eta = 0.0001With the learning rate fixed, we can now compute a CV estimate of the performance (using all data bound to mach) and compare this with performance on the test set:
# custom MLJ loss:
+multi_loss(yhat, y) = l2(MLJ.matrix(yhat), MLJ.matrix(y)) |> mean
+
+# CV estimate, based on `(X, y)`:
+evaluate!(mach, resampling=CV(nfolds=5), measure=multi_loss)
+
+# loss for `(Xtest, test)`:
+fit!(mach) # trains on all data `(X, y)`
+yhat = predict(mach, Xtest)
+multi_loss(yhat, ytest)See also NeuralNetworkRegressor
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
MLJFlux.NeuralNetworkRegressor — TypeNeuralNetworkRegressorA model type for constructing a neural network regressor, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkRegressor = @load NeuralNetworkRegressor pkg=MLJFluxDo model = NeuralNetworkRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkRegressor(builder=...).
NeuralNetworkRegressor is for training a data-dependent Flux.jl neural network to predict a Continuous target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Linear(σ=Flux.relu): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux documentation for more on builders, and the example below for using the @builder convenience macro.
optimiser::Flux.Adam(): A Flux.Optimise optimiser. The optimiser performs the updating of the weights of the network. For further reference, see the Flux optimiser documentation. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.mse: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a regression task, natural loss functions are:
Flux.mse
Flux.mae
Flux.msle
Flux.huber_loss
Currently MLJ measures are not supported as loss functions here.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞).
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training.
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above.Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network.Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalized if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we build a regression model for the Boston house price dataset.
using MLJ
+import MLJFlux
+using FluxFirst, we load in the data: The :MEDV column becomes the target vector y, and all remaining columns go into a table X, with the exception of :CHAS:
data = OpenML.load(531); # Loads from https://www.openml.org/d/531
+y, X = unpack(data, ==(:MEDV), !=(:CHAS); rng=123);
+
+scitype(y)
+schema(X)Since MLJFlux models do not handle ordered factors, we'll treat :RAD as Continuous:
X = coerce(X, :RAD=>Continuous)Splitting off a test set:
(X, Xtest), (y, ytest) = partition((X, y), 0.7, multi=true);Next, we can define a builder, making use of a convenience macro to do so. In the following @builder call, n_in is a proxy for the number input features (which will be known at fit! time) and rng is a proxy for a RNG (which will be passed from the rng field of model defined below). We also have the parameter n_out which is the number of output features. As we are doing single target regression, the value passed will always be 1, but the builder we define will also work for MultitargetNeuralRegressor.
builder = MLJFlux.@builder begin
+ init=Flux.glorot_uniform(rng)
+ Chain(
+ Dense(n_in, 64, relu, init=init),
+ Dense(64, 32, relu, init=init),
+ Dense(32, n_out, init=init),
+ )
+endInstantiating a model:
NeuralNetworkRegressor = @load NeuralNetworkRegressor pkg=MLJFlux
+model = NeuralNetworkRegressor(
+ builder=builder,
+ rng=123,
+ epochs=20
+)We arrange for standardization of the the target by wrapping our model in TransformedTargetModel, and standardization of the features by inserting the wrapped model in a pipeline:
pipe = Standardizer |> TransformedTargetModel(model, target=Standardizer)If we fit with a high verbosity (>1), we will see the losses during training. We can also see the losses in the output of report(mach).
mach = machine(pipe, X, y)
+fit!(mach, verbosity=2)
+
+# first element initial loss, 2:end per epoch training losses
+report(mach).transformed_target_model_deterministic.model.training_lossesExperimenting with learning rate
We can visually compare how the learning rate affects the predictions:
using Plots
+
+rates = rates = [5e-5, 1e-4, 0.005, 0.001, 0.05]
+plt=plot()
+
+foreach(rates) do η
+ pipe.transformed_target_model_deterministic.model.optimiser.eta = η
+ fit!(mach, force=true, verbosity=0)
+ losses =
+ report(mach).transformed_target_model_deterministic.model.training_losses[3:end]
+ plot!(1:length(losses), losses, label=η)
+end
+
+plt
+
+pipe.transformed_target_model_deterministic.model.optimiser.eta = 0.0001With the learning rate fixed, we compute a CV estimate of the performance (using all data bound to mach) and compare this with performance on the test set:
# CV estimate, based on `(X, y)`:
+evaluate!(mach, resampling=CV(nfolds=5), measure=l2)
+
+# loss for `(Xtest, test)`:
+fit!(mach) # train on `(X, y)`
+yhat = predict(mach, Xtest)
+l2(yhat, ytest) |> meanThese losses, for the pipeline model, refer to the target on the original, unstandardized, scale.
For implementing stopping criterion and other iteration controls, refer to examples linked from the MLJFlux documentation.
See also MultitargetNeuralNetworkRegressor
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
MLJFlux provides four model types, for use with input features X and targets y of the scientific type indicated in the table below. The parameters n_in, n_out and n_channels refer to information passed to the builder, as described under Defining a new builder below.
| Model Type | Prediction type | scitype(X) <: _ | scitype(y) <: _ |
|---|---|---|---|
NeuralNetworkRegressor | Deterministic | Table(Continuous) with n_in columns | AbstractVector{<:Continuous) (n_out = 1) |
MultitargetNeuralNetworkRegressor | Deterministic | Table(Continuous) with n_in columns | <: Table(Continuous) with n_out columns |
NeuralNetworkClassifier | Probabilistic | <:Table(Continuous) with n_in columns | AbstractVector{<:Finite} with n_out classes |
ImageClassifier | Probabilistic | AbstractVector(<:Image{W,H}) with n_in = (W, H) | AbstractVector{<:Finite} with n_out classes |
In MLJ a model is a mutable struct storing hyper-parameters for some learning algorithm indicated by the model name, and that's all. In particular, an MLJ model does not store learned parameters.
In Flux the term "model" has another meaning. However, as all Flux "models" used in MLJFLux are Flux.Chain objects, we call them chains, and restrict use of "model" to models in the MLJ sense.
Any AbstractMatrix{<:AbstractFloat} object Xmat can be forced to have scitype Table(Continuous) by replacing it with X = MLJ.table(Xmat). Furthermore, this wrapping, and subsequent unwrapping under the hood, will compile to a no-op. At present this includes support for sparse matrix data, but the implementation has not been optimized for sparse data at this time and so should be used with caution.
Instructions for coercing common image formats into some AbstractVector{<:Image} are here.
MLJ machines cache state enabling the "warm restart" of model training, as demonstrated in the incremental training example. In the case of MLJFlux models, fit!(mach) will use a warm restart if:
only model.epochs has changed since the last call; or
only model.epochs or model.optimiser have changed since the last call and model.optimiser_changes_trigger_retraining == false (the default) (the "state" part of the optimiser is ignored in this comparison). This allows one to dynamically modify learning rates, for example.
Here model=mach.model is the associated MLJ model.
The warm restart feature makes it possible to apply early stopping criteria, as defined in EarlyStopping.jl. For an example, see /examples/mnist/. (Eventually, this will be handled by an MLJ model wrapper for controlling arbitrary iterative models.)
All models share the following hyper-parameters:
| Hyper-parameter | Description | Default |
|---|---|---|
builder | Default builder for models. | MLJFlux.Linear(σ=Flux.relu) (regressors) or MLJFlux.Short(n_hidden=0, dropout=0.5, σ=Flux.σ) (classifiers) |
optimiser | The optimiser to use for training. | Flux.ADAM() |
loss | The loss function used for training. | Flux.mse (regressors) and Flux.crossentropy (classifiers) |
n_epochs | Number of epochs to train for. | 10 |
batch_size | The batch size for the data. | 1 |
lambda | The regularization strength. Range = [0, ∞). | 0 |
alpha | The L2/L1 mix of regularization. Range = [0, 1]. | 0 |
rng | The random number generator (RNG) passed to builders, for weight initialization, for example. Can be any AbstractRNG or the seed (integer) for a MersenneTwister that is reset on every cold restart of model (machine) training. | GLOBAL_RNG |
acceleration | Use CUDALibs() for training on GPU; default is CPU1(). | CPU1() |
optimiser_changes_trigger_retraining | True if fitting an associated machine should trigger retraining from scratch whenever the optimiser changes. | false |
The classifiers have an additional hyperparameter finaliser (default = Flux.softmax) which is the operation applied to the unnormalized output of the final layer to obtain probabilities (outputs summing to one). Default = Flux.softmax. It should return a vector of the same length as its input.
Currently, the loss function specified by loss=... is applied internally by Flux and needs to conform to the Flux API. You cannot, for example, supply one of MLJ's probabilistic loss functions, such as MLJ.cross_entropy to one of the classifier constructors.
That said, you can only use MLJ loss functions or metrics in evaluation meta-algorithms (such as cross validation) and they will work even if the underlying model comes from MLJFlux.
As in the table, when instantiating a model for training on a GPU, specify acceleration=CUDALibs(), as in
using MLJ
+ImageClassifier = @load ImageClassifier
+model = ImageClassifier(epochs=10, acceleration=CUDALibs())
+mach = machine(model, X, y) |> fit!In this example, the data X, y is copied onto the GPU under the hood on the call to fit! and cached for use in any warm restart (see above). The Flux chain used in training is always copied back to the CPU at then conclusion of fit!, and made available as fitted_params(mach).
As for the builder argument, the following builders are provided out-of-the-box:
| Builder | Description |
|---|---|
MLJFlux.MLP(hidden=(10,)) | General multi-layer perceptron |
MLJFlux.Short(n_hidden=0, dropout=0.5, σ=sigmoid) | Fully connected network with one hidden layer and dropout |
MLJFlux.Linear(σ=relu) | Vanilla linear network with no hidden layers and activation function σ |
See the following sections to learn more about the interface for the builders and models.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.
import Random.seed!; seed!(123)
+mach = machine(clf, X, y)
+fit!(mach)
+
+julia> training_loss = cross_entropy(predict(mach, X), y) |> mean
+0.9064070459118777
+
+# Increasing learning rate and adding iterations:
+clf.optimiser.eta = clf.optimiser.eta * 2
+clf.epochs = clf.epochs + 5
+
+julia> fit!(mach, verbosity=2)
+[ Info: Updating Machine{NeuralNetworkClassifier{Short,…},…} @804.
+[ Info: Loss is 0.8686
+[ Info: Loss is 0.8228
+[ Info: Loss is 0.7706
+[ Info: Loss is 0.7565
+[ Info: Loss is 0.7347
+Machine{NeuralNetworkClassifier{Short,…},…} @804 trained 2 times; caches data
+ args:
+ 1: Source @985 ⏎ `Table{AbstractVector{Continuous}}`
+ 2: Source @367 ⏎ `AbstractVector{Multiclass{3}}`
+
+julia> training_loss = cross_entropy(predict(mach, X), y) |> mean
+0.7347092796453824julia> fitted_params(mach).chain
+Chain(Chain(Dense(4, 3, σ), Flux.Dropout{Float64}(0.5, false), Dense(3, 3)), softmax)r = range(clf, :epochs, lower=1, upper=200, scale=:log10)
+curve = learning_curve(
+ clf, X, y;
+ range=r,
+ resampling=Holdout(fraction_train=0.7),
+ measure=cross_entropy,
+ )
+using Plots
+plot(
+ curve.parameter_values,
+ curve.measurements,
+ xlab=curve.parameter_name,
+ xscale=curve.parameter_scale,
+ ylab = "Cross Entropy",
+ )
+
This is still work-in-progress. Expect more workflow examples and tutorials soon.
Settings
This document was generated with Documenter.jl version 1.4.1 on Saturday 25 May 2024. Using Julia version 1.10.3.