Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| Row | mlp | measurement |
|---|---|---|
| MLP… | Float64 | |
| 1 | MLP(hidden = (25, 53, 45), …) | 0.0865692 |
| 2 | MLP(hidden = (49, 41, 49), …) | 0.0870145 |
| 3 | MLP(hidden = (25, 61, 21), …) | 0.0870776 |
| 4 | MLP(hidden = (45, 21, 41), …) | 0.0921284 |
| 5 | MLP(hidden = (49, 13, 33), …) | 0.0941658 |
| 6 | MLP(hidden = (21, 49, 53), …) | 0.100384 |
| 7 | MLP(hidden = (33, 57, 61), …) | 0.101213 |
| 8 | MLP(hidden = (33, 49, 9), …) | 0.10241 |
| 9 | MLP(hidden = (17, 37, 17), …) | 0.10542 |
| 10 | MLP(hidden = (29, 49, 17), …) | 0.108438 |
| Row | mlp | measurement |
|---|---|---|
| MLP… | Float64 | |
| 1 | MLP(hidden = (21, 57, 25), …) | 0.0867019 |
| 2 | MLP(hidden = (45, 17, 13), …) | 0.0929803 |
| 3 | MLP(hidden = (33, 13, 49), …) | 0.0973896 |
| 4 | MLP(hidden = (21, 41, 61), …) | 0.0981502 |
| 5 | MLP(hidden = (57, 49, 61), …) | 0.100331 |
| 6 | MLP(hidden = (25, 25, 29), …) | 0.101083 |
| 7 | MLP(hidden = (29, 61, 21), …) | 0.101466 |
| 8 | MLP(hidden = (29, 61, 5), …) | 0.107513 |
| 9 | MLP(hidden = (21, 61, 17), …) | 0.107874 |
| 10 | MLP(hidden = (45, 49, 61), …) | 0.111292 |
This demonstration is available as a Jupyter notebook or julia script here.
Neural Architecture Search (NAS) is an instance of hyperparameter tuning concerned with tuning model hyperparameters defining the architecture itself. Although it's typically performed with sophisticated search algorithms for efficiency, in this example we will be using a simple random search.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
+using Flux # For more flexibility
+using RDatasets: RDatasets # Dataset source
+using DataFrames # To view tuning results in a table
+import Optimisers # native Flux.jl optimisers no longer supportediris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng = 123);
+X = Float32.(X); # To be compatible with type of network network parameters
+first(X, 5)| Row | SepalLength | SepalWidth | PetalLength | PetalWidth |
|---|---|---|---|---|
| Float32 | Float32 | Float32 | Float32 | |
| 1 | 6.7 | 3.3 | 5.7 | 2.1 |
| 2 | 5.7 | 2.8 | 4.1 | 1.3 |
| 3 | 7.2 | 3.0 | 5.8 | 1.6 |
| 4 | 4.4 | 2.9 | 1.4 | 0.2 |
| 5 | 5.6 | 2.5 | 3.9 | 1.1 |
Now let's construct our model. This follows a similar setup the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg = "MLJFlux"
+clf = NeuralNetworkClassifier(
+ builder = MLJFlux.MLP(; hidden = (1, 1, 1), σ = Flux.relu),
+ optimiser = Optimisers.ADAM(0.01),
+ batch_size = 8,
+ epochs = 10,
+ rng = 42,
+)NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (1, 1, 1),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))We know that the MLP builder takes a tuple of the form $(z_1, z_2, ..., z_k)$ to define a network with $k$ hidden layers and where the ith layer has $z_i$ neurons. We will proceed by defining a function that can generate all possible networks with a specific number of hidden layers, a minimum and maximum number of neurons per layer and increments to consider for the number of neurons.
function generate_networks(
+ ;min_neurons::Int,
+ max_neurons::Int,
+ neuron_step::Int,
+ num_layers::Int,
+ )
+ # Define the range of neurons
+ neuron_range = min_neurons:neuron_step:max_neurons
+
+ # Empty list to store the network configurations
+ networks = Vector{Tuple{Vararg{Int, num_layers}}}()
+
+ # Recursive helper function to generate all combinations of tuples
+ function generate_tuple(current_layers, remaining_layers)
+ if remaining_layers > 0
+ for n in neuron_range
+ # current_layers =[] then current_layers=[(min_neurons)],
+ # [(min_neurons+neuron_step)], [(min_neurons+2*neuron_step)],...
+ # for each of these we call generate_layers again which appends
+ # the n combinations for each one of them
+ generate_tuple(vcat(current_layers, [n]), remaining_layers - 1)
+ end
+ else
+ # in the base case, no more layers to "recurse on"
+ # and we just append the current_layers as a tuple
+ push!(networks, tuple(current_layers...))
+ end
+ end
+
+ # Generate networks for the given number of layers
+ generate_tuple([], num_layers)
+
+ return networks
+endgenerate_networks (generic function with 1 method)Now let's generate an array of all possible neural networks with three hidden layers and number of neurons per layer ∈ [1,64] with a step of 4
networks_space =
+ generate_networks(
+ min_neurons = 1,
+ max_neurons = 64,
+ neuron_step = 4,
+ num_layers = 3,
+ )
+
+networks_space[1:5]5-element Vector{Tuple{Int64, Int64, Int64}}:
+ (1, 1, 1)
+ (1, 1, 5)
+ (1, 1, 9)
+ (1, 1, 13)
+ (1, 1, 17)Let's use this array to define the range of hyperparameters and pass it along with the model to the TunedModel constructor.
r1 = range(clf, :(builder.hidden), values = networks_space)
+
+tuned_clf = TunedModel(
+ model = clf,
+ tuning = RandomSearch(),
+ resampling = CV(nfolds = 4, rng = 42),
+ range = [r1],
+ measure = cross_entropy,
+ n = 100, # searching over 100 random samples are enough
+);Similar to the last workflow example, all we need now is to fit our model and the search will take place automatically:
mach = machine(tuned_clf, X, y);
+fit!(mach, verbosity = 0);
+fitted_params(mach).best_modelNeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (25, 29, 45),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))Let's analyze the search results by converting the history array to a dataframe and viewing it:
history = report(mach).history
+history_df = DataFrame(
+ mlp = [x[:model].builder for x in history],
+ measurement = [x[:measurement][1] for x in history],
+)
+first(sort!(history_df, [order(:measurement)]), 10)| Row | mlp | measurement |
|---|---|---|
| MLP… | Float64 | |
| 1 | MLP(hidden = (25, 29, 45), …) | 0.0812277 |
| 2 | MLP(hidden = (21, 9, 41), …) | 0.0817329 |
| 3 | MLP(hidden = (33, 25, 57), …) | 0.0857974 |
| 4 | MLP(hidden = (21, 9, 45), …) | 0.0905676 |
| 5 | MLP(hidden = (29, 57, 49), …) | 0.0927865 |
| 6 | MLP(hidden = (49, 13, 61), …) | 0.0957713 |
| 7 | MLP(hidden = (21, 9, 13), …) | 0.0966514 |
| 8 | MLP(hidden = (25, 41, 45), …) | 0.0971129 |
| 9 | MLP(hidden = (21, 41, 61), …) | 0.0981502 |
| 10 | MLP(hidden = (49, 53, 29), …) | 0.0998882 |
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| Row | mlp | measurement |
|---|---|---|
| Probabil… | Float64 | |
| 1 | BayesianLDA(method = gevd, …) | 0.0610826 |
| 2 | RandomForestClassifier(max_depth = -1, …) | 0.0996176 |
| 3 | NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …) | 0.113266 |
| 4 | ProbabilisticTunedModel(model = XGBoostClassifier(test = 1, …), …) | 0.221056 |
| Row | mlp | measurement |
|---|---|---|
| Probabil… | Float64 | |
| 1 | BayesianLDA(method = gevd, …) | 0.0610826 |
| 2 | NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …) | 0.0857014 |
| 3 | RandomForestClassifier(max_depth = -1, …) | 0.102881 |
| 4 | ProbabilisticTunedModel(model = XGBoostClassifier(test = 1, …), …) | 0.221056 |
This demonstration is available as a Jupyter notebook or julia script here.
In this workflow example, we see how we can compare different machine learning models with a neural network from MLJFlux.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
+using Flux # For more flexibility
+import RDatasets # Dataset source
+using DataFrames # To visualize hyperparameter search results
+import Optimisers # native Flux.jl optimisers no longer supportediris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
+
+clf1 = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Optimisers.Adam(0.01),
+ batch_size=8,
+ epochs=50,
+ rng=42
+ )NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 50,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))Let's as well load and construct three other classical machine learning models:
BayesianLDA = @load BayesianLDA pkg=MultivariateStats
+clf2 = BayesianLDA()
+RandomForestClassifier = @load RandomForestClassifier pkg=DecisionTree
+clf3 = RandomForestClassifier()
+XGBoostClassifier = @load XGBoostClassifier pkg=XGBoost
+clf4 = XGBoostClassifier();[ Info: For silent loading, specify `verbosity=0`.
+import MLJMultivariateStatsInterface ✔
+[ Info: For silent loading, specify `verbosity=0`.
+import MLJDecisionTreeInterface ✔
+[ Info: For silent loading, specify `verbosity=0`.
+import MLJXGBoostInterface ✔Instead of just comparing with four models with the default/given hyperparameters, we will give XGBoostClassifier an unfair advantage By wrapping it in a TunedModel that considers the best learning rate η for the model.
r1 = range(clf4, :eta, lower=0.01, upper=0.5, scale=:log10)
+tuned_model_xg = TunedModel(
+ model=clf4,
+ ranges=[r1],
+ tuning=Grid(resolution=10),
+ resampling=CV(nfolds=5, rng=42),
+ measure=cross_entropy,
+);Of course, one can wrap each of the four in a TunedModel if they are interested in comparing the models over a large set of their hyperparameters.
We simply pass the four models to the models argument of the TunedModel construct
tuned_model = TunedModel(
+ models=[clf1, clf2, clf3, tuned_model_xg],
+ tuning=Explicit(),
+ resampling=CV(nfolds=5, rng=42),
+ measure=cross_entropy,
+);Then wrapping our tuned model in a machine and fitting it.
mach = machine(tuned_model, X, y);
+fit!(mach, verbosity=0);┌ Warning: Layer with Float32 parameters got Float64 input.
+│ The input will be converted, but any earlier layers may be very slow.
+│ layer = Dense(4 => 5, relu) # 25 parameters
+│ summary(x) = "4×8 Matrix{Float64}"
+└ @ Flux ~/.julia/packages/Flux/Wz6D4/src/layers/stateless.jl:60Now let's see the history for more details on the performance for each of the models
history = report(mach).history
+history_df = DataFrame(
+ mlp = [x[:model] for x in history],
+ measurement = [x[:measurement][1] for x in history],
+)
+sort!(history_df, [order(:measurement)])| Row | mlp | measurement |
|---|---|---|
| Probabil… | Float64 | |
| 1 | BayesianLDA(method = gevd, …) | 0.0610826 |
| 2 | NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …) | 0.0857014 |
| 3 | RandomForestClassifier(max_depth = -1, …) | 0.102431 |
| 4 | ProbabilisticTunedModel(model = XGBoostClassifier(test = 1, …), …) | 0.221056 |
This is Occam's razor in practice.
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
This tutorial is available as a Jupyter notebook or julia script here.
In this workflow example, we see how MLJFlux enables composing MLJ models with MLJFlux models. We will assume a class imbalance setting and wrap an oversampler with a deep learning model from MLJFlux.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
+using Flux # For more flexibility
+import RDatasets # Dataset source
+import Random # To create imbalance
+import Imbalance # To solve the imbalance
+import Optimisers # native Flux.jl optimisers no longer supportediris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
+X = Float32.(X); # To be compatible with type of network network parametersTo simulate an imbalanced dataset, we will take a random sample:
Random.seed!(803429)
+subset_indices = rand(1:size(X, 1), 100)
+X, y = X[subset_indices, :], y[subset_indices]
+Imbalance.checkbalance(y)versicolor: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 28 (65.1%)
+virginica: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 29 (67.4%)
+setosa: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 43 (100.0%)Let's load BorderlineSMOTE1 to oversample the data and Standardizer to standardize it.
BorderlineSMOTE1 = @load BorderlineSMOTE1 pkg=Imbalance verbosity=0
+NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFluxNeuralNetworkClassifierWe didn't need to load Standardizer because it is a local model for MLJ (see localmodels())
clf = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Optimisers.Adam(0.01),
+ batch_size=8,
+ epochs=50,
+ rng=42,
+)NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 50,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))First we wrap the oversampler with the neural network via the BalancedModel construct. This comes from MLJBalancing And allows combining resampling methods with MLJ models in a sequential pipeline.
oversampler = BorderlineSMOTE1(k=5, ratios=1.0, rng=42)
+balanced_model = BalancedModel(model=clf, balancer1=oversampler)
+standarizer = Standardizer()Standardizer(
+ features = Symbol[],
+ ignore = false,
+ ordered_factor = false,
+ count = false)Now let's compose the balanced model with a standardizer.
pipeline = standarizer |> balanced_modelProbabilisticPipeline(
+ standardizer = Standardizer(
+ features = Symbol[],
+ ignore = false,
+ ordered_factor = false,
+ count = false),
+ balanced_model_probabilistic = BalancedModelProbabilistic(
+ model = NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …),
+ balancer1 = BorderlineSMOTE1(m = 5, …)),
+ cache = true)By this, any training data will be standardized then oversampled then passed to the model. Meanwhile, for inference, the standardizer will automatically use the training set's mean and std and the oversampler will be transparent.
It's indistinguishable from training a single model.
mach = machine(pipeline, X, y)
+fit!(mach)
+cv=CV(nfolds=5)
+evaluate!(mach, resampling=cv, measure=accuracy)PerformanceEvaluation object with these fields:
+ model, measure, operation,
+ measurement, per_fold, per_observation,
+ fitted_params_per_fold, report_per_fold,
+ train_test_rows, resampling, repeats
+Extract:
+┌────────────┬──────────────┬─────────────┐
+│ measure │ operation │ measurement │
+├────────────┼──────────────┼─────────────┤
+│ Accuracy() │ predict_mode │ 0.99 │
+└────────────┴──────────────┴─────────────┘
+┌────────────────────────────┬─────────┐
+│ per_fold │ 1.96*SE │
+├────────────────────────────┼─────────┤
+│ [1.0, 1.0, 0.95, 1.0, 1.0] │ 0.0219 │
+└────────────────────────────┴─────────┘
+This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
This demonstration is available as a Jupyter notebook or julia script here.
In this workflow example, we learn how MLJFlux enables us to easily use early stopping when training MLJFlux models.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
+using Flux # For more flexibility
+import RDatasets # Dataset source
+using Plots # To visualize training
+import Optimisers # native Flux.jl optimisers no longer supportediris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
+X = Float32.(X); # To be compatible with type of network network parametersto the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
+
+clf = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Optimisers.Adam(0.01),
+ batch_size=8,
+ epochs=50,
+ rng=42,
+)NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 50,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))Let's start by defining the condition that can cause the model to early stop.
stop_conditions = [
+ Step(1), # Repeatedly train for one iteration
+ NumberLimit(100), # Don't train for more than 100 iterations
+ Patience(5), # Stop after 5 iterations of disimprovement in validation loss
+ NumberSinceBest(9), # Or if the best loss occurred 9 iterations ago
+ TimeLimit(30/60), # Or if 30 minutes passed
+]5-element Vector{Any}:
+ IterationControl.Step(1)
+ EarlyStopping.NumberLimit(100)
+ EarlyStopping.Patience(5)
+ EarlyStopping.NumberSinceBest(9)
+ EarlyStopping.TimeLimit(Dates.Millisecond(1800000))We can also define callbacks. Here we want to store the validation loss for each iteration
validation_losses = []
+callbacks = [
+ WithLossDo(loss->push!(validation_losses, loss)),
+]1-element Vector{IterationControl.WithLossDo{Main.var"#3#4"}}:
+ IterationControl.WithLossDo{Main.var"#3#4"}(Main.var"#3#4"(), false, nothing)Construct the iterated model and pass to it the stop_conditions and the callbacks:
iterated_model = IteratedModel(
+ model=clf,
+ resampling=Holdout(fraction_train=0.7); # loss and stopping are based on out-of-sample
+ measures=log_loss,
+ iteration_parameter=:(epochs),
+ controls=vcat(stop_conditions, callbacks),
+ retrain=false # no need to retrain on all data at the end
+);You can see more advanced stopping conditions as well as how to involve callbacks in the documentation
At this point, all we need is to fit the model and iteration controls will be automatically handled
mach = machine(iterated_model, X, y)
+fit!(mach)
+# We can get the training losses like so
+training_losses = report(mach)[:model_report].training_losses;[ Info: Training machine(ProbabilisticIteratedModel(model = NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …), …), …).
+[ Info: final loss: 0.05287897645527522
+[ Info: final training loss: 0.045833383
+[ Info: Stop triggered by EarlyStopping.NumberLimit(100) stopping criterion.
+[ Info: Total of 100 iterations.We can see that the model converged after 100 iterations.
plot(training_losses, label="Training Loss", linewidth=2)
+plot!(validation_losses, label="Validation Loss", linewidth=2, size=(800,400))
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
This demonstration is available as a Jupyter notebook or julia script here.
In this workflow example we learn how to tune different hyperparameters of MLJFlux models with emphasis on training hyperparameters.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
+using Flux # For more flexibility
+import RDatasets # Dataset source
+using Plots # To plot tuning results
+import Optimisers # native Flux.jl optimisers no longer supportediris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
+X = Float32.(X); # To be compatible with type of network network parametersNow let's construct our model. This follows a similar setup the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
+clf = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Optimisers.Adam(0.01),
+ batch_size=8,
+ epochs=10,
+ rng=42,
+)NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))Let's tune the batch size and the learning rate. We will use grid search and 5-fold cross-validation.
We start by defining the hyperparameter ranges
r1 = range(clf, :batch_size, lower=1, upper=64)
+etas = [10^x for x in range(-4, stop=0, length=4)]
+optimisers = [Optimisers.Adam(eta) for eta in etas]
+r2 = range(clf, :optimiser, values=optimisers)NominalRange(optimiser = Adam(0.0001, (0.9, 0.999), 1.0e-8), Adam(0.00215443, (0.9, 0.999), 1.0e-8), Adam(0.0464159, (0.9, 0.999), 1.0e-8), ...)Then passing the ranges along with the model and other arguments to the TunedModel constructor.
tuned_model = TunedModel(
+ model=clf,
+ tuning=Grid(goal=25),
+ resampling=CV(nfolds=5, rng=42),
+ range=[r1, r2],
+ measure=cross_entropy,
+);Then wrapping our tuned model in a machine and fitting it.
mach = machine(tuned_model, X, y);
+fit!(mach, verbosity=0);Let's check out the best performing model:
fitted_params(mach).best_modelNeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.0464159, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 1,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))With learning curves, it's possible to center our focus on the effects of a single hyperparameter of the model
First define the range and wrap it in a learning curve
r = range(clf, :epochs, lower=1, upper=200, scale=:log10)
+curve = learning_curve(
+ clf,
+ X,
+ y,
+ range=r,
+ resampling=CV(nfolds=4, rng=42),
+ measure=cross_entropy,
+)(parameter_name = "epochs",
+ parameter_scale = :log10,
+ parameter_values = [1, 2, 3, 4, 5, 6, 7, 9, 11, 13 … 39, 46, 56, 67, 80, 96, 116, 139, 167, 200],
+ measurements = [0.9231712033780419, 0.7672938542047157, 0.6736075721456418, 0.6064130950372606, 0.5595521804926612, 0.5270759259385482, 0.5048969423979114, 0.47993815474701584, 0.46130985568830307, 0.4449225600160762 … 0.1621185148276446, 0.12283639917434747, 0.09543014842693512, 0.07850181447968614, 0.06950203807005066, 0.063248279208185, 0.060053521895940286, 0.05921442672620914, 0.05921052970422136, 0.060379476300399186],)Then plot the curve
plot(
+ curve.parameter_values,
+ curve.measurements,
+ xlab=curve.parameter_name,
+ xscale=curve.parameter_scale,
+ ylab = "Cross Entropy",
+)
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
In this workflow example we explore how to incrementally train MLJFlux models.
Julia version is assumed to be 1.10.* This tutorial is available as a Jupyter notebook or julia script here.
using MLJ # Has MLJFlux models
+using Flux # For more flexibility
+import RDatasets # Dataset source
+import Optimisers # native Flux.jl optimisers no longer supportediris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
+X = Float32.(X) # To be compatible with type of network network parameters
+(X_train, X_test), (y_train, y_test) = partition(
+ (X, y), 0.8,
+ multi = true,
+ shuffle = true,
+ rng=42,
+);Now let's construct our model. This follows a similar setup to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
+clf = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Optimisers.Adam(0.01),
+ batch_size=8,
+ epochs=10,
+ rng=42,
+)NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))Now let's train the model. Calling fit! will automatically train it for 100 epochs as specified above.
mach = machine(clf, X_train, y_train)
+fit!(mach)trained Machine; caches model-specific representations of data
+ model: NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …)
+ args:
+ 1: Source @519 ⏎ ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}
+ 2: Source @768 ⏎ AbstractVector{ScientificTypesBase.Multiclass{3}}
+Let's evaluate the training loss and validation accuracy
training_loss = cross_entropy(predict(mach, X_train), y_train)0.4392339631006042val_acc = accuracy(predict_mode(mach, X_test), y_test)0.9Poor performance it seems.
Now let's train it for another 30 epochs at half the original learning rate. All we need to do is changes these hyperparameters and call fit again. It won't reset the model parameters before training.
clf.optimiser = Optimisers.Adam(clf.optimiser.eta/2)
+clf.epochs = clf.epochs + 30
+fit!(mach, verbosity=2);[ Info: Updating machine(NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …), …).
+[ Info: Loss is 0.4393
+[ Info: Loss is 0.4317
+[ Info: Loss is 0.4244
+[ Info: Loss is 0.4171
+[ Info: Loss is 0.4096
+[ Info: Loss is 0.4017
+[ Info: Loss is 0.3931
+[ Info: Loss is 0.3838
+[ Info: Loss is 0.3737
+[ Info: Loss is 0.3626
+[ Info: Loss is 0.3505
+[ Info: Loss is 0.3382
+[ Info: Loss is 0.3244
+[ Info: Loss is 0.3095
+[ Info: Loss is 0.2954
+[ Info: Loss is 0.2813
+[ Info: Loss is 0.2654
+[ Info: Loss is 0.25
+[ Info: Loss is 0.235
+[ Info: Loss is 0.2203
+[ Info: Loss is 0.2118
+[ Info: Loss is 0.196
+[ Info: Loss is 0.179
+[ Info: Loss is 0.1674
+[ Info: Loss is 0.1586
+[ Info: Loss is 0.1469
+[ Info: Loss is 0.1353
+[ Info: Loss is 0.1251
+[ Info: Loss is 0.1173
+[ Info: Loss is 0.1102Let's evaluate the training loss and validation accuracy
training_loss = cross_entropy(predict(mach, X_train), y_train)0.10519664737051289training_acc = accuracy(predict_mode(mach, X_test), y_test)0.9666666666666667That's much better. If we are rather interested in resetting the model parameters before fitting, we can do fit(mach, force=true).
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
This tutorial is available as a Jupyter notebook or julia script here.
Julia version is assumed to be 1.10.*
using MLJ
+using Flux
+import RDatasets
+import Optimisersusing Plotsiris = RDatasets.dataset("datasets", "iris");
+y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
+X = Float32.(X); # To be compatible with type of network network parametersNow let's construct our model. This follows a similar setup to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
+
+clf = NeuralNetworkClassifier(
+ builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
+ optimiser=Optimisers.Adam(0.01),
+ batch_size=8,
+ epochs=50,
+ rng=42,
+)NeuralNetworkClassifier(
+ builder = MLP(
+ hidden = (5, 4),
+ σ = NNlib.relu),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 50,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))Now let's wrap this in an iterated model. We will use a callback that makes a plot for validation losses each iteration.
stop_conditions = [
+ Step(1), # Repeatedly train for one iteration
+ NumberLimit(100), # Don't train for more than 100 iterations
+]
+
+validation_losses = []
+gr(reuse=true) # use the same window for plots
+function plot_loss(loss)
+ push!(validation_losses, loss)
+ display(plot(validation_losses, label="validation loss", xlim=(1, 100)))
+ sleep(.01) # to catch up with the plots while they are being generated
+end
+
+callbacks = [ WithLossDo(plot_loss),]
+
+iterated_model = IteratedModel(
+ model=clf,
+ resampling=Holdout(),
+ measures=log_loss,
+ iteration_parameter=:(epochs),
+ controls=vcat(stop_conditions, callbacks),
+ retrain=true,
+)ProbabilisticIteratedModel(
+ model = NeuralNetworkClassifier(
+ builder = MLP(hidden = (5, 4), …),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 50,
+ batch_size = 8,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 42,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing)),
+ controls = Any[IterationControl.Step(1), EarlyStopping.NumberLimit(100), IterationControl.WithLossDo{typeof(Main.plot_loss)}(Main.plot_loss, false, nothing)],
+ resampling = Holdout(
+ fraction_train = 0.7,
+ shuffle = false,
+ rng = Random._GLOBAL_RNG()),
+ measure = LogLoss(tol = 2.22045e-16),
+ weights = nothing,
+ class_weights = nothing,
+ operation = MLJModelInterface.predict,
+ retrain = true,
+ check_measure = true,
+ iteration_parameter = :epochs,
+ cache = true)Simply fitting the model is all we need
mach = machine(iterated_model, X, y)
+fit!(mach, force=true)trained Machine; does not cache data
+ model: ProbabilisticIteratedModel(model = NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …), …)
+ args:
+ 1: Source @963 ⏎ ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}
+ 2: Source @713 ⏎ AbstractVector{ScientificTypesBase.Multiclass{3}}
+This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
This section assumes familiarity with the MLJ model API
If one subtypes a new model type as either MLJFlux.MLJFluxProbabilistic or MLJFlux.MLJFluxDeterministic, then instead of defining new methods for MLJModelInterface.fit and MLJModelInterface.update one can make use of fallbacks by implementing the lower level methods shape, build, and fitresult. See the classifier source code for an example.
One still needs to implement a new predict method.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
This section assumes familiarity with the MLJ model API
If one subtypes a new model type as either MLJFlux.MLJFluxProbabilistic or MLJFlux.MLJFluxDeterministic, then instead of defining new methods for MLJModelInterface.fit and MLJModelInterface.update one can make use of fallbacks by implementing the lower level methods shape, build, and fitresult. See the classifier source code for an example.
One still needs to implement a new predict method.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
This tutorial is available as a Jupyter notebook or julia script here.
Julia version is assumed to be 1.10.*
using MLJ
+using Flux
+import MLJFlux
+import MLUtils
+import MLJIteration # for `skip`If running on a GPU, you will also need to import CUDA and import cuDNN.
using Plots
+gr(size=(600, 300*(sqrt(5)-1)));Downloading the MNIST image dataset:
import MLDatasets: MNIST
+
+ENV["DATADEPS_ALWAYS_ACCEPT"] = true
+images, labels = MNIST(split=:train)[:];In MLJ, integers cannot be used for encoding categorical data, so we must force the labels to have the Multiclass scientific type. For more on this, see Working with Categorical Data.
labels = coerce(labels, Multiclass);
+images = coerce(images, GrayImage);Checking scientific types:
@assert scitype(images) <: AbstractVector{<:Image}
+@assert scitype(labels) <: AbstractVector{<:Finite}Looks good.
For general instructions on coercing image data, see Type coercion for image data
images[1]
We start by defining a suitable Builder object. This is a recipe for building the neural network. Our builder will work for images of any (constant) size, whether they be color or black and white (ie, single or multi-channel). The architecture always consists of six alternating convolution and max-pool layers, and a final dense layer; the filter size and the number of channels after each convolution layer is customisable.
import MLJFlux
+struct MyConvBuilder
+ filter_size::Int
+ channels1::Int
+ channels2::Int
+ channels3::Int
+end
+
+function MLJFlux.build(b::MyConvBuilder, rng, n_in, n_out, n_channels)
+ k, c1, c2, c3 = b.filter_size, b.channels1, b.channels2, b.channels3
+ mod(k, 2) == 1 || error("`filter_size` must be odd. ")
+ p = div(k - 1, 2) # padding to preserve image size
+ init = Flux.glorot_uniform(rng)
+ front = Chain(
+ Conv((k, k), n_channels => c1, pad=(p, p), relu, init=init),
+ MaxPool((2, 2)),
+ Conv((k, k), c1 => c2, pad=(p, p), relu, init=init),
+ MaxPool((2, 2)),
+ Conv((k, k), c2 => c3, pad=(p, p), relu, init=init),
+ MaxPool((2 ,2)),
+ MLUtils.flatten)
+ d = Flux.outputsize(front, (n_in..., n_channels, 1)) |> first
+ return Chain(front, Dense(d, n_out, init=init))
+endNotes.
There is no final softmax here, as this is applied by default in all MLJFLux classifiers. Customisation of this behaviour is controlled using using the finaliser hyperparameter of the classifier.
Instead of calculating the padding p, Flux can infer the required padding in each dimension, which you enable by replacing pad = (p, p) with pad = SamePad().
We now define the MLJ model.
ImageClassifier = @load ImageClassifier
+clf = ImageClassifier(
+ builder=MyConvBuilder(3, 16, 32, 32),
+ batch_size=50,
+ epochs=10,
+ rng=123,
+)ImageClassifier(
+ builder = Main.MyConvBuilder(3, 16, 32, 32),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.001, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 50,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 123,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))You can add Flux options optimiser=... and loss=... in the above constructor call. At present, loss must be a Flux-compatible loss, not an MLJ measure. To run on a GPU, add to the constructor acceleration=CUDALib() and omit rng.
For illustration purposes, we won't use all the data here:
train = 1:500
+test = 501:1000501:1000Binding the model with data in an MLJ machine:
mach = machine(clf, images, labels);Training for 10 epochs on the first 500 images:
fit!(mach, rows=train, verbosity=2);[ Info: Training machine(ImageClassifier(builder = Main.MyConvBuilder(3, 16, 32, 32), …), …).
+[ Info: Loss is 2.28
+[ Info: Loss is 2.171
+[ Info: Loss is 1.942
+[ Info: Loss is 1.505
+[ Info: Loss is 0.9922
+[ Info: Loss is 0.6912
+[ Info: Loss is 0.5584
+[ Info: Loss is 0.4542
+[ Info: Loss is 0.3809
+[ Info: Loss is 0.3272Inspecting:
report(mach)(training_losses = Float32[2.3174262, 2.280439, 2.1711705, 1.9420795, 1.5045885, 0.99224484, 0.69117606, 0.5583703, 0.45424515, 0.38085267, 0.3271538],)chain = fitted_params(mach)(chain = Chain(Chain(Chain(Conv((3, 3), 1 => 16, relu, pad=1), MaxPool((2, 2)), Conv((3, 3), 16 => 32, relu, pad=1), MaxPool((2, 2)), Conv((3, 3), 32 => 32, relu, pad=1), MaxPool((2, 2)), flatten), Dense(288 => 10)), softmax),)Flux.params(chain)[2]16-element Vector{Float32}:
+ 0.003225543
+ 0.019304937
+ 0.062040687
+ 0.024518687
+ 0.05317823
+ 0.069572166
+ 0.044410173
+ 0.024950704
+ 0.015806748
+ 0.015081032
+ 0.017513964
+ 0.02133927
+ 0.040562775
+ 0.0018777152
+ 0.055122323
+ 0.057923194Adding 20 more epochs:
clf.epochs = clf.epochs + 20
+fit!(mach, rows=train);[ Info: Updating machine(ImageClassifier(builder = Main.MyConvBuilder(3, 16, 32, 32), …), …).
+
Optimising neural net: 10%[==> ] ETA: 0:00:07
Optimising neural net: 14%[===> ] ETA: 0:00:08
Optimising neural net: 19%[====> ] ETA: 0:00:08
Optimising neural net: 24%[=====> ] ETA: 0:00:08
Optimising neural net: 29%[=======> ] ETA: 0:00:07
Optimising neural net: 33%[========> ] ETA: 0:00:07
Optimising neural net: 38%[=========> ] ETA: 0:00:07
Optimising neural net: 43%[==========> ] ETA: 0:00:06
Optimising neural net: 48%[===========> ] ETA: 0:00:05
Optimising neural net: 52%[=============> ] ETA: 0:00:05
Optimising neural net: 57%[==============> ] ETA: 0:00:04
Optimising neural net: 62%[===============> ] ETA: 0:00:04
Optimising neural net: 67%[================> ] ETA: 0:00:04
Optimising neural net: 71%[=================> ] ETA: 0:00:03
Optimising neural net: 76%[===================> ] ETA: 0:00:03
Optimising neural net: 81%[====================> ] ETA: 0:00:02
Optimising neural net: 86%[=====================> ] ETA: 0:00:02
Optimising neural net: 90%[======================> ] ETA: 0:00:01
Optimising neural net: 95%[=======================> ] ETA: 0:00:01
Optimising neural net: 100%[=========================] Time: 0:00:10Computing an out-of-sample estimate of the loss:
predicted_labels = predict(mach, rows=test);
+cross_entropy(predicted_labels, labels[test])0.4883231265583621Or to fit and predict, in one line:
evaluate!(mach,
+ resampling=Holdout(fraction_train=0.5),
+ measure=cross_entropy,
+ rows=1:1000,
+ verbosity=0)PerformanceEvaluation object with these fields:
+ model, measure, operation,
+ measurement, per_fold, per_observation,
+ fitted_params_per_fold, report_per_fold,
+ train_test_rows, resampling, repeats
+Extract:
+┌──────────────────────┬───────────┬─────────────┐
+│ measure │ operation │ measurement │
+├──────────────────────┼───────────┼─────────────┤
+│ LogLoss( │ predict │ 0.488 │
+│ tol = 2.22045e-16) │ │ │
+└──────────────────────┴───────────┴─────────────┘
+Any iterative MLJFlux model can be wrapped in iteration controls, as we demonstrate next. For more on MLJ's IteratedModel wrapper, see the MLJ documentation.
The "self-iterating" classifier, called iterated_clf below, is for iterating the image classifier defined above until one of the following stopping criterion apply:
Patience(3): 3 consecutive increases in the lossInvalidValue(): an out-of-sample loss, or a training loss, is NaN, Inf, or -InfTimeLimit(t=5/60): training time has exceeded 5 minutesThese checks (and other controls) will be applied every two epochs (because of the Step(2) control). Additionally, training a machine bound to iterated_clf will:
For a complete list of controls, see this table.
Some helpers
To extract Flux params from an MLJFlux machine
parameters(mach) = vec.(Flux.params(fitted_params(mach)));To store the traces:
losses = []
+training_losses = []
+parameter_means = Float32[];
+epochs = []Any[]To update the traces:
update_loss(loss) = push!(losses, loss)
+update_training_loss(losses) = push!(training_losses, losses[end])
+update_means(mach) = append!(parameter_means, mean.(parameters(mach)));
+update_epochs(epoch) = push!(epochs, epoch)update_epochs (generic function with 1 method)The controls to apply:
save_control =
+ MLJIteration.skip(Save(joinpath(tempdir(), "mnist.jls")), predicate=3)
+
+controls=[
+ Step(2),
+ Patience(3),
+ InvalidValue(),
+ TimeLimit(5/60),
+ save_control,
+ WithLossDo(),
+ WithLossDo(update_loss),
+ WithTrainingLossesDo(update_training_loss),
+ Callback(update_means),
+ WithIterationsDo(update_epochs),
+];The "self-iterating" classifier:
iterated_clf = IteratedModel(
+ clf,
+ controls=controls,
+ resampling=Holdout(fraction_train=0.7),
+ measure=log_loss,
+)ProbabilisticIteratedModel(
+ model = ImageClassifier(
+ builder = Main.MyConvBuilder(3, 16, 32, 32),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.001, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 30,
+ batch_size = 50,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = 123,
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing)),
+ controls = Any[IterationControl.Step(2), EarlyStopping.Patience(3), EarlyStopping.InvalidValue(), EarlyStopping.TimeLimit(Dates.Millisecond(300000)), IterationControl.Skip{MLJIteration.Save{typeof(Serialization.serialize)}, IterationControl.var"#8#9"{Int64}}(MLJIteration.Save{typeof(Serialization.serialize)}("/tmp/mnist.jls", Serialization.serialize), IterationControl.var"#8#9"{Int64}(3)), IterationControl.WithLossDo{IterationControl.var"#20#22"}(IterationControl.var"#20#22"(), false, nothing), IterationControl.WithLossDo{typeof(Main.update_loss)}(Main.update_loss, false, nothing), IterationControl.WithTrainingLossesDo{typeof(Main.update_training_loss)}(Main.update_training_loss, false, nothing), IterationControl.Callback{typeof(Main.update_means)}(Main.update_means, false, nothing, false), MLJIteration.WithIterationsDo{typeof(Main.update_epochs)}(Main.update_epochs, false, nothing)],
+ resampling = Holdout(
+ fraction_train = 0.7,
+ shuffle = false,
+ rng = Random._GLOBAL_RNG()),
+ measure = LogLoss(tol = 2.22045e-16),
+ weights = nothing,
+ class_weights = nothing,
+ operation = MLJModelInterface.predict,
+ retrain = false,
+ check_measure = true,
+ iteration_parameter = nothing,
+ cache = true)mach = machine(iterated_clf, images, labels);fit!(mach, rows=train);[ Info: Training machine(ProbabilisticIteratedModel(model = ImageClassifier(builder = Main.MyConvBuilder(3, 16, 32, 32), …), …), …).
+[ Info: No iteration parameter specified. Using `iteration_parameter=:(epochs)`.
+[ Info: loss: 2.195050130190149
+[ Info: loss: 1.8450074691283658
+[ Info: Saving "/tmp/mnist1.jls".
+[ Info: loss: 1.1388123685158849
+[ Info: loss: 0.702997545486733
+[ Info: loss: 0.5778269559910739
+[ Info: Saving "/tmp/mnist2.jls".
+[ Info: loss: 0.5222495075757826
+[ Info: loss: 0.49847208228951995
+[ Info: loss: 0.4897800580510804
+[ Info: Saving "/tmp/mnist3.jls".
+[ Info: loss: 0.4893840844808948
+[ Info: loss: 0.49094569068535143
+[ Info: loss: 0.49593260647952264
+[ Info: Saving "/tmp/mnist4.jls".
+[ Info: loss: 0.5062357308150314
+[ Info: final loss: 0.5062357308150314
+[ Info: final training loss: 0.059303638
+[ Info: Stop triggered by EarlyStopping.Patience(3) stopping criterion.
+[ Info: Total of 24 iterations.plot(
+ epochs,
+ losses,
+ xlab = "epoch",
+ ylab = "cross entropy",
+ label="out-of-sample",
+)
+plot!(epochs, training_losses, label="training")
+
+savefig(joinpath(tempdir(), "loss.png"))"/tmp/loss.png"n_epochs = length(losses)
+n_parameters = div(length(parameter_means), n_epochs)
+parameter_means2 = reshape(copy(parameter_means), n_parameters, n_epochs)'
+plot(
+ epochs,
+ parameter_means2,
+ title="Flux parameter mean weights",
+ xlab = "epoch",
+)
Note. The higher the number in the plot legend, the deeper the layer we are **weight-averaging.
savefig(joinpath(tempdir(), "weights.png"))"/tmp/weights.png"mach2 = machine(joinpath(tempdir(), "mnist3.jls"))
+predict_mode(mach2, images[501:503])3-element CategoricalArrays.CategoricalArray{Int64,1,UInt32}:
+ 7
+ 9
+ 5Mutating iterated_clf.controls or clf.epochs (which is otherwise ignored) will allow you to restart training from where it left off.
iterated_clf.controls[2] = Patience(4)
+fit!(mach, rows=train)
+
+plot(
+ epochs,
+ losses,
+ xlab = "epoch",
+ ylab = "cross entropy",
+ label="out-of-sample",
+)
+plot!(epochs, training_losses, label="training")
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| file | description |
|---|---|
notebook.ipynb | Juptyer notebook (executed) |
notebook.unexecuted.ipynb | Jupyter notebook (unexecuted) |
notebook.md | static markdown (included in MLJFlux.jl docs) |
notebook.jl | executable Julia script annotated with comments |
generate.jl | maintainers only: execute to generate first 3 from 4th |
Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| Row | Category | Message |
|---|---|---|
| String7 | String | |
| 1 | ham | Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat... |
| 2 | ham | Ok lar... Joking wif u oni... |
| 3 | spam | Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's |
| 4 | ham | U dun say so early hor... U c already then say... |
| 5 | ham | Nah I don't think he goes to usf, he lives around here though |
This demonstration is available as a Jupyter notebook or julia script here.
In this demo we use a custom RNN model from Flux with MLJFlux to classify text messages as spam or ham. We will be using the SMS Collection Dataset from Kaggle.
Warning. This demo includes some non-idiomatic use of MLJ to allow use of the Flux.jl Embedding layer. It is not recommended for MLJ beginners.
using MLJ
+using MLJFlux
+using Flux
+import Optimisers # Flux.jl native optimisers no longer supported
+using CSV # Read data
+using DataFrames # Read data
+using WordTokenizers # For tokenization
+using Languages # For stop wordsWe assume the SMS Collection Dataset has been downloaded and is in a file called "sms.csv" in the same directory as the this script.
df = CSV.read(joinpath(@__DIR__, "sms.csv"), DataFrame);Display the first 5 rows with DataFrames
first(df, 5)| Row | Category | Message |
|---|---|---|
| String7 | String | |
| 1 | ham | Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat... |
| 2 | ham | Ok lar... Joking wif u oni... |
| 3 | spam | Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's |
| 4 | ham | U dun say so early hor... U c already then say... |
| 5 | ham | Nah I don't think he goes to usf, he lives around here though |
Let's define a function that given an SMS message would:
Tokenize it (i.e., convert it into a vector of words)
Remove stop words (i.e., words that are not useful for the analysis, like "the", "a", etc.)
Return the filtered vector of words
const STOP_WORDS = Languages.stopwords(Languages.English())
+
+function preprocess_text(text)
+ # (1) Splitting texts into words (so later it can be a sequence of vectors)
+ tokens = WordTokenizers.tokenize(text)
+
+ # (2) Stop word removal
+ filtered_tokens = filter(token -> !(token in STOP_WORDS), tokens)
+
+ return filtered_tokens
+endpreprocess_text (generic function with 1 method)Define the vocabulary to be the set of all words in our training set. We also need a function that would map each word in a given sequence of words into its index in the dictionary (which is equivalent to representing the words as one-hot vectors).
Now after we do this the sequences will all be numerical vectors but they will be of unequal length. Thus, to facilitate batching of data for the deep learning model, we need to decide on a specific maximum length for all sequences and:
If a sequence is longer than the maximum length, we need to truncate it
If a sequence is shorter than the maximum length, we need to pad it with a new token
Lastly, we must also handle the case that an incoming text sequence may involve words never seen in training by represent all such out-of-vocabulary words with a new token.
We will define a function that would do this for us.
function encode_and_equalize(text_seq, vocab_dict, max_length, pad_val, oov_val)
+ # (1) encode using the vocabulary
+ text_seq_inds = [get(vocab_dict, word, oov_val) for word in text_seq]
+
+ # (2) truncate sequence if > max_length
+ length(text_seq_inds) > max_length && (text_seq_inds = text_seq_inds[1:max_length])
+
+ # (3) pad with pad_val
+ text_seq_inds = vcat(text_seq_inds, fill(pad_val, max_length - length(text_seq_inds)))
+
+ return text_seq_inds
+endencode_and_equalize (generic function with 1 method)Splitting the data
x_data, y_data = unpack(df, ==(:Message), ==(:Category))
+y_data = coerce(y_data, Multiclass);
+
+(x_train, x_val), (y_train, y_val) = partition(
+ (x_data, y_data),
+ 0.8,
+ multi = true,
+ shuffle = true,
+ rng = 42,
+);Now let's process the training and validation sets:
x_train_processed = [preprocess_text(text) for text in x_train]
+x_val_processed = [preprocess_text(text) for text in x_val];sanity check
println(x_train_processed[1], " is ", y_data[1])["Que", "pases", "un", "buen", "tiempo"] is hamDefine the vocabulary from the training data
vocab = unique(vcat(x_train_processed...))
+vocab_dict = Dict(word => idx for (idx, word) in enumerate(vocab))
+vocab_size = length(vocab)
+pad_val, oov_val = vocab_size + 1, vocab_size + 2
+max_length = 12 # can choose this more smartly if you wish12Encode and equalize training and validation data:
x_train_processed_equalized = [
+ encode_and_equalize(seq, vocab_dict, max_length, pad_val, oov_val) for
+ seq in x_train_processed
+ ]
+x_val_processed_equalized = [
+ encode_and_equalize(seq, vocab_dict, max_length, pad_val, oov_val) for
+ seq in x_val_processed
+ ]
+x_train_processed_equalized[1:5] # all sequences are encoded and of the same length5-element Vector{Vector{Int64}}:
+ [1, 2, 3, 4, 5, 10404, 10404, 10404, 10404, 10404, 10404, 10404]
+ [6, 7, 8, 9, 10, 11, 12, 13, 11, 14, 15, 16]
+ [36, 37, 38, 39, 36, 40, 41, 42, 10404, 10404, 10404, 10404]
+ [43, 24, 36, 44, 45, 46, 10404, 10404, 10404, 10404, 10404, 10404]
+ [43, 47, 48, 49, 50, 51, 52, 53, 54, 55, 44, 45]Convert both structures into matrix form:
matrixify(v) = reduce(hcat, v)'
+x_train_processed_equalized_fixed = matrixify(x_train_processed_equalized)
+x_val_processed_equalized_fixed = matrixify(x_val_processed_equalized)
+size(x_train_processed_equalized_fixed)(4458, 12)For the model, we will use a RNN from Flux. We will average the hidden states corresponding to any sequence then pass that to a dense layer for classification.
For this, we need to define a custom Flux layer to perform the averaging operation:
struct Mean end
+Flux.@layer Mean
+(m::Mean)(x) = mean(x, dims = 2)[:, 1, :] # [batch_size, seq_len, hidden_dim] => [batch_size, 1, hidden_dim]=> [batch_size, hidden_dim]For compatibility, we will also define a layer that simply casts the input to integers as the embedding layer in Flux expects integers but the MLJFlux model expects floats:
struct Intify end
+Flux.@layer Intify
+(m::Intify)(x) = Int.(x)Here we define our network:
builder = MLJFlux.@builder begin
+ Chain(
+ Intify(), # Cast input to integer
+ Embedding(vocab_size + 2 => 300), # Embedding layer
+ RNN(300, 50, tanh), # RNN layer
+ Mean(), # Mean pooling layer
+ Dense(50, 2), # Classification dense layer
+ )
+endGenericBuilder(apply = #15)
+Notice that we used an embedding layer with input dimensionality vocab_size + 2 to take into account the padding and out-of-vocabulary tokens. Recall that the indices in our input correspond to one-hot-vectors and the embedding layer's purpose is to learn to map them into meaningful dense vectors (of dimensionality 300 here).
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg = MLJFlux
+clf = NeuralNetworkClassifier(
+ builder = builder,
+ optimiser = Optimisers.Adam(0.1),
+ batch_size = 128,
+ epochs = 10,
+)NeuralNetworkClassifier(
+ builder = GenericBuilder(
+ apply = Main.var"#15#16"()),
+ finaliser = NNlib.softmax,
+ optimiser = Adam(0.1, (0.9, 0.999), 1.0e-8),
+ loss = Flux.Losses.crossentropy,
+ epochs = 10,
+ batch_size = 128,
+ lambda = 0.0,
+ alpha = 0.0,
+ rng = Random.TaskLocalRNG(),
+ optimiser_changes_trigger_retraining = false,
+ acceleration = CPU1{Nothing}(nothing))x_train_processed_equalized_fixed = coerce(x_train_processed_equalized_fixed, Continuous)
+mach = machine(clf, x_train_processed_equalized_fixed, y_train)untrained Machine; caches model-specific representations of data
+ model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)
+ args:
+ 1: Source @759 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}
+ 2: Source @873 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}
+fit!(mach)trained Machine; caches model-specific representations of data
+ model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)
+ args:
+ 1: Source @759 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}
+ 2: Source @873 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}
+ŷ = predict_mode(mach, x_val_processed_equalized_fixed)
+balanced_accuracy(ŷ, y_val)0.8840999384477648Acceptable performance. Let's see some live examples:
using Random: Random;
+Random.seed!(99);
+
+z = rand(x_val)
+z_processed = preprocess_text(z)
+z_encoded_equalized =
+ encode_and_equalize(z_processed, vocab_dict, max_length, pad_val, oov_val)
+z_encoded_equalized_fixed = matrixify([z_encoded_equalized])
+z_encoded_equalized_fixed = coerce(z_encoded_equalized_fixed, Continuous)
+z_pred = predict_mode(mach, z_encoded_equalized_fixed)
+
+print("SMS: `$(z)` and the prediction is `$(z_pred)`")SMS: `Hi elaine, is today's meeting confirmed?` and the prediction is `CategoricalArrays.CategoricalValue{InlineStrings.String7, UInt32}[InlineStrings.String7("ham")]`This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
An expanded version of this example, with early stopping and snapshots, is available here.
We define a builder that builds a chain with six alternating convolution and max-pool layers, and a final dense layer, which we apply to the MNIST image dataset.
First we define a generic builder (working for any image size, color or gray):
using MLJ
-using Flux
-using MLDatasets
-
-# helper function
-function flatten(x::AbstractArray)
- return reshape(x, :, size(x)[end])
-end
-
-import MLJFlux
-mutable struct MyConvBuilder
- filter_size::Int
- channels1::Int
- channels2::Int
- channels3::Int
-end
-
-function MLJFlux.build(b::MyConvBuilder, rng, n_in, n_out, n_channels)
-
- k, c1, c2, c3 = b.filter_size, b.channels1, b.channels2, b.channels3
-
- mod(k, 2) == 1 || error("`filter_size` must be odd. ")
-
- # padding to preserve image size on convolution:
- p = div(k - 1, 2)
-
- front = Chain(
- Conv((k, k), n_channels => c1, pad=(p, p), relu),
- MaxPool((2, 2)),
- Conv((k, k), c1 => c2, pad=(p, p), relu),
- MaxPool((2, 2)),
- Conv((k, k), c2 => c3, pad=(p, p), relu),
- MaxPool((2 ,2)),
- flatten,
- )
- d = Flux.outputsize(front, (n_in..., n_channels, 1)) |> first
- return Chain(front, Dense(d, n_out))
-endNext, we load some of the MNIST data and check scientific types conform to those is the table above:
N = 500
-Xraw, yraw = MNIST(split=:train)[:];
-Xraw = Xraw[:,:,1:N];
-yraw = yraw[1:N];
-
-scitype(Xraw)scitype(yraw)Inputs should have element scitype GrayImage:
X = coerce(Xraw, GrayImage);For classifiers, target must have element scitype <: Finite:
y = coerce(yraw, Multiclass);Instantiating an image classifier model:
ImageClassifier = @load ImageClassifier
-clf = ImageClassifier(
- builder=MyConvBuilder(3, 16, 32, 32),
- epochs=10,
- loss=Flux.crossentropy,
- )And evaluating the accuracy of the model on a 30% holdout set:
mach = machine(clf, X, y)
-
-evaluate!(
- mach,
- resampling=Holdout(rng=123, fraction_train=0.7),
- measure=misclassification_rate,
- )Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this tutorial we use a custom RNN model from Flux with MLJFlux to classify text messages as spam or ham. We will be using the SMS Collection Dataset from Kaggle.
using MLJ
-using MLJFlux
-using Flux
-using CSV # Read data
-using DataFrames # Read data
-using ScientificTypes # Type coercion
-using WordTokenizers # For tokenization
-using Languages # For stop wordsdf = CSV.read("./sms.csv", DataFrame);Display the first 5 rows with DataFrames
first(df, 5) |> pretty┌──────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
-│ Category │ Message │
-│ String7 │ String │
-│ Textual │ Textual │
-├──────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
-│ ham │ Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat... │
-│ ham │ Ok lar... Joking wif u oni... │
-│ spam │ Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's │
-│ ham │ U dun say so early hor... U c already then say... │
-│ ham │ Nah I don't think he goes to usf, he lives around here though │
-└──────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
-Let's define a function that given an SMS message would:
Tokenize it (i.e., convert it into a vector of words)
Remove stop words (i.e., words that are not useful for the analysis, like "the", "a", etc.)
Return the filtered vector of words
function preprocess_text(text)
- # (1) Splitting texts into words (so later it can be a sequence of vectors)
- tokens = WordTokenizers.tokenize(text)
-
- # (2) Stop word removal
- stop_words = Languages.stopwords(Languages.English())
- filtered_tokens = filter(token -> !(token in stop_words), tokens)
-
- return filtered_tokens
-endpreprocess_text (generic function with 1 method)Define the vocabulary to be the set of all words in our training set. We also need a function that would map each word in a given sequence of words into its index in the dictionary (which is equivalent to representing the words as one-hot vectors).
Now after we do this the sequences will all be numerical vectors but they will be of unequal length. Thus, to facilitate batching of data for the deep learning model, we need to decide on a specific maximum length for all sequences and:
If a sequence is longer than the maximum length, we need to truncate it
If a sequence is shorter than the maximum length, we need to pad it with a new token
Lastly, we must also handle the case that an incoming text sequence may involve words never seen in training by represent all such out-of-vocabulary words with a new token.
We will define a function that would do this for us.
function encode_and_equalize(text_seq, vocab_dict, max_length, pad_val, oov_val)
- # (1) encode using the vocabulary
- text_seq_inds = [get(vocab_dict, word, oov_val) for word in text_seq]
-
- # (2) truncate sequence if > max_length
- length(text_seq_inds) > max_length && (text_seq_inds = text_seq_inds[1:max_length])
-
- # (3) pad with pad_val
- text_seq_inds = vcat(text_seq_inds, fill(pad_val, max_length - length(text_seq_inds)))
-
- return text_seq_inds
-endencode_and_equalize (generic function with 1 method)Splitting the data
x_data, y_data = unpack(df, ==(:Message), ==(:Category))
-y_data = coerce(y_data, Multiclass);
-
-(x_train, x_val), (y_train, y_val) = partition((x_data, y_data), 0.8,
- multi = true,
- shuffle = true,
- rng = 42);Now let's process the training and validation sets:
x_train_processed = [preprocess_text(text) for text in x_train]
-x_val_processed = [preprocess_text(text) for text in x_val];sanity check
println(x_train_processed[1], " is ", y_data[1])["Que", "pases", "un", "buen", "tiempo"] is ham
-Define the vocabulary from the training data
vocab = unique(vcat(x_train_processed...))
-vocab_dict = Dict(word => idx for (idx, word) in enumerate(vocab))
-vocab_size = length(vocab)
-pad_val, oov_val = vocab_size + 1, vocab_size + 2
-max_length = 12 # can choose this more smartly if you wish12Encode and equalize training and validation data:
x_train_processed_equalized = [
- encode_and_equalize(seq, vocab_dict, max_length, pad_val, oov_val) for
- seq in x_train_processed
-]
-x_val_processed_equalized = [
- encode_and_equalize(seq, vocab_dict, max_length, pad_val, oov_val) for
- seq in x_val_processed
-]
-x_train_processed_equalized[1:5] # all sequences are encoded and of the same length5-element Vector{Vector{Int64}}:
- [1, 2, 3, 4, 5, 10404, 10404, 10404, 10404, 10404, 10404, 10404]
- [6, 7, 8, 9, 10, 11, 12, 13, 11, 14, 15, 16]
- [36, 37, 38, 39, 36, 40, 41, 42, 10404, 10404, 10404, 10404]
- [43, 24, 36, 44, 45, 46, 10404, 10404, 10404, 10404, 10404, 10404]
- [43, 47, 48, 49, 50, 51, 52, 53, 54, 55, 44, 45]Convert both structures into matrix form:
matrixify(v) = reduce(hcat, v)'
-x_train_processed_equalized_fixed = matrixify(x_train_processed_equalized)
-x_val_processed_equalized_fixed = matrixify(x_val_processed_equalized)
-size(x_train_processed_equalized_fixed)(4458, 12)For the model, we will use a RNN from Flux. We will average the hidden states corresponding to any sequence then pass that to a dense layer for classification.
For this, we need to define a custom Flux layer to perform the averaging operation:
struct Mean end
-Flux.@layer Mean
-(m::Mean)(x) = mean(x, dims = 2)[:, 1, :] # [batch_size, seq_len, hidden_dim] => [batch_size, 1, hidden_dim]=> [batch_size, hidden_dim]For compatibility, we will also define a layer that simply casts the input to integers as the embedding layer in Flux expects integets but the MLJFlux model expects floats:
struct Intify end
-Flux.@layer Intify
-(m::Intify)(x) = Int.(x)Here we define out network:
builder = MLJFlux.@builder begin
- Chain(
- Intify(), # Cast input to integer
- Embedding(vocab_size + 2 => 300), # Embedding layer
- RNN(300, 50, tanh), # RNN layer
- Mean(), # Mean pooling layer
- Dense(50, 2) # Classification dense layer
- )
-endGenericBuilder(apply = #15)
-Notice that we used an embedding layer with input dimensionality vocab_size + 2 to take into account the padding and out-of-vocabulary tokens. Recall that the indices in our input correspond to one-hot-vectors and the embedding layer's purpose is to learn to map them into meaningful dense vectors (of dimensionality 300 here).
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg = MLJFlux
-clf = NeuralNetworkClassifier(
- builder = builder,
- optimiser = Flux.ADAM(0.1),
- batch_size = 128,
- epochs = 10,
-)NeuralNetworkClassifier(
- builder = GenericBuilder(
- apply = Main.var"##445".var"#15#16"()),
- finaliser = NNlib.softmax,
- optimiser = Flux.Optimise.Adam(0.1, (0.9, 0.999), 1.0e-8, IdDict{Any, Any}()),
- loss = Flux.Losses.crossentropy,
- epochs = 10,
- batch_size = 128,
- lambda = 0.0,
- alpha = 0.0,
- rng = Random._GLOBAL_RNG(),
- optimiser_changes_trigger_retraining = false,
- acceleration = ComputationalResources.CPU1{Nothing}(nothing))x_train_processed_equalized_fixed = coerce(x_train_processed_equalized_fixed, Continuous)
-mach = machine(clf, x_train_processed_equalized_fixed, y_train)untrained Machine; caches model-specific representations of data
- model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)
- args:
- 1: Source @029 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}
- 2: Source @942 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}
-fit!(mach)trained Machine; caches model-specific representations of data
- model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)
- args:
- 1: Source @029 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}
- 2: Source @942 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}
-ŷ = predict_mode(mach, x_val_processed_equalized_fixed)
-balanced_accuracy(ŷ, y_val)0.9370418555201171Acceptable performance. Let's see some live examples:
using Random: Random;
-Random.seed!(99);
-
-z = rand(x_val)
-z_processed = preprocess_text(z)
-z_encoded_equalized =
- encode_and_equalize(z_processed, vocab_dict, max_length, pad_val, oov_val)
-z_encoded_equalized_fixed = matrixify([z_encoded_equalized])
-z_encoded_equalized_fixed = coerce(z_encoded_equalized_fixed, Continuous)
-z_pred = predict_mode(mach, z_encoded_equalized_fixed)
-
-print("SMS: `$(z)` and the prediction is `$(z_pred)`")SMS: `Hi elaine, is today's meeting confirmed?` and the prediction is `CategoricalArrays.CategoricalValue{InlineStrings.String7, UInt32}[InlineStrings.String7("ham")]`This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
A Julia package integrating deep learning Flux models with MLJ.
Provide a user-friendly and high-level interface to fundamental Flux deep learning models while still being extensible by supporting custom models written with Flux
Make building deep learning models more convenient to users already familiar with the MLJ workflow
Make it easier to apply machine learning techniques provided by MLJ, including: out-of-sample performance evaluation, hyper-parameter optimization, iteration control, and more, to deep learning models
MLJFlux support is focused on fundamental and widely used deep learning models. Sophisticated architectures or techniques such as online learning, reinforcement learning, and adversarial networks are currently beyond its scope.
Also note that MLJFlux is limited to training models only when all training data fits into memory, though it still supports automatic batching of data.
import Pkg
+Introduction · MLJFlux MLJFlux.jl
A Julia package integrating deep learning Flux models with MLJ.
Objectives
Provide a user-friendly and high-level interface to fundamental Flux deep learning models while still being extensible by supporting custom models written with Flux
Make building deep learning models more convenient to users already familiar with the MLJ workflow
Make it easier to apply machine learning techniques provided by MLJ, including: out-of-sample performance evaluation, hyper-parameter optimization, iteration control, and more, to deep learning models
MLJFlux Scope MLJFlux support is focused on fundamental deep learning models for common supervised learning tasks. Sophisticated architectures and approaches, such as online learning, reinforcement learning, and adversarial networks, are currently outside its scope. Also, MLJFlux is limited to tasks where all (batches of) training data fits into memory.
Installation
import Pkg
Pkg.activate("my_environment", shared=true)
-Pkg.add(["MLJ", "MLJFlux", "Flux"])
You only need Flux if you need to build a custom architecture or experiment with different optimizers, loss functions and activations.
Quick Start
using MLJ, Flux, MLJFlux
+Pkg.add(["MLJ", "MLJFlux", "Optimisers", "Flux"])
You only need Flux if you need to build a custom architecture, or experiment with different loss or activation functions. Since MLJFlux 0.5, you must use optimisers from Optimisers.jl, as native Flux.jl optimisers are no longer supported.
Quick Start
For the following demo, you will need to additionally run Pkg.add("RDatasets").
using MLJ, Flux, MLJFlux
import RDatasets
+import Optimisers
# 1. Load Data
iris = RDatasets.dataset("datasets", "iris");
@@ -12,15 +13,31 @@
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg="MLJFlux"
clf = NeuralNetworkClassifier(
builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
+ optimiser=Optimisers.Adam(0.01),
batch_size=8,
- epochs=100,
+ epochs=100,
acceleration=CUDALibs() # For GPU support
)
-# 3. Wrap it in a machine
+# 3. Wrap it in a machine
mach = machine(clf, X, y)
# 4. Evaluate the model
cv=CV(nfolds=5)
-evaluate!(mach, resampling=cv, measure=accuracy)
As you can see we were able to use MLJ functionality (i.e., cross validation) with a Flux deep learning model. All arguments provided also have defaults.
Notice that we were also able to define the neural network in a high-level fashion by only specifying the number of neurons in each hidden layer and the activation function. Meanwhile, MLJFlux was able to infer the input and output layer as well as use a suitable default for the loss function and output activation given the classification task. Notice as well that we did not need to implement a training or prediction loop as in Flux.
Basic idea
As in the example above, any MLJFlux model has a builder hyperparameter, an object encoding instructions for creating a neural network given the data that the model eventually sees (e.g., the number of classes in a classification problem). While each MLJ model has a simple default builder, users may need to define custom builders to get optimal results, and this will require familiarity with the Flux API for defining a neural network chain.
Flux or MLJFlux?
Flux is a deep learning framework in Julia that comes with everything you need to build deep learning models (i.e., GPU support, automatic differentiation, layers, activations, losses, optimizers, etc.). MLJFlux wraps models built with Flux which provides a more high-level interface for building and training such models. More importantly, it empowers Flux models by extending their support to many common machine learning workflows that are possible via MLJ such as:
Estimating performance of your model using a holdout set or other resampling strategy (e.g., cross-validation) as measured by one or more metrics (e.g., loss functions) that may not have been used in training
Optimizing hyper-parameters such as a regularization parameter (e.g., dropout) or a width/height/nchannnels of convolution layer
Compose with other models such as introducing data pre-processing steps (e.g., missing data imputation) into a pipeline. It might make sense to include non-deep learning models in this pipeline. Other kinds of model composition could include blending predictions of a deep learner with some other kind of model (as in “model stacking”). Models composed with MLJ can be also tuned as a single unit.
Controlling iteration by adding an early stopping criterion based on an out-of-sample estimate of the loss, dynamically changing the learning rate (eg, cyclic learning rates), periodically save snapshots of the model, generate live plots of sample weights to judge training progress (as in tensor board)
- Comparing your model with a non-deep learning models
A comparable project, FastAI/FluxTraining, also provides a high-level interface for interacting with Flux models and supports a set of features that may overlap with (but not include all of) those supported by MLJFlux.
Many of the features mentioned above are showcased in the workflow examples that you can access from the sidebar.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
PerformanceEvaluation object with these fields:
+ model, measure, operation,
+ measurement, per_fold, per_observation,
+ fitted_params_per_fold, report_per_fold,
+ train_test_rows, resampling, repeats
+Extract:
+┌────────────┬──────────────┬─────────────┐
+│ measure │ operation │ measurement │
+├────────────┼──────────────┼─────────────┤
+│ Accuracy() │ predict_mode │ 0.973 │
+└────────────┴──────────────┴─────────────┘
+┌─────────────────────────────┬─────────┐
+│ per_fold │ 1.96*SE │
+├─────────────────────────────┼─────────┤
+│ [1.0, 1.0, 0.967, 0.9, 1.0] │ 0.0426 │
+└─────────────────────────────┴─────────┘
+As you can see we are able to use MLJ meta-functionality (i.e., cross validation) with a Flux deep learning model. All arguments provided have defaults.
Notice that we are also able to define the neural network in a high-level fashion by only specifying the number of neurons in each hidden layer and the activation function. Meanwhile, MLJFlux is able to infer the input and output layer as well as use a suitable default for the loss function and output activation given the classification task. Notice as well that we did not need to manually implement a training or prediction loop.
As in the example above, any MLJFlux model has a builder hyperparameter, an object encoding instructions for creating a neural network given the data that the model eventually sees (e.g., the number of classes in a classification problem). While each MLJ model has a simple default builder, users may need to define custom builders to get optimal results, and this will require familiarity with the Flux API for defining a neural network chain.
Flux is a deep learning framework in Julia that comes with everything you need to build deep learning models (i.e., GPU support, automatic differentiation, layers, activations, losses, optimizers, etc.). MLJFlux wraps models built with Flux which provides a more high-level interface for building and training such models. More importantly, it empowers Flux models by extending their support to many common machine learning workflows that are possible via MLJ such as:
Estimating performance of your model using a holdout set or other resampling strategy (e.g., cross-validation) as measured by one or more metrics (e.g., loss functions) that may not have been used in training
Optimizing hyper-parameters such as a regularization parameter (e.g., dropout) or a width/height/nchannnels of convolution layer
Compose with other models such as introducing data pre-processing steps (e.g., missing data imputation) into a pipeline. It might make sense to include non-deep learning models in this pipeline. Other kinds of model composition could include blending predictions of a deep learner with some other kind of model (as in “model stacking”). Models composed with MLJ can be also tuned as a single unit.
Controlling iteration by adding an early stopping criterion based on an out-of-sample estimate of the loss, dynamically changing the learning rate (eg, cyclic learning rates), periodically save snapshots of the model, generate live plots of sample weights to judge training progress (as in tensor board)
A comparable project, FastAI/FluxTraining, also provides a high-level interface for interacting with Flux models and supports a set of features that may overlap with (but not include all of) those supported by MLJFlux.
Many of the features mentioned above are showcased in the workflow examples that you can access from the sidebar.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
MLJFlux.Linear — TypeLinear(; σ=Flux.relu)MLJFlux builder that constructs a fully connected two layer network with activation function σ. The number of input and output nodes is determined from the data. Weights are initialized using Flux.glorot_uniform(rng), where rng is inferred from the rng field of the MLJFlux model.
MLJFlux.Short — TypeShort(; n_hidden=0, dropout=0.5, σ=Flux.sigmoid)MLJFlux builder that constructs a full-connected three-layer network using n_hidden nodes in the hidden layer and the specified dropout (defaulting to 0.5). An activation function σ is applied between the hidden and final layers. If n_hidden=0 (the default) then n_hidden is the geometric mean of the number of input and output nodes. The number of input and output nodes is determined from the data.
Each layer is initialized using Flux.glorot_uniform(rng), where rng is inferred from the rng field of the MLJFlux model.
MLJFlux.MLP — TypeMLP(; hidden=(100,), σ=Flux.relu)MLJFlux builder that constructs a Multi-layer perceptron network. The ith element of hidden represents the number of neurons in the ith hidden layer. An activation function σ is applied between each layer.
Each layer is initialized using Flux.glorot_uniform(rng), where rng is inferred from the rng field of the MLJFlux model.
MLJFlux.@builder — Macro@builder neural_netCreates a builder for neural_net. The variables rng, n_in, n_out and n_channels can be used to create builders for any random number generator rng, input and output sizes n_in and n_out and number of input channels n_channels.
Examples
julia> import MLJFlux: @builder;
+Builders · MLJFlux MLJFlux.Linear — TypeLinear(; σ=Flux.relu)
MLJFlux builder that constructs a fully connected two layer network with activation function σ. The number of input and output nodes is determined from the data. Weights are initialized using Flux.glorot_uniform(rng), where rng is inferred from the rng field of the MLJFlux model.
sourceMLJFlux.Short — TypeShort(; n_hidden=0, dropout=0.5, σ=Flux.sigmoid)
MLJFlux builder that constructs a full-connected three-layer network using n_hidden nodes in the hidden layer and the specified dropout (defaulting to 0.5). An activation function σ is applied between the hidden and final layers. If n_hidden=0 (the default) then n_hidden is the geometric mean of the number of input and output nodes. The number of input and output nodes is determined from the data.
Each layer is initialized using Flux.glorot_uniform(rng), where rng is inferred from the rng field of the MLJFlux model.
sourceMLJFlux.MLP — TypeMLP(; hidden=(100,), σ=Flux.relu)
MLJFlux builder that constructs a Multi-layer perceptron network. The ith element of hidden represents the number of neurons in the ith hidden layer. An activation function σ is applied between each layer.
Each layer is initialized using Flux.glorot_uniform(rng), where rng is inferred from the rng field of the MLJFlux model.
sourceMLJFlux.@builder — Macro@builder neural_net
Creates a builder for neural_net. The variables rng, n_in, n_out and n_channels can be used to create builders for any random number generator rng, input and output sizes n_in and n_out and number of input channels n_channels.
Examples
julia> import MLJFlux: @builder;
julia> nn = NeuralNetworkRegressor(builder = @builder(Chain(Dense(n_in, 64, relu),
Dense(64, 32, relu),
@@ -11,4 +11,4 @@
Chain(front, Dense(d, n_out));
end
-julia> conv_nn = NeuralNetworkRegressor(builder = conv_builder);
sourceSettings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
MLJFlux.NeuralNetworkClassifier — TypeNeuralNetworkClassifierA model type for constructing a neural network classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFluxDo model = NeuralNetworkClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkClassifier(builder=...).
NeuralNetworkClassifier is for training a data-dependent Flux.jl neural network for making probabilistic predictions of a Multiclass or OrderedFactor target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any AbstractVector whose element scitype is Multiclass or OrderedFactor; check the scitype with scitype(y)
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Short(): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux.jl documentation for examples of user-defined builders. See also finaliser below.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.crossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.crossentropy: Standard multiclass classification loss, also known as the log loss.
Flux.logitcrossentopy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with softmax and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default softmax finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.softmax: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.softmax.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we build a classification model using the Iris dataset. This is a very basic example, using a default builder and no standardization. For a more advanced illustration, see NeuralNetworkRegressor or ImageClassifier, and examples in the MLJFlux.jl documentation.
using MLJ
+Classification · MLJFlux MLJFlux.NeuralNetworkClassifier — TypeNeuralNetworkClassifier
A model type for constructing a neural network classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
Do model = NeuralNetworkClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkClassifier(builder=...).
NeuralNetworkClassifier is for training a data-dependent Flux.jl neural network for making probabilistic predictions of a Multiclass or OrderedFactor target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)
Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any AbstractVector whose element scitype is Multiclass or OrderedFactor; check the scitype with scitype(y)
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Short(): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux.jl documentation for examples of user-defined builders. See also finaliser below.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.crossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.crossentropy: Standard multiclass classification loss, also known as the log loss.
Flux.logitcrossentopy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with softmax and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default softmax finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights.] Typically, batch size is between 8 and 512. Increassing batch size may accelerate training if acceleration=CUDALibs() and a GPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.softmax: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.softmax.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).
Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.
Examples
In this example we build a classification model using the Iris dataset. This is a very basic example, using a default builder and no standardization. For a more advanced illustration, see NeuralNetworkRegressor or ImageClassifier, and examples in the MLJFlux.jl documentation.
using MLJ
using Flux
-import RDatasets
First, we can load the data:
iris = RDatasets.dataset("datasets", "iris");
+import RDatasets
+import Optimisers
First, we can load the data:
iris = RDatasets.dataset("datasets", "iris");
y, X = unpack(iris, ==(:Species), rng=123); # a vector and a table
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
clf = NeuralNetworkClassifier()
Next, we can train the model:
mach = machine(clf, X, y)
-fit!(mach)
We can train the model in an incremental fashion, altering the learning rate as we go, provided optimizer_changes_trigger_retraining is false (the default). Here, we also change the number of (total) iterations:
clf.optimiser.eta = clf.optimiser.eta * 2
+fit!(mach)
We can train the model in an incremental fashion, altering the learning rate as we go, provided optimizer_changes_trigger_retraining is false (the default). Here, we also change the number of (total) iterations:
clf.optimiser = Optimisers.Adam(clf.optimiser.eta * 2)
clf.epochs = clf.epochs + 5
fit!(mach, verbosity=2) # trains 5 more epochs
We can inspect the mean training loss using the cross_entropy function:
training_loss = cross_entropy(predict(mach, X), y)
And we can access the Flux chain (model) using fitted_params:
chain = fitted_params(mach).chain
Finally, we can see how the out-of-sample performance changes over time, using MLJ's learning_curve function:
r = range(clf, :epochs, lower=1, upper=200, scale=:log10)
@@ -19,7 +20,7 @@
xlab=curve.parameter_name,
xscale=curve.parameter_scale,
ylab = "Cross Entropy")
-
See also ImageClassifier, NeuralNetworkBinaryClassifier.
sourceMLJFlux.NeuralNetworkBinaryClassifier — TypeNeuralNetworkBinaryClassifier
A model type for constructing a neural network binary classifier, based on unknown.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkBinaryClassifier = @load NeuralNetworkBinaryClassifier pkg=unknown
Do model = NeuralNetworkBinaryClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkBinaryClassifier(builder=...).
NeuralNetworkBinaryClassifier is for training a data-dependent Flux.jl neural network for making probabilistic predictions of a binary (Multiclass{2} or OrderedFactor{2}) target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)
Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any AbstractVector whose element scitype is Multiclass{2} or OrderedFactor{2}; check the scitype with scitype(y)
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Short(): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux.jl documentation for examples of user-defined builders. See also finaliser below.
optimiser::Flux.Adam(): A Flux.Optimise optimiser. The optimiser performs the updating of the weights of the network. For further reference, see the Flux optimiser documentation. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.binarycrossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.binarycrossentropy: Standard binary classification loss, also known as the log loss.
Flux.logitbinarycrossentropy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with σ and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default sigmoid finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.binary_focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
- Increassing batch size may accelerate training if
acceleration=CUDALibs() and a
GPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞).
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training.
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.σ: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.σ.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).
Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.
Examples
In this example we build a classification model using the Iris dataset. This is a very basic example, using a default builder and no standardization. For a more advanced illustration, see NeuralNetworkRegressor or ImageClassifier, and examples in the MLJFlux.jl documentation.
using MLJ, Flux
+
See also ImageClassifier, NeuralNetworkBinaryClassifier.
sourceMLJFlux.NeuralNetworkBinaryClassifier — TypeNeuralNetworkBinaryClassifier
A model type for constructing a neural network binary classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkBinaryClassifier = @load NeuralNetworkBinaryClassifier pkg=MLJFlux
Do model = NeuralNetworkBinaryClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkBinaryClassifier(builder=...).
NeuralNetworkBinaryClassifier is for training a data-dependent Flux.jl neural network for making probabilistic predictions of a binary (Multiclass{2} or OrderedFactor{2}) target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)
Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any AbstractVector whose element scitype is Multiclass{2} or OrderedFactor{2}; check the scitype with scitype(y)
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Short(): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux.jl documentation for examples of user-defined builders. See also finaliser below.
optimiser::Flux.Adam(): A Flux.Optimise optimiser. The optimiser performs the updating of the weights of the network. For further reference, see the Flux optimiser documentation. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.binarycrossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.binarycrossentropy: Standard binary classification loss, also known as the log loss.
Flux.logitbinarycrossentropy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with σ and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default sigmoid finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.binary_focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and 512. Increassing batch size may accelerate training if acceleration=CUDALibs() and a GPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞).
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training.
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.σ: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.σ.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).
Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.
Examples
In this example we build a classification model using the Iris dataset. This is a very basic example, using a default builder and no standardization. For a more advanced illustration, see NeuralNetworkRegressor or ImageClassifier, and examples in the MLJFlux.jl documentation.
using MLJ, Flux
import Optimisers
import RDatasets
First, we can load the data:
mtcars = RDatasets.dataset("datasets", "mtcars");
y, X = unpack(mtcars, ==(:VS), in([:MPG, :Cyl, :Disp, :HP, :WT, :QSec]));
Note that y is a vector and X a table.
y = categorical(y) # classifier takes catogorical input
@@ -47,4 +48,4 @@
xscale=curve.parameter_scale,
ylab = "Cross Entropy",
)
-
See also ImageClassifier.
sourceSettings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
See also ImageClassifier.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
Following is an example defining a new builder for creating a simple fully-connected neural network with two hidden layers, with n1 nodes in the first hidden layer, and n2 nodes in the second, for use in any of the first three models in Table 1. The definition includes one mutable struct and one method:
mutable struct MyBuilder <: MLJFlux.Builder
+Custom Builders · MLJFlux Defining Custom Builders
Following is an example defining a new builder for creating a simple fully-connected neural network with two hidden layers, with n1 nodes in the first hidden layer, and n2 nodes in the second, for use in any of the first three models in Table 1. The definition includes one mutable struct and one method:
mutable struct MyBuilder <: MLJFlux.Builder
n1 :: Int
n2 :: Int
end
@@ -11,5 +11,5 @@
Dense(nn.n1, nn.n2, init=init),
Dense(nn.n2, n_out, init=init),
)
-end
Note here that n_in and n_out depend on the size of the data (see Table 1.
For a concrete image classification example, see the Image Classification Example.
More generally, defining a new builder means defining a new struct sub-typing MLJFlux.Builder and defining a new MLJFlux.build method with one of these signatures:
MLJFlux.build(builder::MyBuilder, rng, n_in, n_out)
-MLJFlux.build(builder::MyBuilder, rng, n_in, n_out, n_channels) # for use with `ImageClassifier`
This method must return a Flux.Chain instance, chain, subject to the following conditions:
chain(x) must make sense:
- for any
x <: Array{<:AbstractFloat, 2} of size (n_in, batch_size) where batch_size is any integer (for use with one of the first three model types); or - for any
x <: Array{<:Float32, 4} of size (W, H, n_channels, batch_size), where (W, H) = n_in, n_channels is 1 or 3, and batch_size is any integer (for use with ImageClassifier)
The object returned by chain(x) must be an AbstractFloat vector of length n_out.
Alternatively, use MLJFlux.@builder(neural_net) to automatically create a builder for any valid Flux chain expression neural_net, where the symbols n_in, n_out, n_channels and rng can appear literally, with the interpretations explained above. For example,
builder = MLJFlux.@builder Chain(Dense(n_in, 128), Dense(128, n_out, tanh))
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
Note here that n_in and n_out depend on the size of the data (see Table 1.
For a concrete image classification example, see Using MLJ to classifiy the MNIST image dataset.
More generally, defining a new builder means defining a new struct sub-typing MLJFlux.Builder and defining a new MLJFlux.build method with one of these signatures:
MLJFlux.build(builder::MyBuilder, rng, n_in, n_out)
+MLJFlux.build(builder::MyBuilder, rng, n_in, n_out, n_channels) # for use with `ImageClassifier`This method must return a Flux.Chain instance, chain, subject to the following conditions:
chain(x) must make sense:
x <: Array{<:AbstractFloat, 2} of size (n_in, batch_size) where batch_size is any integer (for use with one of the first three model types); orx <: Array{<:Float32, 4} of size (W, H, n_channels, batch_size), where (W, H) = n_in, n_channels is 1 or 3, and batch_size is any integer (for use with ImageClassifier)The object returned by chain(x) must be an AbstractFloat vector of length n_out.
Alternatively, use MLJFlux.@builder(neural_net) to automatically create a builder for any valid Flux chain expression neural_net, where the symbols n_in, n_out, n_channels and rng can appear literally, with the interpretations explained above. For example,
builder = MLJFlux.@builder Chain(Dense(n_in, 128), Dense(128, n_out, tanh))Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
MLJFlux.ImageClassifier — TypeImageClassifierA model type for constructing a image classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
ImageClassifier = @load ImageClassifier pkg=MLJFluxDo model = ImageClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in ImageClassifier(builder=...).
ImageClassifier classifies images using a neural network adapted to the type of images provided (color or gray scale). Predictions are probabilistic. Users provide a recipe for constructing the network, based on properties of the image encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is any AbstractVector of images with ColorImage or GrayImage scitype; check the scitype with scitype(X) and refer to ScientificTypes.jl documentation on coercing typical image formats into an appropriate type.
y is the target, which can be any AbstractVector whose element scitype is Multiclass; check the scitype with scitype(y).
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder: An MLJFlux builder that constructs the neural network. The fallback builds a depth-16 VGG architecture adapted to the image size and number of target classes, with no batch normalization; see the Metalhead.jl documentation for details. See the example below for a user-specified builder. A convenience macro @builder is also available. See also finaliser below.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.crossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.crossentropy: Standard multiclass classification loss, also known as the log loss.
Flux.logitcrossentopy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with softmax and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default softmax finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.softmax: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.softmax.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we use MLJFlux and a custom builder to classify the MNIST image dataset.
using MLJ
+Image Classification · MLJFlux MLJFlux.ImageClassifier — TypeImageClassifier
A model type for constructing a image classifier, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
ImageClassifier = @load ImageClassifier pkg=MLJFlux
Do model = ImageClassifier() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in ImageClassifier(builder=...).
ImageClassifier classifies images using a neural network adapted to the type of images provided (color or gray scale). Predictions are probabilistic. Users provide a recipe for constructing the network, based on properties of the image encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)
Here:
X is any AbstractVector of images with ColorImage or GrayImage scitype; check the scitype with scitype(X) and refer to ScientificTypes.jl documentation on coercing typical image formats into an appropriate type.
y is the target, which can be any AbstractVector whose element scitype is Multiclass; check the scitype with scitype(y).
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder: An MLJFlux builder that constructs the neural network. The fallback builds a depth-16 VGG architecture adapted to the image size and number of target classes, with no batch normalization; see the Metalhead.jl documentation for details. See the example below for a user-specified builder. A convenience macro @builder is also available. See also finaliser below.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.crossentropy: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a classification task, the most natural loss functions are:
Flux.crossentropy: Standard multiclass classification loss, also known as the log loss.
Flux.logitcrossentopy: Mathematically equal to crossentropy, but numerically more stable than finalising the outputs with softmax and then calculating crossentropy. You will need to specify finaliser=identity to remove MLJFlux's default softmax finaliser, and understand that the output of predict is then unnormalized (no longer probabilistic).
Flux.tversky_loss: Used with imbalanced data to give more weight to false negatives.
Flux.focal_loss: Used with highly imbalanced data. Weights harder examples more than easier examples.
Currently MLJ measures are not supported values of loss.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
- Increassing batch size may accelerate training if
acceleration=CUDALibs() and a
GPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
finaliser=Flux.softmax: The final activation function of the neural network (applied after the network defined by builder). Defaults to Flux.softmax.
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above. Predictions are probabilistic but uncalibrated.
predict_mode(mach, Xnew): Return the modes of the probabilistic predictions returned above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network. This includes the final layer specified by finaliser (eg, softmax).
Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.
Examples
In this example we use MLJFlux and a custom builder to classify the MNIST image dataset.
using MLJ
using Flux
import MLJFlux
+import Optimisers
import MLJIteration # for `skip` control
First we want to download the MNIST dataset, and unpack into images and labels:
import MLDatasets: MNIST
data = MNIST(split=:train)
images, labels = data.features, data.targets
In MLJ, integers cannot be used for encoding categorical data, so we must coerce them into the Multiclass scitype:
labels = coerce(labels, Multiclass);
Above images is a single array but MLJFlux requires the images to be a vector of individual image arrays:
images = coerce(images, GrayImage);
@@ -45,4 +46,4 @@
resampling=Holdout(fraction_train=0.5),
measure=cross_entropy,
rows=1:1000,
- verbosity=0)
See also NeuralNetworkClassifier.
sourceSettings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
See also NeuralNetworkClassifier.
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
MLJFlux.MultitargetNeuralNetworkRegressor — TypeMultitargetNeuralNetworkRegressorA model type for constructing a multitarget neural network regressor, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
MultitargetNeuralNetworkRegressor = @load MultitargetNeuralNetworkRegressor pkg=MLJFluxDo model = MultitargetNeuralNetworkRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in MultitargetNeuralNetworkRegressor(builder=...).
MultitargetNeuralNetworkRegressor is for training a data-dependent Flux.jl neural network to predict a multi-valued Continuous target, represented as a table, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any table or matrix of output targets whose element scitype is Continuous; check column scitypes with schema(y). If y is a Matrix, it is assumed to have columns corresponding to variables and rows corresponding to observations.
Hyper-parameters
builder=MLJFlux.Linear(σ=Flux.relu): An MLJFlux builder that constructs a neural network. Possible builders include: Linear, Short, and MLP. See MLJFlux documentation for more on builders, and the example below for using the @builder convenience macro.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.mse: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a regression task, natural loss functions are:
Flux.mse
Flux.mae
Flux.msle
Flux.huber_loss
Currently MLJ measures are not supported as loss functions here.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew having the same scitype as X above. Predictions are deterministic.Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network.Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we apply a multi-target regression model to synthetic data:
using MLJ
+Multi-Target Regression · MLJFlux MLJFlux.MultitargetNeuralNetworkRegressor — TypeMultitargetNeuralNetworkRegressor
A model type for constructing a multitarget neural network regressor, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
MultitargetNeuralNetworkRegressor = @load MultitargetNeuralNetworkRegressor pkg=MLJFlux
Do model = MultitargetNeuralNetworkRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in MultitargetNeuralNetworkRegressor(builder=...).
MultitargetNeuralNetworkRegressor is for training a data-dependent Flux.jl neural network to predict a multi-valued Continuous target, represented as a table, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)
Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.
y is the target, which can be any table or matrix of output targets whose element scitype is Continuous; check column scitypes with schema(y). If y is a Matrix, it is assumed to have columns corresponding to variables and rows corresponding to observations.
Hyper-parameters
builder=MLJFlux.Linear(σ=Flux.relu): An MLJFlux builder that constructs a neural network. Possible builders include: Linear, Short, and MLP. See MLJFlux documentation for more on builders, and the example below for using the @builder convenience macro.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.mse: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a regression task, natural loss functions are:
Flux.mse
Flux.mae
Flux.msle
Flux.huber_loss
Currently MLJ measures are not supported as loss functions here.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and 512. Increassing batch size may accelerate training if acceleration=CUDALibs() and a GPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew having the same scitype as X above. Predictions are deterministic.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network.
Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalised if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.
Examples
In this example we apply a multi-target regression model to synthetic data:
using MLJ
import MLJFlux
-using Flux
First, we generate some synthetic data (needs MLJBase 0.20.16 or higher):
X, y = make_regression(100, 9; n_targets = 2) # both tables
+using Flux
+import Optimisers
First, we generate some synthetic data (needs MLJBase 0.20.16 or higher):
X, y = make_regression(100, 9; n_targets = 2) # both tables
schema(y)
schema(X)
Splitting off a test set:
(X, Xtest), (y, ytest) = partition((X, y), 0.7, multi=true);
Next, we can define a builder, making use of a convenience macro to do so. In the following @builder call, n_in is a proxy for the number input features and n_out the number of target variables (both known at fit! time), while rng is a proxy for a RNG (which will be passed from the rng field of model defined below).
builder = MLJFlux.@builder begin
init=Flux.glorot_uniform(rng)
@@ -15,7 +16,7 @@
fit!(mach, verbosity=2)
# first element initial loss, 2:end per epoch training losses
-report(mach).transformed_target_model_deterministic.model.training_losses
For experimenting with learning rate, see the NeuralNetworkRegressor example.
pipe.transformed_target_model_deterministic.model.optimiser.eta = 0.0001
With the learning rate fixed, we can now compute a CV estimate of the performance (using all data bound to mach) and compare this with performance on the test set:
# custom MLJ loss:
+report(mach).transformed_target_model_deterministic.model.training_losses
For experimenting with learning rate, see the NeuralNetworkRegressor example.
pipe.transformed_target_model_deterministic.model.optimiser = Optimisers.Adam(0.0001)
With the learning rate fixed, we can now compute a CV estimate of the performance (using all data bound to mach) and compare this with performance on the test set:
# custom MLJ loss:
multi_loss(yhat, y) = l2(MLJ.matrix(yhat), MLJ.matrix(y))
# CV estimate, based on `(X, y)`:
@@ -24,4 +25,4 @@
# loss for `(Xtest, test)`:
fit!(mach) # trains on all data `(X, y)`
yhat = predict(mach, Xtest)
-multi_loss(yhat, ytest)
See also NeuralNetworkRegressor
sourceSettings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
See also NeuralNetworkRegressor
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
MLJFlux.NeuralNetworkRegressor — TypeNeuralNetworkRegressorA model type for constructing a neural network regressor, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkRegressor = @load NeuralNetworkRegressor pkg=MLJFluxDo model = NeuralNetworkRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkRegressor(builder=...).
NeuralNetworkRegressor is for training a data-dependent Flux.jl neural network to predict a Continuous target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Linear(σ=Flux.relu): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux documentation for more on builders, and the example below for using the @builder convenience macro.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.mse: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a regression task, natural loss functions are:
Flux.mse
Flux.mae
Flux.msle
Flux.huber_loss
Currently MLJ measures are not supported as loss functions here.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and
acceleration=CUDALibs() and aGPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above.Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network.Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalized if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.Examples
In this example we build a regression model for the Boston house price dataset.
using MLJ
+Regression · MLJFlux MLJFlux.NeuralNetworkRegressor — TypeNeuralNetworkRegressor
A model type for constructing a neural network regressor, based on MLJFlux.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
NeuralNetworkRegressor = @load NeuralNetworkRegressor pkg=MLJFlux
Do model = NeuralNetworkRegressor() to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in NeuralNetworkRegressor(builder=...).
NeuralNetworkRegressor is for training a data-dependent Flux.jl neural network to predict a Continuous target, given a table of Continuous features. Users provide a recipe for constructing the network, based on properties of the data that is encountered, by specifying an appropriate builder. See MLJFlux documentation for more on builders.
Training data
In MLJ or MLJBase, bind an instance model to data with
mach = machine(model, X, y)
Here:
X is either a Matrix or any table of input features (eg, a DataFrame) whose columns are of scitype Continuous; check column scitypes with schema(X). If X is a Matrix, it is assumed to have columns corresponding to features and rows corresponding to observations.y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)
Train the machine with fit!(mach, rows=...).
Hyper-parameters
builder=MLJFlux.Linear(σ=Flux.relu): An MLJFlux builder that constructs a neural network. Possible builders include: MLJFlux.Linear, MLJFlux.Short, and MLJFlux.MLP. See MLJFlux documentation for more on builders, and the example below for using the @builder convenience macro.
optimiser::Optimisers.Adam(): An Optimisers.jl optimiser. The optimiser performs the updating of the weights of the network. To choose a learning rate (the update rate of the optimizer), a good rule of thumb is to start out at 10e-3, and tune using powers of 10 between 1 and 1e-7.
loss=Flux.mse: The loss function which the network will optimize. Should be a function which can be called in the form loss(yhat, y). Possible loss functions are listed in the Flux loss function documentation. For a regression task, natural loss functions are:
Flux.mse
Flux.mae
Flux.msle
Flux.huber_loss
Currently MLJ measures are not supported as loss functions here.
epochs::Int=10: The duration of training, in epochs. Typically, one epoch represents one pass through the complete the training dataset.
batch_size::int=1: the batch size to be used for training, representing the number of samples per update of the network weights. Typically, batch size is between 8 and 512. Increasing batch size may accelerate training if acceleration=CUDALibs() and a GPU is available.
lambda::Float64=0: The strength of the weight regularization penalty. Can be any value in the range [0, ∞). Note the history reports unpenalized losses.
alpha::Float64=0: The L2/L1 mix of regularization, in the range [0, 1]. A value of 0 represents L2 regularization, and a value of 1 represents L1 regularization.
rng::Union{AbstractRNG, Int64}: The random number generator or seed used during training. The default is Random.default_rng().
optimizer_changes_trigger_retraining::Bool=false: Defines what happens when re-fitting a machine if the associated optimiser has changed. If true, the associated machine will retrain from scratch on fit! call, otherwise it will not.
acceleration::AbstractResource=CPU1(): Defines on what hardware training is done. For Training on GPU, use CUDALibs().
Operations
predict(mach, Xnew): return predictions of the target given new features Xnew, which should have the same scitype as X above.
Fitted parameters
The fields of fitted_params(mach) are:
chain: The trained "chain" (Flux.jl model), namely the series of layers, functions, and activations which make up the neural network.
Report
The fields of report(mach) are:
training_losses: A vector of training losses (penalized if lambda != 0) in historical order, of length epochs + 1. The first element is the pre-training loss.
Examples
In this example we build a regression model for the Boston house price dataset.
using MLJ
import MLJFlux
-using Flux
First, we load in the data: The :MEDV column becomes the target vector y, and all remaining columns go into a table X, with the exception of :CHAS:
data = OpenML.load(531); # Loads from https://www.openml.org/d/531
+using Flux
+import Optimisers
First, we load in the data: The :MEDV column becomes the target vector y, and all remaining columns go into a table X, with the exception of :CHAS:
data = OpenML.load(531); # Loads from https://www.openml.org/d/531
y, X = unpack(data, ==(:MEDV), !=(:CHAS); rng=123);
scitype(y)
-schema(X)
Since MLJFlux models do not handle ordered factors, we'll treat :RAD as Continuous:
X = coerce(X, :RAD=>Continuous)
Splitting off a test set:
(X, Xtest), (y, ytest) = partition((X, y), 0.7, multi=true);
Next, we can define a builder, making use of a convenience macro to do so. In the following @builder call, n_in is a proxy for the number input features (which will be known at fit! time) and rng is a proxy for a RNG (which will be passed from the rng field of model defined below). We also have the parameter n_out which is the number of output features. As we are doing single target regression, the value passed will always be 1, but the builder we define will also work for MultitargetNeuralRegressor.
builder = MLJFlux.@builder begin
+schema(X)
Since MLJFlux models do not handle ordered factors, we'll treat :RAD as Continuous:
X = coerce(X, :RAD=>Continuous)
Splitting off a test set:
(X, Xtest), (y, ytest) = partition((X, y), 0.7, multi=true);
Next, we can define a builder, making use of a convenience macro to do so. In the following @builder call, n_in is a proxy for the number input features (which will be known at fit! time) and rng is a proxy for a RNG (which will be passed from the rng field of model defined below). We also have the parameter n_out which is the number of output features. As we are doing single target regression, the value passed will always be 1, but the builder we define will also work for MultitargetNeuralNetworkRegressor.
builder = MLJFlux.@builder begin
init=Flux.glorot_uniform(rng)
Chain(
Dense(n_in, 64, relu, init=init),
@@ -27,7 +28,7 @@
plt=plot()
foreach(rates) do η
- pipe.transformed_target_model_deterministic.model.optimiser.eta = η
+ pipe.transformed_target_model_deterministic.model.optimiser = Optimisers.Adam(η)
fit!(mach, force=true, verbosity=0)
losses =
report(mach).transformed_target_model_deterministic.model.training_losses[3:end]
@@ -36,10 +37,10 @@
plt
-pipe.transformed_target_model_deterministic.model.optimiser.eta = 0.0001
With the learning rate fixed, we compute a CV estimate of the performance (using all data bound to mach) and compare this with performance on the test set:
# CV estimate, based on `(X, y)`:
+pipe.transformed_target_model_deterministic.model.optimiser.eta = Optimisers.Adam(0.0001)
With the learning rate fixed, we compute a CV estimate of the performance (using all data bound to mach) and compare this with performance on the test set:
# CV estimate, based on `(X, y)`:
evaluate!(mach, resampling=CV(nfolds=5), measure=l2)
# loss for `(Xtest, test)`:
fit!(mach) # train on `(X, y)`
yhat = predict(mach, Xtest)
-l2(yhat, ytest)
These losses, for the pipeline model, refer to the target on the original, unstandardized, scale.
For implementing stopping criterion and other iteration controls, refer to examples linked from the MLJFlux documentation.
See also MultitargetNeuralNetworkRegressor
sourceSettings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
These losses, for the pipeline model, refer to the target on the original, unstandardized, scale.
For implementing stopping criterion and other iteration controls, refer to examples linked from the MLJFlux documentation.
See also MultitargetNeuralNetworkRegressor
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
MLJFlux provides four model types, for use with input features X and targets y of the scientific type indicated in the table below. The parameters n_in, n_out and n_channels refer to information passed to the builder, as described under Defining a new builder below.
| Model Type | Prediction type | scitype(X) <: _ | scitype(y) <: _ |
|---|---|---|---|
NeuralNetworkRegressor | Deterministic | Table(Continuous) with n_in columns | AbstractVector{<:Continuous) (n_out = 1) |
MultitargetNeuralNetworkRegressor | Deterministic | Table(Continuous) with n_in columns | <: Table(Continuous) with n_out columns |
NeuralNetworkClassifier | Probabilistic | <:Table(Continuous) with n_in columns | AbstractVector{<:Finite} with n_out classes |
NeuralNetworkBinaryClassifier | Probabilistic | <:Table(Continuous) with n_in columns | AbstractVector{<:Finite{2}} (n_out = 2) |
ImageClassifier | Probabilistic | AbstractVector(<:Image{W,H}) with n_in = (W, H) | AbstractVector{<:Finite} with n_out classes |
In MLJ a model is a mutable struct storing hyper-parameters for some learning algorithm indicated by the model name, and that's all. In particular, an MLJ model does not store learned parameters.
In Flux the term "model" has another meaning. However, as all Flux "models" used in MLJFLux are Flux.Chain objects, we call them chains, and restrict use of "model" to models in the MLJ sense.
Any AbstractMatrix{<:AbstractFloat} object Xmat can be forced to have scitype Table(Continuous) by replacing it with X = MLJ.table(Xmat). Furthermore, this wrapping, and subsequent unwrapping under the hood, will compile to a no-op. At present this includes support for sparse matrix data, but the implementation has not been optimized for sparse data at this time and so should be used with caution.
Instructions for coercing common image formats into some AbstractVector{<:Image} are here.
MLJ machines cache state enabling the "warm restart" of model training, as demonstrated in the incremental training example. In the case of MLJFlux models, fit!(mach) will use a warm restart if:
only model.epochs has changed since the last call; or
only model.epochs or model.optimiser have changed since the last call and model.optimiser_changes_trigger_retraining == false (the default) (the "state" part of the optimiser is ignored in this comparison). This allows one to dynamically modify learning rates, for example.
Here model=mach.model is the associated MLJ model.
The warm restart feature makes it possible to apply early stopping criteria, as defined in EarlyStopping.jl. For an example, see /examples/mnist/. (Eventually, this will be handled by an MLJ model wrapper for controlling arbitrary iterative models.)
All models share the following hyper-parameters:
| Hyper-parameter | Description | Default |
|---|---|---|
builder | Default builder for models. | MLJFlux.Linear(σ=Flux.relu) (regressors) or MLJFlux.Short(n_hidden=0, dropout=0.5, σ=Flux.σ) (classifiers) |
optimiser | The optimiser to use for training. | Flux.ADAM() |
loss | The loss function used for training. | Flux.mse (regressors) and Flux.crossentropy (classifiers) |
n_epochs | Number of epochs to train for. | 10 |
batch_size | The batch size for the data. | 1 |
lambda | The regularization strength. Range = [0, ∞). | 0 |
alpha | The L2/L1 mix of regularization. Range = [0, 1]. | 0 |
rng | The random number generator (RNG) passed to builders, for weight initialization, for example. Can be any AbstractRNG or the seed (integer) for a MersenneTwister that is reset on every cold restart of model (machine) training. | GLOBAL_RNG |
acceleration | Use CUDALibs() for training on GPU; default is CPU1(). | CPU1() |
optimiser_changes_trigger_retraining | True if fitting an associated machine should trigger retraining from scratch whenever the optimiser changes. | false |
The classifiers have an additional hyperparameter finaliser (default = Flux.softmax) which is the operation applied to the unnormalized output of the final layer to obtain probabilities (outputs summing to one). Default = Flux.softmax. It should return a vector of the same length as its input.
Currently, the loss function specified by loss=... is applied internally by Flux and needs to conform to the Flux API. You cannot, for example, supply one of MLJ's probabilistic loss functions, such as MLJ.cross_entropy to one of the classifier constructors.
That said, you can only use MLJ loss functions or metrics in evaluation meta-algorithms (such as cross validation) and they will work even if the underlying model comes from MLJFlux.
As in the table, when instantiating a model for training on a GPU, specify acceleration=CUDALibs(), as in
using MLJ
+Summary · MLJFlux Models
MLJFlux provides four model types, for use with input features X and targets y of the scientific type indicated in the table below. The parameters n_in, n_out and n_channels refer to information passed to the builder, as described under Defining Custom Builders.
Model Type Prediction type scitype(X) <: _scitype(y) <: _NeuralNetworkRegressorDeterministicTable(Continuous) with n_in columnsAbstractVector{<:Continuous) (n_out = 1)MultitargetNeuralNetworkRegressorDeterministicTable(Continuous) with n_in columns<: Table(Continuous) with n_out columnsNeuralNetworkClassifierProbabilistic<:Table(Continuous) with n_in columnsAbstractVector{<:Finite} with n_out classesNeuralNetworkBinaryClassifierProbabilistic<:Table(Continuous) with n_in columnsAbstractVector{<:Finite{2}} (n_out = 2)ImageClassifierProbabilisticAbstractVector(<:Image{W,H}) with n_in = (W, H)AbstractVector{<:Finite} with n_out classes
What exactly is a "model"?
In MLJ a model is a mutable struct storing hyper-parameters for some learning algorithm indicated by the model name, and that's all. In particular, an MLJ model does not store learned parameters.
Difference in Definition In Flux the term "model" has another meaning. However, as all Flux "models" used in MLJFLux are Flux.Chain objects, we call them chains, and restrict use of "model" to models in the MLJ sense.
Dealing with non-tabular input
Any AbstractMatrix{<:AbstractFloat} object Xmat can be forced to have scitype Table(Continuous) by replacing it with X = MLJ.table(Xmat). Furthermore, this wrapping, and subsequent unwrapping under the hood, will compile to a no-op. At present this includes support for sparse matrix data, but the implementation has not been optimized for sparse data at this time and so should be used with caution.
Instructions for coercing common image formats into some AbstractVector{<:Image} are here.
Fitting and warm restarts
MLJ machines cache state enabling the "warm restart" of model training, as demonstrated in the incremental training example. In the case of MLJFlux models, fit!(mach) will use a warm restart if:
only model.epochs has changed since the last call; or
only model.epochs or model.optimiser have changed since the last call and model.optimiser_changes_trigger_retraining == false (the default) (the "state" part of the optimiser is ignored in this comparison). This allows one to dynamically modify learning rates, for example.
Here model=mach.model is the associated MLJ model.
The warm restart feature makes it possible to externally control iteration. See, for example, Early Stopping with MLJFlux and Using MLJ to classifiy the MNIST image dataset.
Model Hyperparameters.
All models share the following hyper-parameters. See individual model docstrings for a full list.
Hyper-parameter Description Default builderDefault builder for models. MLJFlux.Linear(σ=Flux.relu) (regressors) or MLJFlux.Short(n_hidden=0, dropout=0.5, σ=Flux.σ) (classifiers)optimiserThe optimiser to use for training. Optimiser.Adam()lossThe loss function used for training. Flux.mse (regressors) and Flux.crossentropy (classifiers)n_epochsNumber of epochs to train for. 10batch_sizeThe batch size for the data. 1lambdaThe regularization strength. Range = [0, ∞). 0alphaThe L2/L1 mix of regularization. Range = [0, 1]. 0rngThe random number generator (RNG) passed to builders, for weight initialization, for example. Can be any AbstractRNG or the seed (integer) for a Xoshirio that is reset on every cold restart of model (machine) training. GLOBAL_RNGaccelerationUse CUDALibs() for training on GPU; default is CPU1(). CPU1()optimiser_changes_trigger_retrainingTrue if fitting an associated machine should trigger retraining from scratch whenever the optimiser changes. false
The classifiers have an additional hyperparameter finaliser (default is Flux.softmax, or Flux.σ in the binary case) which is the operation applied to the unnormalized output of the final layer to obtain probabilities (outputs summing to one). It should return a vector of the same length as its input.
Loss Functions Currently, the loss function specified by loss=... is applied internally by Flux and needs to conform to the Flux API. You cannot, for example, supply one of MLJ's probabilistic loss functions, such as MLJ.cross_entropy to one of the classifier constructors.
That said, you can only use MLJ loss functions or metrics in evaluation meta-algorithms (such as cross validation) and they will work even if the underlying model comes from MLJFlux.
More on accelerated training with GPUs
As in the table, when instantiating a model for training on a GPU, specify acceleration=CUDALibs(), as in
using MLJ
ImageClassifier = @load ImageClassifier
model = ImageClassifier(epochs=10, acceleration=CUDALibs())
-mach = machine(model, X, y) |> fit!
In this example, the data X, y is copied onto the GPU under the hood on the call to fit! and cached for use in any warm restart (see above). The Flux chain used in training is always copied back to the CPU at then conclusion of fit!, and made available as fitted_params(mach).
Built-in builders
As for the builder argument, the following builders are provided out-of-the-box:
Builder Description MLJFlux.MLP(hidden=(10,))General multi-layer perceptron MLJFlux.Short(n_hidden=0, dropout=0.5, σ=sigmoid)Fully connected network with one hidden layer and dropout MLJFlux.Linear(σ=relu)Vanilla linear network with no hidden layers and activation function σ
See the following sections to learn more about the interface for the builders and models.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this example, the data X, y is copied onto the GPU under the hood on the call to fit! and cached for use in any warm restart (see above). The Flux chain used in training is always copied back to the CPU at then conclusion of fit!, and made available as fitted_params(mach).
| Builder | Description |
|---|---|
MLJFlux.MLP(hidden=(10,)) | General multi-layer perceptron |
MLJFlux.Short(n_hidden=0, dropout=0.5, σ=sigmoid) | Fully connected network with one hidden layer and dropout |
MLJFlux.Linear(σ=relu) | Vanilla linear network with no hidden layers and activation function σ |
MLJFlux.@builder | Macro for customized builders |
Settings
This document was generated with Documenter.jl version 1.4.1 on Wednesday 12 June 2024. Using Julia version 1.10.4.
| Row | mlp | measurement |
|---|---|---|
| Probabil… | Float64 | |
| 1 | BayesianLDA(method = gevd, …) | 0.0610826 |
| 2 | RandomForestClassifier(max_depth = -1, …) | 0.106565 |
| 3 | NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …) | 0.113266 |
| 4 | ProbabilisticTunedModel(model = XGBoostClassifier(test = 1, …), …) | 0.221056 |
Neural Architecture Search is (NAS) is an instance of hyperparameter tuning concerned with tuning model hyperparameters defining the architecture itself. Although it's typically performed with sophisticated search algorithms for efficiency, in this example we will be using a simple random search.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-using RDatasets: RDatasets # Dataset source
-using DataFrames # To view tuning results in a tableiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng = 123);
-X = Float32.(X); # To be compatible with type of network network parameters
-first(X, 5)Now let's construct our model. This follows a similar setup the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg = "MLJFlux"
-clf = NeuralNetworkClassifier(
- builder = MLJFlux.MLP(; hidden = (1, 1, 1), σ = Flux.relu),
- optimiser = Flux.ADAM(0.01),
- batch_size = 8,
- epochs = 10,
- rng = 42,
-)We know that the MLP builder takes a tuple of the form $(z_1, z_2, ..., z_k)$ to define a network with $k$ hidden layers and where the ith layer has $z_i$ neurons. We will proceed by defining a function that can generate all possible networks with a specific number of hidden layers, a minimum and maximum number of neurons per layer and increments to consider for the number of neurons.
function generate_networks(;
- min_neurons::Int,
- max_neurons::Int,
- neuron_step::Int,
- num_layers::Int,
-)
- # Define the range of neurons
- neuron_range = min_neurons:neuron_step:max_neurons
-
- # Empty list to store the network configurations
- networks = Vector{Tuple{Vararg{Int, num_layers}}}()
-
- # Recursive helper function to generate all combinations of tuples
- function generate_tuple(current_layers, remaining_layers)
- if remaining_layers > 0
- for n in neuron_range
- # current_layers =[] then current_layers=[(min_neurons)],
- # [(min_neurons+neuron_step)], [(min_neurons+2*neuron_step)],...
- # for each of these we call generate_layers again which appends
- # the n combinations for each one of them
- generate_tuple(vcat(current_layers, [n]), remaining_layers - 1)
- end
- else
- # in the base case, no more layers to "recurse on"
- # and we just append the current_layers as a tuple
- push!(networks, tuple(current_layers...))
- end
- end
-
- # Generate networks for the given number of layers
- generate_tuple([], num_layers)
-
- return networks
-endgenerate_networks (generic function with 1 method)Now let's generate an array of all possible neural networks with three hidden layers and number of neurons per layer ∈ [1,64] with a step of 4
networks_space =
- generate_networks(min_neurons = 1, max_neurons = 64, neuron_step = 4, num_layers = 3)
-
-networks_space[1:5]5-element Vector{Tuple{Int64, Int64, Int64}}:
- (1, 1, 1)
- (1, 1, 5)
- (1, 1, 9)
- (1, 1, 13)
- (1, 1, 17)Let's use this array to define the range of hyperparameters and pass it along with the model to the TunedModel constructor.
r1 = range(clf, :(builder.hidden), values = networks_space)
-
-tuned_clf = TunedModel(
- model = clf,
- tuning = RandomSearch(),
- resampling = CV(nfolds = 4, rng = 42),
- range = [r1],
- measure = cross_entropy,
- n = 100, # searching over 100 random samples are enough
-);
-nothing #hideSimilar to the last workflow example, all we need now is to fit our model and the search will take place automatically:
mach = machine(tuned_clf, X, y);
-fit!(mach, verbosity = 0);
-fitted_params(mach).best_modelLet's analyze the search results by converting the history array to a dataframe and viewing it:
history = report(mach).history
-history_df = DataFrame(
- mlp = [x[:model].builder for x in history],
- measurement = [x[:measurement][1] for x in history],
-)
-first(sort!(history_df, [order(:measurement)]), 10)This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this workflow example, we see how we can compare different machine learning models with a neural network from MLJFlux.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-import RDatasets # Dataset source
-using DataFrames # To visualize hyperparameter search resultsiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng=123);Now let's construct our model. This follows a similar setup to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
-
-clf1 = NeuralNetworkClassifier(
- builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
- batch_size=8,
- epochs=50,
- rng=42
- )NeuralNetworkClassifier(
- builder = MLP(
- hidden = (5, 4),
- σ = NNlib.relu),
- finaliser = NNlib.softmax,
- optimiser = Adam(0.01, (0.9, 0.999), 1.0e-8, IdDict{Any, Any}()),
- loss = Flux.Losses.crossentropy,
- epochs = 50,
- batch_size = 8,
- lambda = 0.0,
- alpha = 0.0,
- rng = 42,
- optimiser_changes_trigger_retraining = false,
- acceleration = CPU1{Nothing}(nothing))Let's as well load and construct three other classical machine learning models:
BayesianLDA = @load BayesianLDA pkg=MultivariateStats
-clf2 = BayesianLDA()
-RandomForestClassifier = @load RandomForestClassifier pkg=DecisionTree
-clf3 = RandomForestClassifier()
-XGBoostClassifier = @load XGBoostClassifier pkg=XGBoost
-clf4 = XGBoostClassifier();[ Info: For silent loading, specify `verbosity=0`.
-import MLJMultivariateStatsInterface ✔
-[ Info: For silent loading, specify `verbosity=0`.
-import MLJDecisionTreeInterface ✔
-[ Info: For silent loading, specify `verbosity=0`.
-import MLJXGBoostInterface ✔
-Instead of just comparing with four models with the default/given hyperparameters, we will give XGBoostClassifier an unfair advantage By wrapping it in a TunedModel that considers the best learning rate η for the model.
r1 = range(clf4, :eta, lower=0.01, upper=0.5, scale=:log10)
-tuned_model_xg = TunedModel(
- model=clf4,
- ranges=[r1],
- tuning=Grid(resolution=10),
- resampling=CV(nfolds=5, rng=42),
- measure=cross_entropy,
-);Of course, one can wrap each of the four in a TunedModel if they are interested in comparing the models over a large set of their hyperparameters.
We simply pass the four models to the models argument of the TunedModel construct
tuned_model = TunedModel(
- models=[clf1, clf2, clf3, tuned_model_xg],
- tuning=Explicit(),
- resampling=CV(nfolds=5, rng=42),
- measure=cross_entropy,
-);Then wrapping our tuned model in a machine and fitting it.
mach = machine(tuned_model, X, y);
-fit!(mach, verbosity=0);┌ Warning: Layer with Float32 parameters got Float64 input.
-│ The input will be converted, but any earlier layers may be very slow.
-│ layer = Dense(4 => 5, relu) # 25 parameters
-│ summary(x) = "4×8 Matrix{Float64}"
-└ @ Flux ~/.julia/packages/Flux/Wz6D4/src/layers/stateless.jl:60
-Now let's see the history for more details on the performance for each of the models
history = report(mach).history
-history_df = DataFrame(mlp = [x[:model] for x in history], measurement = [x[:measurement][1] for x in history])
-sort!(history_df, [order(:measurement)])| Row | mlp | measurement |
|---|---|---|
| Probabil… | Float64 | |
| 1 | BayesianLDA(method = gevd, …) | 0.0610826 |
| 2 | RandomForestClassifier(max_depth = -1, …) | 0.106565 |
| 3 | NeuralNetworkClassifier(builder = MLP(hidden = (5, 4), …), …) | 0.113266 |
| 4 | ProbabilisticTunedModel(model = XGBoostClassifier(test = 1, …), …) | 0.221056 |
This is Occam's razor in practice.
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this workflow example, we see how MLJFlux enables composing MLJ models with MLJFlux models. We will assume a class imbalance setting and wrap an oversampler with a deep learning model from MLJFlux.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-import RDatasets # Dataset source
-import Random # To create imbalance
-import Imbalance # To solve the imbalanceiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
-X = Float32.(X); # To be compatible with type of network network parameters
-nothing #hideTo simulate an imbalanced dataset, we will take a random sample:
Random.seed!(803429)
-subset_indices = rand(1:size(X, 1), 100)
-X, y = X[subset_indices, :], y[subset_indices]
-Imbalance.checkbalance(y)Let's load BorderlineSMOTE1 to oversample the data and Standardizer to standardize it.
BorderlineSMOTE1 = @load BorderlineSMOTE1 pkg=Imbalance verbosity=0
-NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
-# We didn't need to load Standardizer because it is a local model for MLJ (see `localmodels()`)
-
-clf = NeuralNetworkClassifier(
- builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
- batch_size=8,
- epochs=50,
- rng=42
- )First we wrap the oversampler with the neural network via the BalancedModel construct. This comes from MLJBalancing And allows combining resampling methods with MLJ models in a sequential pipeline.
oversampler = BorderlineSMOTE1(k=5, ratios=1.0, rng=42)
-balanced_model = BalancedModel(model=clf, balancer1=oversampler)
-standarizer = Standardizer()Now let's compose the balanced model with a standardizer.
pipeline = standarizer |> balanced_modelBy this, any training data will be standardized then oversampled then passed to the model. Meanwhile, for inference, the standardizer will automatically use the training set's mean and std and the oversampler will be transparent.
It's indistinguishable from training a single model.
mach = machine(pipeline, X, y)
-fit!(mach)
-cv=CV(nfolds=5)
-evaluate!(mach, resampling=cv, measure=accuracy)This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this workflow example, we learn how MLJFlux enables us to easily use early stopping when training MLJFlux models.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-import RDatasets # Dataset source
-using Plots # To visualize trainingiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
-X = Float32.(X); # To be compatible with type of network network parameters
-nothing #hideNow let's construct our model. This follows a similar setup to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
-
-clf = NeuralNetworkClassifier(
- builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
- batch_size=8,
- epochs=50,
- rng=42
- )Let's start by defining the condition that can cause the model to early stop.
stop_conditions = [
- Step(1), # Repeatedly train for one iteration
- NumberLimit(100), # Don't train for more than 100 iterations
- Patience(5), # Stop after 5 iterations of disimprovement in validation loss
- NumberSinceBest(9), # Or if the best loss occurred 9 iterations ago
- TimeLimit(30/60), # Or if 30 minutes passed
-]We can also define callbacks. Here we want to store the validation loss for each iteration
validation_losses = []
-callbacks = [
- WithLossDo(loss->push!(validation_losses, loss)),
-]Construct the iterated model and pass to it the stop_conditions and the callbacks:
iterated_model = IteratedModel(model=clf,
- resampling=CV(nfolds=6), # Split the data internally into 0.7 training and 0.3 validation
- measures=log_loss,
- iteration_parameter=:(epochs),
- controls=vcat(stop_conditions, callbacks),
- retrain=false # no need to retrain on all data at the end
- );
-nothing #hideYou can see more advanced stopping conditions as well as how to involve callbacks in the documentation
At this point, all we need is to fit the model and iteration controls will be automatically handled
mach = machine(iterated_model, X, y)
-fit!(mach)
-# We can get the training losses like so
-training_losses = report(mach)[:model_report].training_losses;
-nothing #hideWe can see that the model converged after 100 iterations.
plot(training_losses, label="Training Loss", linewidth=2)
-plot!(validation_losses, label="Validation Loss", linewidth=2, size=(800,400))using Literate #srcThis page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this workflow example we learn how to tune different hyperparameters of MLJFlux models with emphasis on training hyperparameters.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-import RDatasets # Dataset source
-using Plots # To plot tuning resultsiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
-X = Float32.(X); # To be compatible with type of network network parameters
-nothing #hideNow let's construct our model. This follows a similar setup the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
-clf = NeuralNetworkClassifier(
- builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
- batch_size=8,
- epochs=10,
- rng=42
- )Let's tune the batch size and the learning rate. We will use grid search and 5-fold cross-validation.
We start by defining the hyperparameter ranges
r1 = range(clf, :batch_size, lower=1, upper=64)
-r2 = range(clf, :(optimiser.eta), lower=10^-4, upper=10^0, scale=:log10)Then passing the ranges along with the model and other arguments to the TunedModel constructor.
tuned_model = TunedModel(
- model=clf,
- tuning=Grid(goal=25),
- resampling=CV(nfolds=5, rng=42),
- range=[r1, r2],
- measure=cross_entropy,
-);
-nothing #hideThen wrapping our tuned model in a machine and fitting it.
mach = machine(tuned_model, X, y);
-fit!(mach, verbosity=0);
-nothing #hideLet's check out the best performing model:
fitted_params(mach).best_modelWe can visualize the hyperparameter search results as follows
plot(mach)With learning curves, it's possible to center our focus on the effects of a single hyperparameter of the model
First define the range and wrap it in a learning curve
r = range(clf, :epochs, lower=1, upper=200, scale=:log10)
-curve = learning_curve(clf, X, y,
- range=r,
- resampling=CV(nfolds=4, rng=42),
- measure=cross_entropy)Then plot the curve
plot(curve.parameter_values,
- curve.measurements,
- xlab=curve.parameter_name,
- xscale=curve.parameter_scale,
- ylab = "Cross Entropy")This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
In this workflow example we explore how to incrementally train MLJFlux models.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-import RDatasets # Dataset sourceiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
-X = Float32.(X) # To be compatible with type of network network parameters
-(X_train, X_test), (y_train, y_test) = partition((X, y), 0.8,
- multi = true,
- shuffle = true,
- rng=42);
-nothing #hideNow let's construct our model. This follows a similar setup to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
-clf = NeuralNetworkClassifier(
- builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
- batch_size=8,
- epochs=10,
- rng=42
- )Now let's train the model. Calling fit! will automatically train it for 100 epochs as specified above.
mach = machine(clf, X_train, y_train)
-fit!(mach)Let's evaluate the training loss and validation accuracy
training_loss = cross_entropy(predict(mach, X_train), y_train)val_acc = accuracy(predict_mode(mach, X_test), y_test)Poor performance it seems.
Now let's train it for another 30 epochs at half the original learning rate. All we need to do is changes these hyperparameters and call fit again. It won't reset the model parameters before training.
clf.optimiser.eta = clf.optimiser.eta / 2
-clf.epochs = clf.epochs + 30
-fit!(mach, verbosity=2);
-nothing #hideLet's evaluate the training loss and validation accuracy
training_loss = cross_entropy(predict(mach, X_train), y_train)training_acc = accuracy(predict_mode(mach, X_test), y_test)That's much better. If we are rather interested in resetting the model parameters before fitting, we can do fit(mach, force=true).
This page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.
Julia version is assumed to be 1.10.*
using MLJ # Has MLJFlux models
-using Flux # For more flexibility
-import RDatasets # Dataset source
-using Plots # For training plotiris = RDatasets.dataset("datasets", "iris");
-y, X = unpack(iris, ==(:Species), colname -> true, rng=123);
-X = Float32.(X); # To be compatible with type of network network parameters
-nothing #hideNow let's construct our model. This follows a similar setup to the one followed in the Quick Start.
NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux
-
-clf = NeuralNetworkClassifier(
- builder=MLJFlux.MLP(; hidden=(5,4), σ=Flux.relu),
- optimiser=Flux.ADAM(0.01),
- batch_size=8,
- epochs=50,
- rng=42
- )Now let's wrap this in an iterated model. We will use a callback that makes a plot for validation losses each iteration.
stop_conditions = [
- Step(1), # Repeatedly train for one iteration
- NumberLimit(100), # Don't train for more than 100 iterations
-]
-
-validation_losses = []
-gr(reuse=true) # use the same window for plots
-function plot_loss(loss)
- push!(validation_losses, loss)
- display(plot(validation_losses, label="validation loss", xlim=(1, 100)))
- sleep(.01) # to catch up with the plots while they are being generated
-end
-
-callbacks = [ WithLossDo(plot_loss),]
-
-iterated_model = IteratedModel(model=clf,
- resampling=Holdout(), # Split the data internally into 0.7 training and 0.3 validation
- measures=log_loss,
- iteration_parameter=:(epochs),
- controls=vcat(stop_conditions, callbacks),
- retrain=true # no need to retrain on all data at the end
- )Simply fitting the model is all we need
mach = machine(iterated_model, X, y)
-fit!(mach, force=true)using Literate #srcThis page was generated using Literate.jl.
Settings
This document was generated with Documenter.jl version 1.4.1 on Tuesday 11 June 2024. Using Julia version 1.10.4.