From 6746b8b51e79b9b7d24e7a01b3786d776d233ce0 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Wed, 12 Jun 2024 16:32:07 +1200 Subject: [PATCH] final doc tweaks; no more substantive warnings from Documenter --- docs/Project.toml | 10 + .../architecture_search/README.md | 2 +- .../src/common_workflows/comparison/README.md | 2 +- .../common_workflows/composition/README.md | 2 +- .../common_workflows/early_stopping/README.md | 2 +- .../early_stopping/notebook.jl | 2 +- .../hyperparameter_tuning/README.md | 2 +- .../incremental_training/README.md | 2 +- .../common_workflows/live_training/README.md | 2 +- docs/src/extended_examples/MNIST/README.md | 6 +- .../extended_examples/MNIST/notebook.ipynb | 1815 +++++++++++++++-- docs/src/extended_examples/MNIST/notebook.jl | 20 +- docs/src/extended_examples/MNIST/notebook.md | 17 +- .../MNIST/notebook.unexecuted.ipynb | 8 +- .../spam_detection/Project.toml | 1 - .../spam_detection/README.md | 2 +- .../spam_detection/notebook.ipynb | 16 +- .../spam_detection/notebook.jl | 1 - .../spam_detection/notebook.md | 1 - .../spam_detection/notebook.unexecuted.ipynb | 1 - docs/src/index.md | 47 +- docs/src/interface/Custom Builders.md | 4 +- docs/src/interface/Summary.md | 98 +- 23 files changed, 1731 insertions(+), 332 deletions(-) diff --git a/docs/Project.toml b/docs/Project.toml index b6718d0d..c86ddfde 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -1,10 +1,20 @@ [deps] +CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b" DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" DocumenterTools = "35a29f4d-8980-5a13-9543-d66fff28ecb8" Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c" +Imbalance = "c709b415-507b-45b7-9a3d-1767c89fde68" +Languages = "8ef0a80b-9436-5d2c-a485-80b904378c43" MLDatasets = "eb30cadb-4394-5ae3-aed4-317e484a6458" MLJ = "add582a8-e3ab-11e8-2d5e-e98b27df1bc7" +MLJDecisionTreeInterface = "c6f25543-311c-4c74-83dc-3ea6d1015661" MLJFlux = "094fc8d1-fd35-5302-93ea-dabda2abf845" +MLJIteration = "614be32b-d00c-4edb-bd02-1eb411ab5e55" +MLJMultivariateStatsInterface = "1b6a4a23-ba22-4f51-9698-8599985d3728" +MLJXGBoostInterface = "54119dfa-1dab-4055-a167-80440f4f7a91" +MLUtils = "f1d291b0-491e-4a28-83b9-f70985020b54" Optimisers = "3bd65402-5787-11e9-1adc-39752487f4e2" +Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" RDatasets = "ce6b1742-4840-55fa-b093-852dadbb1d8b" +WordTokenizers = "796a5d58-b03d-544a-977e-18100b691f6e" diff --git a/docs/src/common_workflows/architecture_search/README.md b/docs/src/common_workflows/architecture_search/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/architecture_search/README.md +++ b/docs/src/common_workflows/architecture_search/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/common_workflows/comparison/README.md b/docs/src/common_workflows/comparison/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/comparison/README.md +++ b/docs/src/common_workflows/comparison/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/common_workflows/composition/README.md b/docs/src/common_workflows/composition/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/composition/README.md +++ b/docs/src/common_workflows/composition/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/common_workflows/early_stopping/README.md b/docs/src/common_workflows/early_stopping/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/early_stopping/README.md +++ b/docs/src/common_workflows/early_stopping/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/common_workflows/early_stopping/notebook.jl b/docs/src/common_workflows/early_stopping/notebook.jl index 7cbf9094..a6c59da3 100644 --- a/docs/src/common_workflows/early_stopping/notebook.jl +++ b/docs/src/common_workflows/early_stopping/notebook.jl @@ -1,4 +1,4 @@ -# # Early Stopping with MLJFlux +# # Early Stopping with MLJ # This demonstration is available as a Jupyter notebook or julia script # [here](https://github.com/FluxML/MLJFlux.jl/tree/dev/docs/src/common_workflows/early_stopping). diff --git a/docs/src/common_workflows/hyperparameter_tuning/README.md b/docs/src/common_workflows/hyperparameter_tuning/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/hyperparameter_tuning/README.md +++ b/docs/src/common_workflows/hyperparameter_tuning/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/common_workflows/incremental_training/README.md b/docs/src/common_workflows/incremental_training/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/incremental_training/README.md +++ b/docs/src/common_workflows/incremental_training/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/common_workflows/live_training/README.md b/docs/src/common_workflows/live_training/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/common_workflows/live_training/README.md +++ b/docs/src/common_workflows/live_training/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/extended_examples/MNIST/README.md b/docs/src/extended_examples/MNIST/README.md index 4e6605d1..b68a07e7 100644 --- a/docs/src/extended_examples/MNIST/README.md +++ b/docs/src/extended_examples/MNIST/README.md @@ -1,9 +1,5 @@ # Contents -**Important.** This folder was updated in June 2024 but will no longer be updated. - -For the lastest version of this example see [here](/docs/src/full\ tutorials/MNIST). - | file | description | |:----------------------------|:---------------------------------------------------------| | `notebook.ipynb` | Juptyer notebook (executed) | @@ -15,5 +11,5 @@ For the lastest version of this example see [here](/docs/src/full\ tutorials/MNI # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/extended_examples/MNIST/notebook.ipynb b/docs/src/extended_examples/MNIST/notebook.ipynb index 6f955f3e..617be38e 100644 --- a/docs/src/extended_examples/MNIST/notebook.ipynb +++ b/docs/src/extended_examples/MNIST/notebook.ipynb @@ -2,159 +2,999 @@ "cells": [ { "cell_type": "markdown", + "metadata": {}, "source": [ "# Using MLJ to classifiy the MNIST image dataset" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "This tutorial is available as a Jupyter notebook or julia script\n", "[here](https://github.com/FluxML/MLJFlux.jl/tree/dev/docs/src/extended_examples/MNIST)." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32m\u001b[1m Activating\u001b[22m\u001b[39m project at `~/GoogleDrive/Julia/MLJ/MLJFlux/docs/src/extended_examples/MNIST`\n" + ] + } + ], "source": [ "using Pkg\n", "const DIR = @__DIR__\n", "Pkg.activate(DIR)\n", "Pkg.instantiate()" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "**Julia version** is assumed to be 1.10.*" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], "source": [ "using MLJ\n", "using Flux\n", "import MLJFlux\n", "import MLUtils\n", "import MLJIteration # for `skip`" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "If running on a GPU, you will also need to `import CUDA` and `import cuDNN`." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], "source": [ "using Plots\n", "gr(size=(600, 300*(sqrt(5)-1)));" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Basic training" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Downloading the MNIST image dataset:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], "source": [ "import MLDatasets: MNIST\n", "\n", "ENV[\"DATADEPS_ALWAYS_ACCEPT\"] = true\n", "images, labels = MNIST(split=:train)[:];" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "In MLJ, integers cannot be used for encoding categorical data, so we\n", "must force the labels to have the `Multiclass` [scientific\n", "type](https://juliaai.github.io/ScientificTypes.jl/dev/). For\n", "more on this, see [Working with Categorical\n", "Data](https://alan-turing-institute.github.io/MLJ.jl/dev/working_with_categorical_data/)." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], "source": [ "labels = coerce(labels, Multiclass);\n", "images = coerce(images, GrayImage);" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Checking scientific types:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], "source": [ "@assert scitype(images) <: AbstractVector{<:Image}\n", "@assert scitype(labels) <: AbstractVector{<:Finite}" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Looks good." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "For general instructions on coercing image data, see [Type coercion\n", "for image\n", "data](https://juliaai.github.io/ScientificTypes.jl/dev/#Type-coercion-for-image-data)" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "" + ], + "text/plain": [ + "28×28 Array{Gray{Float32},2} with eltype Gray{Float32}:\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) … Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) … Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) … Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " ⋮ ⋱ \n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) … Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) … Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)\n", + " Gray{Float32}(0.0) Gray{Float32}(0.0) Gray{Float32}(0.0)" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "images[1]" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We start by defining a suitable `Builder` object. This is a recipe\n", "for building the neural network. Our builder will work for images of\n", @@ -163,12 +1003,13 @@ "alternating convolution and max-pool layers, and a final dense\n", "layer; the filter size and the number of channels after each\n", "convolution layer is customisable." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], "source": [ "import MLJFlux\n", "struct MyConvBuilder\n", @@ -194,19 +1035,18 @@ " d = Flux.outputsize(front, (n_in..., n_channels, 1)) |> first\n", " return Chain(front, Dense(d, n_out, init=init))\n", "end" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "**Notes.**" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "- There is no final `softmax` here, as this is applied by default in all MLJFLux\n", " classifiers. Customisation of this behaviour is controlled using using the `finaliser`\n", @@ -214,19 +1054,56 @@ "\n", "- Instead of calculating the padding `p`, Flux can infer the required padding in each\n", " dimension, which you enable by replacing `pad = (p, p)` with `pad = SamePad()`." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "We now define the MLJ model." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mFor silent loading, specify `verbosity=0`. \n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "import MLJFlux ✔\n" + ] + }, + { + "data": { + "text/plain": [ + "ImageClassifier(\n", + " builder = MyConvBuilder(3, 16, 32, 32), \n", + " finaliser = NNlib.softmax, \n", + " optimiser = Adam(0.001, (0.9, 0.999), 1.0e-8, IdDict{Any, Any}()), \n", + " loss = Flux.Losses.crossentropy, \n", + " epochs = 10, \n", + " batch_size = 50, \n", + " lambda = 0.0, \n", + " alpha = 0.0, \n", + " rng = 123, \n", + " optimiser_changes_trigger_retraining = false, \n", + " acceleration = CPU1{Nothing}(nothing))" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "ImageClassifier = @load ImageClassifier\n", "clf = ImageClassifier(\n", @@ -235,184 +1112,303 @@ " epochs=10,\n", " rng=123,\n", ")" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "You can add Flux options `optimiser=...` and `loss=...` in the above constructor\n", "call. At present, `loss` must be a Flux-compatible loss, not an MLJ measure. To run on a\n", "GPU, add to the constructor `acceleration=CUDALib()` and omit `rng`." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "For illustration purposes, we won't use all the data here:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "501:1000" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "train = 1:500\n", "test = 501:1000" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Binding the model with data in an MLJ machine:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], "source": [ "mach = machine(clf, images, labels);" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Training for 10 epochs on the first 500 images:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mTraining machine(ImageClassifier(builder = MyConvBuilder(3, 16, 32, 32), …), …).\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 2.291\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 2.208\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 2.049\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 1.685\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 1.075\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 0.628\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 0.4639\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 0.361\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 0.2921\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mLoss is 0.2478\n" + ] + } + ], "source": [ "fit!(mach, rows=train, verbosity=2);" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Inspecting:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(training_losses = Float32[2.3242702, 2.2908378, 2.20822, 2.0489829, 1.6850392, 1.0751165, 0.6279615, 0.46388212, 0.36103815, 0.29207793, 0.2478443],)" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "report(mach)" - ], - "metadata": {}, - "execution_count": null + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(chain = Chain(Chain(Chain(Conv((3, 3), 1 => 16, relu, pad=1), MaxPool((2, 2)), Conv((3, 3), 16 => 32, relu, pad=1), MaxPool((2, 2)), Conv((3, 3), 32 => 32, relu, pad=1), MaxPool((2, 2)), flatten), Dense(288 => 10)), softmax),)" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "chain = fitted_params(mach)" - ], - "metadata": {}, - "execution_count": null + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "16-element Vector{Float32}:\n", + " 0.011803599\n", + " 0.05579675\n", + " 8.461591f-5\n", + " 0.013422165\n", + " -0.001925053\n", + " 0.011568692\n", + " -0.00051727734\n", + " -0.0003228416\n", + " 0.03614383\n", + " 0.06365696\n", + " -0.0005846103\n", + " -0.004092362\n", + " 0.0036211032\n", + " 0.0031117066\n", + " 0.02764553\n", + " 0.05152524" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "Flux.params(chain)[2]" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Adding 20 more epochs:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mUpdating machine(ImageClassifier(builder = MyConvBuilder(3, 16, 32, 32), …), …).\n", + "\u001b[33mOptimising neural net: 100%[=========================] Time: 0:00:30\u001b[39m\n" + ] + } + ], "source": [ "clf.epochs = clf.epochs + 20\n", "fit!(mach, rows=train);" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Computing an out-of-sample estimate of the loss:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.36284237158113225" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "predicted_labels = predict(mach, rows=test);\n", "cross_entropy(predicted_labels, labels[test])" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Or to fit and predict, in one line:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "PerformanceEvaluation object with these fields:\n", + " model, measure, operation,\n", + " measurement, per_fold, per_observation,\n", + " fitted_params_per_fold, report_per_fold,\n", + " train_test_rows, resampling, repeats\n", + "Extract:\n", + "┌──────────────────────┬───────────┬─────────────┐\n", + "│\u001b[22m measure \u001b[0m│\u001b[22m operation \u001b[0m│\u001b[22m measurement \u001b[0m│\n", + "├──────────────────────┼───────────┼─────────────┤\n", + "│ LogLoss( │ predict │ 0.363 │\n", + "│ tol = 2.22045e-16) │ │ │\n", + "└──────────────────────┴───────────┴─────────────┘\n" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "evaluate!(mach,\n", " resampling=Holdout(fraction_train=0.5),\n", " measure=cross_entropy,\n", " rows=1:1000,\n", " verbosity=0)" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "## Wrapping the MLJFlux model with iteration controls" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Any iterative MLJFlux model can be wrapped in *iteration controls*,\n", "as we demonstrate next. For more on MLJ's `IteratedModel` wrapper,\n", "see the [MLJ\n", "documentation](https://alan-turing-institute.github.io/MLJ.jl/dev/controlling_iterative_models/)." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The \"self-iterating\" classifier, called `iterated_clf` below, is for\n", "iterating the image classifier defined above until one of the\n", "following stopping criterion apply:" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "- `Patience(3)`: 3 consecutive increases in the loss\n", "- `InvalidValue()`: an out-of-sample loss, or a training loss, is `NaN`, `Inf`, or `-Inf`\n", @@ -425,98 +1421,121 @@ "- save a snapshot of the machine every three control cycles (every six epochs)\n", "- record traces of the out-of-sample loss and training losses for plotting\n", "- record mean value traces of each Flux parameter for plotting" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "For a complete list of controls, see [this\n", "table](https://alan-turing-institute.github.io/MLJ.jl/dev/controlling_iterative_models/#Controls-provided)." - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Wrapping the classifier" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Some helpers" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "To extract Flux params from an MLJFlux machine" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], "source": [ "parameters(mach) = vec.(Flux.params(fitted_params(mach)));" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "To store the traces:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Any[]" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "losses = []\n", "training_losses = []\n", "parameter_means = Float32[];\n", "epochs = []" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "To update the traces:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "update_epochs (generic function with 1 method)" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "update_loss(loss) = push!(losses, loss)\n", "update_training_loss(losses) = push!(training_losses, losses[end])\n", "update_means(mach) = append!(parameter_means, mean.(parameters(mach)));\n", "update_epochs(epoch) = push!(epochs, epoch)" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The controls to apply:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [], "source": [ "save_control =\n", - " MLJIteration.skip(Save(joinpath(DIR, \"mnist.jls\")), predicate=3)\n", + " MLJIteration.skip(Save(joinpath(tempdir(), \"mnist.jls\")), predicate=3)\n", "\n", "controls=[\n", " Step(2),\n", @@ -530,20 +1549,56 @@ " Callback(update_means),\n", " WithIterationsDo(update_epochs),\n", "];" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "The \"self-iterating\" classifier:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ProbabilisticIteratedModel(\n", + " model = ImageClassifier(\n", + " builder = MyConvBuilder(3, 16, 32, 32), \n", + " finaliser = NNlib.softmax, \n", + " optimiser = Adam(0.001, (0.9, 0.999), 1.0e-8, IdDict{Any, Any}()), \n", + " loss = Flux.Losses.crossentropy, \n", + " epochs = 30, \n", + " batch_size = 50, \n", + " lambda = 0.0, \n", + " alpha = 0.0, \n", + " rng = 123, \n", + " optimiser_changes_trigger_retraining = false, \n", + " acceleration = CPU1{Nothing}(nothing)), \n", + " controls = Any[Step(2), Patience(3), InvalidValue(), TimeLimit(Dates.Millisecond(300000)), IterationControl.Skip{Save{typeof(Serialization.serialize)}, IterationControl.var\"#8#9\"{Int64}}(Save{typeof(Serialization.serialize)}(\"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/mnist.jls\", Serialization.serialize), IterationControl.var\"#8#9\"{Int64}(3)), WithLossDo{IterationControl.var\"#20#22\"}(IterationControl.var\"#20#22\"(), false, nothing), WithLossDo{typeof(update_loss)}(update_loss, false, nothing), WithTrainingLossesDo{typeof(update_training_loss)}(update_training_loss, false, nothing), Callback{typeof(update_means)}(update_means, false, nothing, false), WithIterationsDo{typeof(update_epochs)}(update_epochs, false, nothing)], \n", + " resampling = Holdout(\n", + " fraction_train = 0.7, \n", + " shuffle = false, \n", + " rng = Random._GLOBAL_RNG()), \n", + " measure = LogLoss(tol = 2.22045e-16), \n", + " weights = nothing, \n", + " class_weights = nothing, \n", + " operation = MLJModelInterface.predict, \n", + " retrain = false, \n", + " check_measure = true, \n", + " iteration_parameter = nothing, \n", + " cache = true)" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "iterated_clf = IteratedModel(\n", " clf,\n", @@ -551,52 +1606,90 @@ " resampling=Holdout(fraction_train=0.7),\n", " measure=log_loss,\n", ")" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Binding the wrapped model to data:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], "source": [ "mach = machine(iterated_clf, images, labels);" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Training" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mTraining machine(ProbabilisticIteratedModel(model = ImageClassifier(builder = MyConvBuilder(3, 16, 32, 32), …), …), …).\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mNo iteration parameter specified. Using `iteration_parameter=:(epochs)`. \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 2.2247422992833092\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 1.9681479167178544\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mSaving \"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/mnist1.jls\". \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 1.220910971646785\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.5940933327640742\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.46833501799372196\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mSaving \"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/mnist2.jls\". \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.4241402839593314\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.40840895980242126\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.404754883332919\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mSaving \"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/mnist3.jls\". \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.4097772917650752\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.420399235463716\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.43216415903189187\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mfinal loss: 0.43216415903189187\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mfinal training loss: 0.043363843\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mStop triggered by Patience(3) stopping criterion. \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mTotal of 22 iterations. \n" + ] + } + ], "source": [ "fit!(mach, rows=train);" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Comparison of the training and out-of-sample losses:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/loss.png\"" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "plot(\n", " epochs,\n", @@ -607,21 +1700,154 @@ ")\n", "plot!(epochs, training_losses, label=\"training\")\n", "\n", - "savefig(joinpath(DIR, \"loss.png\"))" - ], - "metadata": {}, - "execution_count": null + "savefig(joinpath(tempdir(), \"loss.png\"))" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Evolution of weights" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAFyCAIAAACm2zNGAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3xTVf8H8HNH9mjTRVs6aFktZW+QPRUFBAEVER4VUZYoKqDIgwwRfUR8/DErIKKobJmPIEuGRWiBUlo2lLbQPTNvcu89vz9OGmLKpk3a5vt++fKV3Nwk34Q0n5xzzzmXwhgjAAAAwFvRni4AAAAA8CQIQnB38fHxr7322vXr1z1dCABV67E/6p9//vlbb72l1+uroirgThCE3mj+/PnsPZw5c4bs8+eff65duzYvL8+zpdZimzdvjo+Pt9lsni7E2z32R/23336Lj483m8333+3o0aPx8fFZWVmPWyCocqynCwAeIIqiIAgRERENGzZ0uUmj0XikJC80d+7clJSUl19+WSKReLoWr9a0adPevXv7+PhU0eP/9NNP8fHxe/fuDQsLq6KnAE8IgtB7vfDCC19//bWnqwDAwz766KOPPvrI01UAT4IgBA+F5/nk5GSVShUTE+O8PS8vLzMzMzw8PCgoCCGUnp5eWFgYFBQUHh7uvFtGRkZ+fn5AQEBkZOS9nuLs2bMIoZYtW5aWlu7atevWrVvBwcHPPfecn5+fy55Go/Ho0aM3b94sKSmpU6dOjx496tWr57yDIAhnz55VKpWxsbHFxcV79uy5detWx44du3XrRl7LiRMnrl27lpOTo9FoOnTo0KZNG5enSEtLM5vNrVq1slqtu3fvvnHjRlBQ0KBBg3x9fckOiYmJCQkJPM/37NmzZcuWd33Hjh07du7cOY7joqOj+/Xr52htm0ymCxcukC41UidCSCqVNmvWzHF3juMOHTp08eJFQRAaNWrUt29fuVzu/PinT59mWbZ58+YGg2HPnj0ZGRlxcXHPPPPMXd/b1NRUi8XSunVri8Wye/fu9PT0OnXqDB48WKvVkh1Onjx54sQJQRB69+7dvHnzu76cI0eOpKSkWK3W+vXr9+vXT61Wu+xTXFx8/PjxmzdvGgyG8PDwnj17hoSEOO9gNpvT0tJ8fX3r16+fm5u7e/fu/Pz8qKioAQMGVHw0Zy4fM8c7gDFu0KCBozFntVpTUlI0Gk2jRo0cu2GMExMTT548aTAYwsLC+vXrFxgY6Pzg165dKygoiIuLc64BY3zs2LGkpCSaptu3b9+xY8f8/PyMjIywsLA6depUfHN+//33S5cuyeXyvn37Oj/7mTNn8vPzEUJXrlzx9/cnG5s1ayaVSskdDx48eO3aNYPBEBAQ0KhRow4dOrAsfC27HQbeZ86cOQih99577z77jBw5EiGUkJBArhYUFCCE2rdv77Lb4sWLEULffvstuXrq1CmpVBoYGHjr1i3HPpmZmQEBATKZLCkp6T7P6OPjo9Pp/vjjD51O5/h86nS6Xbt2Oe/25ZdfymQy588wTdOTJ08WBMGxT1FREUKoVatWmzZtcny7TZ48GWN84MAB58cn+vTpU1hY6PwsTZs2RQidOHEiKirKsZufn9+pU6eMRuMLL7zg2EhR1Jw5c1xey99//924cWPnpwgICNi+fTu59fTp0xX/EsPCwhx3//33311+SYSHhx87dsz5KWQyWUhIyB9//BEQEED2GT58+L3eW/LVfOrUqYiICMdjBgYGnjlzRq/XDx482PnlLFiwwOXux48fb9CggXM9gYGBe/bscd7nvffec/kGl0ql8+fPd97n3LlzCKFBgwZ9//33zrkeFhaWlpZ2r+Ixxn/88QdC6K233nJsSU9PJ/f9+uuvHRv/97//IYQmTpzo2HL9+vWOHTs6V6VUKpcuXer84C4fdYxxWVlZv379nO/1/PPPL1q0CCG0aNEix27t27cnHxLyaSEYhvnPf/7j2Mfl5wuRmZmJMT527Fh0dLTLTV26dLnP+wCqCAShN6q6IHRs6d69O8/zGGObzdalSxeE0LJly+5flY+Pj1wu9/X1feeddy5evJiRkfH1119LpVK5XO78LfnRRx+99dZbu3btSk1NvXz58saNG2NjY12+EEkQBgQEKJXKqVOn/v7773/99deRI0cwxps3bx48ePCGDRtOnz59/fr1/fv39+/fHyE0ZMgQ52LIV1t4ePjIkSMPHTqUlJQ0ceJEhFCTJk1ef/31xo0bb968OSUlZc2aNVqtlqIo0johUlNT1Wq1QqH49NNPExMTL126tGLFCl9fX6lUmpiYiDE2Go2JiYkkWo4cOZKYmJiYmHju3Dly98OHD7Ms6+Pjs2jRorNnz6ampv7nP/9RKBRarfb69euOZ5HJZGq12tfX980339y1a1dCQsIff/xxr/eWBGF4ePjo0aMPHz6cmJg4btw4hFCLFi1effXVJk2abNmyJSUl5bvvvlOr1TRNp6SkOO6bnJysVCqVSuW8efOSkpIuXry4dOlSHx8fmUx29uxZx26vv/761KlT9+3bd+nSpbS0tDVr1pAs37Rpk2MfEoRhYWEqleqzzz47efLk8ePHhw4dihDq3LnzfT4bJpNJLpfXr1/fseW7775DCNE0PWDAAMfGDz74ACG0detWcjUvLy8sLIxhmAkTJiQkJFy5cmXDhg3kp4BzVRWDkJTUp0+fEydO5OTk7N27NyYmJjg4+K5BGBkZ+eyzz+7du/fcuXPffPONQqFwfgNPnz49ZMgQhNCSJUsSy3Ecx/N8WFiYRCL56quvLl68mJ2dnZSUtGLFCucUB24DQeiNSBAGBAQ0+yfnP/LHDkJRFMlf/r///W+M8fTp0xFCw4YNe2BVpIPLpVmzYMEChNCIESPuc8dbt26p1eqoqCjHFhKECKFZs2Y98HltNlubNm0oisrIyHBsJEHoUgzpAvXz8ysoKHBs/Pzzz12eqE+fPgihX375xfm+v//+O0Lo2WefdWwhHaFlZWXOu4mi2KRJE5qmDx8+7Lx99erVCKFx48Y5tpBmsfOW+yBB+Morrzg/UVxcHEIoMDCwuLjYsZ18NubNm+fYQvqTt2zZ4vyAO3bsqPjrwcX58+cpiurWrZtjCwlClxyyWq0knJx7ESrq2bMnQsjxU+Cll15SKpWDBw9Wq9VWq5VsbNWqFcMwjsb9hAkTEEJz5851fpyLFy9KpdKGDRs6trh81JOSksiPBrPZ7Njn2rVrZExTxSB8+umnRVF0bPzkk09c3kDym2Pv3r0uZZCG5n1eMnAbmD7hvcxmc8E/GY3GJ39YiqJWr14dGRk5f/78Tz755D//+U90dPSqVase8u7vvfee89UJEybI5fJdu3bxPF9xZ6vVWlxcrFAomjdvfuPGDZLWDgzDvP/++/d5LlEUi4uL9Xp99+7dMcaJiYkuO7z77rvOV7t27YoQGjVqlONgj2PjjRs3yNWcnJz9+/fXr1//pZdecr5v//79o6KiDhw4YLVa71NSUlJSWlpaly5dunfv7rx9zJgxCoWCdP05+/DDD+/zaPd5ORRFkZb6mDFjHAc+K76cjIyMI0eOxMbGkkaSw8CBA+vWrbtv3z5BECo+kcViKS4uDg0NjYiIOHXqlMutkZGRw4YNc1yVSCQkax29nXfVu3dvhNDBgwcRQhjjw4cPd+3adcCAAQaD4e+//0YIFRYWJicnt2rVihxUFkXx559/lkqlpJno0Lhx4549e165cuXq1at3faJdu3YhhN566y3nXs3o6OiBAwfedf+pU6dSFOW4Sn4GOd7AeyFFpqamunxogUfAUVnvNW7cuCoaNarT6TZu3Ni1a9fPPvtMIpGsX7/+4cemu4zU8PHxiYqKunDhQnp6OulL5Hl++fLlP//8c3p6ek5OjvPOhYWFjgNmCKHQ0NC7Pu/OnTuXLVt2/vz5nJwc53wtLCx02dNlegkZZHHXjY5ZaGTID03TM2bMcHk0QRAsFktubq7L8T9n5O56vb7i3eVy+a1btwRBYBiGbJFKpRUPMt3HY78cjHHFehBCRqOxoKCADB4xmUyLFy/eunVrRkaGy5e7yWQiA4LuWgZCiDyCy7+mi969e3/yyScHDhx44403UlJScnJypk6dSlLnwIEDXbp0OXjwoCiKZAtC6MaNGyUlJf7+/qSN6yw3NxchdPPmTZcDn8Tly5cRQqS/3VlcXNzWrVsr7u88NAYhRIbzkKe4j8DAwOeff/63336rV69ev379evbs+fTTT1d8Z4B7QBCCKtG4ceOgoKCsrKwWLVqQHqSHoVQqVSqVy8bAwMALFy441u94+eWXN2/eHBkZOXTo0Lp16+p0OoZhli5deu7cOZdWo3MoOnz77bdTpkzx9fUdOHBggwYNdDqdTCbbt2/fli1bKjY6FQqF81Xyw/+uG3H5mr0lJSUIodzc3E2bNrk8Gsuy0dHRd23aOpSWliKEbt68WfHuOp1Op9NZrVZHAf7+/jT9CJ06LpWT+z7My8nOzq5Yj0wmi46OJgsC8Dzfr1+/48ePx8TEvPLKK8HBwTqdjqKo+fPnZ2Zmurxk51B0rgTfd93jdu3a+fr6HjhwAGN84MABhFCfPn2io6NJO3v27NlkI2k4ovJ30mQyVawcIRQdHe3cjHNmsVgQQhV/Qt3rx9xd39X7vxbi119//eqrr9atW7dt27Zt27YhhNq2bbt8+fK2bds+8L6gckEQgodC/rwr9oMZDIa77v/mm29mZWUFBQUlJiYuWLCAHDh5IJPJpNfrXSb1k4YC+Ro6derU5s2b27Rpc/ToUecvoIfseuU4btasWVqt9vTp087DQckBm0pBJiR06dJl9+7dj333YcOGrVy5srJKehKknp49e5Jv6nvZuXPn8ePHn3nmmZ07dzoarAih2bNnV1YlDMN069Ztx44dKSkpBw4c8Pf3b9GiBUKod+/ea9eu1ev1Bw4ckMlkTz31lHPlYWFhpIX38Ein5e3bt12237p1qxJehhOZTDZz5syZM2feuHHj4MGDv/766/79+5955pkLFy7c9TccqDpwjBA8FF9fX4VCUbHzKjU1teLOy5cv37RpU8eOHVNSUiIjI2fPnk0O7TwMMlTBoaCgID09XaPRkAmIKSkpCKFnn33WOQUNBsNDJllGRkZZWVmbNm2cU7Dikz6JVq1aURR18uTJB66dRgZfuLSWWrdujRA6fvx4ZdXzhEg9f//9912PBTqQf5chQ4Y4p2BWVtb9ezsfFWnt/f7770ePHu3duzf5cda7d2+e59evX3/16tXOnTs7PhhRUVE6ne7q1asP7KV0QRpkLp9YjPHDf4Zd3PUf2llUVNQbb7zxxx9/DBo0qKCg4OjRo4/3ROCxQRCCh0JRVFRU1K1btxyLkSKEUlNTt2zZ4rJnSkrK+++/r9Ppfv3116CgoB9//JGm6dGjRz/koIAvv/zSuVtp8eLFVqv1hRdeIN+w5GDStWvXnO+ycOHCh1z4OCgoiKKomzdvOn8r7d2798iRIw9z94cREhLyzDPPFBQUfPHFFxVvLS4udlyuW7cuQujmzZvOO7Ru3bply5apqalr1qy5/93dIyIiom/fvtnZ2WQW3b3queu/y8yZMyu3GBKE33zzTVlZWa9evcjGXr16URQ1d+5c5NQvihBiGGbMmDEY42nTplXsqLzPO/nCCy+o1eqff/7Z+aP+008/JScnP17Z5B86MzPTeaPBYKj4U4kMwrp/5zmoCtA1Ch7WqFGjPv744+eff37OnDl169Y9ffr0woULGzZsmJaW5tjHYDCMGDHCYrH88ssvpA3XtWvX2bNnz5o1a/To0bt3777XgRlCoVCcOXNm5MiRb7/9tlKp3Lx581dffaXVaj/99FOyQ8eOHcmXVGRk5MCBA3me/+WXX1atWhUVFfXAcXoIIR8fnzZt2iQmJr744ouTJ09Wq9UHDhz47LPPGjdufOnSpSd6d5wsXbq0Q4cOs2bNSk5OHjJkSHR0dFFR0bVr17Zs2aJUKvfs2UN269Chw86dO0ePHj106FCFQqHRaCZMmEBR1Pfff9+1a9c333zzr7/+6tevX2RkZF5e3uXLl3/99ddWrVrFx8dXVp0Pafny5R07dpw+ffqZM2cGDRoUHR1dUFBw7dq1TZs2+fv7//bbbwihHj16MAzzzTff+Pn59ejRw2g0fvfdd7t37w4ODq7ERmGTJk1CQkKys7NR+eBMhFBQUFCzZs3IxAznIEQIzZkzZ9++fevWrcvIyBg9enTjxo2NRmN6evquXbsuXbp0r14Ef3//xYsXjxs3rkuXLq+88kpUVNSZM2e2bt367LPPPvADfFcdOnRACM2ePfvGjRtkMYcJEyacOHFi9OjRY8aMadeuXWRkpMVi+f3339etWxcYGNi3b99HfQrwpDwxZwN42GNMqMcYcxw3YsQIxyeHYZjZs2e7zCN89dVXEUJTp051fihBEMjftvOKGxWRlWVOnjzpPKgyPDz8+PHjzrvt2LHDediCTqfbsmXLoEGDEELnz58n+zhWlqn4LJcuXXIe5sey7L///e8vv/wSIbR8+XLHbmQeoV6vd77vvHnzEEKrVq1y3kgOQfXt29d5440bNyqudhYSEkLau4TJZHr99dcdr8V5ZZlz586RuQ3O6tWr9/333zv2ISvL3Of9dEZeMsdxzhv//e9/I4TWrVvnvPH8+fPon/MdMcbXrl1zWWkFIRQaGrp48WLHPqtXr3burw4NDf3zzz9btWqFECotLXW8LoTQoEGDXMojk0A2b978wBdCPpaRkZHOG8mUG41G45hQ6FBUVDR69GjnDluEkK+vL1lmyPkxnT/qGONff/3VMYYzLi5u+/bt8+fPRwjFx8c79iGjwHJzc53vSPLVeZo/xvjzzz8PDQ11FJCZmZmUlFRx8HCzZs3uv/oSqCIUhjPUex+9Xl9WVqbRaBxLTVZUXFxsMpmCgoJczo1w7ty55ORkhULx1FNPhYSElJaWkkkLWq1WFEUyFSw8PNzlXkajMTc3l2VZ5yW+XPj6+tI0XVRUZLVajx49mpWVFRIS0r17d5cF1RBCZWVliYmJWVlZwcHBXbt2JQcvTSZTWFgYWcKRVCKTyUivlAue5xMTEy9fvuzr69u5c+eAgICSkpKioiLyKsg+WVlZVqu1Xr16zsMyyW6BgYHOw3lsNltmZqZCoXBZVxMhdOvWrVOnThUXFwcGBkZERDRt2vSugzz1en1+fn7FN+fatWvJycmlpaV16tSJjo52WeX1xo0bNE3fZ+1WZ5mZmTabLSoqyrlBU1xcXFxcHBQU5LzGptVqzcrKUiqVZCEVlwdJTEwsKSkJDAysV69eXFycS/OooKDg9OnTOTk5ERERTz31lEQiuXXrFsdxjrfxXg9eVFRElo2tOGbYBanZ5d0uKysrKCiQSqX3Or1DQUHBiRMn8vPz/f39Q0NDW7Ro4fz5vNdHHSFUWloqkUjIMNcxY8asW7du3759jhYbeXWRkZHOQUs+D3d9Ay0WS05OjiiKERERZDm6GzduXLx4MS8vz8/PLzo6mixxANwPghBUF44g9HQhALi6detWXFycKIq3b9++//rgoCaCY4QAAPAPO3fu/Omnn0aMGNGgQQOLxXL27NnPP/+8tLT0008/hRSslSAIAQDgH2Qy2bZt2zZu3OjYIpfLP/nkk1mzZnmwKlB1IAhBdfHNN988xpA8ACpdv379ioqKTp06lZ2dbbFYQkJCOnXq5LwiK6hl4BghAAAArwYT6gEAAHg1CEIAAABeDYIQAACAV4MgBAAA4NUgCAEAAHi1mhGE77//vssi/d4MY/zAU/yAqmC1Wj1dgpeCd94jbDabl0wrqBlB+Ndff5El5wFCSBRFCEKP4DjO0yV4KXjnPYLneVEUPV2FO9SMIAQAAACqCAQhAAAArwZBCAAAwKtBEAIAAPBqEIQAAAC8GgQhAAAArwZBCAAAoGYo5tDB2/hiSSXPboTzEQIAAKimbhrw2UJ8thCdLcRni3Axh1v4Ue/E0TG+lXnu0lobhIIgtGnTRq/Xe7qQSvD0008vXbrU01UAAEDVEjC6acCpxTipACcV4JP5mKFQnI5q4ksNqUfNb0vH+lJ0FZy9u9YGoc1mu3jxYlpamqcLeVJ///13fHy8p6sAAIDKZxXRlVJ77CUV4DOF2EeK2gRQbQKocTH0d13pYIU7yqi1QYgQoigqOjra01U8qYyMDE+XAAAAlaPEis4X3Um+63ocraFI8g2PolsFUCpPhFJtDkIAAACeddt0J/bSilERh+N0VJsAqk9dakpTupkfJa0GQzYhCAEAAFQOXkSXSnFSAU4rwanFOCEXSxl7V+fohnQTX6qJjqqCY3xPCoIQAADAY9LbUHKhPfaSCnByEY5QUXE6qokOjYuh13SjA+WeLvEhQBACAAB4WLdNOK0YOQZ2ZhlxQx+qia/9IF+bAEpRA1OlBpYMAADAXZwP8iXmY05EJPb61KWmt6iq+QxuBkEIAADAzibaW3unC3FiHpNWKoYocSt/qpU/NbEJ3dKfcs98BjeDIPSw/Pz877777vTp02VlZfv27fN0OQAA78IJ6FwRPl2ITxfg0wU4rQTX11Kt/anWAdTQumKbOhJfOePpGqscBKGH5eXl3bp1q1mzZnPmzPF0LQCA2s9lDjsZ3kIGdr7agG4dQCnLY8FsxlKJR2t1FwhCN9Hr9dOmTfv6668VCgVCKC8vb968eYsXL46Li1u6dOmFCxcgCAEAVYEM7LTP5CvBF0twpPrOHPYaOrylcnn9G+AuGo3mwoULW7ZsGTVqFEJo7dq1eXl5LAvvPwCgkpXZ0LnCu6/eMi6GbhdIyWp/Z+ej8aIvYhOPGm7kzUIln7/jXrb2YXuE/GM01fjx45csWTJq1CiM8apVq1asWOGeSgAAtZvLumVZRtzM787Azhhfiqn5AzurlBcFoZJFF4ezvOimp/OVuW4ZOnTo1KlTU1NTc3JyRFHs2bOnm0oBANQuxdydmXxJBfiWCTfV1bYpDe7kRUGIENJ49MCvRCJ57bXXvvvuu+zs7LfffpuqjisNAQCqI+fJfEkF2MyjOKfkq57rltUg3hWEHjdu3LjWrVtbrVbH+QVFUUxPT8/KykIIXb9+nWGYyMhIj9YIAPA85+Q7lY/p8tPyDY+iF7arpit21lwQhG4VERHRvn17X1/fgIAAsqWsrKxv374IoaioqL59+wYGBp44ccKjNQIAPMA5+f7Owyx957R88V3oEKWn66vVIAjdymKxpKamrl+/3rHF19f32rVrHiwJAOB+Zh6dL8ZnCvHZQpxchFOKcJCCahNAtfan3omjW3en/CsMMgBVB4LQfTZv3rx06dLY2NguXbp4uhYAgFs5D+x0nswXp6OeCac6BtWMszTUVhCE7lOvXr0PP/ywV69eni4EAFDlHF2d5FwNZGAnGeEyLoZuG0h5wcplNQYEofu0bdvW0yUAAKqETUSXS+80+M4U2E9IG6ejnougZreGgZ3VGgShh23fvv3rr79OS0tTKpXDhg1bsGCBTAYHBwCo7kqtKKXozqnYzxbiSLX9hLTvxNEdAumg2niWhtoKgtDD8vLypk2b1rlz57y8vGHDhqnValh0FIBqyKWrk6zeQqY0vNoAVuys2eCfzk30ev24ceO+++47tVqNELp9+/aMGTPWrFnz5ptvkh10Ot2oUaMOHz7sySoBAAghhHgRXSq90+A7kYcltH0mH+nqhNVbahMIQjfRaDRFRUW//vrr2LFjEUKrV69GCDkvuo0xPnDgQKdOnTxWIgBezLFQNUk+cnIi0tU5LoZe1ZWuA12dtZcXBSG2crdnvYQ5s3ueLnDiF7KGLZy3jB8/fsGCBWPHjhVFcfXq1c6zCRFCixYtunnz5rZt29xTHgBejnR1kn5OslB1Qx+qie9dTssHaj0v+qempLK6X3gyZgYOHPjuu++ePn06JydHqVR27tzZcVN8fPySJUv+/PNPlUrlwQoBqK2KOHShBF8oH9hythCrJFRLP9TCnxoYSf27NV1fC6M6vZcXBaHHMQzzxhtvkEW3x48f71h0e+3atfPnzz906BCsMgpApcgy4osl6EKJfep6Wgm2CijGl2riS8X4UgPC6Zb+FExgBw4QhG41duzYuLg4QRDWrl1Ltvzyyy8TJ05ctWpVSUlJUlKSUqmMjY31aI0A1DC3TfYeTnJ4L6UIMzRq4kvF6ahoDTUgnIrTUVEaaPCBe4IgdKuQkJAuXbr4+/v7+vqSLWfPno2NjV20aBG52qhRo59//tlzBQJQrdlElGnEqcV3ku9CCfaR2sdztgmghkfRzfwoGNgCHgkEoVsZjcbExMTt27c7tnzxxRcerAeA6qzUiq6W4et6fCaXvWoUUovxdT0OVVJNdChOR3UJpsbF0K0CKBV8jYEnA58g91m3bt1///vfzp07t2vXztO1AFDtkLOup5Xg63p7m++2CUdrqDgd1UCFhkdT0zV0Ux0lgyU6QWWjn/whSkpKXnjhBa1WGxUVda9uvfnz59epU8fPz2/y5Mk8zzvfdOLEifr16ztOVFuLtW3bdsmSJRs2bPB0IQB43m0T3n8Lx18UpyQIff/HB6+31d9oe/eEcCwH66TUuBj6jwGM8V+S1GHsxt7MR3H88Ci6TQCkIKgSldAinDFjBkVRubm5Z8+e7d+/f+fOnevVq+e8w549e1asWPH3339rtdo+ffqsXLly4sSJ5CaO4yZOnKjRaIqLi5+8kmquSZMm97pJFEWaroQfJQBUQ84H9khrL7kIaySIDGZp4ksNjKDjdBScexZ4ypMGIcdx69evP3LkiEKh6NSpU//+/X/44YfZs2c770MWEiPpOHXq1K+//toRhHPmzBk+fPipU6eesIya648//hg/fvytW7coiurRo0d8fHxYWJiniwLg8eWZ0ZUyfKUUXy7FF0tRWjHOMNpXaWnsg7oGU+Ni6BhfOLAHqpEn/TDevn3baDQ2bdqUXG3WrNmlS5dc9rl8+fLLL79ccYezZ8/u27cvISHhpZdeesIyaq7GjRv//vvvDRo0MBqNr7/++pQpU7Zs2eLpogB4KDlmdKUUXy0j/6GrZfhqKcb1DO0AACAASURBVJYxqIGWauhDNfKhXqmPYnV0Qy0lgf4OUI09aRAWFRXJZDKJREKuarXawsLCivtoNBpyWaPRmEwmjuMYhhk3btyKFSsc972Pq1evOi/C2aJFi6NHj97/LhaL5RFeRtXT6/UvvvjiL7/84uPjgxBKT09/9913N2/eHBERQXZQqVRPP/30ihUrKt5XEAS9Xu981Wq1CoLgnsqBg8Fg8HQJHlNipdKN1A0DSjfQF0qpi2XUNT3F0KieCkepUT21+JQffqMejvPFWgl2ua/FiJ7wr9Gb33kPMpvNUqmUYWr2gVm5XP7AlHnSIAwICLBYLFarVSqVIoRKSkqCgoIq7lNWVkYul5aWarVamUy2aNGi0NBQiqKSkpKKi4tv37595cqVhg0b3vVZGjRosHPnzo4dOz58YXd95SabWcBuyg+1VEWhO1N4NRoNy7Lr16+fMGECQmjVqlWBgYFk0W2DwfC///0vKytrzZo1CxYsqPhQDMM4fkkghARB4DhOqYQjKh7g/A9RWxVz6Lr+ztDN63p8uRSzNIrWUNEaKlqLnqlHfaCjGmgpH6n7qvKGd766YVm2FgThw3jSIAwJCdFqtefOnSOnXz937lzr1q1d9omJiUlOTh42bBhCKDk5uXHjxgghmUxmNBpnzJiBEEpNTc3IyPD19b1rDFQWC8+9umsCx3NV9xQODMXM6Ta9ZVBT543jx4+fMWPGhAkTeJ7//vvvHbMJTSbT/v37s7KyEEKkvQiAezww856LoOJ0dEMfSvvgjhsAaiz8xCZNmvTss88WFRXt3btXo9FkZGRgjK9evTp48GCbzYYx3rdvX3BwcEpKSmZmZlxc3MqVK10eYejQofPmzbvPU3Ts2DEhIeGRqjKbzXK5/BFfStUSRbFhw4YJCQlbt25t0aJFxR1+/vnnsLAwl42HDh3q0aOH8xae541GYxUWCu6hrKzM0yU8viILTswXf7gsTD/JD9/Pt9lmU6+16tZZ22yzDd/Pz07iN14XEvPFMqunC72bGv3O11wmk4nneU9X4Q6VMHJrwYIFkydPjo2NDQwM/PHHH8PDwxFCPM/n5OSQHfr27TtjxozBgwdbrdZXX32VnJDPWXR0dEhIyJNXUs1RFDV27Nj4+Pjs7GzSQeoiLi4uPz+f53nn8xQC8KicZ6ZfL0PX9fhiCZYyiMxViNNRw6NRtIZu5ENpoJ0HAEIUxq5HtquhTp06LV68+JGOEVosFp1OZza76eyDD6mgoKBRo0aiKGZmZpJjHps2bQoODo6KisrOzv744481Gs3WrVud73L48OE5c+YcOnTIsQWOEXqKXq+vbkeqDDZEziVLxm1eKcU39DhIQTXQooZaqoEP1UCLGmipBlpKXpMP9FTDd94b1I7BMg8DWh5uFRAQ0K1bt+DgYMdfNcdxM2fOzMrK8vPzI01nz1YIqjmbiFKK8Ml8fCofn8zH6Qbc3I9q7kc10FLdglFDH7q+BpZfAeDRQBC6VWlp6fHjxw8cOODYMmrUqFGjRnmwJFD9kXOpH8/Fx3LwuSIcrqLaBFBtAqjXGtHtAiH2AHhSEITus3z58sWLFw8aNKh58+aergVUayVWlJiPj+WKSQX47zzM0ogk3/QWdJc6tE7m6foAqF0gCN1nwIABPXv2jImJ8XQhoNox2NDZQpxUYP/vlgk3LT/N0KquNJxdD4AqBUHoPpGRkZ4uAVQXvIguldpj73guTi3GcTrqqTpUn7rU9BZ0Ex2cTh0A94EgrC5GjRq1Z8+etLS04OBgT9cCqsRtk/04X1IBTi7CEeWH+oZHwaE+ADwJgrBa2LJlS2FhYXFxsSiKnq4FVJpsE0osEEmz70QelpQf6vu0NdOpDpx+AYDqAv4W3USv1z/99NM7duzw9/dHCF25cuXtt9/eu3cvy7KFhYWzZs3atm0bHD6s6fQ2lHyPQ32ru9JBcKgPgGrJu4KQK7FhwU0LCMh0Eor+x6LbdevW/eGHH6ZOnYoQio+Pj42NJSvITJkyZfr06XXq1HFPYaASOQ71kT7PTCNu7ke1CYBDfQDUJF4UhKJVTF2ZLvLuCEKKphq+GKqNVjlvHD9+/Pjx49977z2bzbZu3bq9e/cihHbv3n379u3Ro0eXlpa6oTDw5O51qO/VBnT7IEoKJ94DoKbxoiCkpXTr6Xc/zZN79OzZk2XZI0eOZGdn169fv2XLlgihSZMmvf3225s3bzYajQihXbt2DRgwAE5SXw1dKcUfJEj/zLMFyql2gVT7QGpENN3Kn1J60d8QALUT/BG71bhx48ii22+99RbZ0qVLl6SkpKSkJJvNhhDauXNn8+bNIQirFTOPPk8WVlwQp8aKa3rK/GE+OwC1CwShW40ePXr27Nk0TY8YMYJs+fHHH8mFkpKS3377beXKlaGhoZ4rELjalYHfSRDidChpCOsrWjSQggDUOhCEbuXr69ujR4+IiAiFwnUEoUQiGT58eMXtwFNuGfFHp8QT+Xj5U0z/MAohpNd7uiYAQBWAIHSrgoKCP//8MyEhoeJNKpVq48aN7i8JVGQT0bI0cUGyMD6W/q4rC1PdAajdYIib+yxatKhZs2b/+te/Gjdu7OlawD0dzsYtt/L7b4t/D2I/bc1ACgJQ60GL0H1Gjx49atQomC9YbWWb0PSTwl95+NtOzIBwmAEIgLeAIHSfwMBAT5cA7o4X0dI0cd4ZYWwMff4FtkafzB0A8Kiga9TDDhw4wLKsX7kdO3Z4uiKvczQHt/6N35EhHhvILmzHQAoC4G2gReh5zZs3P336tKer8EaFHPr4lLA7Ey9oS49uCD8KAfBS8MfvJnq9vmnTpjk5OeRqampq+/bteZ4nV8vKysiEeuAeIkbrrohNN9vkDLowjIUUBMCbeVeLsLTggii4KW+0/o0Z9s7sa41G06pVq7Vr186YMQMhtHLlyq5du5JFt8+fP9+wYcOysrIBAwbEx8eT01OAqnO6AI8/LihYtH8AG6eDQTEAeDsvCkKBN585MF0UODc8F0WxzXvM8Q9p67xx/Pjxo0aNmjZtGsdx69evP3r0KEKodevWOTk5fn5+RUVFI0aMmDp16g8//OCGCr1TMYc+PS1sSccL2tKvNqQhAwEAyKuCkGEVPV705FCUzp0763S6/fv3Z2dnN2/evEmTJgghnU5HbvXz85s2bdq//vUvD1ZYi2GEfrwizjglDI+i04axWomnCwIAVBteFITVgWPR7cmTJ1e8NTc318fHx/1V1XpnC/HEvwSrgLb3ZdsFQjsQAPAPEIRu9corr3z88ccMwwwZMoRsWbFihVwuj4qKunz58qxZs8gRRFBZSq3o30nCL9fEmS2ZyXE0dIYCACqCIHQrtVrdvXv3Bg0ayGT2cTRhYWG//PJLXl5eUFDQsmXLHAEJntzODHHSX2L3YCptmCRA7ulqAADVFQShW92+ffvgwYOLFi1ybHnuueeee+45D5ZUK10uxZP+EgosaEMvpmMQNAMBAPcD06fcZ+7cubGxsVOnTo2KivJ0LbWWiUefnha67eKfDadPPc9CCgIAHghahO7zwQcfTJ8+3dEpCirdzgzxnQSxax3q3FBJEJzYEQDwcCAI3UepVN7rJovFcu7cOYVCERsbS2bZg0dytQy/kyDcMqKfejBP1YFWIADgEcB3ruft3r379ddfDw0NRQg1bNgQTs/7SMw8+uKcsCRVnBRHf9yXkUJnPwDgEUEQelhmZubLL7+8bdu23r17I4TMZrOnK6pJdmaIUxLEOB06M5QNV0FDEADwOOD3s5vo9fqIiIjMzExy9fTp07GxsYIgrF+/vnv37j179szMzOR5XqGAQ1sPJcuIRx8W3v9bXNGF2dkPUhAA8Pi8qEWIETqaX2AVRfc8XQd/P43T0T6NRtOvX7/Vq1d/+umnCKGVK1cOGTKEYZjLly8LghAbG6tWqzMzM+Pj459//nn3VFhD2US0LE387KwwoQn9XVdWBqcPBAA8GS8KQrMg/OfSFYtbgpClqPlNm7TR+TpvnDRp0sCBA2fNmmU2mzds2JCYmIgQKikpSUhIOH/+fN26dXfs2DFmzJg+ffqo1Wo3FFkTHcrGk44L0Vp0cjBbTwOtQABAJfCiIFQyzM4unTxYQMuWLUNDQ/fs2ZOdnd2pU6cGDRoghEJCQp566qm6desihAYOHMhx3MWLF9u2bfugB/M6t014xknxrzz8bSdmQDhEIACg0lTOMcKffvqpc+fO7du3X7Zs2V13SElJee6551q0aDFhwgS9Xk82rlq1qm/fvi1atOjTp8+GDRsqpZJqbvz48fHx8fHx8W+99RbZ0rFjx9zcXHK5pKSE4zg4H6ELXkT/PS8238KHqtD5F1hIQQBA5aqEIDxy5Mi77747f/78JUuWLFy4cMuWLS47cBzXv3//Hj16bNy4MScnZ9KkSWS7j4/Pxx9//Ouvv7711lvjxo3bv3//kxdTzb344osnTpzIzs52LKs2YsSIwsLCmTNn7t27d8yYMU8//TSsO+PsaA5utY3fmSEeG8gubMfI4YggAKCyVUIQLl++/O233+7Vq1f79u0//PDDio3CrVu3+vv7f/DBB40bN168ePGGDRsKCwsRQsOHD+/Zs2dsbOzw4cO7d+9OjpnVbgqFokuXLm+88YZj1rxMJjt69KjFYvnhhx+6d+9e8WeE18oxo9GHhZGHhA+b0/sHsDG+0BAEAFSJSjhGmJKS8uKLL5LLbdu2nTdvnssO58+fdxz0ioyM9PHxuXz5cqdOnRBCJSUlOTk5ycnJZ8+e/fzzz5+8mGru2rVrBw8eXLJkifPGunXrOi/DDRBCy9LET08LYxvTl4azSi86kA0A8IBK+I7Jz893nE5Wp9MVFBSIokjTd9qaeXl5vr53xk/qdDrHUbFt27b997//vXHjxr/+9a9GjRrd6ylu3LgxdOhQxyqd0dHR27dvv39VFovl8V5O1ZkxY0Z8fPzcuXPJ0JiHJ4qiwWBwXBUEgeM40V3zQNxvzTVmxWXm9162RlosWpDhwfdwE6PRSFHQMPUAeOc9wmw2S6VShqnZByTkcvkD162shCDUarVGo5Fc1uv1Pj4+zimIEPLx8SkrK3Nc1ev1Op2OXH7ttddee+01vV7fv3//L7/8cubMmXd9ivDw8BkzZrRq1Ypc1el0D5xgUA1X7Fy4cOHChQsf4440TTu/XkEQJBLJfVYurdFO5uN5KfzR59gYX6mna3GFMYaZLR4B77xHMAxTC4LwYVRCWkRHR1++fJlcvnz5csWxHlFRUY71M0tKSvLy8lz20Wg0/fv3P3v27D2rZNm6detGR0c/ebXVTWFhYWlpqeMqRVHePFimiEMvHRRWdmHgiCAAwG0qYbDMqFGjVq1aVVZWxnHcsmXLRo0aRbbPmzcvLS0NITRixIikpKS///4bIfTtt98+9dRTERERCKFDhw5hjBFCGRkZGzduJEcNvc33338/olz37t27dOni6Yo8RsTolUP8y/WpofVg5T8AgPtUQotw5MiRx44di4iIoGm6X79+EydOJNt/+umntm3bNmnSJDAwMD4+/tlnn5VIJAEBAZs2bSI7fPzxx8nJyRqNxmKxjB079p133nnyYmqcDz744IMPPiCXhwwZ0rhxY8/W40GzkgSriOa2qf39MACAaqUSgpBhmJUrV37zzTcYY+cDV5cuXXJcHjly5IgRI4xGo2NYDUIoISHBZrOZTCbnjbWVwWAIDw8/deoUWVDmr7/+evHFF9PT0x3977m5uXv27PGGobN3tSsD/3QVJz7PMtAnCgBwr0obUfLA0yawLFsx8CQSiftSECPbmRuYF9zzbJK4MEold1xVq9UjR45cs2bNggULEELx8fFjx451Pgq9bt26Dh06xMTEuKe8aiVdj988ym/uwwbKH7wzAABUrmo3tLLqYBtvTbyKbG4JQppmQnSM6h/f6xMnTuzdu/ecOXNMJtPWrVvPnz/vfOvatWs//PBDd9RWzVgE9MIBYXZrOLM8AMAzvCgIKSmrGtfXgwU0adKkYcOG27dvz87O7tWrFxkxRPz1118ZGRnDhg3zYHmeMuG4EONDvR0LA2QAAJ7hRUFYHZBFt/Py8lyOBa5evfrll1/2wplSKy6IiQX4xCD4HAIAPAZ+hrvVCy+8cO7cuZKSkv79+zs2GgyGTZs2vfHGGx4szCNO5uNPTwtb+zCwiBoAwIPgG8itpFJpx44d27Zt67IE3bx58zp06ODBwtyPzJ1f9hTTQAuHBgEAngQtQrc6c+bMoUOHxo0b57wxOjp6ypQpnirJI0SMRsLceQBA9QAtQveZPHnyjh07/vvf/wYFBXm6Fg+blSTYYO48AKB6gCB0n//7v//7v//7P09X4Xkwdx4AUK1AEHqeyWQ6fPiwXq9v2bJlrV9iDebOAwAeGcai2YgtJpEzYYuR0dVhfAMq8eEhCD0sKyurc+fOzZs3r1ev3pQpUz788MP333/f00VVFZg7DwAQLSbMmUWLEVtMosWEzUbRYhQtJvtVi0m0GLHZKNqvGsn+tFJNyZW0TEnJlaqOT6s69KvEkiAI3UQQBL1e7zhBMca4pKTEx8fn559/btiw4a5duxBCAwYMeP3112txEMLceQBqGWyziia9aDaIJgM260WTAdus2GYVzYY7221WbLOSW0VjGaJpSiKlWCmtVFMKDa1U00o1rVBTEhktV9K6ILKFUmgoiZSSSGmFmtH4IroKhxRAELqJ1Wpt0KDBkSNHmjRpghA6fPjwG2+8cfXqVT8/P4vFQvYxm81+fn4eLbMKLYe58wBUV9hqES1mzJU30TgTtpg5Q5nVZkGcmbTVRIsJc0bRYsJmk2g2iBYTxTCUTEnLlbRCRSlUtFxJyVV0ebtNEuxHyZW0XEXJlbRcSS7TChWiql2HkBd9K2Es5Cf/hkXePU+na9RLqg50XFUoFGPGjFm9evWiRYsQQvHx8ePGjaNpesyYMWfOnOnYsWN4ePiVK1d++ukn95TnZifz8ZzTwrGBLMydB6AKiaJoMYlmA+ZIqpkxZxZNepEzY5JnnFk0G8gFXN5FKZqNlERKy5SUTEErVJRcRcsVlEwhMlJKpWFUGkoXVJ5kSlqupBQqEm8UU0v+nmvJy3gYWBRMuRdF3uKOJ6MYbUQ75BSECKEJEyZ07Njxs88+MxqNu3bt+vrrrxFCKSkpu3fvnjRpUmhoaHx8/A8//NC6dWt3VOhGMHcegEdl7060cY5eR2yzYRsnmgyk1xHz5VcdfZK8VTQbKVZCK9S0Uk1JZIiVOnc8UnKlxNHx6LhVoaZVWoqVVKzBbDZLpVLnk+TUVl4UhDQjrdd/pgcLqF+/fqtWrbZs2ZKdnT1gwICQkBCE0JdffvnSSy+Rc/P27du3Tp0606ZNq1u3rgfrrFxk7vxImDsPvB4WeNFYJhrLRKNeNJaKhlKBXDWViYYy0owTORO2mESzkZLIaLmCdDxSChUtU1JyBS1TUDIlrVSzQeG0TEHJFbRMae+TlClpuYKSwmjsx+FFQVgdvP322998801ubu6yZcvIFoZhrFYruWy1WjHGtez3F5k7PwfmzoPaCwu8aNTb88xUJhpKBWOpPe2cYg/brLRKW/6fD6P2oVUa1i+IDm9IqzT2zkaZotoeSKvFIAjdatCgQVOmTJHJZL169SJbxo4dO2jQIIVCERoa+v333w8ZMiQ4ONizRVYimDsPajpsswplhUJpkX3Qo8kgmg1CWZFQWujokxT0xbRCxWj9nMdA0kqNJCiM8SnfqFAzWj+It+oJgtCtWJZt2bJl9+7dqfK/h549e546dWrnzp25ubnTp08fOnSoZyusRDB3HlRnD5VwhhKKYRmtP631c044aVgDJq49JFytAUHoPiaTKSEh4dixY+vWrXPeHhMTExMT46mqqgjMnQeehLGgLxH1xUJpoWAoEUsLBX2xUFYklhUJ+hLRWCpazMydXkoNrfKh1T6MSsv6B9MqLa3U0motrfah5SpPvxLgDhCE7jN37ty9e/euXr1ap9N5upYqNx7mzoOqhG1WR7AJpQWioUQoKU87fbFoKKVVGlrty/gEMBpf2sefDQiVRTeltX6MxpdW+9AKrzsJNrgPCEL3Wbhw4cKFCz1dhTssvyCeLsAJMHcePAFH16VYVmTNu13KGYWyQtFkEEnvpdlAK9SMj73TktH6S0Kj5D5taa0f4+PH+AbWmiluwA3gswIqGZk7fxzmzoMHEU0GoayQBJtw5/9F2Kzni/ORKJCDc4yPnyBXSwNDJMERtFJNa/0ZrR8clgOVCL6rQGUic+eXP8XUh7nzXs95NIpQWiSUFdmHopDMM5TQciWj9aO1/oyPH6P1Z/1DSO8lrVSzuiBKpnA8lF6v12g0HnwtoHaDIASVxjF3fgjMnfcSGAtkQEppoVCSL5QWCiUFQkmBUFYoFOchimZ8Axi1L+PjT2t0jNZPFtWE1ugYH39Go6PVPp6uHgA7CEJQaWDufK2EBV40lAplRWJZoVBaxBfmlB+6K+SL8iiWJa06NiCERB3jY78KA1JATVGbg5Dn+S+++MLTVTyp69eve7qEhwJz52s055Ep5FgdX5hNrvLFubRcyfqH0Fo/xsef9Q+WBEeQUSqsXx1Y0wvUArU2COVy+dy5c4uLiz1dyJPS6XRTpkzxdBUPAHPnawR72hXmlEddjuOInWg2OEamMFo/xsdfGd6NXGV1dRANfd2gNqu1QYgQ+uijjzxdgleAufPVimgy8IXZpBuTjE/hC7LFskLHOEzGP5gMTpHUiZA3asX4B8MgTODlanMQAvcYf1yI9YW58+4jGkrJEBV7N6a+WCjJF8qKhOJ80VBKq30Y30Ayl47xCZBFN1W26clo/RjfQEoi9XTtAFRHEITgicDc+cqHsWAoEfUlQkmBYCgRivNFQ4lQUuAIP0quJL2XtNaP0fqxAaGy+s0YH39GF8RofBENg5UAeDTw/QUeH8ydf2z3mksulhUJJfmUTE5OZWA/YucbII1oZB+f8s8JdgCAJwdfYOAxwdz5+3uYqKs4lxyWBwPA/eDvDTwOmDuP7ht1fHEeLVdA1AFQI8BfI3gcnyR619x5oTiPu3befPksZygWSgpEfYlo0tNqH8YngNHqaB9/RusnCY2SN25tP3Sn0cEgTABqCghC8Mh2ZeD112r73HmMbXmZ1mvnuevnuevnEW+TRjdl6jZUtexqXzBM4wtRB0DtAEEIHk1tnjsvira8TOv1VMvlM9zVcxTNSKPj5I1aafq8KKkTgShKr9fLYelnAGqdygnCmzdvfvXVV3l5eb17937zzTepCr+UOY775ptvkpKSoqKipk+f7ufnhxBKTU3duHHjlStXfHx8Xn311c6dO1dKMaDqkLnzn7apRXPnRcF66zp3+Qx3PdV6I5VWaGSNWyniOvgMGsv61fF0cQAAd6iEIDSbzd26dRs2bNjIkSNnzpxZWFhYcUmXSZMmXbly5f3339+4ceOzzz6bkJCAEPryyy/r1Knz/PPPp6en9+3bd/v27X369HnyekDVIXPn34qp2QNkMGe23rzIXU/lrqdab16UBIVJo5qo2vXxG/k+rdJ6ujoAgLtVQhBu2rTJ399/0aJFCCE/P78RI0Z88MEHEonEsUN+fv6PP/54+fLliIiIZ555JiQk5NixY126dFm7dq2j7Zienr5hwwYIwuqsRs+dFw0l1psXuetp3PVUPjudDakni47TdB8srf8JLVd5ujoAgCdVwpfaqVOnunbtSi537tw5Pz//5s2bDRo0cOxw9uzZkJCQiIgIhBDLsp07dz558mSXLl2ce1CzsrJatGjx5MWAKlIT584LpYXWG2nc9fPWG2l8/i1JRGNZdJzPM6Ok0U0pVvLg+wMAvEMlfKvl5OQ0a9aMXGYYRqfT5eTkOAdhbm6uv7+/46q/v392drbzI2zZsuXEiRNr1qy511NkZWVNmjRJq7V3W4WFhS1btuzJK6+hBEHgOE4URbc9Y7GVGrFfsrgNX4fmDAa3Pe3jEItz+avn+IyL/I00JNjo0PqSerHS58YqQ6PJIE8bQjYLhxD3GA9uNBorHv8GbgDvvEeYzWapVMowNXuWlFwuZ9kHJF0lBKFSqeS4O18rZrNZpVLdZweLxeK8w4EDB8aPH79jx46AgIB7PYW/v/+YMWNiY2PJVT8/P7Xae8/5KQiCRCJRKpXueToRo2HH+FENqZdjquWSzaJovXXNej2Vu5HKXUmm5SppdBNVwxbS/iMlwZGV+1QYY2/+4HkQvPMewTBMLQjCh1EJQRgWFpaenk4u5+fnm0ymsLAwlx2ysrJ4niexnJ6e3rt3b3LTkSNHRo4cuXnz5o4dO97nKRQKRbt27e6/D6ginyQKfDWbO4+tnDXrqvVGKnc91XojjdHqpNFNFXEdfAe/yeiCPF0dAKCGqYQgHD58eK9evXJycoKDg9euXdujR4/AwECE0MGDBzUaTbt27dq2bavT6bZt2zZ8+PC0tLTk5ORBgwYhhBISEoYPH/7LL79069btycsAVWFXBv75Gj5VDebO/2OoZ/oFSZ1w+1DPVz6glTC3DwDw+CohCFu2bPnaa6+1atWqUaNGV69e3blzJ9m+dOnS6Ojodu3a0TS9ZMmSMWPGLFu27Pz5859//jlJymnTphUXFw8bNozs37t3702bNj15PaCyXC3Drx/hf+vrsbnzQlmxLfMSdz2Nu3zGlpclCY2WRcdp+4yQRsXBqfUAAJWFwhhXygPdvHkzNze3efPmcrn9W1Ov1zMM4ziUVVpaevHixcjIyODgYMcOPM87HkEikdzrMECnTp0WL14MXaMEGSxT1ccILQJ6aic/LoZ2/6xBvjBb/8cG7mqyaDHJouNk9ZtJo+Okdet7/Ex7er1eAyvLeAK88x5ROwbLPIxKGwsfGRkZGfmPsQkuH1wfH58OHTrcZwdQrXhk7jy2QUhuKgAAIABJREFUWfUHNhqO7lD3GOLfc44kKBzW8wQAVLWaMykMuNGyNA/MnbeknijZupINjqjz/hLGD8a8AADcBIIQuDqZj+eecevceb4wu2TrCj4vy3fYRHlsWzc9KwAAIIQgCIELN5933t4XemS7uttg/9c+gQVfAADuB0EI7iDnnX+lgZvOO29JPVGydQUbHFnng6XQFwoA8BQIQnAHmTv/aesqHyTGF2SXbF3O59/yHT5ZHtOmqp8OAADuA4IQ2Lln7vw/+kJfnwV9oQAAj4MgBAi5a+78nb7QD5fCWmgAgGoCghAgM49ePCjMa8t0rrLzzkNfKADeSeSxaBMRQpjHglVECGERC5yIEEIi4i0C2Y03Oy6ICGOEkGARyXovAidiASGERJso8hghpItR+zerzHNoQxACNOGvKpw7D32hAHiQI4cEi4hFjEUkcAJCSLTZt/MWEWGMBaftvIgQ4k0iz/M0TYscRgghjHmz/dRvd0LLIiARIZJVIkYICVYR8xg5hRbNUrSERghRLMVIaYQQRVOMjEYIIRqxcvuIBFbhuECTZTQYOU3OvcXIaEqGEEK0hKVZCiGkCKjkFRYhCL1dlc6dh75QAFwIVhELWOBELGDny66JhZFgcSQWRiR1MCL7Ox4HIcSbBIQQeTRkbz9hVB5Xjhxi5DRFU4iyZw8tsW9n5TSiKIpBjKx8O0sjhFgFjXmKZRlJIIsQQhTFKuy/le+ElpxBNEIkq2gKIcRIaYqlEEK0hCahVSNAEHq1E3lVNXeeL8gu2bqMz78NfaGgJiKxRJpNpJF038uYNIBEq2i/4z8vk3YSiS5aStMMxchoiqEYKU2VX3ZJLIpCTHlisUoaIST3kyAKUTTFyGnklDSskkEIkUdDJJMYCjnF1WODtUZB7XfbhIcfENZ0Yyt37vw/+kLfmE0x8BkDVYt00PFmgRx8IrFkb2xZRCxi3ixgEQkWEQuiYMUiL4rW8gyzJ5ZztmHRJpJYIs2me16WUDRLk6AivXb/2E1avvM/L3v63aoubCJv4S1Gm4njOYvAGaxGTrBaBaveauB4KydwBqvRInBWwWqwGi08xwmcyWY282ZOsA5t/NxLsUMqsRj4kvJSFgEN3S9MakIPCK/Mv0xL6omSLcvZkHrQFwruyp5VJH6songnq0SEMW8WsIAEq0CaWeVhJtisPLIVYhELFnuPIgktsgPpoGMVDDn4ZA8eCUWzNGldkcNOrJKmGIaR0jRL01J7htl3ZsqzqjyxPP0+VXdWwcqR0BKsVt5qsNmzymgzmXmLlbcabSaLYOF4K9nCCVazzWzizRzPmXmL0WaiKUrBKpQSpYyRKli5SqqUMVIZI1NLVTJGJmOkGpk6gPGTMVK1VE22qKRKBSuXMVJ/hX/lvhwIQi816S8hTEVNa1Fpf/B8we2Srcv5/Nu+I96BvtDahDcL5LgUaSqRQ1BOuYUFK2lgYcEikEgrb5mJzmGGRcSbBZJVZNyEPatkNMVQrJxGNMUqGIpGjIyhWYpV0lIflmYpRsZYrGaNn5ocpiJ9gKTtRboWPf0O1TwCFkw2s8lm4gSrmbeQ1hjHc/Y84+15ZrSaOIEz8xZO4Cw8Z7CV72Y1sjQrZ0loOaUXK1VLVHLWHmOBTICMlaok9vRSSBRKViFjZQpWrpQoGKoa9bhCEHqjb86LSQX42MDK6aaBvtBqhQQPbxbI2HR7htmwyIsu/YSkP5AcvuItAiIZZg82UeQxGQpoDyc5Q5pKrrlljyWKVdIynYSiEassb5k5ZRUjZyj68Y9a6fW0RnP3k5V6J6PNZL2TYRzHWw02o5m3WAWr0WYy2cykBWa0mewZZjWSC0abycybRYxVEqVSopAxUgWrUEmVckYmZaQaqdoRY0GqAEqgtAqNWqqSslIFK1dJlDJGKmflKomKrl3nR4PvLK9z4DZemCwkDGJVlfGPf6cvdNoyxjewEh4RlPcf8mZBsIgCJwoWgVzlzaLA3dnIW0TeIggWUbSJgkUUBSxaRftYDHuXIEPGYvyj7aWgKZpilTTNsvagYilGxlCMU+8iybDyoYCgcll4zipYDTYjx3OccKfz0GQzmWxmC8+Rg2cWgeN4Tm81kKgjGcYJVpPNrJQoZIyMhJOcdc4wmUqiVEjkGqlaqVUoJUo5I5OzMrVEJWNlMkaqlqrkrFxCP9QfPwyWAbVTuh6/epj/tRcbpXnSLzi+4HbJluV8YTb0hd4HaZDxZoG32PNMsIgCJ5CQ4y13go03CeQyz4miTWTlDKtgGDnNyGhGzjAymlUwrJxmZLTEX0o2sgqakTOsnKYlNCOnHeMGQZVytMaMNpNjWAcnOC7Y22ekF1FvNZDDafabBKvBarzTncjKpIyEdCfKGZlSolRIFHJGppGpg9VBsgoZpmDlMkamlCg8/R7UNhCEXsRgQ4P+EGa1YnqEPFEKYiunP7jJ3hc61ov6QgWLaLFanRtqPPm/2SXkRN4sCJz9cBojo1kliTR7kjHlIafQsIycZmUMI6fLY49h5DTkWRURsUiGKXLCnfEdJpvZ0alotlkcW0gTzcybOd4eeyT/HD2KSntXoUwlUdpDS6qSMlKNTB2iriNlpOUxJiUtNhkjkzISjRT6eKsdb/kKAxihN44KbQKo8bFP9CXrFX2hGHElNkuh1VxgtRSW/1dgFUVRqpawCoaR0Y7QYuUMq6BlPpLy1htputn3gfGHT0JvNZDoKjWU0bZcq2CzClYLz9lEm8lmFrBgsBpFLJL/G20mXuTNvMUq2DiB4wTOJvAm3iyIgsFm342maHKgy55MrEzGSEnvIulUlLNylVQZrq2rYOVSxj7QQ8pIlRKFozfS0+8KqHwQhN5i3hkxy4gPDnj8f/E7faEvTpE3bl2JtXkQFjDJPEuh1VJov2DK42iGkvtLyX++DVXyjjq5v9Qm4TRajadLrhbIlC9y2WA1YoQRQiSHyEYzbxFEgexGostsM/NYMFqNIsaOhCPDF+0JJ3A2wZ5wRptRxFgjVVMUpZaoaESppCoJw5KGl4SWKCQKlmLUUhWFKI1MTSEqTBvKUoxCopDQEhJsEoZVsAqWZtQSFUVR0BQD9wJB6BV23BTjL4onB5NFlB5Z7egL5c3CneZdeeZZS21SH0l55kkCwrVyf6kiUGZfC/GfbHrObdWaeQsv8gghXhTMvJlsNNpMIhYRQiQ5UHlfH7mV4zmraCOXHeFkE3hLeTiR5hFCCCNssBrJRnIEy34vmxFhhBCyiTYLz5VXYuZFweW5SKOKXFZJlDRFI4RYmlGw9sNXClbO0IyUkcgYmYyVScujSyVR0jQdqgmmEKWWqhiKUUoUEpqVsTLSc0iiy/GYhF6v12jgJwioKjXyGw08kosleNwxYWc/NlT5OIcGLaknircsk4RE1ZS+UCxirti1kWcpsCIKORp56jB5QAut3F8q00keZmBkKVeWZyq4WZDJFrOkZYNIxgj2jDFYjWSlfEdTyTlsHKkmiIKjzWSwOe7CWQUb+mfSKFg5S7Pon+niiAcpIyE5RBpM5FaSN+QyaSohhCQMq5HZW0JBqgDymCSEyEYpI5WVR5paokIUQgixNOvoAyTJhBAi/YoP/e8AQI0BQVjL/X979x3fVnnvD/x7pvayZMsrdpzEsbOnybjZITeDMBIopUDhhkILBQp0hFmgLaWUMloo65YfZRXKCpdASEIWSZw0y3Z24gzHTuIhydpbZ/3+OLIsj0wPeXzfL168jo7OOXqsyPr4ec4z3FG4Zp3wpxKqJP2SU5B31HpWvME7G9JuekgxdFxXFK+DRF6Keds0bDZESIaUA0+dqbCM1cvbFxzExotCY9hpCzpsQbst6LAFHbaQwx50NATtLMlmaNLTWKNaoZJrNtAyLTRsfGSVmWQVNAstwyaRahRJJU6R2+sgKYqST0EIdRsMwr5MlODW7/lr84hlQy+ty0bLttCne0Jb6MU3bKozFOR5e13GhFhj2FUfsNUFGhpDLmfYVRdoqA/YbEGHmlFlazOztNZsbeZAQ97ErLFZWmuuLluuCWEDHUJ9Uuq/4FDXeXiXwInwp5JLuzGY1Bb6BmW0dFHZzkMSJO/JYNgeTeq0yVFKUiUHnoXVF6gzJhqVFpbVne8D7I8F5IRLTrvGsNMfC1pUaVlaq1mVZlGlFZmHzM6flqW1WjXpPWraJ4RQ98Ag7LP+dUL8olradS1NX3RtMN4W6rKlqi3UXxO2l7kbK7yqDIUmS6m0sIbBGjn8zjW0jhM4R9hZH7A1hpzOiLvO31AXaHCGXQ1BO0My2dpMs8pkVqVl6zJnm6fJ4WdWmeT7ZwghBBiEfVWFU/rVTmH9ItpycaOe4m2hpd/o592knX41kN1aMYp5ucZ9Ptsut8hL6eMMYx4crDS3XoHaHws0hl2usLsu0JBIO2fY3Rh2WVRp8bTTZsrVuzSVKUuTocQhXwihi4BB2AfZwnDdOuHvU6mRpouq98hj5NnBI60Pv0HpTF1dvASRE12H/fbdHl9NKG2YruCaTGOhFgjwxwJbq7fX+uvlHiuOUGND0KGmVVZNeoYm3aqxZGqswyxDM9QWqybdpDR2W4ERQn0SBmFfw4nww438nUXkDQUXbhLlG+vcn74iBn2mW5crBo3ohuIBAEjgqw7Z93gaK7zaPFXGRGPx7QNIhhQlcVd9+ZqqDTvry8dZRw8y5o2wFM/Jn27VpFs1GYku/ggh1LkwCPua+7cLJpZ4YtyFUlCSAltX+r77SD/vR93WFhqyRRv3eu27PSRDZJQYJzxWyGhpADjjq11/ZMuaqg1KWjF/0JwHSn5mUOi7oTwIIQQYhH3MG0fELQ3SjmsvsNAg77K5P35J4mIZD7xEp+d0dan4sNC412vf44m4OMto/bBleZocJQAEudDGqi1rT22q9p6elTftjzMfG2Ia1NWFQQihVjAI+45tNun35ULp1bSeOfdBkhT8z2rvt+/pZl+vm30DkF04JbTIS57KgH2Px13pNxXpcuekm4ZpCZIQJemA4/Daqk2bT28fmV58TeGCabmT6e7tnoMQQgkYhH3E6YD0gw38P2fSg/XnrA0KLrvr3y9JsUjG/S/Q1gFdV5jA2bB9t8dR4VWlsxkTjYU35chTd9pDjeurN399fC1LMfMHzfngmteNCkPXFQMhhC4GBmFfEObh+vXCw6OpBbnnTMHQ7vWer/7RpRXBqIdzlHtsOz1AQPo4w5gHBynTWACICrGtp3evPL7muKtqZt7U301/eGja4K4oAEIIXQYMwl5PXmhwmJF4YGT78Sb4XO5PXhH97vT7/8JY8zq9AEJEdB702fd4AmfDljGGwpty9AXxqZkrXSe+Pr52Y83WYnMhNoEihHomDMJe7/l9YqVXKl3c/j9leO9Wzxevq6+Yp7/jiU6eMlQCz/GAfY/HddivH6jOnGIyj8wnKAIAHKHGdUlNoP+65k0c7YcQ6rE655vR6XS+//77brf76quvLikpaXuAKIqffPLJgQMHRo8efeONN5IkKe+srKysqKhQKpVLly7tlJL0N9/VSq8eFndcQ6na/EsKPrfns1d4Z4PlZ88wuZ3ZFCmPgrDtcjMaOmOiseCaTHkUREyIbU9qAn1s6oOj0od34usihFBX6IQgDAaDkyZNmjJlyvDhwxcsWPDBBx8sWrSo1TH33HNPWVnZzTff/MILL2zevPmNN94AgNdee+1Pf/qT2WxmGAaD8DIc80o//p7/fC6dq2l9azBREUz7n8c7qyLIh4TGfV77Hk/UzaWPN468e6AqXSE/JTeBbjpdWpQ2ZH7B7D/N+i2Of0cI9RaEvDRoR7z11lvvvffe9u3bAeAf//jHu+++u23btuQDzp49O2TIkFOnTmVlZdXX1w8aNOjkyZPZ2dkcxzEM8/HHH//lL38pLy8/z0tMmTLl5Zdfnjx5cgeL2jcIghCNRgVGPWUl/+BI8s6iFrcGBb/b89mrvKMu7ZZfM7lDOv5yrUZBZEw0yqMgAKAx5Pyu+vtVJ9aRBLlw8NwFBXPSVN03Q1v3w2WYUgXf+ZQIh8Msy1JU37+v3wl1hU2bNi1YsEDeXrBgwU9/+tNIJKJUNs93vHXr1hEjRmRlZQFAVlbWsGHDtm7d+sMf/pBhzjPeDZ2PKMEtm4S52USrFAzv3epZ8bq6ZF7a7Y91vCJ4rlEQchPo2qqNBxuPzhww9ZEpv8AmUIRQ79UJQdjQ0DBr1ix522q1AkBdXd2gQc1ThNTX12dkZCQeWq3W+vr6S3oJm8327LPPJi6Sl5e3fPnyDha79xIE4clyyRURPpwmRiKcvFMKev1fvik6anW3P07nDIlyPHD85V0/5uVd+/yO3T6SIkyjtMX35ChMDABwUuxwQ9X6mi2bTpcOMuZfmTfj4Ym/UNIKAIhEIp310/VkkUgE/3pLCXznUyISiYii2NtrhAzDXPBH6IQgpChKEAR5W95o9ZGlaVoUxcRDQRBo+tJeV6VSFRcXFxQUyA8tFktv/7fpiK/OEF+cJf6zmFAx8Tchun+b/6u3lBPmam75DUFf5vcFHxE8h4ON5b5gXSRtpG7wjVm6gSr5KWfEvfn09jWnNnAiNztv+uv//bxVnd45P0yvQlFUf/7gpRC+8ylBNUl1QTqEIC68CE8nBGF2dnZdXZ28XVtbS5KkXC9MPqC2tjbxsLa2Njs7+5JeQq/XL126FO8RAsA+l3TfTv7/ZvHZOhUAiKGA95t3YlUHzXf+js0behkXlETJeyKYGAWRNTXNPFIvj4LgBG53Q8Xaqk3ltv2TsyfeN/HO8Zmj+/OStgzDYL0kJfCdTwme5y+mOtUHdEIQLl68+A9/+MPTTz/NMMwXX3wxf/58lmUBYP/+/WazOScnZ+7cubfffvuRI0eGDRt25MiRmpqaOXPmdPx1+yFXFK5fL7wymRiXJgFA5PAu96evKIdfkfHLVwlWcalXE3mpbouzbnOj0sJmTDQOWpJFq+Kf+FPe099VbVpdtT5Xlz1/0JxHpzyAi9wihC6bIEm+NjdrIqIQbmpNTPBzPN+yC6coSV6OS95TpNMNUKs6sXidEIRLly79+9//PmvWrKFDh3799dfffvutvP+ee+5ZsmTJr3/9a5PJ9Pjjj8+fP3/RokXffvvtE088YTQaAWDv3r2/+c1vGhoaampq5s2bV1JS8uyzz3a8PH0VL8IN6/kfDSZuLICIN+j+5u3osYq0Hz+sGDzqkq8lgX2Pp2aNTTdANereAlVGPERdYffGmtLVVRv8Mf/c/BlvzH8hS2s9/5UQQpctLAgRIX7bKCjwsaRbSD6OF5LywB2LJbYFSfLxzaESE8Ug3xwnYUGIJKVLgOe5pOt4OU5seihKkBwwnCQGkq4TEYQgxxEEkWha5EQpwLcOs5AgRMXWYebleLFlmFEEoWdax42SpFRtqps6hqZbNmYSQBjZFu0BP84fcFt+Z06S1QnDJwCA47gNGzY4nc45c+bIvUMB4MCBA2azOdEKWl5efujQoZEjR44bN07e4/f7jx07lriIXq8vLCxs9/o4fAIA7t8unPBJ38yno4d3ej59VTXiCuN1PyXYS66oeY4Fqr9uIGiy4GqrfpAGADiR311fvrZq0+76ipKscdcULujnTaDngp34U+Uy3nk/z/OiBABRUQgJAgBIEniavvcjQnNdJMALnBRPIE+Mk78QeUn0N9VgIqKYdDDPNcWVOxa/Gi9J/qaESM6hRBkAwM3Fkyw5TlQUpaTiHb/VFK1ImgS4VR6Y2OaBuRRB6JO6WbAkqaGb40RFUcqkdNHSNJN0HQPDkE0PCYDkgKEJQpd0WSVFkTzPMAzZVCqGJLRtuneoKUrRZt5EA0OTF3FnrufonCDsahiE7x8Xn90r7pgfldb8v2hlhXrpz/Uj2pnB5/xC9ZHqVbZwYyx/YYZltAEIqPae+er46o3VWwuMeQsHzZ2ZNxWbQM8Dg7AtTmyuRng5TgQJACJCPDmSG7X8fLzJK7kGk6joBAVBrg9xopjICQ8Xr8z4olGBJAGAF8VE5CTqTIkWtuQqjo6maZIAOSeahhKZmr73FSSpbvpO19IUQ8S/640sI39/0wSpa6rBKEkyUXHR0jTTFAyJqyVHiJJqruUkygAAJiaeZDTZIm96MhxHiHqQ/9ilX+8USov3hV9+RTlsouVXr8bg0paPiHq4M+sczoO+AXPTh92RJhGwvW73F5VfV3tOLx4y/62FL2ZqMi58FZRSiVQI8Lxcg0k0rCWHh5fjm9JICAsiAEggeZrqLom84UXJz8d3JhIlKoohoXVEhQQhKogQb5GLnxLgBblixJCktqk6YmAYEggAUFLx5Ehu1ErkTXINJlHR0VCUhmGaLkg3XbCpYhGLmTQaiLewxS+oZ2iKICCphY0kwIB9atClwyDs6epDsOw7/7rYu/rV5aabf6koHCsIAkSjF3k6HxLObmxs2OnKnJQ28bGhUTL6TdV3nx1dqaQUVxfOnz/zCRbnQrsQOVG8HOcMBoEXIH7nPx4J7kSNp+kmf0jg5eRI5EoidSSARCZ5OU5uYgvy8ZtDiYoUJ8WDLbmKY2AYuXahpWk5URINa63DI55GlIoioWUaJfKGJomhtFbemUgUBUmqm/78N7KM3DyeeBUSiETMaGmK6cpVnVvBujjqUhiEPVpEgKc+3f3ViVczRk80Ln+DUFxCRylJkOq3uc6sd5hH6ccvL2wkGt8+/ME3J78baSl+YOJPJ2SO6bpi9xwejosKYlDgfRwfFUU/xwUFISIIXo6X7+W4OS4qiCGB93F8RBQCvBDg+Ygg+Dhe7gXgjnEKklTTlIFhGAA9ywIASTRHgpGJN6YlbuqoKVpBkQCgIElTm9RJtKfp6Xj8aGiaJUlIqkgxRDzYCAKMWMVBqIthEPZcUiy66n//3/11O/P/5yHl0HGXciY07vdWf21TZypG3z/oBJz4494Pym375w2c9fbCv1o1vWMsfFgQ3DFOvv0jV5USD90xTu7s4ObkDbHlw/iRXo5nSVJFkXLTmZIiTSyroiglKW+QSooyMYyJYZSU2sQyKopSklR8f9NDs4JNdGHAeglCfRIGYQ8VrTp06t0XQ6riwY+/qdRcQkXQcyxw6usGkiYH/ShzN7HnL7te4EX+2sJFj055UJ4OrQcKCcIRn/+wz3fI5z/k9R32+esiERVF6mhGSZE6mtbQtJIkDQwj55mRYZQUpaYoq1KppEgtTWtpWkGSBoZRU5SCIk0Mm6hdIYTQ+WEQ9jgSF/Ot+dC9c8OTWT//821T1W2WWDqXwNlw9dcNUS+fNlf9vWLLkxXfDrcMvW9Cj5sOJiaKxwOBw02Zd8jnqwqGBmnUI/T64XrdbQPzhut1xTod1au6XyOEei8Mwp4lduqw66MXY9ZBC4a++tY8U4HuosIg6ubOrHe4j/jZqdJa1Zr/VO+eVzCrh/QF5UTxWJvYy1Yqh+t1E0zGHwzIeUpfjLGHEEohDMKeQq4IhvZsUC6577+rSh4aTs7MunA2JDqFxoaFPp/4hTPkWpJ71S8n3Z2q4YDnib0RBv3i7MyHi4eONOgV3djhECGEzg+DsEeIVR9xffQik12QsfyNm3dqJlrgZ8UXiAq5U+jpdXbPANfHIz82W0w3FV83JaekO1tBOVE8Ew4f8vrL3G6MPYRQL4VBmGKJiqDx+ntVo6f+rlw8GxQ3XXXefxcJ3AcD+9acalQ3fl70+ajC4t8VPzzQMKCri5qIPblXy2Gf76jfb1W0iL0Rep0Su6gghHoVDMJUitUcdX30IpM10Lr8DVKj/6pG/EeluOtaij13Dcpd6T+04pSTc20u+H7ihFGvDnlWz3ZJh/62sVfpD2QoFHLsXWlNf6BwMMYeQhdDFDk+Fkrew8d8kpS0SisfFfjWq1vzXFASuVY7uag/+UQAAJC4qK+9Vwy23im0+yohUYi12snF/CCJgiCQJHme9fxEkW/7KheDi3ou4yw+FhRFHgAKx981eOxPLuMK54JBmBqSwPvX/Tu4/Vvj9T9XjZkGAEc90s9Kha//m85Wt/+xqz/pOPzlqaA7cqTowNgpw14Y8iRJdFqTIy9Jp0Oh88fecL0OBySgbsPHAmLTsgZhfy0pagBAEgUuFpB3ikJU4MPxg7mQKMQzg4t64weIPN/i4EjT/tYhIX/vJx62jaVWX9w8F04OD0kS+Ji/5QWDkti8UANJMjSrTj6AZvVE0u8vRSuoNvf1KVpNtpn4iVHoiNa/+ASj0Lc6jCBphtW22klSCkZhaLVTpc1q+yo0qyUIKhaL0TRNnvvWBknSNKs517PnwSiMl3EWzWpIkgYAhdpyGaef78qdezl0MbjaKtdHL9Lp2dblb5BaAwC4o3D1d8JzJVRJejspWFlddWjVSdUZTf3Is7PuuOIqw39Fo9EOpqAgSRUezxaHc4fTdcjnOxUM5avVIwy64Xr94qzM5UWFQ3VaFu/t9UKiwPFcc+VD5MOC0DwhX3JgAAAf80tS8xo6sYi3+UKSyCV9ubcKj1Z1C54Li8mvEgtKEp/0VKzVy0lScyVGFDmBi19Z4CMCH78OzWrJpmUNKEYvfx0TJJX4ficpBUXHh9jSjJqk4lPwMAo9ANF0sK7pYDaRASTJ0KYWX98Mq4PkWKJYimkxeLfVFzfNqJLDgyAoumXDDM2qSbLXTwmEk26jLiEJfOD7Ff5NKwyLbtNMXSTvFCS45Xt+6UDif4a2CB5RknZU7Tm6pmbA6XymmJ782AiTfioACG2WsrxInCjucXs2Oxq3OBq3O115atXMdMuSnOzfDi8uwti7dIl6gyQ11zwSWcJzAblOkPhyl8R4REmSlKi18LGAnA2JLBFFTogfJnLReBRxUR+ACElJlqgPSWKLxCIphmaaKx8kraKo5okUkgMDAGhWRxDNX3OsMqm6QJBM0pc7STLJf/u3qluotJlk8quwGoKIf7dQtJJqmskh8XIE0VygH7Q3AAAgAElEQVSJIUmGYjRtD06Gc/qgLoVB2H24+mrXv/5C6c3W37xGGcyJ/ct3CbwIz5Y0fx8FudCayo0nN50dXztxUOHASY+NUhkuc1IYXpL2ebzrbfbSRud2pytTqZhmMS8ryH//iokWRX+cbltOF4EPCXyYiwZ4LijyES4W4LmAwEf4WJCPBQQ+zHMhLuYX+IjAhbmoR+AjAh+JRTwEQfCxkChyAEBSLM2ooGWFgFHo5XsqNKMhSAaSWr0IorkdiVXGaxg0oyVICgCUGmv8MJKmGQ20TItEhCTqIiTFUrS61WEIocuAQdgdBK/Tv+mLcNkmw3V3qSfMSX7qwxPi/1VLu66jKQIA4Ky/7suj39aU1c0/u3BW3qDRvx6itFxyXIUEodzt2dboXG937HC6inTa/7KYbxuY994VE9LY3h5+UiziE/iwwIe5qP/iYswrx1hTnsVYpZ6iVRStYhQ6ilZTtJJR6GhGQ9JKhtXSjJZVpdGMmmF1FK2kGBWjMFC0kqKVUY7WarWtKlUIod4Og7Br8c56/4bPwvtKNVfMy1j+OqUzJT9b4ZR+vVPYsIg2sVJZw/7Pj64MV3GLaxdP1c0o/EmuvkB9rsu2FeSF/zidpY3ObU7Xf5zOYp3uSmvGL4YM/mzKFb1o+YJIoMHTeNjrOORrPMpxwfZiLMoqDReIMaWJZjU0q6VoJc2oGYUce0pGYTxXy9tFEv1+VokNdAj1NRiEXYVrqPFv+DRyaJdmysLMx94mNa0br2xhuG6d8NfJXJVz4/PbVmYHs//77CI9r8u/xmoZ07pnV7sCPL/D6Vpvd5Q2Ovd6PHL4PVxU+F+Wyb2ie6ck8n7XCTn5PI5DXvtBADBkjDSmj8jIn8UodPEYo1pUy1JdaoRQX4NB2Pm42pP+71dEK8s1U6/K/O0/SVXrTswAwIlw47raWcZ1H1Wsn6gZd7ftbrqWHTAv3TrJRJDnmxrGx/E7GhvX1jfs9PoOen0jDfppFvPTw4unWcw9f0gfF/N7HUf8rmM+Z6Xbts9jP8go9CbraGPG6CFj79CbizSG/FSXESHU72AQdqZo1SH/hk+5+lO6mUtNNz5AMO3fkDvgOPxo6UoicmCSed7M8B/82yOZk9IG3JpOKdrvt2mLRHe53NuczvU2+/FAoMRkmmTQPz28eEa6pYd39YwEGtz2/T7nMZ+z0mPfH/TWaAz5xozRJuvonMKrjRkjk/s3IoRQSmAQdo7osQrv6g9Ev1s7/VrzsicIup3bcpzAbazZ+u8jXzZGeE9s4Tvm291rPIrRiqHLBzC61v8Q9ZFIaaOztNG5rdF5PBC4Ii3tyoz0v44dPSnNREpSNBpVq3tchIgiF3BXuW375Tqfq76cIGm9uUhvHpqRN6Oo5D5dWmGbscAIIZRiGIQdI0mRwzt9az6SBE43+3r1hDnQpooW4sJlDft211dsPr19VMawuYPuWrEh9wVXoxjjxzw4WGlurjXWhSNyta+00dkYjV2RZppmMb81Ydw4o4FMmuXosscRdjou6vM2HvXY98l1Pq/jkFqfqzcX6dKGFoy6dcK8FxXq9FSXESGELgCD8DJJAh8u/963/hNSrdcvuFk5fBIkZZUoScddJ3fXV+yqLz/urhphKb4ia9xbC1/0nVDt+VfDQ0bP8B/HO4VWBYNytW+dzR4WxOkW839ZzD8dVDDeZOyBC/TJTZ1u236Pfb/PeSwWcevNRXKdL2/YD0zW0diZBSHU62AQXjKJ50K71vm++5gyZRiv+YlyxOTEU+6Id5/94J76vf+p3c1S7ITMMUuLFl+RNV5NqzwnAjX/dByzeblp1pzp1L9t9tJdzs2ORk4Up1nMV1ozflE4eLhe36PCTxS4gKdKjj23bZ/XcZhmtSbraL25KKfw6pHTRuvNQ6EbV31CCKGugEF4CaRoOLhjrX/jZ0zuEPMdT7B5RQAgSMLhxsrtZ3eXNew7668bkzFiau4VPx55o1WTDgBRN2fb6G7YffqUgXvXLB4eEArFjpm2sDPSzfOs6c+MHJbXk271RQINvniXzv2t+rZkDZpnSB+pUKWluowIIdTJMAgvihj0BbauDGxdqSgcY7nnT0xmXl2goezE2j31e8sa9mVrMydkjvnZuNvHZIykSQoARF5y7PNuL6vf4ncdzOb2jPADoWTEtL+MzZ2bMTZb1SPaDyVJ8LtOeOwHPY6DXsdhr+MgEJQxfbghfWTmwNlFJffpTIMJEj8hCKE+Dr/mLkDwuwPffxn8z7fKEZP09z9XCf5tZ9du27EzJnITM8fOzp/260n36pKWOzlU7V6598xWR2OZPqwyUdOHW2bqs2zHzAM1mndmUGmXP6tJJ5BE3uc65rEdcNv3e+wHvI7DKm2mMWOUIX3E0Al3G9KHKzXWVJYPIYRSAYPwnHhnQ2Dzl4GyTQ3jxh+94dpy97GjWx4rNhdOzBr7+xmPFKUNSRxZFQxurnNsPN6w3tfIS9IVCsNVk/PeGJydr1avqBbv3SY8Mob6xcjzjpPvGpLI+90nEzf5kgewF5XcZ84uYZWmC18FIYT6NAzCdnD11Wc3frT3bNm+gsyyMbSWqZrA6W4ovnpi5li2aRGyxFCH1WdskRg/yqucqTb/eszkMSMscveRiAAP/EdYc1b6dgE9ztxNIZgYyZfo3qLW5+AAdoQQOg8MwmZRIbb38IYd+7/eG6l3quixk8aX5E68N3tCetNqyPWRSGl97XqbfZ3NHuT4ElE/ppZ9WRowZUxmxmIjrW6e4eyoR/rhRqHIQOy6ljZ05XoP508+k3VUYuVShBBC7cIghLpAQ1nDvl0ntpQ5D1ujxHjL8Ifm/Gxk5mh5DHtDJPLZ2drEIPcZFsu4mOa5ugH5Z8n0MQbrUpMmu3XPl/ePi7/aKTwxlnpgZOfPoiIKnN91wlFbHvIcddv2yWPYk5IPR/IhhNCl6adB6Iv6y23799Tv3VVfTvL8CI8wwS/dX3J7xtSrgaTs0egXtXXyOPfTofCkNNM0i/lWbXbmfsG5yqsdoMqcYjIv0xNU6wZPPwd3lwp7ndKmq+iRps5pDuVjAY/jsMe+L3lIgzZteHrO+JzCq03WMR1ZVwghhFA/CkJREo+7q8oa9u2p33vUebzYXDiWTHvoNDUoqtDNWhoc+V873N5th47KE1vLc3u+NWHcCErj2uuzfeMWeZ/qCuOERwrbzgsqK2uUbtooTMkgdl9HqzvwvspLNLRKPrnOVzDqx2mZYyWgeuZcowgh1Bv1/SB0hd276yu21+4ua9hnVpmm5lxx87AlhWcc4Q2fhXTRo5NueVltWW+zH1uzfpI5Hn7jjAZCAu+JYMNK994T9WnDdAXXZBoLteeaREUCeOWg+Me9wqtTqR8OuuTm0MSMnXLyhf31ekuxyTomI2/GkHF3GSzDWq2H3nPmGkUIoT6gbwZhVIgddByRR7vbgo6x1pETs8beP+FOM6t37Pzu+399+L6lYNvoJcc5cVKUvtLAyKs6MCQJAGF79PQqu223W2FgMqeYCm/KOdfqSLLGCCzbwtvCsONaepDuoppDWydfoEFvLpKTD5doQAihbtYHg/Cf+z/+9Oj/FZsLS7LGLZ9832BTQYgXtzfUPfv91lJHY6XKOHHMwuk5uX+2mJPX8xMiov2gx77HE7JFMyYYx9w/SGm5cHfPzfXSrd8LSwcSX1xJsecIL54LBj3VAU+1z1npsR9w2/cLXCgxb9mwyb/UGgfijJ0IIZQqnROEDodj3bp1Go1m/vz5SmU7vRY5jvvuu+9cLtfcuXOzs7MT+6urqzdv3pydnT1nzhyqkxZYv27owh8NXyISzPZG54d1ztK9W3Y7XUWBxqkq6veTJs8eOlyRvFKSBL7qkH2Pp3GfVz9QnTnFZB6lP/8a8TJRgr/sF/96UHhnBr1wQPz4WMQd8FQHPacC3pqgpzrgORXwVPMxv9ZYoDHk681FecNuGD3zdxpDXqf8pAghhDquE4Lw8OHDM2fOXLBgQX19/dNPP11aWqrRaJIP4Dhu9uzZADB06NAHH3xw9erVV1xxBQCsXbv25ptvXrp0aUVFRVZW1sqVKwmiEypGq+3+107s2+/1jddrp/oafn5k65T8QemLb6Yt2cmHxbycvczT8B83SRMZJcYJjxQy2ot9N2xhuHODUx09/d3oM2zD6fLKGp+z0uesFIWoHHgaQ745uyRv2PUaQ75aPwCbOhFCqMfqhCD84x//uGzZsueff14UxRkzZrz//vv33HNP8gErVqzweDwVFRUMwzz33HNPP/30t99+CwCPP/74n//85zvvvDMUChUVFW3atGnOnDkdL89AjeZ3eZkj9x8RN25UT5itu/cPlMGceFbkJdchn323x1cTsow2FN8+QJt7gSHn8poMQe/poLcm6K2pb6wJuqt+TFPmtPzQ6XzCkG+yjskderXGkK8x5He8/AghhLpTJwThN998s27dOgAgSXLJkiWrVq1qFYSrVq269tprGYYBgBtuuOHxxx+PxWJOp7OsrOz6668HALVavXDhwm+++aZTgnBU+Xf+zf+nnn619vF3SHXzdNihhqh9j8e2y63JVmZMNBbfNoBseVtPFLhwoC7orUlkXtBb43Meo2iFxpCvSxuqTSvaTl/1BZn3x+sHzxyg73hREUIIpVxHgzAQCPh8vpycHPlhTk5ObW1tq2Nqa2snT56cOEAUxYaGBofDodFoTCZTYv+RI0fO9Soej+e9997buHGj/DA3N/dHP/rRuQ5mx8+2TFlEKFQCgMBxfFhw7ffbd3q5AG8ZZxhxX57CxIhCzO8/FfTW+F3H/K7jId+ZoLcmGrIp1Fa5MVOtz8sctEitz9OaBsvzc54Owq2bwaKEL5ZAmgI4jrvc96yjBEHgOC6FBei38G1PFXznU4LjOIIgRFFMdUE6hKIokrzAzamOBqE8pi1xb4+iKJ7nWx0jimKiHHKPGJ7nBUFIviPY7okJHMf5fD6Xy5U4+Hxj6TQGCQB4wXcy5Kzwu47alANdyrFuVmVz+k+f2Xwm5DstZ55anydnXnrebF3aUJU2u93l9wRB+OoM8cBO4tcjpHuHSQRAagfyCU1SWYh+Cd/2VMF3PiX6xtt+wRSEjgehwWBQq9V2u13uC2qz2ZI7hcoyMzNtNpu83dDQAADZ2dkMwwQCgVAoJM+Q0u6JCenp6ffff3+iWnl+9Ue3n9q52ms7KbJ1HFlH5ZAazUAhWqBR5afnXqE1/lBryFdqMy/yB4wI8PAuYc1ZafVCqtsWkTg/+W+Idnvnoi7FcRy+7SmB73xKSJLEsmxn9efvyTrhHuHs2bPXrFkzduxYAFizZo3cQVSSJLvdbrFYKIqaPXv2u++++9RTTwHA2rVrp06dqlQqc3NzCwsL165du2TJEkEQ1q1b96c//anjhQEA++FqINjCWdel5RVqjQM7suRety0igRBCKFU6IQgfeeSRxYsXx2Kx2traioqKd999FwBcLldmZuaxY8cKCwtvvvnm559//tZbbx0xYsQLL7zw/vvvAwBBEI8//vjPf/7zY8eO7dixQ61WX3311R0vDACMWXpzp1ynSxeRQAgh1EN0wlf8tGnTtmzZAgCFhYVlZWUWiwUAtFrtO++8Y7Va5e1du3ZNnDgxEomsXbv2qquukk+8/fbbP/nkk0AgMG/evC1bttB0T5nmxs/BLZuEP+8TN11FYwoihFDfRkiSlOoyXNiUKVNefvnli7xH2EGJRSTenEZ1ZBGJriMIAq4+kRJ+v1+n06W6FP0RvvMpEQ6H8R5hbxXz1XMhN6OxMJq0dnuBnksHF5FACCHUG/XBIGw8vLrxwEou6OSCTkqhZbQWRp3GatMZjZlObGjiG2TTeu6XsYgEQgihPqAPBmH25DuyJ98hb4t8JOZ3cAE7H/bGAo6Y3x6yV8YCjpjfxkd8MZ9NEmK0Uh9RWitClpsNhskDrcoj6Q6lgdVlMNp0Wmlgdem4NARCCPVhfTAIk5G0UmkaoDQNONcBPBd5tdzzxaGGp8c4x2j9sYA96j4TjByK+e2xgF0Ie/mIn1bqKJWB1WawugxaqWd0Gaw2g1bqWV0Go81gtGaC6Ptt6Agh1Ff18SA8P1sYbvue5kTLpzdZs9XtV/tEIcaHPHzEywUcMb+dj3g5v8PrOCFXMTm/jQu6KIWW0aXTSj2rs7LadEppYHUZrDadVhkYbYbCkHVJtyoRQgh1p/77Bb2+VvqfLcKdRcRvx1HUuds+SYpldRmsLgPSC9s9QBJ5Lujigk4u4OBCLi7ojAXs/jNn5A0+6OJCLlplZDRmWqknKJZW6gmapRgVyapJiqWUOpJiSUZFsRqSZimFlqSVJM1SCh1BMZRCQzIqksLB/Agh1FX6YxDyIjyzV3j3mPTxbGp6Zkfv/xEkHU9KGHaOQyQ5KfmIVxI4PuyVBE7gwmIsJAoxIezj+KjIR4RoQBQ4IRoQ+YjIx4SITxQ4MRYSuJAkcHKIUqyKZNUEQROsllGo5PgkKJZWaElGQdAKWo5PVkMyKpJm5bNIRkWxaoJiaCWumIEQQq31uyCsCUg/2iikq4jyJXSaontek2A0ZkZjvvCB5yaHqBALi7EQz0UiARdDgchFhGhAEjk+GhC5qBj2Rd1nJZETYiExFhIFjo/4JCEmcmEhGhQFToj647VPhYag5NqngqQVBEVTrAYA5IcAQCm0BEEBQVBKHQCQJEOy6pYHaOT23ni4kk1XYJQkzQIAxWoJEm+dIoR6gf4VhCuqxXu3CY+MoX4xkuxdPUFppQEAGA0AgCAI9OUOqBebap9SvPYZFfmoXPUEAJGLiEIMAIRoQJIEkCQh7AMATmw6gI+KfDR+gCgASHzEDwCSmHQFPgYAQkw+IF5ygqQohQYACFohD1mhWHVzlBIEQVJJYRwf00KyKoJikstPNCVuMkqhJYgW4z5JWkEyreZoJmhl6xHZJKNuc/14OdtcH0O9vxO5sPzb0X8I0QgvMhJFAYD8h3iqSxTHai0kc4EF1S9JfwnCxCIS3y6ge8giEikhV+nkcOoefMQLAJIoCNEgAIh8RJKjNBaUk5IPNx0Qkw+IinxEPleMhcWWv3uSyEdiwVYvIUQDktRiyTSRj4pcpOVR8cxucRgXkoQWq9wlytniR4j6Qbq0JdkoVtNVPaRIklZoL3xYG3w0AD1+YTkhFpTEcy7Hllr98G69JEmJxfIIiqHYzsyejsiceGvW5GWdeMF+EYS4iEQKJUKXUaeltiQdd/ETfXXhF7oo8tHAZZxHK7RwEQuzpda5/oDAKdZSAqdY6ztwEQnU/dq233YiWn35K4shhNrqy0Ho5+DuUmGvU9p0FT3S1H+bQxFCCJ1Hn60klTVK47/kKQJ2X4cpiBBC6Jz6YI0QF5FACCF08fpgEH7y//ZlnGncN8pgbjQKpJ5MNxCabhowiBBCqNfpg0E4Z9YAXQ0FDh+383jE7hUdPiBJMkNPZRjIdD2ZYaAy9GS6gTRrgcAmU4QQ6u/6YBBmDE6DwS166kvBqOjwCnaf6PAJVTZu53HB7pU8QcKoSaQjma6nMvSk1UiocIAFQgj1I30wCNsiNApKk0ENzGixlxdEd1Cwe0WHT7R7uWp7xO4TG9xAkWS6XGtMBKSBtOiw+ogQQn1SvwjC9tEUma4n01vPQ92i+njKzu06cc7qY5aJUDDtXhshhFBv0Y+D8BwuofpY5wKGaqf62CZcEUII9VgYhBfnQtVHodbFHzoj2H2i3QscH68+ZqdRuWnxdEzTAoVjORBCqMfBIOyQdquPUjAiJ6Jo9wpVttiO46LdK3pDlEVHZhhIq4GyGkmrgbIaSIse0xEhhFILg7DzERolXaCEgtaNq0KjX7R5RZtHaPBw+6oFu1dyBQijRs5F0mqgMgxkppFK1wPd92e5RQihHgKDsLvQFJVppDKNAPnNOwVRdAUStx5jJxqEpluPVHYalZPW3DEn00QosWMOQgh1PgzClKLIc916FGpdQp2r3XEdRJZJSNcKA9Jx1CNCCHUcBmFPRGgU9NAsemhW8s54x5xaF3/WKVVUB9cfEhs8QBHJ3Vap7DRqgBnTESGELh4GYa+R6JhDCQJEo2q1GlqOeuSP1UU3HRJtHiCJ1oM6ctJIYxeukIcQQr0XBmHvdo5uq23mBDjrlHihvRlz9IAT5iCE+jcMwj6o/XQMRASbV7R7RJuXP1onbjks2L3Ai6TVSCX6rGYYSKsB644IoX4Fg7C/ILRKWquEwdbknVIoKtq8ckBylXXilsOizStFuORcpKwGMsNApumw7ogQ6pMwCPs1Qq2gCjKoNkMe4/PJ1brON9tqThqVacQJARBCvR0GIWojMZ/ciAHNO1vOtho7VifUutqfizw7jWDxc4UQ6jXwCwtdnHZnW205IUBiLnJCraBy0lqkI04IgBDqqTAIUQecY0IA0RMUal0t0rHeDXTTQo/ZaVROGi6DjBDqITAIUecjjZq2XU8TEwIItS5u94mI3df+hAC5aYRakZJiI4T6p04LwuPHj7tcrnHjxrFs+3/jOxyOEydOFBYWWiyW5P3BYFAQBL0e1/Dr49od1CF6gqLNK9q9gs3LHzoTnxCAJql0A6FXEToVqVUSOhWpVxFapfwfqVcRGmWqfgqEUN/TCUEoiuJtt922devWAQMGnD17dv369UOGDGl1zAcffPDQQw+NHj16//79f/vb32655RYAWLly5S9/+cuTJ0+OGzeuvLy84yVBvU687liUnbxT8ocFu0/yhyV/WAxEJF+Yb/CI/rAUiEiBiOgPS6EoqWuKxnhMqkhd00OdkpCfVeBdSYTQhXVCEK5bt27r1q0HDhzQ6/W/+tWvnnzyyY8++ij5gFAo9MADD3z11VfTp0/fsmXL0qVLly5dqlKphg0b9sknnxw6dOivf/1rx4uB+gxCp6J1qvMfI8V4yRsS3UEpGJFCUSkYFT1BsdouBaNSMCrvkUJRQq0gNAr5/6RRQxg1pCaxR0maNKRJg4teIdTPdUIQfvrppzfccIPctrls2bKJEycKgkBRzV8u69evT0tLmz59OgDMmDHDYDBs3LjxqquuKiwsBIBjx451vAyovyFYmmivn04yiROkYCQejcGI6A2J7qBg90mh+E7RExTdQYIAQqNsSko1oVYSGkU8L40a0qghNApSrwYSJxRAqG/qhCA8c+bM2LFj5e2BAwdGo1GbzZadnZ18QEFBQeJhQUHB6dOnL+klwuHw7t27A4GA/NBkMk2YMKHDBUd9HMFQhFEDF5oxTgrHJF9IDEQkfyTeGOsPC2cCbRtjJTXrU7KkWgEsTTAUoVYATREKmlCywFCEiiUUNNAUoVYQDE2wFKFSAEMRCoZQMjjzAEI91kUF4aFDh5566qm2+//617/m5uaGw2GFIt7NT6lUAkAoFEo+LBwOJ/egUSqVrQ64IKfT+d577yU61BQUFPztb3+7pCv0JYIgRKNRURRTXZA+REODRgtWbavdBAABQAKAKEmBSMjhVtGsGI4RnCByAoRjwAnACVIgRAiiFI5BTACOhwgnxXjgRSkUJQRRinIQ5UGSQMkQLC1RJKFRAk0SLA1KBhgKWJpQMhJFEmoFMBQwFKhYgqEkhiJULFAkoWKBpYGhoL+OxQwGgwSBNfLuJn91Jzfv9UZKpZKmL5B0FxWEVqv11ltvbbvfYDAAQGZmptPplPc0NjbKe1qdnjhAPiYrq8VKexeUm5v78ssvT548+ZLO6qsEQWAYRl6GCXUfvQ50Sp1Od5mnC6IU4aQoB7wghaISJ0gxXgrHgOOlKC+FY8ALUjgmBWPAxQ8gYrwUikq8IMoncoIUjhEsDYxc6aSAZQgVS9CkPBwzPuxEQRM0BRRJKFkAIFQskATBUMDQQBLxI5UsQRFAU3J/IvlEuTrbOe9VZ5MkSatt/WcK6moURfWBILwYFxWEFovluuuuO9ezJSUlmzZtevTRRwFg69atI0aMaPWRLSkp2bdvn9/v1+l0Pp9v3759JSUlHSw3Qr0MRRIaBaHp6BBJKcYnkhJinBSOSZwgRTgAkEJRAJCiHPAiCKIUjACA2OgDUZI4HjhBEiUIxwBAisQkQQJekKJJJ0Y4EESgSHkOoHg6sjQwNNGUoKBiCZIAhiIYGgiCUMuxygBFAU22G6uEWiHX5Qi1AggicUAilRFKuU64R3jHHXc8//zzv//974cPH/6b3/wm0Yi6YMGCxYsX33fffcXFxfPmzbvlllvuuuuu//3f/50/f77cTaampubf//73vn37bDbbn//854KCghtvvLHj5UGoDyNYGli644F6ToLYJlYFSZCkSAwAIByTRAk4QeJ4ECWpKVYhwgEvitE2eQwghaKSBPH9kpQ4IHF6cyKSTRlMUyBnKkPJ89YKhBRUKwEAGJpgKICmim8ihuMbJAAQSlbu2USoWTl6yfjbRSTet8SkDYljUH/WCUFosVhKS0tfeeWVysrK55577qabbpL3L1myZPjw4fL2Rx999NJLL33wwQeTJk166KGH5J08z7vd7ry8vB//+Mdutzs9Pb3jhUEIdQhFymnRhVnbSiIRxaYM5gWQM5UTpBgPAKIvwFAMAADHS5wAyUEb4SDCAYDo9IMgAoAUjjUnrgQAIAajAAAgSfGNpjAGkELxgwklCxQB8p8acsSyNDA0ABBNld3kWmxSlCrkFcoIJRtP4kRtWG6Rjkc7DQDNdesWqcwCEMknom5GSPJfaz3blClT8B5hgtxZBu8Rdj+5eT/VpeiPuuGdlyIxECSQI1YO1BgPHA8AEi/KbcjNmZ0cpYlwjcTiJ0Z54AUAkFukIR7tPABITRXulqkcA5Ag0ToNkGigBgBC1RS0iWAmmmMYlAwhd0imKEIRr9g0Z3PTKS0iXMkASUJSPENzpRmAZeJ1bpKIgsgwDJXU7t1X4VyjCCEEct8i6M6q8Hk05yVI4Xi9tjmYpeYYhggnyfibzLgAAAsiSURBVNkpCFKUj5+SiNgYB8EIAIAoiQ2e+M4IB6IIABInQCx+SlOlGSDKSbwQPyUcjcrtxk1Vc4AWIZ1ozY5vJ7I2OaqTt6HNNpG8HX8g9/CKbzfFNiS1flM5aW1nM+4IDEKEEOphmhqoIaXB3P7wCb45cUFoqisDgCg2V5eTozp5G1puB1vub2qelHt4xfc3xTYk1ZiZyYWKGcM7/gMmYBAihBC6aDRFJN3IJOACsyH2CjjbBUIIoX4Ng7D3OXXq1IYNG1Jdiv7o888/93g8qS5Fv+Pz+T777LNUl6I/2rRp04kTJ1Jdiu6AQdj7bNu27eOPP051KfqjN9544+jRo6kuRb9z4sSJV155JdWl6I8+++yzLVu2pLoU3QGDECGEUL+GQYgQQqhfwyBECCHUr/WOmWVyc3P1ej3OpSJzuVw+n2/gwIGpLki/U1lZOWDAAPwcdrNwOFxdXT1s2LBUF6Tfqamp0Wq1ZrM51QXpkKVLlz722GPnP6Z3BOHRo0cDgQAuSCYTBCESiWg0nTmxAroYPp9Pp9Ph57D7+Xy+xHKkqNuEQiGWZS+4mF8PV1BQkJaWdv5jekcQIoQQQl0E7xEihBDq1zAIEUII9WsYhAghhPo1DEKEEEL9Wu/uDtTfnD59es2aNYmHCxYsyMvLS2F5+rZwOLxv377Dhw9PmDBhzJgxif1er/fDDz+02+2LFi2aNGlSCkvYVzU2NpaVldXU1PzgBz8wmUzyzh07duzfvz9xzB133NHbezP2NG63e/Xq1ceOHUtPT7/xxhvT09MTT61atWrHjh1Dhgy5+eabGaYPrtCLNcLe5MCBA08++WRZE6/Xm+oS9WXz5s1btmzZo48+unbt2sTOaDQ6derUTZs2sSy7ePHiL7/8MoUl7JPcbveAAQOeeuqpu+++u66uLrH/888/f/PNNxMffrFpjTrUWZYuXfr5559TFFVaWlpUVHT8+HF5/7PPPvvAAw/odLp33nnnhhtuSG0huwj+SdXLDBo06K233kp1KfqFjRs3six7zTXXJO/8/PPPGYb59NNPSZLMy8v7wx/+sGTJklSVsE8yGo0+n49hmNbrwQIsWrTomWeeSUmp+oOVK1fqdDp5+5prrvnnP//57LPPhkKhF154Ye3atSUlJffee29ubm5FRcW4ceNSW9ROhzXCXsblcr322msff/yxw+FIdVn6OJZl2+7cuHHj/PnzSZIEgIULF1ZUVLjd7m4vWl9GEMS5Gt8OHTr06quvrly5kuO4dg9AHZFIQZlCoQCAiooKmqZLSkoAQKPRzJgxY+PGjakpX1fCIOxNlEplUVHRyZMnP/jgg+Li4t27d6e6RP1OfX19RkaGvG02m2maTm6+Q13HbDbrdLqqqqonn3xy/PjxeF+g66xfv760tPTOO++Elh94ALBarX3yA49No73J3Llz586dK28/8sgjDz/8cJ/866wnoygqcXdKkiRJkrDLRvd49NFH5Q1BEKZNm/bqq68+8cQTqS1Sn1RRUXHrrbd++OGHOTk5AECSZPLtWEEQsLMM6kFmzpyZuJuNuk12dnbiL+KGhgZBELKyslJbpP6Goqjp06fjh78rHDhwYNGiRa+//vqiRYvkPTk5OQ0NDYksrKur65MfeAzC3iQWiyW2165di/Pxd7+rrrpq1apV0WgUAFasWDFjxgycDLp7JD78HMdt2LChuLg4teXpeyorKxcuXPjiiy8uXbo0sXP8+PFKpVJueWpsbNy6dWsiI/sSnHS7N7nttttsNlt+fv6RI0dOnDixevXqsWPHprpQfdZLL720evXqvXv3mkym/Pz85cuXz5s3TxCEK6+8MhqNjhw58osvvlixYsXMmTNTXdK+5oc//KHL5dqwYcPkyZM1Gs3nn39uMBiKiopGjx6t1+tLS0tNJtP69eu1Wm2qS9qnlJSUVFdXJ75SZs2a9fjjjwPAO++889hjj11//fWbNm2aMWPGm2++mdJidgkMwt6ksbFx586dDQ0N2dnZM2bMwJWYutSRI0dqa2sTD0eMGCE3CnEct27dOofDMXv2bJzQoCts3bpVrnPLZs6cyTDMyZMny8vLg8Hg4MGDp02bhothdbodO3YEAoHEQ6vVOmrUKHn74MGDu3btGjJkyIwZM1JUuq6FQYgQQqhfw3uECCGE+jUMQoQQQv0aBiFCCKF+DYMQIYRQv4ZBiBBCqF/DIEQIIdSvYRAi1O+89NJLn376aapLgVBPgUGIUL/z9ttvr1q1KtWlQKinwCBECCHUr+EKMgil3rFjx06fPm0ymcaOHZtYmV0URa/Xq9FoaJouKysLBALjxo0zGo2tzj1z5kxlZaVWqx0/fnzbxYTdbvf+/fsJghg8eLC8sE6y6urqqqqqIUOG4FxxqF+TEEKpc/LkyUmTJgGAvK7h0KFD9+/fLz91+vRpAHjmmWeGDx8uLwKn0Wg+/PDDxLl+v//6668HADk7MzMzV61alXg2Go3ed9998ony/3/xi1/ITw0bNuyWW275yU9+kpixc9myZd35UyPUo2DTKEIpEwwG582bF4lEysvLY7FYVVWVxWJZvHhxOBxOHPPMM8/cddddPp/P4XDMmTNn2bJlBw8elJ+65557vvnmm/fffz8UClVXVw8ePPiGG25ILNT34IMPvvnmm88++6zb7Q6FQvv3758+fXrisitWrIjFYmfPng0EAo888sg///nPTZs2defPjlAPkuokRqj/euuttwDg0KFDiT1Hjx4FgK+++kpqqhHOnz8/8WxjY6NGo7n//vvlbYqi7rvvvsSzp06domn6oYcekiSpvr6epum777673dcdNmxYfn5+NBqVH0ajUY1G89vf/rYLfkSEegG8R4hQymzdulWr1ZaWlpaWliZ2Mgxz9OjRa665Rn545ZVXJp4ym83jxo07fPgwAFRWVgqCMH/+/MSzAwcOLC4u3r17NwDs3LmT5/nrrrvuXC89YcKExA1FlmWzs7Pr6uo69YdDqNfAIEQoZTwejyRJn332WfLOmTNnmkymxMPkbQAwm81VVVUA4HA4ACAtLa3Vsx6PBwD8fj8AWCyWc710q1VtaZoWBOGyfxCEejUMQoRSxmq1AsCaNWsSPUXbSl4cGADOnj0r9/CU/9/q2TNnzgwePBgAMjMzAaCmpmbChAldUHCE+hTsLINQyixatCgYDL7//vvnOebTTz8VRVHerqysrKioKCkpAYCioiKdTvfhhx8mjty2bVtVVdXcuXMBYMqUKUaj8c0335Rw5W2ELoR6+umnU10GhPqpoqKisrKyv//978FgUJKkmpqatWvXLl++fNSoUVlZWT6f7+WXX2YYZtu2benp6eXl5XfccQdN0++++65arWYYhiCI119/vba2VqvVbt68+ac//anFYnn77beVSiXLsmaz+W9/+1tZWZlarXY4HN98883q1atnzZoFAK+99lpOTs6SJUsSJXnttddyc3PPc08RoT4Mm0YRShmSJL/88svnn3/+3Xfffe6551iWzcvLmz9/fvLw9qeffnrPnj3XXnttKBSaNm3aG2+8kbjz98gjj+j1+pdeeuntt99Wq9ULFy586aWXEiPu77rrrrS0tD/+8Y/XX389SZIFBQXLly+XnxoxYsTAgQOTSzJixIj8/Pzu+JkR6nkIbDlBqCfgeV4eU59w5syZvLy8Dz744NZbbwUAQRDOdSux7bnJ5JZVksT7IAi1D2uECPUI50ky2Xk61Jz/XIxAhM4Pf0MQQgj1a9g0ilAPFYvFDhw4UFBQ0GqwIEKoc2EQIoQQ6tewaRQhhFC/9v8BELxwdd8TzMYAAAAASUVORK5CYII=", + "image/svg+xml": [ + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ], + "text/html": [ + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ] + }, + "execution_count": 27, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "n_epochs = length(losses)\n", "n_parameters = div(length(parameter_means), n_epochs)\n", @@ -632,62 +1858,217 @@ " title=\"Flux parameter mean weights\",\n", " xlab = \"epoch\",\n", ")" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "**Note.** The higher the number in the plot legend, the deeper the layer we are\n", "**weight-averaging." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", - "source": [ - "savefig(joinpath(DIR, \"weights.png\"))" - ], + "execution_count": 28, "metadata": {}, - "execution_count": null + "outputs": [ + { + "data": { + "text/plain": [ + "\"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/weights.png\"" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "savefig(joinpath(tempdir(), \"weights.png\"))" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Retrieving a snapshot for a prediction:" - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", - "source": [ - "mach2 = machine(joinpath(DIR, \"mnist3.jls\"))\n", - "predict_mode(mach2, images[501:503])" - ], + "execution_count": 29, "metadata": {}, - "execution_count": null + "outputs": [ + { + "data": { + "text/plain": [ + "3-element CategoricalArrays.CategoricalArray{Int64,1,UInt32}:\n", + " 7\n", + " 9\n", + " 5" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "mach2 = machine(joinpath(tempdir(), \"mnist3.jls\"))\n", + "predict_mode(mach2, images[501:503])" + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "### Restarting training" - ], - "metadata": {} + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "Mutating `iterated_clf.controls` or `clf.epochs` (which is otherwise\n", "ignored) will allow you to restart training from where it left off." - ], - "metadata": {} + ] }, { - "outputs": [], "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mUpdating machine(ProbabilisticIteratedModel(model = ImageClassifier(builder = MyConvBuilder(3, 16, 32, 32), …), …), …).\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.4449181129617429\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.4575672614002921\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mSaving \"/var/folders/4n/gvbmlhdc8xj973001s6vdyw00000gq/T/mnist1.jls\". \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.4693455717095324\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.48012884529192995\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mloss: 0.49023152105995377\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mfinal loss: 0.49023152105995377\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mfinal training loss: 0.010609009\n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mStop triggered by Patience(4) stopping criterion. \n", + "\u001b[36m\u001b[1m[ \u001b[22m\u001b[39m\u001b[36m\u001b[1mInfo: \u001b[22m\u001b[39mTotal of 32 iterations. \n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAFyCAIAAACm2zNGAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3gU1foH8PfMzvbd9ISEJEASSEJIaNISUECKUq4NiEhRBK6A7WK9/MCG7YIKKogCykVRLFQhV0SaSgmC9N5bKIH0ZHuZ+f2xIYRk6bs7Sfb7eXx8ZmdmZ98JkO/OmTPnMFEUCQAAwF9xUhcAAAAgJQQhAAD4NQQhAAD4NQQhAAD4NQQhAAD4NQQhAAD4NQQhAAD4NQQhAAD4NQQhAAD4NQQhAAD4tZoehFlZWatXr5a6ijtlt9ulLkEyOHf/hHP3T7X03Gt6EP71119///231FXcKYvFInUJksG5+yecu3+qpede04MQAADAqxCEAADg1xCEAADg1xCEAADg1xCEAADg1xCEAADg1xCEAADg1+pUEFpP7C9Z/pXodEhdCAAA1Bq81AV4kiI63vDHkrzPXg0dNkEWGCp1OQBQWzmdzj179giCcBvvNZlMGo3G4yXVCl4998jIyOjoaG8cuU4F4XmHKvcfExL3LLo05dmQoeOUTVpIXREA1EqrVq167LHHGjdufBvvFUWRMebxkmoF7527zWYzGo3Hjx/3xsHrVBDmW6n3KueCewd0aJhc+O1k3T0P6rtlSl0UANQ+DofjnnvuWb58udSFQLkzZ87cfffdXjp4nbpH2CKELenOZ65z/KVNi3jhE/PujYXffiDarFLXBQAANVedCkIi6liPLerGZ65zZFvCwp/7iPH8pU9fdBTkSl0XAADUUHUtCIno7kj2XRd+wFrH38Xy4Mde1HXsk/fpC5bDO6SuCwAAaqI6GIRE1COazevCP7DKsS1f1Gb0Dhn2WtH3U8rWLiBRlLo0AACoWepmEBJRz2j2dWe+72+O7fmiMr5ZxIvTzHuzC+ZMFCwmqUsDAIAapM4GIRHdH8O+6Cj7xyrHviJRFhga/uyHspB6l6Y+b794RurSAACgpqjLQUhEDzfipqfL7vvVub9IZLw86JExAT0G5k1/1bwnW+rSAABqkG3btn355ZcbN2705YceOHBgxYoVvvxEt+p4EBJRvzjuk3Su56/OA8UiEWnadg8f/W7Jz7NKsv6LW4YAUPfk5+f37dv3lt6ybt26fv365eXlmUw+vXmUnZ09Z84cX36iW3U/CIloQBw3tQPX81fnoWKRiOQxjSNemm7LOZL/5RuC2SB1dQAAN8tms91wH4vF8vvvv9/ScTZs2PDII4+MHz++Z8+eVfYUBKF6OprNZreHrbzeanXzDLcgCNc/hZs5QY/ziyAkokfjuXfv4u5b6TxRJhIRpw0IH/2ePCru0tR/2S+ckrg4AIAb2bFjR8uWLRMSEqKjo2fPnu1amZmZuWjRItfyokWLMjMziahfv35mszkhISEhIaH6mGR//PFH06ZN4+PjGzVq5HrvpEmTPvnkk7lz5yYkJKxbt65iT6fTOWLEiOjo6JYtW8bGxu7du5eIJk+e3LBhw2bNmkVHR1dczE2ePHnEiBFdunSJj49v0qTJwYMHhwwZkpCQEBUV9eeff7r2iY2NnThxYmJiYkxMzMiRI6sH3urVq5s1a5aYmBgXF1dxUr5Rp4ZYu75hiZxAdO8vzt/7yOL0jDhZ4D+Gy6Pj82aMC+7/jLqltwbvAYA6wOSg7Is+upmilFF6BOMrXadYrdb+/fuPHz9+5MiRR48ebdeuXcuWLdu1a2cwGCoSxWazGQwGIlq8eHFSUpLbYTmLior69+//9ddf9+3bd9u2bV26dGnVqtW4cePKyspsNtuHH35Yeef169dv27YtJyeH5/nS0lLXygcffPCll17ief7kyZPp6en33XdfTEyM2WxetmzZli1bEhISnn766bvvvvvHH3/87rvvvvrqq3Hjxm3evJmIiouLDxw4cPjwYYvF0q1bt1mzZj333HMVn3XmzJmhQ4euXLmyZcuWx48f79SpU7t27Ro0aODhn+w1+FEQEtHwRM4p0L0rnL/3ljXSMyLStO4ir9egYO471hP7gh56ijiZ1DUCQE10sFicvMfpm89iRF/dLWuguzJ69d69e41G44gRI4ioSZMmjzzyyPLly9u1a3czRzt8+LAoikSUkJCwadOm+vXru+4gtmnTpnPnzr/++uuzzz5bsfPZs2ddaRoZGVmvXr2cnJxp06Y99NBD8fHxrh3i4uIWL1586NAhs9msUql27doVExNDRH379k1ISCCie++9d+PGjd27d3ctv/rqqxUHHzt2rEwm02q1Y8aMmT9/fuUgXLZsWWJiYn5+/po1a4iocePG69evHzJkyO3+CG+NfwUhEf0zmTM5qOdK5x99ZPU1jIjk0fERL04rnDcp7/NxocMmcLogqWsEgBrnrjC2updkvzALCgrCwsIqJnaIiIgoKCioso94jd5/Tz75pMPhIKKlS5e6jlOxqfpxpkyZsmnTJiIaN27cI488snjx4rlz57733nvx8fGLFi1q2LDh/fff74pSnudXrFhRVlbmemNgYKBrQaFQVF6ufKcwNLR8drywsLAqn5ubm5ufn79w4ULXy5SUlMjIyJv72XiA3wUhEf0rlROIuv7i/KMPH6UhIuI0+rBR75atW3hxyvOhT76maJAodY0AAFfEx8efOnWqrKxMr9cT0e7du++9914iCgwMLCoqcu1T0RYql8udzisXr9nZ2ZWPc+jQIbvdLpfLXcfp1q1b5Q/6+OOPK7/s2rVr165d7Xb74MGDZ82a9dJLL23atMlisXAc53A4XnjhhVs6i/379ycmJhLRvn37XJePFZKTk/V6/axZs27pgJ7ij0FIRC+kcoJIXX9x/NGXj1QTERFj+m6Z8sgGBV++GdB3mLb9fRKXCABwWZMmTbp16zZs2LCxY8du2rTp77//njdvHhF17dr1448/Tk5OPnPmzLffftukSRMiCg8PDwwMnDRpUkJCwv333+/KTpdOnTo1aNBg9OjRw4cPX758eVFRUb9+/a71oWvXrt27d2/79u1tNtvRo0cfeOCBoKCgkJCQTz/9ND09febMmdfqO3otkyZN0mq1paWlU6dOXbBgQeVNmZmZH3744ejRowcPHmy329evXz9o0CBXavqAnwYhEb2Uxhns1PNXx7refJiqfKWqWYfw52ML5ky0nT4c1O9pJvPfnw8A1CgLFy785JNPpk2bFh0dvW3bNlcL54gRI4xG42effdasWbNvvvnm8OHDRMRx3Nq1axcvXrx9+/YuXbpUDkLG2OrVq6dMmTJ16tTGjRtv2bJFrVYT0T333FP5ItKlUaNGq1atmjx5slarfemll1x37FauXDl16tRNmzY99thjnTt3TklJIaJOnTpVPGLRpEmTwYMHu5b1ev2LL75YccD333//hx9+MJlM8+bN69y5MxGlpaW5rk2VSuXmzZunT58+bdo0hUJx1113RUVFeetHWQ27VrNyDTFhwgStVjt+/HgvHf/17c6s0+LaPnyo8spK0WounP+RYCgOGTZBFhBy559S0aDhh3DuUlchjdp+7llZWV9++SUm5vUgvV6/f//+2+4I6pqY9/Tp056tysVfniO8lnfukvVpwLqvcBRWevSTKdWhT76matb+0tR/2U4dlK46AADwOjT90XttZE7R2eNXx5pefHDFdaHrlmH9uIL/vq3v9qiu80NSlggAUMtV9C+tgfz9itBlUltZt/qsz2+OMvtV61VN24Y/P8X418rC+R+KdgkG/gEAuG3ffvvtwYPXa9P64IMPiouLfVZPjYUgLDe5naxVGOu10mG4Ogv5sPoRL04jYpc+fdFReFGi6gDA302fPn369Om39JZ169adOXO9WecWLVrkenzezyEIyzGizzJkaSGs128Oo+PqTXJFyOCXdRm98z55wXpkl0QFAoBfO3fu3Llz56qsNJvNFSsNBsP58+crb507d+599115EkwUxfPnz9vtV77sb9261TUujIvD4Th//nyVHpRmszkvL89TZ1EzIQivYESfd5SlBLGHVjvMjqpbtRm9Q4a+WvjdB4Y/l0pRHQD4r1WrVs2aNWvWrFkJCQlPPPEEEUVGRr788sspKSkDBgwgos6dO7dt27ZXr14NGjRYv369613333//kiVLiOjdd9995JFHOnXq1LNnz8jIyLVr17p2iIiIOHnyJBH169dv9OjRLVu27NatW1xcXMWz+a+//nrDhg179uw5aNCguLi4bdu2+f7cfQCdZa7CiL7oKBv2p/Oh1Y5lPXnV1SOPKpu0jHjhk7wvxvPhMaqUthLVCAASECxG66EdPvowXq5Kvovx8ooVPXv2HDVqFBFNmjSpYmVubu6xY8dkMhkRzZ8/33Vtt3LlytGjRx84cICIHA6HIAiunTds2LBjx47Y2Ng5c+a8/vrrrgFl7HZ7xfXfjh07/vrrL51O9/TTT0+dOnXGjBkbN26cM2fOvn37IiIifvjhhx9++MFHp+9zXgnCsrKy7OzskpKS1NRU1+OWbvdZs2YNY6xHjx5ardYbZdwejtHczrIn/nQ+vNrxcw9eeXUWyoIj9N0fNWz6BUEI4FechZdMu9b76MMYU8Q2kQWGXn+vMWPGuFKQiGw22/vvv3/u3Dmbzeaa4UGlUlXeuVevXrGxsUTUuXPnf//739WPNmjQIJ1OR0RdunT56quviGjt2rUPPfRQREQEEQ0cOHD06NGeOLeayPNBePLkyZYtW7Zt2zYyMvKZZ5558sknP/jggyr75Obmpqenp6WlCYLw8ssvZ2dnu37WNYSM0TedZUP+cA5c51zQTSa/uv1Y06pzybIvnYWXZCE1qGYA8Cp5/bjQYROkruIqISHlw32cPn26Y8eO//73vx9++GG73f7VV1+ZTKYqQVgxvoFcLq98m7CCKwUr72A0GjUajWslY6xiue7x/D3CsLCwgwcPrlmz5rvvvtuwYcOHH3549uzZKvt8+umn7dq1W758+f/+978WLVrMmDHD42XcIRmjeZ1lRDRwndMhXLWJyRWaNt2MW1ZKUxkA+CWFQuE2wIho48aNrVu3Hjt2bPfu3SvS8c6lpqa6phIkomPHjl28WGe7zXs+CPV6ff369V3L9evXl8lkFoulyj5ZWVmumZSJaMCAATVzHCM5Rz91kzlEeuz3qlmoy+ht3LxSdFbrUQMA4B0tW7Zcvnz5Z599tmLFiiqbUlNTs7Ozv/vuuwULFrht9rw9jz76qNFoHDBgwAcffDBixIiQkBCOq5v9K73bWeb999/v2LFjlek2iOjs2bPR0dGu5ZiYmOp9gisUFBTs379/5syZrpccxw0ePLjKJb/3yIh+7EID1gmDfhe/68xkl6fJZGH1ZeHRpj3ZquYdb+Y4Tqez+oC2fgLnLnUV0qjt517Rx6TmePjhh1Uq1f79+11P/r322msVd5RatGjx/fffL168OCgoaM6cOYsWLXI1Yz755JPNmjUjonvvvbfiecHg4OCJEye6lidMmOC6ghwyZEjFL+pmzZo99dRTRKRUKjdu3LhkyZKSkpL58+enpKT4ciBst27jLxXHcRXzOF6LFwfd/uabb1577bUNGzY0atSoyia9Xr9u3bq2bdsS0ebNm/v06VNYWOj2IEOHDj106FCLFi1cL1Uq1WuvvRYU5NO5c61OGrieC1fRzA4Cd/nnad2z0bJ1deDIiTdzhNo+APGdwLlLXYU0avu5//LLL19//XVWVpbUhUhs69atrVu3FkVx8uTJS5Ys2b59+w1DxUvOnDnTqVOnI0eO3Oob5XJ5RZeia/HWFeGPP/44fvz4tWvXVk9BIoqKirp06ZJr+eLFi9f5ltGgQYOmTZt6b/aJm6EiWtKD+q5yPPe37Mu7Za4sVN7VxfS/OXxZAR8efcMj2O12n13F1jQ4d6mrkEZtP3eFQiHVb/wa5dtvvx04cKAgCC1atFiwYIG0PxPGmJf+UnmlwXfJkiUvvvjib7/9lpycXLHS4XBUDGrXuXPnVatWuZZXr17dpUsXb5ThQWqesnryJ8rEURudritoJuM17XoaN/8qcWUAAF4zffr0EydOnDp1atmyZY0bN5a6HG/x/BXhkSNHBg4c2KFDh4ph8caOHdu0adN169b179+/tLSUiF544YWMjIyAgABBEL7//vstW7Z4vAyP0/D0v/v43isdYzc7P02XEZGuY5+LU54P6PU4kyukrg4AAG6T54MwKCjos88+q7wmICCAiJo1azZt2jTXmpSUlK1bt/7www+Msb///ru2fNHQ8pR1H99hmWP1ObFHNJMFRyhim5h3b9S0uVfq0gAA4DZ5PggjIiJcPY6qiI6OHjZsWMXLxMTEN9980+Of7m0BcvpnMvfTCaFHtIyItB37lK1diCAEAKi96uZDIV41II4tPSXYBCIidUo7Z0me/dxxqYsCAIDbhEG3b1mMljUNYmvPib1iGXGctv19xs0rg/o/I3VdAOBJp0+fnj17ttRVQLlrPWLnEQjC25EZzy04KfSKlRGRNr3XxUmjAv8xnCnVUtcFAJ7RvHnzDh06bN++/Tbea7fb5XL5jferi7x67mPGjPHSkRGEtyMznpu4w251ypQykgWEKBs3N+34Q5veS+q6AMAzGjZsOGvWrNt7b20fTOBO1NJzxz3C2xGpptRgtvpc+aA82ozeho3/k7YkAAC4PQjC2zQgnlt4snw0QlVSa9FmsZ255bF/AABAcgjC29Q/jlt+WrC4BoBlTJvey7jpF4lrAgCAW4cgvE2RamoVyladLb8o1Lbvad67STAZpK0KAABuFYLw9g2I5xacLL9NyGkDVE3bmbatlbYkAAC4VQjC29c/jluRI5gvz86r7djbsOkX8tq0VgAA4A0IwtsXrqLWoWzl5dZRZXwq4zjriX3SVgUAALcEQXhHMuO5hSevXAJqM3qjywwAQO2CILwjjzTiVuQIxsuto5q23SyHtgmGYkmLAgCAW4AgvCNhKmofzlbmlLeOciqtunkn45ZV0lYFAAA3D0F4pyr3HSUi3d3/MGb/ii4zAAC1BYLwTvVrxK06e6V1VB6dwGn0lsM7JC0KAABuFoLwTgUrqUME++WMULFG27EPuswAANQWCEIPqNJ3VHNXV+vJ/c7ifAlLAgCAm4Qg9ICHG3Grzwml9vKXTK7QtO5i/GulpEUBAMBNQRB6QJCCOtWr1jq6+VfR6bjOuwAAoCZAEHrGgKtbR+X1GvChUZYDWyUsCQAAbgaC0DMebsT9fv5K6ygRaTtilBkAgFoAQegZAXK6J4rLOn2ldVTd8h77uROOvHMSVgUAADeEIPSYAXGs8pP1TMZr2nU3/vWbhCUBAMANIQg95sGG3PoLQontyhptRm/jllWiw37tNwEAgMQQhB6jl1OXKG5ZpdZRPjRKEZNg3rNJwqoAAOD6EISelBnPFp4UKq/RZvQ2ZqPLDABAzYUg9KQHGnIbc8VC65U16tQOjvwLzotnpCsKAACuB0HoSVqeukVzyyu1jhIn03a437ZtjXRFAQDA9SAIPWxAHFtQpXW0w/32PRtFm0WqkgAA4DoQhB72jwbc5otiQaXWUVlQGN8w2bTzT+mKAgCAa0IQepiGpx7R3M+nrrooVLTtYdy0QqqSAADgOhCEnle97yjfuKVgKrPlHJWqJAAAuBYEoef1juW2XBIvmSutYkzb4T5jNi4KAQBqHASh52l4uj+W+/n01V1m0nuZ92wSLEapqgIAALcQhF4xIK5q6yinDVAltTJtWydVSQAA4BaC0Ct6x3I78q9uHSXSZvQxbsySqCIAAHAPQegVKhn1iuWWXN13VNm4ORGzndwvVVUAAFAdgtBbBsSxBSeEKiu16b0MmK0XAKAmQRB6y/0x3K5C8bxJrLxS06675cBWwVAiVVUAAFAFgtBblDLqG8stPXVVEHJqnTotw7h1tVRVAQBAFQhCL8qM56r0HaWKiZlE0e1bAADAxxCEXnRfDNtfVLV1VNEwmam01qO7pKoKAAAqQxB6kZyjvg24xSerXvzpMnqjywwAQA2BIPSuAXHcgmqto5q7ulqP7naWFEhSEgAAVIYg9K4e0exgkXjWxCqvZEq1pnVn41+/SVUVAABUQBB6l5yjBxtyy8/KqqzXZvQxbv6VhKoXiwAA4GMIQq/LjOeW5lT9Ocvrx8mCwy0Ht0pSEgAAVEAQel23+uyEgZ02uO0yg4mZAAAkhiD0Op6jPtHComp9R9WtOttzjjoKciWpCgAAXBCEvvBwrFD9yXrGyzVt7jX+tVKSkgAAwAVB6Av3RDhPlYknyqpeFGo79jVtWSU6HZJUBQAAhCD0DRmjhxu5ebKeD4viIxtY9mRLUhUAABCC0GcGxHHVZ2UiIl1GH0M2uswAAEgGQegjXaLYOZN4rLRal5nmGY5LOfbcM5JUBQAACEIf4Rg93Iir3neUOJm2XU90mQEAkApffdWMGTM6derUokWL2z5oWVnZjh07du/eXb9+/f79+1ffIScn5/vvv694+cADDzRt2vS2P662yIzjXtziHNei6pcPbcfeFz98JrD3E0yhlKQwAAB/5uaKcOHChS1btkxPT587d67JZLqNg37wwQf/+te//vvf/37zzTdudzhx4sSUKVOKLrPZbLfxKbXO3ZHsopmOllS9KJQFhSsaNTXvWi9JVQAAfs5NEP72228LFizQ6XQjRoyIiooaNWrUrl23NnneO++8s2vXriFDhlxnn8jIyEmX3cnVZy3CMXqkEVtYvXXUNcoMuswAAEjBTRAqlcoBAwasXr364MGDY8aMWbJkSatWrdq0aTN79myj0eipDy4sLHz77bc//fTTY8eOeeqYNd+1+o6qUtoJpYX2s8d9XxIAgJ9joujmAqUyi8Uybdq0CRMmOByOoKCgp5566qWXXoqIiLjhoT/66KM///wzKyur+qZt27ZNnz49KSnp2LFjCxcu/Omnn3r37u32IIMGDdqzZ09SUpLrpUKh+PDDD0NCQm746TWKwWDQ6XREJIiUtEz+v3sdSQFVf+zmPxYJJQXaB0dJUaAXVZy7H8K5S12FNHDuUldxFYVCwfNuesNUdr3NTqdz5cqVs2fPXrFihU6ne+KJJ4ho5syZ33777ZYtW2JjY2+7sjZt2lTcPkxNTZ0wYcK1gjAkJCQ1NfXhhx92veQ4LjQ0VKFQ3PZHS8JmsymV5R1h+seJy8/JXwtnVfbh03vnfzhG/sBITqXxeYFeVPnc/Q3OXeoqpIFzl7qKq3DcjR+OcB+EZ8+enTNnzpw5c3Jyctq0aTNr1qyBAwdqNBoiGj9+fOvWrRcsWPDSSy95pMq2bdu+995719oaGBhYv379Rx991COfJRWZTCaTlU9J+Gi8OHqT8827qs5QKAsOUya2tO1ar+3Yx+cFelHlc/c3OHepq5AGzl3qKm6Zm6h87rnnGjVq9MEHH/To0WPr1q1///338OHDXSlIRBERES1atCgsLLzVTxIEISsrq6SkhIhKS0sr1i9cuNBPOsu4pNdjpTY6UOy2y0wfw6b/+b4kAAB/5uaK0GAwTJky5YknnggKCnL7njlz5ly/cXLFihVvvPHGxYsXy8rK2rRp8+CDD77++ut2u/2BBx7Yvn1769atX3zxxe3bt8fFxR07dsxgMCxbtswzZ1MbMKJ+cWzBCeGt1lW/NymbtBDtNtupQ4pGyZLUBgDgh9wE4dy5c6//nsjIyOvvcM899yxYsKDipV6vJyKFQrF///74+Hgi+vzzz3fu3Hnx4sV69eq1atWq1t3zu0MD4rgn1zvfal1tA2PajN7G7F8QhAAAPuP+HqEgCCtWrNi6deupU6dCQ0OTk5P79+8fGhp6kwfV6XTVOw4xxlJSUlzLCoWiffv2t110bdc+glmctK9ITA2u2mVG2/6+3PeGB5rKOI1ektoAAPyNmyAsKip68MEHN2zYoFAoIiMji4uLS0tLx48f/9NPP3Xv3t33JdY9jGhAHFt4Qkit1mWG0+hUKe1MW1frujwiSW0AAP7GTWeZsWPH7tixY968eQaD4fTp0yUlJTt37kxKSsrMzCwuLvZ9iXXSgDjupxPun+AsH2XmRs93AgCAR7gJwmXLlr399ttDhw6Vy+WuNS1btly2bFlJScn69RgP0zPahjOHSLsL3aSdIi6FKZTWY3t8XxUAgB9yE4SMsYqRXCqEh4fXuvFcarj+cWyhu+HWiEib3suIoUcBAHzCTRD27dt30aJFVVb++eefVqu1U6dOPqnKL2TGcT9eo3VU06qL5eA20Wb1cUkAAH7ITWeZ0aNHP/bYYw8++ODQoUNjY2NdLaJffPHFhAkTTp48efLkSSKKjIyMjo72ebV1SuswxjHaWSC2Cq3ad5TT6BQNkyyHt6vTMiSpDQDAf7gJwldeeSUnJycnJ2f58uWV148bN65iefz48dcZFw1u0oA4tuCE0CrUzYhEqrR0897NCEIAAG9zE4SffvqpayC062jUqJFXyvEzA+K4R9Y4329LVS8JidRpGaW/fkuCk7jaN3AfAEAt4iYI27Zt6/s6/FPLUKaU0fZ8sU1Y1SiUBYbyoZHW4/uUTfxoIFYAAN+75vwUFotlz549y5Yt27RpU35+vi9r8ivX6TuqTssw7832cT0AAP7GfRDOmDEjKiqqRYsWDz30UKdOnerVq5eZmVlUVOTj4vyBq++o286jqrQM897NeLIeAMCr3AThvHnznn322datW8+fP3/jxo1ZWVnPP//8L7/8kpmZ6fv66ry0EKbj6e88N2knj2zA5Ar7ueO+rwoAwH+4uUc4ZcqU/v37L1iwgLHyG1d9+/a99957H3jggd27d/vV3IG+MSCeLTghtAt30ylGnZZu3rtZHtPY91UBAPgJN1eER44cGTRoUEUKuvTt2zcwMPDw4cO+KsyPDIznFlyjdVSdlmHes8nXBQEA+BM3QRgQEJCTk1NlZXFxscFgCAgI8ElV/iU5iAUo6K9L7sYdbZgsGEsdeed8XxUAgJ9wE4S9evV666231q5dW7HmwoULgwcP1ul0HSNtem8AACAASURBVDt29GFtfiQznnPfd5QxVWoH876/fF4RAIC/cBOEkyZNCg8P7969e1xcXOfOnZs3bx4XF7d27drZs2e75poHj3s0ni08KQrumkfVaemWvZt9XhEAgL9wE4SRkZHbt2//+OOPO3ToYLfbGzRo8Nxzz+3atQu9Rr0nKZCFKCn7opskVCa2sueedpbi2RUAAK9w02v08ccff+SRR8aOHev7avzZgDhu4UmhU2TVvqNMxquSW1sObNV2uE+SwgAA6jY3V4RLly41m82+L8XPPZbAFp4UnO5aR1Wp6RhiBgDAS9wEYfv27ffswfTovpYQwCLVbJO71lF1s/a24/tEK76dAAB4npsgnDp16vz58+fOnWuxWHxfkD8bEM8tcNd3lCnVirimlkPbfF8SAECd5yYIX3nllfz8/OHDh2s0mpCrnTp1yucV+pFH49nia7SOqtMyzHvQdxQAwPPcdJZ58MEHW7Vq5XZvPFDvVfF6Fq1l63PFrlFVZ2VSpaWX/O+/otPBZG7+yAAA4La5+a369NNP+74OcHE9Wd81qmrfUZk+mI+ItR7bo0pqLUlhAAB1lZum0cmTJ2/ZsqXKytLS0lGjRmFiQm/LjGOLTgoOd4PMqNMy8GQ9AIDHuQnCZcuWHT16tMpKs9k8e/bs0tJSn1TlvxrpWZye/Znrru9oi47mvdmYnhAAwLOuOUN9FXl5eUQUFBTkzWKA6Np9R/mw+pxaa8s54vuSAADqsCv3CA8cODBv3jwiOn369E8//bRv376KTQaD4ffff4+LiwsJCZGgRj/zaDy7a6lzRoaMr/YtxTVnvaJBkhR1AQDUTVeC8PTp07NnzyaisrKy33//fdOmK9PgBQYGJiYmTpw4UYIC/U+sliUEsHUXxJ7RVfuOqtMyir7/KLDPMCnqAgCom64EYa9evQoLC4koIyPj6aefHjJkiHRV+bvMeG7BCaFndNW+o4rYJoLF7Lh0lo+IkaQwAIC6x809wuzsbKSgtDLj2NJTgtFRbQNj6tQOGHcUAMCDrvl0dmlpaU5OTpVR1lJTU5VKpfer8nfRWtYpkvvphDA8seo3FVVaeumv3+q7YUosAADPcBOEFy5cGDNmzPLly8VqPfWPHz8eHx/vk8L83VPJ3Ns7nNWDUNm4uSPvnLO0UBaAjksAAB7gJgjHjBmzdu3at99+u1WrVlWu/6KionxVmL/rFcOe2UQ7C8RWoVd1mWEyXpXcxrLvL21Gb6lqAwCoS9wE4aZNm956662XXnrJ99VABY7RiCTuy0PC5x2rdplRN083/vUbghAAwCPcdJZhjMXGxvq+FKhiZBL34wmhzF51vSq5je3kAcFskKIoAIC6xk0QDhw4MCsry/elQBVRGronkvup2igzTKlWJKRZDmJ6QgAAD3DTNDps2LChQ4cOHTq0X79+VS4N0WvUx0Ylc2/ucI5Mqvp9xTUAt6Z1FymKAgCoU9wE4bPPPnvgwIEDBw589913VTah16iP3RfDns6mHfli67Crusyo09JLln0p2m1MrpCqNgCAusFNEH766aclJSVu90avUR/jGI1I5GYfEmZ2uqrLDKcNkEc1sh7brWraVqraAADqBjdB2LYtfrfWICOTuZRF9g/aywLkV61XpaWb925GEAIA3KFrTsOUk5OzYsWK2bNnO51OIjIYDAYDuilKIFJNXaK4H49X7TKjbtHJsnczpicEALhDboLQ4XCMGjWqUaNGffr0GTVqlMPhIKJp06a1atXK5+UBEdGoZO6Lg1WDkA+px+mDbKcPSVISAECd4SYI33vvvW+//XbGjBnZ2VcGdx46dOixY8cOHcKvXQn0jGFldtqeX/XiT52WgQG4AQDukJsg/PrrrydMmDB69OiYmCtz/cTGxup0uhMnTviwNijHiIYncrMOVWsdTUs3797k9i0AAHCT3AThhQsX3LaCyuVy3CaUyvAkbtFJofTqUWbkMY1JFOy5ZyQqCgCgLnAThFFRUQcOHKiy8uDBg0VFRXFxcT6pCqqKVNO99bkfqnWZUaV2sKB1FADgDrgJwszMzMmTJ2/ZsoWx8oe4z5w5M3z48MTExNatW/u2PLjiqWTuiwPVWkdT03GbEADgTrh5jvCNN97Izs5OT093Xf9lZGTs27dPo9GsXLlSJqs6EwL4TI9oZnTQ33li2/Aro8woE9IchRedxXmyoHAJawMAqL3cXBFqtdp169Z9+eWXzZs3v+uuuzQazb/+9a/du3e3b9/e9/VBBUY0IombXaXLDMepUtqZ9/0lUVEAALWemytCIpLL5SNGjBgxYoSPq4HrezKRa7rIPqXDVaPMqNPSDRuydJ3+IV1dAAC12DVHloEaqJ6autfn5h+76qJQldzGfuaIYEKHXgCA24EgrGWeSuZmXj3KDJMrlE1aWA5slaokAIBaDUFYy3SLZmYnbbl01Sgz6rR0897NUpUEAFCrIQhrGUY0slqXGVVqB+uRnaLdJlVVAAC1l1eCcNWqVePGjcvMzMzKyrrWPnPmzElPT8/IyPjmm2+8UUMdNjyRW3JKKLJeWcNp9PKYBOuRHdIVBQBQW904CIuKitauXXv69OmbP+jKlSsZY4cOHTp8+LDbHVasWDFhwoSPPvpo8uTJr7766po1a27+4BCmop4x3PdXjzKjTkXrKADA7XAThK+++uqTTz7pWj569GhSUlL37t3j4+O/+OKLmzzo1KlT//Of/zRs2PBaO3z++ecvvPBCx44d77777ueee+7zzz+/jdL92ajkqmNwq9MyzHs3k1B16BkAALg+N0H4888/t2vXzrU8ceJEnudXrVr1yiuvvPrqq0VFRR751L1797ZtWz61etu2bffs2eORw/qPrvWZ1Ul/VeoyIwuJkAWHW09VHSQWAACuz80D9RcuXEhISCAiQRBWrFjxwgsv9OjRo2vXrp9//vnOnTvvvffeO//UvLy8wMBA13JwcPClS5eutefevXs3btz45Zdful7KZLKVK1dGRkbeeQ2+ZDQaKwZu9ZTH4/jP9jpS2zsq1siS2pRt/9Me0cizH3SHvHHutQXOXeoqpIFzl7qKq6hUKp53P3RMBTeb5XK5zWYjouzs7KKiou7duxMRz/MBAQH5+fkeqSwgIMBoNLqWy8rKgoODr7VnSkpKYmLi008/7XqpUqnq16/vkRp8SRRFnU7n2WOOSqUmC+x2uSpYWb5GeVeXgq/e0g14xrMfdIe8ce61Bc5d6iqkgXOXuopb5qZptFmzZvPmzSsrK5s5c2ZYWJirDdNqtebm5oaHe2Zk5/j4+CNHjriWjxw5cp3ZnWQyWUhISPxltTEFvSRMRffHct9VGmVGXj+OZLz9/EkJqwIAqHXcBOEbb7yRlZUVEBAwf/78cePGuS4qV65cKYpiixYtbuagJpOpqKjIbrebzeaioiLX9eXRo0fffPNN1w5DhgyZOXOm2Ww2mUyzZ88eMmSI587Ij4xK5r64epQZdbP2mJUJAOCWuAnCHj167N279+uvv964ceNLL73kWqlUKmfMmBESEnIzBx03blxCQsLWrVs//vjjhISEn3/+mYjOnj1b8cjgP//5z6ZNm0ZHR0dHR7ds2XLYsGGeORs/0yWKEVH2xStdZlTNMxCEAAC3hImieOO9vMNkMjHG1Gr1dfaZMGGCVqsdP368z6ryhrKyMr1e740jT9kr7CkUv+l8eZ5IUbzw5qDwf33Mh9aU/kTeO/eaD+cudRXSwLlLXcUtc3NFuGvXroon3B0Ox+uvv96+ffuhQ4deuHDBs5+t0Wiun4JwQ8MSueWnhcKKUWYYU6W0s2B6QgCAm+YmCMeOHbty5UrX8hdffPHuu+/qdLrff//9gQce8G1tcGOhSuody3179MqdQnVahnkfhpgBALhZboLwwIED6enpruW5c+f2799/7dq1Gzdu3LFjx65du3xbHtzYqKbcrENCRQO3Mqm1/exxwVgqZU0AALWHmyCseLDv4sWLu3bt6tevHxE1atQoKirqxIkTvi4QbuSeSMaINuWWRyHj5crEVub9W6StCgCgtnAThJGRkYcOHSKiRYsWMca6du3qWl9aWiqXy31aHdyckVcPPapOS7dgAG4AgJvjZmSZfv36vfbaawcPHly0aFG3bt3q1atHRMeOHSsrK4uPj/d5hXBjTyZyb++w51tkYSoiIlWzdsWLZog2C1OopC4NAKCmc3NF+Pbbbz/22GNr1qxp06bN7NmzXSsXLlwYHx+fkpLi2/LgpgQpqG+DK6PMcGqdomGS5TCmJwQAuDE3QajRaGbMmHHw4MGsrKxGjRq5Vv7f//3f8ePHa9poqlDhqeSrusyo0jA9IQDATbnmxLyiKJ49e3bDhg0HDx60Wq3X2g1qiLsjmYzRhstdZtRpGZb9W0hwSlsVAEDN5z4If/jhhwYNGsTGxt5zzz0pKSmBgYFjxoypmC8CaqZ/JnGzL3eZkQWG8qGR1uP7pC0JAKDmc9NZZsmSJYMHD27VqtW///3vhg0bFhUVrV27ds6cObm5uUuXLvV9iXCTnkjkJu6051lk4Sqi8jnrs5VNbmqcdAAAv+UmCN9///37778/KytLJisfwfLxxx+/7777Bg8efPDgwaZNm/q2QrhZQQr6RwPu26PCi2kcEanSMvJnvRb08GjCnV0AgGtzP7LMiBEjKlLQZeDAgQEBAfv2oamtRnsqmZt9ucuMPLIBkyvs545LXBMAQM3mvtdoXl5elZUmk8lkMmk0Gp9UBbepYz0m5+jPCxVdZtB3FADgBtzPRzhx4sQdO648hVZaWjpy5EilUpmRkeHD2uB2uC4KXcvqtAzznk3S1gMAUMO5uUc4adKkTp06tWnTplWrVg0aNCgqKtq5c6fBYJg1a5ZrDFKoyR5vwr25w37JLItQk6JhsmAsdeSd48Ojpa4LAKCGcnNF2LBhw127dk2YMEGpVO7Zsyc/P/+hhx7Kzs4eOXKk7+uDWxWooAcacPNco8wwpkrtYMb0hAAA1+bmivD1119PT09/55133nnnHd8XBHduVFNu6B/Ol9I4RqROSy9b/ZO+az+piwIAqKHcXBF+/PHHJSUlvi8FPCU9gunl9McFkYiUia3suaedpUVSFwUAUEO5CcLWrVsfPnzY96WAB41I4mYdFIiIyXhVcmvLAUxPCADgnpsg/OCDD2bPnr1s2TJRFKtvhVphaGNu1TnhkpmISJWKhygAAK7JzT3CWbNmMcYeeughnU7XqFEjpVJZsWnZsmXR0eh/WAsEKuihhtzXR4VXm3PqZu2LF0wXrWamVEtdFwBAjeMmCKOjozt27Oh+b97N/lAzjUrmBv/hfDmN45RqRVxTy6Ft6hZ3S10UAECN4ybY3n33Xd/XAR7XPoIFyOn3C2K3+kydlmHesxlBCABQ3TXnI4Q6YOTlLjOqtHTLwa2i0yF1RQAANY6bIHzzzTcHDRpUZeXixYtbt27tdGKi19pkaBNuzXnhgolk+mA+ItZ6bI/UFQEA1DhugvCnn36qPqZoz549d+/evW3bNp9UBZ6hl9Mjjbh5RwVyzVmPvqMAANW4CcIzZ87Ex8dXWanX68PCwnJycnxSFXiMawxuQSR1i47mvdmER2IAAK7mJgi1Wu2ZM2eqrDQYDPn5+Wo1+t/XMu3CWZCS1p4X+bD6nFpryzkidUUAADWLmyDs0qXLRx99VFR01aBcb775plwu79Chg68KA4/5Z1L5xEyqtAw8WQ8AUIWbxycmTpzYoUOHxMTEoUOHJiQkFBQUrFixYsuWLe+8805oaKjvS4Q7NKQxN2Gb/byJC0vLKPr+o8A+w6SuCACgBnEThCkpKRs3bnzxxRenTZvm6iYaHx8/e/bsf/7znz4vDzxAJ6d+cdw3R8X/a95EsJgdl87yETFSFwUAUFO4HymmefPma9asMZlMubm5Op0uIiLCx2WBZz2VzGWudf67Oa9O7WDem63vlil1RQAANcX1HqjXaDTx8fFIwTqgTRgLUdLqc6IqDQNwAwBcBSPL+AvXcxTKxs0deeecpYVSlwMAUFMgCP3FoATujwvCBatM1bSNZd9fUpcDAFBTIAj9hU5O/eO4uUdEdVq6eW+21OUAANQUCEI/8nQKN/uQIE9qYzt5QDAbpC4HAKBGQBD6kRYhLEJFq/NUioQ0y0EMGwsAQIQg9DeuLjMYgBsAoAKC0L88lsCtzxWKEzpYDm0X7TapywEAkB6C0L/o5JQZz/33rF4e1ch6bLfU5QAASA9B6HfGNOW+PCQoUvFkPQAAEYLQDzUPYVEa2lqvo2XvZkxPCACAIPRHTyVzn50P5/RBtlMHpK4FAEBiCEJ/9FgCtylXsHZ4qPD7qc7Sohu/AQCg7kIQ+iMNT48mcHN1PTTtuud/8X+CsVTqigAAJIMg9FOjkrk5hwVt98dUKe3yZ70uWs1SVwQAIA0EoZ9qHsKitfTuTkHZ60l5bOP8r97CY4UA4J8QhP7rm86yXQVi44XO+U2fJn1IwZyJotMhdVEAAL6GIPRfiYFsaQ/Zyvtl6y9SS27sEYsq75tJJAhS1wUA4FMIQn+XGswWdJP91kcxPfXVP0+b1n8+1WzHw4UA4EcQhEBElBrMfuiuTHnmDVZ4/rOPvpi8WzCjlRQA/AOCEK5oFqHq+Oo7mdz+4M0/xP1kRxwCgD9AEMJVOJU29pn3Hij6Y0PI8u35IuIQAOo8BCFUxemCwsf8R78ta652zZre/P4iMR5xCAB1lxeD0Gw222x4NK1WkgWFhY15v/TXeQmnNszrIluNOASAussrQWixWB599NGoqKiIiIixY8eK1aY42LRpU0glS5Ys8UYZcCf4sPpho98rXvKF5eDfqcEMcQgAdZVXgnDatGnnzp27ePHiiRMnfvnll6VLl1bZweFwREVFHb+sb9++3igD7pA8qlHoyDcL50+xHt9LRIhDAKiTvBKE8+bNe/7555VKZUhIyPDhw+fNm1d9H5lMFnyZQqHwRhlw5xQNk0OfGFcw9z1bzlHXGsQhANQxXgnCEydOJCcnu5aTk5OPHz9efZ/Dhw9rNJr69es/88wzBoPhWodyOp2FhYUnLjt//rw3CobrUDZpGTxwbMGXb9pzz1SsRBwCQJ3Be/yIVqvVbDZrNBrXS51OV1xcXGWf5OTknTt3JiUlHTt27PHHH3/55Zdnzpzp9mj79+/ftGnT4sWLXS85jlu5cmVkZKTHy/aq6yR97dCwmbLnkLwvxmtHTOSCI66s5mnGXXSgMfvkIB+3l3s60Tm6iUN99d+pWn/udwDn7p9w7jWKSqWSy+XX38fzQahUKgMCAkpKSlwvi4uLIyIiquxTr169evXqEVFSUtI777wzbNiwawVh8+bN09PTx48f7/E6fUyv10tdwp3p1NtAgmHee+HPfSgLDK28pb2efoih3YXixB2y1ivlrzbnnkrmVLIrO9T6c78DOHf/hHO/IbtABjuZHKJVoGIr2QQy2MnoEG3VXhZZy3euePlMCvdkoiebMz0fhESUmpq6ffv2u+66i4i2bduWmpp6nZ1NJpNSqfRGGeBZuk59RYsxf+aE8Gc/4LQBVba2CGFLust2F4oTdwgf7HG82pz7ZxKn9srfLwCQjNlBFicV20SLk0wOKrWTxUEGh1hmJ6uT8sp4p8xpFajERjYnlVWKOrtw1UsZR3o5aXim5ChISfKrXyo40slJwzMtT3H6ipeca2vjAObZk/LKL6oxY8a88cYb7dq1Ky4u/uqrr7KysojIbrd36dJl7ty5iYmJP/30U1BQUFxc3NGjR1955ZXBgwd7owzwOH33RwWLKX/Wa+HPTGZKdfUdqsThK2ncY9Hkv9+NAWqSitAy2MnipFIbmRyixUnFtmvGW8WeRododVKxjVQyUvMUpGBKGWl5CpCTUkZ6OdPJSSUjFVEgz4IV1EhHChnp5aSWcSoZBSrKX7qiLlBBnIez7I54JQiHDBmSl5c3fPhwpVL52WefZWRkEBFjLCQkhOd5IhIEYdKkSRcvXoyMjHz++efHjBnjjTLAGwL7Plm81JY3c0L4mPeZQuV2n8px+O5OZdsIR1owSwthqcEsJZgpMJwRwK1zilRqozK7aHKQ0UHFNjI5RLODim1kdJDJQWV2sdRGJgeZHFRsE10LJTYyOESTgwx20stJJSO9nGnl5AonVywFK8vjLVDBIq6KN04nJ+XVe15fWZlDr3fzFbmGY9Wfdq9RJkyYoNVqa/s9wrKysjp1z0AUi378xFmSHzryLcbf4C70wYtlx6zafUW0p1DcVygeLxPjdCw1hDUPYanBlBbC4vSsJn019KS69ud+K3DubjcV28hgF40OMtivWi6ykskhui7FyuxkdpDBIZZUSjWjnZwiBShIxzMNTzq5K5xIw7Og8gUKkDO9nDQ8aeUUpGBqGWl4ClKSlmcanvQ3+Jfq9XOvyXAPB24dY8GP/qvgm/8UzpsUOmwCcde7xIvRUNN63D8alL+0CXSoWNxXJO4pFL88JO4rogKr2CzYlYssNZg1D2Fh7q8zAWqEEhsZHaLBTmX2qsuuYCs0yY2i02gXjQ4qs1OxjYx20eAgg50CFaSTMy1PejkFKUgrZzqedHIKUpCGZxFqahLouhlGOp4LuJxwwQqm4Ukpu3FtcHsQhHBbOC7k8X8XfPVW4Y8fhzz2It30RZ2Co+YhrHkIG5RQvqbERvuKxL2F4p5CcdFJYV+RqOQoLYQ1D2GugEwJYuh0A57lug4zOsQiq2uBDHax2EZll5dLbFRqJ6OdDHax1E6ldjLYRaOdSu0UICetnHRyFiCnADnp5Ewrp4rlSA01UAr1ApiW53Ry0sspUEFanrmWoWbCLxi4TUzGhz75ev7M8cVLZwY9cvt3eQMV1LEe61jvSpTmGMV9hbS3SFx3Xpy2XzhcIsZqWVoISwtmqcHUPITFBzBZXW1OhZtmdbpulYkGOxnsZHBQiU0scy3bqdQultrK15fZxRIbXd5NLLaSmiednHQ8C1KS65pMy7NgpWslhalY4wDSy13ruUBFefhpeRZ4E6NglZU59XrcCa9NEIRw+5hCGfrU23mf/bv0t+8D7hvkqcPGalmslnrFlmedQ6AjpeK+QnFPofjtMdpbKFw0i8lB5a2paSEsLYRF1r7b80CldjI7yOgQi61kcpDJSSU20Wh39fugUrt4+Q5Z+f0zV5iZneW9QmSMtHIKVjCdnLTlDYxM77pc4ylYwRrqytcHyLkARXnI6eWspnVZBMkhCOGOcCpt+Oj3Lk1/mSmU+q79vPERPEcpQSwliGXGl68x2Gl/kbinUNxXJP6SI+wpFM0OClKyIAWV/6dkgeXLLEhBQUoKVrAg5ZU1PL6v3xmH4GpFLO9Sb3GS2UElNtHiJNeNsRITb+OcZXYyOchoL+/EaHZSsZWMDtEVdeU9O3h2ud8HBSlYeb8PBenlLExFWp6CFKTmmatXiJYnDU96OdPL8YcIHoMghDvF6QLDx/wnb/rLnFqr7XC/Dz5RJ6f2Eax9xJVv9a6edcU2KrZSsY2KbWKJjYptlG8Rj5VSkZWKbUKxjYptVGwVS2wk56oGp2shsK4HZ5mdHALZBDJefgLa9TCZVaAi65U8swrlDYlWgUps4uWHzMjiJJNDLLUREQW463yv5EgnJ52cOCeFqVisli53ceQ0lzsxanim4SkAN8ygxkAQgge4JvLNm/4KU6o1rTr7vgBXP/L6mooVN2j5qh6crgW3wVlscz1KxZSX47By25qcI12l3+k6nskv7yY4FaEaZ8UmNU+VR54LUlzpYqSQkbbSv0VXXBGRxUlmp0hEgkglNg9sdV1IyTnS8az6I2JKWXkv/FAZ6eWk5UkpoyAF56o8SEFKGWn5m7oaq6XPk4F/QhCCZ7gm8s3//P84pUaV0lbqcm7gVoPTeLndz6XYShWP39oEMtqv7FlmFx2XtxmMDnul5yxNDrJeiUUqsl15hNfqJFOl6TsqkkbJUbCCERHHqHGAm60annNtrejEcfVWVmUrAFSHIASPcU3km//lW6FPTlAmpEldjidpedLylcLyek8MX9mtrExA70GAmg//SsGTKk3ke0TqWgAAbgqCEDzM7US+AAA1FoIQPE+d2iHwwX/mz5zgKMiVuhYAgBtAEIJXaO66V9/j0fwv/k8oLZS6FgCA60EQgrfoOvbVpvcyfv22eU821exJTgDAn6HXKHiRvlumXRdS9vuikmVf6u55QNP+Pk6lufHbAAB8CEEI3iVPaR/Svrv97LGyP5eW/va9uuXd+s4P8/Vipa4LAKAcghB8QR7TOGTwK87SImP2L5emv6xomKzv/JAysZXUdQEAIAjBh2QBwQH3D9F3f9S888/iJTNJxus69tG07c7kGPgEACSDIARfY7xc07a7pm1364n9ZWsXlK78VtO2h+6eB2WBoVKXBgD+CEEIklHGN1PGT3TknTNsWH5x8ihVs/b6rv3l9eOkrgsA/AuCECTGh0cHPTImoPfjpi2r82e/IQuO0Hd+SN28I3F4tgcAfAFBCDUCp9LqOj+k7dTXsndz2Z9LS/73X93dD2o73MeUmMoHALwLQQg1CJPx6pZ3q1vebcs5alj/c+nE+Zo29+q79JOFREhdGgDUWQhCqIkUsU1CBr/iLCkwbv714pRnFY2a4nELAPASBCHUXLLA0ID7h+i79jNt/71o8RecQqm750F16y5Mhr+3AOAx+IUCNR1TqrUZvbXpvaxHd5X9+XNJ1n+1Gb11dz/AaQNu/GYAgBtBEEItwZgysZUysZXj0lnDxqzc94armrXXd8uURzaUujIAqN0QhFDL8BExQY+MCej5mHHLqvwvxstCIgO6D1CltCfGpC4NAGolBCHUSpwuSN8tU9f5YdOOP0pWfFu0+HNV0l2qpNbKJi3QZAoAtwRBCLUY4+Xadj207Xo4Ci5YD+807VpftGAaHxqpTGylSmqliE9lvFzqmU73GgAAFs9JREFUGgGgpkMQQl3Ah0bxGVHajN4kCLZzx61HdpauWWA/8668QaIqqZUysZUipjHaTgHALQQh1C0cp4htoohtou+WKVrNttOHLId3Fi+Y5ijKUzZOUyW2UqW0kwWFSV0lANQgCEKos5hS7epoSkSutlPLkZ0l/5vLqXXKpFaqxFaqpm0whBsAIAjBL1RvOzVkryj8foo8OkGd2h5tpwD+DEEIfqZy26nNajt14HLb6SVl4+aqxFaq5DYY2hTAryAIwX8xhbKi7dRZVmQ7vs9yZGfpb/MZr3C1nSqTW3MqrdRlAoB3IQgBiIhk+mDXxBd0+Yai8e81hT9MlUfElD+MkZCGMU4B6iT8wwaoquKGomi3WY/vtR7eUfzzbGdxvqJBojymsSKmsTymMR8aiXuKAHUDghDgmphcoUq+S5V8VyCRYCi2nTlqO3fMtOMP+/KvBJNBHpNQkYvyiBjiOKnrBYDbgSAEuCmcLkiV0laV0tb1UrCY7OdP2nOOWg5vL1vzo6Mglw+NlMc2UcQ0kcc2UcQ2YXKFtAUDwE1CEALcDk6lUcY3U8Y3c70UrWb7pbOO3NO2nKOmXRvs50/wweEU2YjFpchjmyhiEphCJW3BAHAtCEIAD2BKteupDE3b7kQkOh2OvHNlR/c68nJcuSjTBfGRDRSxTRSxjRWNUjAyOEDNgSAE8Dwm4+WRDeXaEL1eT0QkOO2Xztpzjtpyjpb9ucw+fwrj5a4WVEVsY3mDJJk+WOqSAfwXghDA+ziZPLKhPLKh63qRRNFx6azt3HH72WNlf/5sP3uMKTWKmAR5TGN5/Th5RKwsLAqPagD4DP6xAfgcY3y9WL5eLLXu4lrhKMi1nz1mP3vc+Ncqx6UcZ3GeLDCUD4/hI6L58Bh5RDQfFi0LDscDGwDegCAEkB4fGsmHRqpbdCp/LTgdRZecBbmO/AuOglzLwb8duWecpQWygFA+soE8siEfGsmHRclCI/nQKEkLB6gLEIQANQ8n40Oj+NAo1/BvLqLT4SzOc+SetueesZ09Ztq1wZF7WnTY+NDyRJRHNuAjG8ojYjClBsAtQRAC1A5MxrvSUdWsQ8VKwWRwFFxwFuQ6Ci5Yjux0rF9mv3SWyWR8aNTla8coWWikPLIhnmsEuBYEIUAtxml0Ck0Tim1yZZUoOovzHHnn7HnnHJfOWY/tdeSdc5bky4LC+YgYeUQMHx4tC46QBYfLgsIwpDgAIQgB6hrGZMERsuCIqs2qBbn2S2cdl87aco4692xyFuc5i/NFQZAFR8gCQ2SBYXxwOBcYygeFyQLDucAQPNEB/gNBCFD3MRnPR8TwETFV1ot2m7O0wFlSKJQWOgouOC7mWI/udr10lhZwap0sMFQWGikLCJEFhpb/PzSSD44gTibJiQB4A4IQwH8xucJ137H6JtHpEAwlztJCZ0Gus6TAWVpoyzlaHplFlziVmg+N4ioCMiCED4viAkL44Ah01YFaB0EIAG4wGS8LDJUFhl51A9JFcDpLi5xFl5ylBc7ifGdRnvVSjmnbWmdxvrOkgNPquYBQURvkCArhtAGcNpDTBXCaAE4bwGn1Mm0ApwnATB1QoyAIAeAWcTJZUJgsKMzNJlF0lhU7S/INuWcVgl0wlgrGUkfeWcFYJhhLnMYywVgqmEo5lZbTBlT8J9PoOW0ApwvktIGcVs9pXcGpx/A64Bv4ewYAnsOYLCBYFhAsD4rUusZZdUcwGQRjiWAsE0ylgqHEaSoTjKWOnEuCoUQwlQquvDSWMrmiWjoGcFq9TBfIaQI4XSCn1DCVhlNrMeYO3AkEIQD4GqfRcRodhd9gN8FiEowlrlC8nI4ljvOnrBUrzUbRahLMRiZXMIWaU6o5jY4pVa5lptZyKg1TqJhKwylUnEbPlCqmVHMKNVNrOJWWKVR4vBLIe0GYl5e3ffv26OjotLQ0tzvYbLbs7GzGWEZGhlwu91IZAFB7cSoNp9LQTQwjJ9osgtUsWi2C2SBazaLVLFjNosUkWEyi1ewszndYzZU2WUSLSbAYRatZFAROpebUOqZQMeXl+FRqyiNTpeFUWuJ5TqkhjuNUGmKMqbSMMabWEhGn1hMRLklrO68E4dq1awcOHNixY8ddu3b17dv3s88+q7JDQUHBPffcExISIopiWVnZH3/8ERyMh5YA4DYxhUqmUNE122KvSXQ6RKtFMJeJVkuV+BRsZsFkcBTkktMpWE0kCILFRKIoWoyiKIpmIxEJ5jIiEsxGEkVOpSWOMZVGFJlJo2WcjClVxPGcUkUynilUjJczuYLxCiZXMLmC8XImVxIv5xQqYoxTuwY3KI9YIuLUOmJERJxKS4wjIqZU4b6pN3jlZ/ryyy//5z//GTly5KVLl5KSkkaNGlXlunD69OkJCQnLly8XRbFPnz6ff/75hAkTvFEJAMB1MBnPXO20d0ywGEkQBYvRaCjTyJgoCKLVTIJTsJrJ6RRsFnLYRbtVdNhFu00wGUSHTbTbRIddtFnKU5aIqDxiiUgwG0i8fGRRICLBYibBSURMoWI8T0RMrmS8goiIl3MKJRG5QpeImEzmepSFMa4iXF1JXFHzVdeynIyr9OgLU6qY7Epb3VU/IleEV2xSaSq6AQtGoyCrenHsCv5b/4n6jueD8OTJk3v37h04cCARRURE9OjRY+nSpVWCcMmSJa7kY4wNGjTok08+QRACQK3mGq+O0+g4uUZ+7Y5CHiHaLKLDQUSi3So6bEQk2u2i3UpE5HSINgsRiU6naDUTkSg4xfKUJdFuc+3v4rqWLX/hdDhslisfYbWITvuVPU2GKx/vuPxZrk0WEwlC+bLw/+3de1BU5f8H8Gf3sMtF4rIY17hfDZRYEAw2QRABF4OMLjM04x9iOoxp2RSEoQ5dTWayGYJwkATLMYaoJONSQNCOIBtCRMbdhlDu7LLAwF7Y8/3j6PntdxH19/txeIDzef3hnPOcR3g/nnWefc55znm0s8pZ/bRqFalW6RXS410ax4Cv27/eLTTewNG77GzAMxXtMQncgZbP8neEg4ODAoHA1PTu1wcnJ6fBwcHFdZycnKhtZ2fnxRVoCoWiv7+/pKSE2uVyufHx8Xz+qv5ysZhWq9Xe+6CwDbQddwo8oO3M/g4DPjLgI4Q4Riar6ubk9PT0Y4/2JYAe79JIjUq3f71bODdLkv9dT6M2sH7i0f+FORyOfle6yPJ3hEql0sDgf34sj8ebn5/Xq6NSqegJMvetQBsfH//zzz9VqrvfJgwNDUNCQqysrJY7NbOUSiVrJwRB23GnwAPajjsFHv+LtnMM9EaEiG+A+Cb61TbcZ/qICiG0dK+hh8/n63ZJ97X8HaGdnd3k5KRWq+VyuQihsbExOzv9SV+2trbj4+PU9tjYmL29/VI/zc3NbfPmzRkZGcuecyUtLCyYmCw6wewAbcedAg9oO+4UeKzRti//i448PT3Nzc2vXbuGENJqtfX19SKRSK9OWFhYbW0ttV1XVxcWFrbsMVaV8+fP446ATWFhIe4I2EDb2Ym1bSdJco22naN/+XU5fPTRR8XFxenp6TU1NW1tba2trQRBVFdXJyUlKRQKhFBra2t4ePjJkydJknzvvfckEslSjxseP358w4YNa3pESJIkQRDsvF+ysLBgaGio0WhwB8FAo9EYGxur1eqHV1131Gr1hg0b6DsarKJSqR577DGlUv9eFxsolUpzc/MH3OpatRh5fCI9Pd3FxaWhocHT0/Ps2bMEQSCEvLy8srKyqAoBAQF1dXUXL17kcDj19fVL9YIAAAAA0xgZES4jGBGuaTAihBEh28CIcC2OCGExFAAAAKy22t/W09XVdf369bKyMtxB/l84HE5QUBDuFHiQJMnOtpMkqdVqWdv2hYUF1rZdo9FA21ePvXv3PvSa4mq/NDo2Ntbb27vmnqDXo1AozMzMcKfAA9qOOwUe0HbcKfBYhW13dXUVCAQPrrPaO0IAAACAUXCPEAAAAKtBRwgAAIDVoCMEAADAatARAgAAYLXV/vjEWnfhwgX6sWJ3d/eoqCi8eZgml8t///33/v7+Z5991tbWli6/devWN998gxB6+eWXXVxcsOVj0szMzI0bN7q7u3fs2OHu7k4V9vT01NXV0XUSExOtra0xBWRQe3v7r7/+Ojk5KRQK4+PjufeWaZXJZF999dXExIRYLN66dSvekAzp7e2trq6mFiFPSkqi1l6Qy+X04nEIodDQUD8/P3wZmUK9RHNqasrLy+uFF14wNLy7muDk5OTFixflcvmePXuEQiHekI8CRoTMeuONN+rq6lpaWlpaWm7duoU7DrNIkrSxsUlPT3/ttdf6+vro8p6eHqFQOD4+Pj4+LhQKe3t7MYZkzpYtWw4fPvzmm282NzfThY2NjR988EHLPbOz+muWrgMdHR27d+/u7u4mCCItLe3FF1+kyufm5p5++mmJRMLj8WJjY8vLy/HmZIJCoQgJCWlrayMI4uzZsxEREdQX36Ghoddff50+76Ojo7iTMuLMmTPj4+N8Pj8vL08kElFtn5mZCQ4Obm5u5nK5O3furKqqwh3zEZCASRYWFr29vbhTrBylUkmSpEAgkEgkdGFqauqhQ4eo7YMHDx4+fBhPOIZRbQ8ICLh06RJdWFRUFB8fjy/USpibm1OpVNT2v//+y+Fw/vnnH5IkCwsLg4KCtFotSZIFBQXBwcE4UzJDo9HMzs5S2zMzM2ZmZvX19SRJ3rx508bGBmu0FTU/P29iYiKVSkmSzMvLCw0Npc57bm7uM888gzvdw8GIkHHffvttfn5+W1sb7iAr4b6vPqitrY2Li6O24+LiampqVjbUClnqtQ/Dw8M5OTklJSVyuXyFI60MIyMjei1W6qKokZERQqiuri42NpZaHDwuLq65uXlmZgZjTiYQBEEvv2dgYLCwsEC1HSGkUqny8/OLiooGBgbwBVwhnZ2dXC7X0dERLTrvEolk9b95FTpCZgUEBAwODkql0oiIiJMnT+KOg8fQ0BB9Y8zW1nZoaAhvnpVkYmLi7Ozc39+fn5/v4+PT2dmJOxGzjh07lpycbGNjg/77vFtbW3M4nDt37mBNx6zMzMzAwEDqVihBEEFBQV1dXeXl5b6+vj/88APudEzZt2+fi4uLSCT6+uuvF593W1tbkiSHh4exZnw4mCzDLHr94SNHjgiFwgMHDjzxxBN4I608LpdLL76h0WgMDFj0qUtKSkpKSqK2Dxw4cOLECd05FOvMO++809PTQ3/mdc87tbGOT31ubm5paWlDQwM1EvLy8qqurqYOnTt37ujRowkJCVgDMuXjjz9WKBSVlZUpKSktLS2Ojo56/9/RWjjvMCJcIVu2bLGwsOjv78cdBAMHBwd6KHDnzh07Ozu8eXCJiIjo6enBnYIpp06dunr1alVVlbm5OVVib2+ve94RQroTideT8+fPnz59uqam5r5fc8PDwwcGBtbrolR2dnbe3t5Hjx718fGh5kPpnXeCIFb/TGnoCBmk+9GXSqVTU1Oenp4Y8+AiFovp9UPKysri4+Px5llJup+ByspKHx8fjGGYk52dffny5erq6o0bN9KFYrG4vLycWpGxrKwsMjKSvp22nhQVFWVmZlZVVbm6utKFuue9qqrKzc1tra8csBg1Q4ranpmZ6e/vd3BwQAiJxeIrV65QY8GysrKYmBj6FvKqBS/dZlB5eXlGRkZgYOD8/PxPP/307rvvvv3227hDMWv//v0DAwP19fX+/v4WFhZFRUXU18OQkJBt27YhhJqbm5uamtbloDAjI0MqlV6/ft3Z2dnW1vbMmTNPPfUUNWXUwcHhjz/+GBkZqampoR8xXDdu3LgRGBjo7+//+OOPUyXZ2dn+/v5qtToyMpIkyU2bNpWVlV25ciUsLAxv1GU3MjLi4ODg6elJjwXfeuutXbt2HT9+XCKReHp63r59u6mp6fLlyzExMXijLrvGxsZ9+/Zt27aNy+XW1tYGBgaWlpYSBKFUKsPDww0NDT09Pb/77ruKiorg4GDcYR8COkIGqVQqqVTa19dnZGQUFBTk5uaGOxHjGhsbdR+VCw0NpQYBU1NTlZWVCKHY2Fj60tk609raOjExQe8KhUKBQDA8PCyVSkdHRx0dHbdv305PKVxPqLco6JYEBgZaWloihNRqdVVV1cTERFRU1Lq8Oz4/Py+RSHRLnnzySXt7e4VC0dTUNDg4uHHjRpFI9NBlgNYikiQ7Ojra29tJkvT19Q0ICKAPqVSqyspKuVy+c+dOe3t7jCEfEXSEAAAAWA3uEQIAAGA16AgBAACwGnSEAAAAWA06QgAAAKwGHSEAAABWg44QAAAAq0FHCAC7XLhw4YsvvsCdAoBVBJ4jBIBd4uPjp6amfvvtN9xBAFgtYEQIAACA1Vb76hgArHsDAwPd3d2mpqZCoVD31cwymczY2NjIyKi9vX1sbMzPz49a703XyMjIX3/9xefzhULh4ldaz87Otra2ajQaJyenxW/4Gx4evnnzppOTk7u7O7V4EAAsxczC9wCAhxsbG9u9eze6t2CbnZ1dTU0NfZTH4x07dkwkEhEEQRAEj8c7ffo0fVStVqempnK5XIIgEEJmZmaFhYX0Ua1Wm5WVRXWN1Lv/9+zZQx0Si8UikejUqVPUX0QIRUdHUysJAMBOcGkUADxIkkxISOjo6Kirq5ubmxsZGQkJCXnuued0l3H//PPPg4OD5XK5TCZLSUlJS0urqKigDmVlZeXl5X3yyScKhWJ0dDQ6Onr//v319fXU0ezs7BMnThw6dGh0dFSpVPb09Dz//PP0j21pabl27VpXV5dSqczJyfn555+Li4tXsu0ArC64e2IAWIpawby8vJwukcvlxsbGn376KbXL4/G8vb0XFhaoXZVK5eTkJBaLSZLUarWWlpbUNmV6elogECQmJlI1LS0td+3add/fKxaLTUxMxsfH6RJvb+/k5OTlbh8AawbcIwQAD2reZl9f37lz5+hCCwuLzs5OejcqKorLvXvZhsfjRUREUH9reHhYJpPpLnFnamoqEomkUilC6O+//5bJZImJiUv9ai8vLysrK3rXxcVFdxgKANtARwgAHnK5nMvl/vjjj7qFvr6+uuu3Uav60aysrEZHRxFCY2NjCCG9Ve4EAsHMzAxCaHp6GiGku1i8HlNTU91dAwODubm5/3NDAFjroCMEAA8bGxutVltcXGxnZ7dUndu3b+vuDg4OOjk5IYSoPxcftba2RgjZ2toihAYGBpiIDcD6A5NlAMCDmi+al5f3gDrl5eWzs7PUtkwmq6ys3Lp1K0LIwsLCw8Pj0qVLWq2WOjowMFBfXx8ZGYkQcnd39/b2LigoUKvVzLYBgHUBOkIA8AgICEhNTf3www8PHjxYUVHR0NDw5ZdfJiQkfP/993QdQ0PD2NjYioqKq1evxsbGajSa9PR06tD777/f3t6+d+/eqqqqkpKS6OhoExOTtLQ06uhnn33W29u7ffv2kpISiURSWFh45MgRDI0EYC2AS6MAYJOTk+Pn55ebm1tQUMDlcu3t7aOiojZv3kxXePXVV1Uq1SuvvDI5Oenv719ZWblp0ybq0EsvvUQQRFZWVmxsLI/H27FjR2lpqaurK3U0Jibml19+yczMTE5O1mq1jo6OKSkp1CEPDw/qViLNw8ODHncCwELwrlEA8KPmcNMTRCl8Pj8zMzMzMxMhtLCwQD//rkej0VDP4y/loRUAYDn47wEAfhwO58EvOVuqF0T33krzANALAvBgcI8QAAAAq8FXRQBWqaampgc8WQEAWC5wjxAAAACrwaVRAAAArPYfjOXSV2JD4EQAAAAASUVORK5CYII=", + "image/svg+xml": [ + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ], + "text/html": [ + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "iterated_clf.controls[2] = Patience(4)\n", "fit!(mach, rows=train)\n", @@ -700,33 +2081,31 @@ " label=\"out-of-sample\",\n", ")\n", "plot!(epochs, training_losses, label=\"training\")" - ], - "metadata": {}, - "execution_count": null + ] }, { "cell_type": "markdown", + "metadata": {}, "source": [ "---\n", "\n", "*This notebook was generated using [Literate.jl](https://github.com/fredrikekre/Literate.jl).*" - ], - "metadata": {} + ] } ], - "nbformat_minor": 3, "metadata": { + "kernelspec": { + "display_name": "Julia 1.10.3", + "language": "julia", + "name": "julia-1.10" + }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "1.10.3" - }, - "kernelspec": { - "name": "julia-1.10", - "display_name": "Julia 1.10.3", - "language": "julia" } }, - "nbformat": 4 + "nbformat": 4, + "nbformat_minor": 4 } diff --git a/docs/src/extended_examples/MNIST/notebook.jl b/docs/src/extended_examples/MNIST/notebook.jl index 83b4a4cf..448f50ee 100644 --- a/docs/src/extended_examples/MNIST/notebook.jl +++ b/docs/src/extended_examples/MNIST/notebook.jl @@ -147,11 +147,13 @@ cross_entropy(predicted_labels, labels[test]) # Or to fit and predict, in one line: -evaluate!(mach, - resampling=Holdout(fraction_train=0.5), - measure=cross_entropy, - rows=1:1000, - verbosity=0) +evaluate!( + mach, + resampling=Holdout(fraction_train=0.5), + measure=cross_entropy, + rows=1:1000, + verbosity=0, +) # ## Wrapping the MLJFlux model with iteration controls @@ -205,7 +207,7 @@ update_epochs(epoch) = push!(epochs, epoch) # The controls to apply: save_control = - MLJIteration.skip(Save(joinpath(DIR, "mnist.jls")), predicate=3) + MLJIteration.skip(Save(joinpath(tempdir(), "mnist.jls")), predicate=3) controls=[ Step(2), @@ -249,7 +251,7 @@ plot( ) plot!(epochs, training_losses, label="training") -savefig(joinpath(DIR, "loss.png")) +savefig(joinpath(tempdir(), "loss.png")) # ### Evolution of weights @@ -266,12 +268,12 @@ plot( # **Note.** The higher the number in the plot legend, the deeper the layer we are # **weight-averaging. -savefig(joinpath(DIR, "weights.png")) +savefig(joinpath(tempdir(), "weights.png")) # ### Retrieving a snapshot for a prediction: -mach2 = machine(joinpath(DIR, "mnist3.jls")) +mach2 = machine(joinpath(tempdir(), "mnist3.jls")) predict_mode(mach2, images[501:503]) diff --git a/docs/src/extended_examples/MNIST/notebook.md b/docs/src/extended_examples/MNIST/notebook.md index 4a43bf2d..ec7cef3f 100644 --- a/docs/src/extended_examples/MNIST/notebook.md +++ b/docs/src/extended_examples/MNIST/notebook.md @@ -245,7 +245,7 @@ The controls to apply: ````@example MNIST save_control = - MLJIteration.skip(Save(joinpath(DIR, "mnist.jls")), predicate=3) + MLJIteration.skip(Save(joinpath(tempdir(), "mnist.jls")), predicate=3) controls=[ Step(2), @@ -299,7 +299,7 @@ plot( ) plot!(epochs, training_losses, label="training") -savefig(joinpath(DIR, "loss.png")) +savefig(joinpath(tempdir(), "loss.png")) ```` ### Evolution of weights @@ -320,16 +320,23 @@ plot( **weight-averaging. ````@example MNIST -savefig(joinpath(DIR, "weights.png")) +savefig(joinpath(tempdir(), "weights.png")) ```` ### Retrieving a snapshot for a prediction: -````@example MNIST -mach2 = machine(joinpath(DIR, "mnist3.jls")) +````julia +mach2 = machine(joinpath(tempdir(), "mnist3.jls")) predict_mode(mach2, images[501:503]) ```` +```` +3-element CategoricalArrays.CategoricalArray{Int64,1,UInt32}: + 7 + 9 + 5 +```` + ### Restarting training Mutating `iterated_clf.controls` or `clf.epochs` (which is otherwise diff --git a/docs/src/extended_examples/MNIST/notebook.unexecuted.ipynb b/docs/src/extended_examples/MNIST/notebook.unexecuted.ipynb index 6f955f3e..f2beaabc 100644 --- a/docs/src/extended_examples/MNIST/notebook.unexecuted.ipynb +++ b/docs/src/extended_examples/MNIST/notebook.unexecuted.ipynb @@ -516,7 +516,7 @@ "cell_type": "code", "source": [ "save_control =\n", - " MLJIteration.skip(Save(joinpath(DIR, \"mnist.jls\")), predicate=3)\n", + " MLJIteration.skip(Save(joinpath(tempdir(), \"mnist.jls\")), predicate=3)\n", "\n", "controls=[\n", " Step(2),\n", @@ -607,7 +607,7 @@ ")\n", "plot!(epochs, training_losses, label=\"training\")\n", "\n", - "savefig(joinpath(DIR, \"loss.png\"))" + "savefig(joinpath(tempdir(), \"loss.png\"))" ], "metadata": {}, "execution_count": null @@ -648,7 +648,7 @@ "outputs": [], "cell_type": "code", "source": [ - "savefig(joinpath(DIR, \"weights.png\"))" + "savefig(joinpath(tempdir(), \"weights.png\"))" ], "metadata": {}, "execution_count": null @@ -664,7 +664,7 @@ "outputs": [], "cell_type": "code", "source": [ - "mach2 = machine(joinpath(DIR, \"mnist3.jls\"))\n", + "mach2 = machine(joinpath(tempdir(), \"mnist3.jls\"))\n", "predict_mode(mach2, images[501:503])" ], "metadata": {}, diff --git a/docs/src/extended_examples/spam_detection/Project.toml b/docs/src/extended_examples/spam_detection/Project.toml index 7d4e772f..da092aaa 100644 --- a/docs/src/extended_examples/spam_detection/Project.toml +++ b/docs/src/extended_examples/spam_detection/Project.toml @@ -8,6 +8,5 @@ MLJ = "add582a8-e3ab-11e8-2d5e-e98b27df1bc7" MLJFlux = "094fc8d1-fd35-5302-93ea-dabda2abf845" MLJText = "5e27fcf9-6bac-46ba-8580-b5712f3d6387" Optimisers = "3bd65402-5787-11e9-1adc-39752487f4e2" -ScientificTypes = "321657f4-b219-11e9-178b-2701a2544e81" TextAnalysis = "a2db99b7-8b79-58f8-94bf-bbc811eef33d" WordTokenizers = "796a5d58-b03d-544a-977e-18100b691f6e" diff --git a/docs/src/extended_examples/spam_detection/README.md b/docs/src/extended_examples/spam_detection/README.md index f64327a2..b68a07e7 100644 --- a/docs/src/extended_examples/spam_detection/README.md +++ b/docs/src/extended_examples/spam_detection/README.md @@ -11,5 +11,5 @@ # Important -Scripts or notebooks in this folder cannot be reliably exectued without the accompanying +Scripts or notebooks in this folder cannot be reliably executed without the accompanying Manifest.toml and Project.toml files. diff --git a/docs/src/extended_examples/spam_detection/notebook.ipynb b/docs/src/extended_examples/spam_detection/notebook.ipynb index 2f0b50c7..eed3ba38 100644 --- a/docs/src/extended_examples/spam_detection/notebook.ipynb +++ b/docs/src/extended_examples/spam_detection/notebook.ipynb @@ -38,7 +38,10 @@ "name": "stdout", "output_type": "stream", "text": [ - " Activating project at `~/GoogleDrive/Julia/MLJ/MLJFlux/docs/src/extended_examples/spam_detection`\n" + " Activating project at `~/GoogleDrive/Julia/MLJ/MLJFlux/docs/src/extended_examples/spam_detection`\n", + "┌ Warning: The project dependencies or compat requirements have changed since the manifest was last resolved.\n", + "│ It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.\n", + "└ @ Pkg.API /Applications/Julia-1.10.app/Contents/Resources/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:1807\n" ] } ], @@ -68,7 +71,6 @@ "import Optimisers # Flux.jl native optimisers no longer supported\n", "using CSV # Read data\n", "using DataFrames # Read data\n", - "using ScientificTypes # Type coercion\n", "using WordTokenizers # For tokenization\n", "using Languages # For stop words" ], @@ -528,7 +530,7 @@ { "output_type": "execute_result", "data": { - "text/plain": "NeuralNetworkClassifier(\n builder = GenericBuilder(\n apply = Main.var\"##364\".var\"#15#16\"()), \n finaliser = NNlib.softmax, \n optimiser = Adam(0.1, (0.9, 0.999), 1.0e-8), \n loss = Flux.Losses.crossentropy, \n epochs = 10, \n batch_size = 128, \n lambda = 0.0, \n alpha = 0.0, \n rng = Random.TaskLocalRNG(), \n optimiser_changes_trigger_retraining = false, \n acceleration = ComputationalResources.CPU1{Nothing}(nothing))" + "text/plain": "NeuralNetworkClassifier(\n builder = GenericBuilder(\n apply = Main.var\"##1022\".var\"#15#16\"()), \n finaliser = NNlib.softmax, \n optimiser = Adam(0.1, (0.9, 0.999), 1.0e-8), \n loss = Flux.Losses.crossentropy, \n epochs = 10, \n batch_size = 128, \n lambda = 0.0, \n alpha = 0.0, \n rng = Random.TaskLocalRNG(), \n optimiser_changes_trigger_retraining = false, \n acceleration = CPU1{Nothing}(nothing))" }, "metadata": {}, "execution_count": 16 @@ -559,7 +561,7 @@ { "output_type": "execute_result", "data": { - "text/plain": "untrained Machine; caches model-specific representations of data\n model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)\n args: \n 1:\tSource @805 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}\n 2:\tSource @855 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}\n" + "text/plain": "untrained Machine; caches model-specific representations of data\n model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)\n args: \n 1:\tSource @796 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}\n 2:\tSource @667 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}\n" }, "metadata": {}, "execution_count": 17 @@ -587,13 +589,13 @@ "output_type": "stream", "text": [ "[ Info: Training machine(NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …), …).\n", - "\rOptimising neural net: 18%[====> ] ETA: 0:00:39\u001b[K\rOptimising neural net: 27%[======> ] ETA: 0:00:27\u001b[K\rOptimising neural net: 36%[=========> ] ETA: 0:00:21\u001b[K\rOptimising neural net: 45%[===========> ] ETA: 0:00:17\u001b[K\rOptimising neural net: 55%[=============> ] ETA: 0:00:13\u001b[K\rOptimising neural net: 64%[===============> ] ETA: 0:00:10\u001b[K\rOptimising neural net: 73%[==================> ] ETA: 0:00:07\u001b[K\rOptimising neural net: 82%[====================> ] ETA: 0:00:05\u001b[K\rOptimising neural net: 91%[======================> ] ETA: 0:00:02\u001b[K\rOptimising neural net: 100%[=========================] Time: 0:00:23\u001b[K\n" + "\rOptimising neural net: 18%[====> ] ETA: 0:00:13\u001b[K\rOptimising neural net: 27%[======> ] ETA: 0:00:13\u001b[K\rOptimising neural net: 36%[=========> ] ETA: 0:00:12\u001b[K\rOptimising neural net: 45%[===========> ] ETA: 0:00:11\u001b[K\rOptimising neural net: 55%[=============> ] ETA: 0:00:09\u001b[K\rOptimising neural net: 64%[===============> ] ETA: 0:00:07\u001b[K\rOptimising neural net: 73%[==================> ] ETA: 0:00:05\u001b[K\rOptimising neural net: 82%[====================> ] ETA: 0:00:04\u001b[K\rOptimising neural net: 91%[======================> ] ETA: 0:00:02\u001b[K\rOptimising neural net: 100%[=========================] Time: 0:00:19\u001b[K\n" ] }, { "output_type": "execute_result", "data": { - "text/plain": "trained Machine; caches model-specific representations of data\n model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)\n args: \n 1:\tSource @805 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}\n 2:\tSource @855 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}\n" + "text/plain": "trained Machine; caches model-specific representations of data\n model: NeuralNetworkClassifier(builder = GenericBuilder(apply = #15), …)\n args: \n 1:\tSource @796 ⏎ AbstractMatrix{ScientificTypesBase.Continuous}\n 2:\tSource @667 ⏎ AbstractVector{ScientificTypesBase.Multiclass{2}}\n" }, "metadata": {}, "execution_count": 18 @@ -618,7 +620,7 @@ { "output_type": "execute_result", "data": { - "text/plain": "0.9556194393150919" + "text/plain": "0.9468762240501374" }, "metadata": {}, "execution_count": 19 diff --git a/docs/src/extended_examples/spam_detection/notebook.jl b/docs/src/extended_examples/spam_detection/notebook.jl index 0abfb3e4..3d712ebf 100644 --- a/docs/src/extended_examples/spam_detection/notebook.jl +++ b/docs/src/extended_examples/spam_detection/notebook.jl @@ -21,7 +21,6 @@ using Flux import Optimisers # Flux.jl native optimisers no longer supported using CSV # Read data using DataFrames # Read data -using ScientificTypes # Type coercion using WordTokenizers # For tokenization using Languages # For stop words diff --git a/docs/src/extended_examples/spam_detection/notebook.md b/docs/src/extended_examples/spam_detection/notebook.md index 0c5acff6..12cd2a7e 100644 --- a/docs/src/extended_examples/spam_detection/notebook.md +++ b/docs/src/extended_examples/spam_detection/notebook.md @@ -23,7 +23,6 @@ using Flux import Optimisers # Flux.jl native optimisers no longer supported using CSV # Read data using DataFrames # Read data -using ScientificTypes # Type coercion using WordTokenizers # For tokenization using Languages # For stop words ```` diff --git a/docs/src/extended_examples/spam_detection/notebook.unexecuted.ipynb b/docs/src/extended_examples/spam_detection/notebook.unexecuted.ipynb index ab999b09..cd5758bf 100644 --- a/docs/src/extended_examples/spam_detection/notebook.unexecuted.ipynb +++ b/docs/src/extended_examples/spam_detection/notebook.unexecuted.ipynb @@ -60,7 +60,6 @@ "import Optimisers # Flux.jl native optimisers no longer supported\n", "using CSV # Read data\n", "using DataFrames # Read data\n", - "using ScientificTypes # Type coercion\n", "using WordTokenizers # For tokenization\n", "using Languages # For stop words" ], diff --git a/docs/src/index.md b/docs/src/index.md index cc4ad8f1..aba818d5 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -10,21 +10,27 @@ A Julia package integrating deep learning Flux models with [MLJ](https://juliaai - Make it easier to apply machine learning techniques provided by MLJ, including: out-of-sample performance evaluation, hyper-parameter optimization, iteration control, and more, to deep learning models -!!! note "MLJFlux Coverage" - MLJFlux support is focused on fundamental and widely used deep learning models. Sophisticated architectures or techniques such as online learning, reinforcement learning, and adversarial networks are currently beyond its scope. +!!! note "MLJFlux Scope" -Also note that MLJFlux is limited to training models only when all training data fits into memory, though it still supports automatic batching of data. + MLJFlux support is focused on fundamental deep learning models for common + supervised learning tasks. Sophisticated architectures and approaches, such as online + learning, reinforcement learning, and adversarial networks, are currently outside its + scope. Also, MLJFlux is limited to tasks where all (batches of) training data + fits into memory. ## Installation ```julia import Pkg Pkg.activate("my_environment", shared=true) -Pkg.add(["MLJ", "MLJFlux", "Flux"]) +Pkg.add(["MLJ", "MLJFlux", "Optimisers", "Flux"]) ``` -You only need `Flux` if you need to build a custom architecture or experiment with different optimizers, loss functions and activations. +You only need `Flux` if you need to build a custom architecture, or experiment with different loss or activation functions. Since MLJFlux 0.5, you must use optimisers from Optimisers.jl, as native Flux.jl optimisers are no longer supported. ## Quick Start + +For the following demo, you will need to additionally run `Pkg.add("RDatasets")`. + ```@example using MLJ, Flux, MLJFlux import RDatasets @@ -51,20 +57,23 @@ mach = machine(clf, X, y) cv=CV(nfolds=5) evaluate!(mach, resampling=cv, measure=accuracy) ``` -As you can see we were able to use MLJ functionality (i.e., cross validation) with a Flux deep learning model. All arguments provided also have defaults. - -Notice that we were also able to define the neural network in a high-level fashion by only specifying the number of neurons in each hidden layer and the activation function. Meanwhile, `MLJFlux` was able to infer the input and output layer as well as use a suitable default for the loss function and output activation given the classification task. Notice as well that we did not need to implement a training or prediction loop as in `Flux`. - -## Basic idea - -As in the example above, any MLJFlux model has a `builder` hyperparameter, an object encoding -instructions for creating a neural network given the data that the -model eventually sees (e.g., the number of classes in a classification -problem). While each MLJ model has a simple default builder, users -may need to define custom builders to get optimal results, -and this will require familiarity with the [Flux -API](https://fluxml.ai/Flux.jl/stable/) for defining a neural network -chain. +As you can see we are able to use MLJ meta-functionality (i.e., cross validation) with a Flux deep learning model. All arguments provided have defaults. + +Notice that we are also able to define the neural network in a high-level fashion by only +specifying the number of neurons in each hidden layer and the activation +function. Meanwhile, `MLJFlux` is able to infer the input and output layer as well as use +a suitable default for the loss function and output activation given the classification +task. Notice as well that we did not need to manually implement a training or prediction +loop. + +## Basic idea: "builders" for data-dependent architecture + +As in the example above, any MLJFlux model has a `builder` hyperparameter, an object +encoding instructions for creating a neural network given the data that the model +eventually sees (e.g., the number of classes in a classification problem). While each MLJ +model has a simple default builder, users may need to define custom builders to get +optimal results, and this will require familiarity with the [Flux +API](https://fluxml.ai/Flux.jl/stable/) for defining a neural network chain. ## Flux or MLJFlux? diff --git a/docs/src/interface/Custom Builders.md b/docs/src/interface/Custom Builders.md index 980d1165..42543ed2 100644 --- a/docs/src/interface/Custom Builders.md +++ b/docs/src/interface/Custom Builders.md @@ -25,8 +25,8 @@ end Note here that `n_in` and `n_out` depend on the size of the data (see [Table 1](@ref Models). -For a concrete image classification example, see -the [Image Classification Example](@ref). +For a concrete image classification example, see [Using MLJ to classifiy the MNIST image +dataset](@ref). More generally, defining a new builder means defining a new struct sub-typing `MLJFlux.Builder` and defining a new `MLJFlux.build` method diff --git a/docs/src/interface/Summary.md b/docs/src/interface/Summary.md index 3a115418..cc607e53 100644 --- a/docs/src/interface/Summary.md +++ b/docs/src/interface/Summary.md @@ -7,26 +7,26 @@ indicated in the table below. The parameters `n_in`, `n_out` and `n_channels` refer to information passed to the builder, as described under [Defining Custom Builders](@ref). -Model Type | Prediction type | `scitype(X) <: _` | `scitype(y) <: _` ------------|-----------------|---------------|---------------------------- -`NeuralNetworkRegressor` | `Deterministic` | `Table(Continuous)` with `n_in` columns | `AbstractVector{<:Continuous)` (`n_out = 1`) -`MultitargetNeuralNetworkRegressor` | `Deterministic` | `Table(Continuous)` with `n_in` columns | `<: Table(Continuous)` with `n_out` columns -`NeuralNetworkClassifier` | `Probabilistic` | `<:Table(Continuous)` with `n_in` columns | `AbstractVector{<:Finite}` with `n_out` classes -`NeuralNetworkBinaryClassifier` | `Probabilistic` | `<:Table(Continuous)` with `n_in` columns | `AbstractVector{<:Finite{2}}` (`n_out = 2`) -`ImageClassifier` | `Probabilistic` | `AbstractVector(<:Image{W,H})` with `n_in = (W, H)` | `AbstractVector{<:Finite}` with `n_out` classes +| Model Type | Prediction type | `scitype(X) <: _` | `scitype(y) <: _` | +|---------------------------------------------|-----------------|-----------------------------------------------------|-------------------------------------------------| +| [`NeuralNetworkRegressor`](@ref) | `Deterministic` | `Table(Continuous)` with `n_in` columns | `AbstractVector{<:Continuous)` (`n_out = 1`) | +| [`MultitargetNeuralNetworkRegressor`](@ref) | `Deterministic` | `Table(Continuous)` with `n_in` columns | `<: Table(Continuous)` with `n_out` columns | +| [`NeuralNetworkClassifier`](@ref) | `Probabilistic` | `<:Table(Continuous)` with `n_in` columns | `AbstractVector{<:Finite}` with `n_out` classes | +| [`NeuralNetworkBinaryClassifier`](@ref) | `Probabilistic` | `<:Table(Continuous)` with `n_in` columns | `AbstractVector{<:Finite{2}}` (`n_out = 2`) | +| [`ImageClassifier`](@ref) | `Probabilistic` | `AbstractVector(<:Image{W,H})` with `n_in = (W, H)` | `AbstractVector{<:Finite}` with `n_out` classes | ```@raw html -
See definition of "model" +
What exactly is a "model"? ``` In MLJ a *model* is a mutable struct storing hyper-parameters for some learning algorithm indicated by the model name, and that's all. In particular, an MLJ model does not store learned parameters. !!! warning "Difference in Definition" - In Flux the term "model" has another meaning. However, as all - Flux "models" used in MLJFLux are `Flux.Chain` objects, we call them - *chains*, and restrict use of "model" to models in the MLJ sense. + In Flux the term "model" has another meaning. However, as all + Flux "models" used in MLJFLux are `Flux.Chain` objects, we call them + *chains*, and restrict use of "model" to models in the MLJ sense. ```@raw html
@@ -67,12 +67,10 @@ models, `fit!(mach)` will use a warm restart if: Here `model=mach.model` is the associated MLJ model. -The warm restart feature makes it possible to apply early stopping -criteria, as defined in -[EarlyStopping.jl](https://github.com/ablaom/EarlyStopping.jl). For an -example, see [/examples/mnist/](/examples/mnist/). (Eventually, this -will be handled by an MLJ model wrapper for controlling arbitrary -iterative models.) +The warm restart feature makes it possible to externally control iteration. See, for +example, [Early Stopping with MLJFlux](@ref) and [Using MLJ to classifiy the MNIST image +dataset](@ref). + ```@raw html
``` @@ -81,35 +79,37 @@ iterative models.) ## Model Hyperparameters. -All models share the following hyper-parameters: +All models share the following hyper-parameters. See individual model docstrings for a full list. -| Hyper-parameter | Description | Default | -|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| -| `builder` | Default builder for models. | `MLJFlux.Linear(σ=Flux.relu)` (regressors) or `MLJFlux.Short(n_hidden=0, dropout=0.5, σ=Flux.σ)` (classifiers) | -| `optimiser` | The optimiser to use for training. | `Flux.ADAM()` | -| `loss` | The loss function used for training. | `Flux.mse` (regressors) and `Flux.crossentropy` (classifiers) | -| `n_epochs` | Number of epochs to train for. | `10` | -| `batch_size` | The batch size for the data. | `1` | -| `lambda` | The regularization strength. Range = [0, ∞). | `0` | -| `alpha` | The L2/L1 mix of regularization. Range = [0, 1]. | `0` | -| `rng` | The random number generator (RNG) passed to builders, for weight initialization, for example. Can be any `AbstractRNG` or the seed (integer) for a `MersenneTwister` that is reset on every cold restart of model (machine) training. | `GLOBAL_RNG` | -| `acceleration` | Use `CUDALibs()` for training on GPU; default is `CPU1()`. | `CPU1()` | -| `optimiser_changes_trigger_retraining` | True if fitting an associated machine should trigger retraining from scratch whenever the optimiser changes. | `false` | +| Hyper-parameter | Description | Default | +|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------| +| `builder` | Default builder for models. | `MLJFlux.Linear(σ=Flux.relu)` (regressors) or `MLJFlux.Short(n_hidden=0, dropout=0.5, σ=Flux.σ)` (classifiers) | +| `optimiser` | The optimiser to use for training. | `Optimiser.Adam()` | +| `loss` | The loss function used for training. | `Flux.mse` (regressors) and `Flux.crossentropy` (classifiers) | +| `n_epochs` | Number of epochs to train for. | `10` | +| `batch_size` | The batch size for the data. | `1` | +| `lambda` | The regularization strength. Range = [0, ∞). | `0` | +| `alpha` | The L2/L1 mix of regularization. Range = [0, 1]. | `0` | +| `rng` | The random number generator (RNG) passed to builders, for weight initialization, for example. Can be any `AbstractRNG` or the seed (integer) for a `Xoshirio` that is reset on every cold restart of model (machine) training. | `GLOBAL_RNG` | +| `acceleration` | Use `CUDALibs()` for training on GPU; default is `CPU1()`. | `CPU1()` | +| `optimiser_changes_trigger_retraining` | True if fitting an associated machine should trigger retraining from scratch whenever the optimiser changes. | `false` | -The classifiers have an additional hyperparameter `finaliser` (default -= `Flux.softmax`) which is the operation applied to the unnormalized -output of the final layer to obtain probabilities (outputs summing to -one). Default = `Flux.softmax`. It should return a vector of the same -length as its input. +The classifiers have an additional hyperparameter `finaliser` (default is `Flux.softmax`, +or `Flux.σ` in the binary case) which is the operation applied to the unnormalized output +of the final layer to obtain probabilities (outputs summing to one). It should return a +vector of the same length as its input. !!! note "Loss Functions" - Currently, the loss function specified by `loss=...` is applied - internally by Flux and needs to conform to the Flux API. You cannot, - for example, supply one of MLJ's probabilistic loss functions, such as - `MLJ.cross_entropy` to one of the classifier constructors. -That said, you can only use MLJ loss functions or metrics in evaluation meta-algorithms (such as cross validation) and they will work even if the underlying model comes from `MLJFlux`. + Currently, the loss function specified by `loss=...` is applied + internally by Flux and needs to conform to the Flux API. You cannot, + for example, supply one of MLJ's probabilistic loss functions, such as + `MLJ.cross_entropy` to one of the classifier constructors. + +That said, you can only use MLJ loss functions or metrics in evaluation meta-algorithms +(such as cross validation) and they will work even if the underlying model comes from +`MLJFlux`. ```@raw html
More on accelerated training with GPUs @@ -134,14 +134,12 @@ CPU at then conclusion of `fit!`, and made available as ``` -## Built-in builders - -As for the `builder` argument, the following builders are provided out-of-the-box: - -|Builder | Description | -|:-------------------------|:-----------------------------------------------------| -| `MLJFlux.MLP(hidden=(10,))` | General multi-layer perceptron | -| `MLJFlux.Short(n_hidden=0, dropout=0.5, σ=sigmoid)` | Fully connected network with one hidden layer and dropout| -| `MLJFlux.Linear(σ=relu)` | Vanilla linear network with no hidden layers and activation function `σ` | +## Builders -See the following sections to learn more about the interface for the builders and models. +| Builder | Description | +|:--------------------------------------------------------------|:-------------------------------------------------------------------------| +| [`MLJFlux.MLP`](@ref)`(hidden=(10,))` | General multi-layer perceptron | +| [`MLJFlux.Short`](@ref)`(n_hidden=0, dropout=0.5, σ=sigmoid)` | Fully connected network with one hidden layer and dropout | +| [`MLJFlux.Linear`](@ref)`(σ=relu)` | Vanilla linear network with no hidden layers and activation function `σ` | +| [`MLJFlux.@builder`](@ref) | Macro for customized builders | +| | |