Skip to content

Commit

Permalink
build based on 005348f
Browse files Browse the repository at this point in the history
  • Loading branch information
Documenter.jl committed Dec 4, 2024
1 parent 7b66e84 commit 8575a8e
Show file tree
Hide file tree
Showing 32 changed files with 111 additions and 111 deletions.
2 changes: 1 addition & 1 deletion previews/PR2535/.documenter-siteinfo.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"documenter":{"julia_version":"1.11.2","generation_timestamp":"2024-12-03T07:39:06","documenter_version":"1.8.0"}}
{"documenter":{"julia_version":"1.11.2","generation_timestamp":"2024-12-04T04:25:54","documenter_version":"1.8.0"}}
2 changes: 1 addition & 1 deletion previews/PR2535/ecosystem/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion previews/PR2535/guide/gpu/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -168,4 +168,4 @@
true</code></pre><p>For Metal GPU:</p><pre><code class="language-julia hljs">julia&gt; using Metal

julia&gt; Metal.functional()
true</code></pre></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../models/recurrence/">« Recurrence</a><a class="docs-footer-nextpage" href="../saving/">Saving &amp; Loading »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.0 on <span class="colophon-date" title="Tuesday 3 December 2024 07:39">Tuesday 3 December 2024</span>. Using Julia version 1.11.2.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
true</code></pre></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../models/recurrence/">« Recurrence</a><a class="docs-footer-nextpage" href="../saving/">Saving &amp; Loading »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.0 on <span class="colophon-date" title="Wednesday 4 December 2024 04:25">Wednesday 4 December 2024</span>. Using Julia version 1.11.2.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
4 changes: 2 additions & 2 deletions previews/PR2535/guide/models/basics/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@

model1(Float32[0.1]) # output is a Float32 number

grad = gradient(|&gt;, [1f0], model1)[2]</code></pre><p>This gradient is starting to be a complicated nested structure. But it works just like before: <code>grad.inner.W</code> corresponds to <code>model1.inner.W</code>.</p><h3 id="[Flux&#39;s-layers](@ref-man-layers)"><a class="docs-heading-anchor" href="#[Flux&#39;s-layers](@ref-man-layers)"><a href="@ref man-layers">Flux&#39;s layers</a></a><a id="[Flux&#39;s-layers](@ref-man-layers)-1"></a><a class="docs-heading-anchor-permalink" href="#[Flux&#39;s-layers](@ref-man-layers)" title="Permalink"></a></h3><p>Rather than define everything from scratch every time, Flux provides a library of commonly used layers. The same model could be defined:</p><pre><code class="language-julia hljs">model2 = Chain(Dense(1 =&gt; 20, σ), Dense(20 =&gt; 1), only)</code></pre><p>How does this <code>model2</code> differ from the <code>model1</code> we had before?</p><ul><li>Flux&#39;s <a href="../../../reference/models/layers/#Flux.Chain"><code>Chain</code></a> works left-to-right, the reverse of Base&#39;s <code></code>. Its contents is stored in a tuple, thus <code>model2.layers[1].weight</code> is an array.</li><li>Flux&#39;s layer <a href="../../../reference/models/layers/#Flux.Dense"><code>Dense</code></a> has only minor differences, mainly that like <code>struct Poly3{T}</code> above it has type parameters for its fields – the compiler does not know exactly what type <code>layer3s.W</code> will be, which costs speed. Its initialisation uses not <code>randn</code> but ?? It also produces more friendly errors on wrong-size input, and has some performance tricks.</li><li>The function <a href="../../../reference/models/activation/#NNlib.sigmoid"><code>σ</code></a> is calculated in a slightly better way, and has a rule telling Zygote how to differentiate it efficiently.</li><li>Flux overloads <code>Base.show</code> so to give pretty printing at the REPL prompt.</li></ul><p>If what you need isn&#39;t covered by Flux&#39;s built-in layers, it&#39;s easy to write your own. There are more details later, but the steps are invariably those shown for <code>struct Layer</code> above:</p><ol><li>Define a <code>struct</code> which will hold the parameters.</li><li>Make it callable, to define how it uses them to transform the input <code>x</code></li><li>Define a constructor which initialises the parameters.</li><li>Annotate with <code>@layer</code> to opt-in to pretty printing, and other enhacements.</li></ol><h3><img src="https://github.com/FluxML/Optimisers.jl/blob/master/docs/src/assets/logo.png?raw=true" width="40px"/><a href="https://github.com/FluxML/Functors.jl">Functors.jl</a></h3><p>To deal with such nested structures, Flux relies heavily on an associated package called Functors. Its basic function is <a href="../../../reference/models/functors/#Functors.fmap"><code>fmap</code></a>, which generalises <code>map(f, x)</code> to work on almost anything.</p><p>For example, this is how <a href="../../../reference/models/functors/#Flux.gpu-Tuple{Any}">gpu</a> moves all arrays within a model to the GPU, reconstructing another <code>only ∘ Layer(...) ∘ Layer(...)</code> around the new <code>CuArray</code>s:</p><pre><code class="language-julia hljs">using CUDA
grad = gradient(|&gt;, [1f0], model1)[2]</code></pre><p>This gradient is starting to be a complicated nested structure. But it works just like before: <code>grad.inner.W</code> corresponds to <code>model1.inner.W</code>.</p><h3 id="[Flux&#39;s-layers](@ref-man-layers)"><a class="docs-heading-anchor" href="#[Flux&#39;s-layers](@ref-man-layers)"><a href="@ref man-layers">Flux&#39;s layers</a></a><a id="[Flux&#39;s-layers](@ref-man-layers)-1"></a><a class="docs-heading-anchor-permalink" href="#[Flux&#39;s-layers](@ref-man-layers)" title="Permalink"></a></h3><p>Rather than define everything from scratch every time, Flux provides a library of commonly used layers. The same model could be defined:</p><pre><code class="language-julia hljs">model2 = Chain(Dense(1 =&gt; 20, σ), Dense(20 =&gt; 1), only)</code></pre><p>How does this <code>model2</code> differ from the <code>model1</code> we had before?</p><ul><li>Flux&#39;s <a href="../../../reference/models/layers/#Flux.Chain"><code>Chain</code></a> works left-to-right, the reverse of Base&#39;s <code></code>. Its contents is stored in a tuple, thus <code>model2.layers[1].weight</code> is an array.</li><li>Flux&#39;s layer <a href="../../../reference/models/layers/#Flux.Dense"><code>Dense</code></a> has only minor differences, mainly that like <code>struct Poly3{T}</code> above, it has type parameters for its fields – the compiler does not know exactly what type <code>layer3s.W</code> will be, which costs speed. Its initialisation uses not <code>randn</code> but ?? It also produces more friendly errors on wrong-size input, and has some performance tricks.</li><li>The function <a href="../../../reference/models/activation/#NNlib.sigmoid"><code>σ</code></a> is calculated in a slightly better way, and has a rule telling Zygote how to differentiate it efficiently.</li><li>Flux overloads <code>Base.show</code> so to give pretty printing at the REPL prompt.</li></ul><p>If what you need isn&#39;t covered by Flux&#39;s built-in layers, it&#39;s easy to write your own. There are more details later, but the steps are invariably those shown for <code>struct Layer</code> above:</p><ol><li>Define a <code>struct</code> which will hold the parameters.</li><li>Make it callable, to define how it uses them to transform the input <code>x</code></li><li>Define a constructor which initialises the parameters.</li><li>Annotate with <code>@layer</code> to opt-in to pretty printing, and other enhacements.</li></ol><h3><img src="https://github.com/FluxML/Optimisers.jl/blob/master/docs/src/assets/logo.png?raw=true" width="40px"/><a href="https://github.com/FluxML/Functors.jl">Functors.jl</a></h3><p>To deal with such nested structures, Flux relies heavily on an associated package called Functors. Its basic function is <a href="../../../reference/models/functors/#Functors.fmap"><code>fmap</code></a>, which generalises <code>map(f, x)</code> to work on almost anything.</p><p>For example, this is how <a href="../../../reference/models/functors/#Flux.gpu-Tuple{Any}">gpu</a> moves all arrays within a model to the GPU, reconstructing another <code>only ∘ Layer(...) ∘ Layer(...)</code> around the new <code>CuArray</code>s:</p><pre><code class="language-julia hljs">using CUDA
fmap(cu, model)</code></pre><p>And this is a very simple gradient update of the parameters:</p><pre><code class="language-julia hljs">fmap((x, dx) -&gt; x isa Array ? (x - dx/100) : x, model, grad)</code></pre><p>Before Flux v0.15 (and Functors v0.5), this exploration of structs was opt-in. After defining <code>struct Layer</code> it was necessary to call <code>@functor Layer</code> (or <code>@layer Layer</code>) before Flux would look inside. This has now changed to be opt-out: Functors (and hence Flux) will explore arbitrary structs, unless told not to (using <code>Functors.@leaf</code>). This is why even &quot;anonymous structs&quot; created by closures like <code>poly3</code> and <code>layer3</code> above are now valid Flux models, although the use of named structs is still recommended practice.</p><h2 id="Curve-Fitting"><a class="docs-heading-anchor" href="#Curve-Fitting">Curve Fitting</a><a id="Curve-Fitting-1"></a><a class="docs-heading-anchor-permalink" href="#Curve-Fitting" title="Permalink"></a></h2><p>Above we took gradients of the output, or sometimes to the first element of the output – it must be a number, not a vector. Adjusting the parameters to make this smaller won&#39;t lead us anywhere interesting. Instead, we should minimise some <em>loss function</em> which compares the actual output to our desired output.</p><p>Perhaps the simplest example is curve fitting. Given a function like <code>f(x) = 2x - x^3</code> evaluated at some points <code>x in -2:0.1:2</code>, we can aim to adjust the parameters of the two-layer <code>model</code> from above so that its output is similar to the truth. Here&#39;s how this might look:</p><pre><code class="language-julia hljs">data = [([x], 2x-x^3) for x in -2:0.1f0:2] # training points (x, y)

for _ in 1:1000 # adjust parameters to minimise the error:
Expand All @@ -87,4 +87,4 @@
end
end</code></pre><p>And here&#39;s how to plot the desired and actual outputs:</p><pre><code class="language-julia hljs">using Plots
plot(x -&gt; 2x-x^3, -2, 2, legend=false)
scatter!(-2:0.1:2, [model([x]) for x in -2:0.1:2])</code></pre><p>If this general idea is unfamiliar, you may want the <a href="#ref ???">tutorial on linear regression</a>.</p><p>More detail about what exactly the function <code>train!</code> is doing, and how to use rules other than simple <a href="../../../reference/training/optimisers/#Optimisers.Descent"><code>Descent</code></a>, is what the next page in this guide is about.</p></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../overview/">« Fitting a Line</a><a class="docs-footer-nextpage" href="../custom_layers/">Custom Layers »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.0 on <span class="colophon-date" title="Tuesday 3 December 2024 07:39">Tuesday 3 December 2024</span>. Using Julia version 1.11.2.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
scatter!(-2:0.1:2, [model([x]) for x in -2:0.1:2])</code></pre><p>If this general idea is unfamiliar, you may want the <a href="#ref ???">tutorial on linear regression</a>.</p><p>More detail about what exactly the function <code>train!</code> is doing, and how to use rules other than simple <a href="../../../reference/training/optimisers/#Optimisers.Descent"><code>Descent</code></a>, is what the next page in this guide is about.</p></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../overview/">« Fitting a Line</a><a class="docs-footer-nextpage" href="../custom_layers/">Custom Layers »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.0 on <span class="colophon-date" title="Wednesday 4 December 2024 04:25">Wednesday 4 December 2024</span>. Using Julia version 1.11.2.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
2 changes: 1 addition & 1 deletion previews/PR2535/guide/models/custom_layers/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -104,4 +104,4 @@
# rms over all the mse
ŷs = model(x)
return sqrt(mean(Flux.mse(y, ŷ) for (y, ŷ) in zip(ys, ŷs)))
end</code></pre><div class="admonition is-info"><header class="admonition-header">Note</header><div class="admonition-body"><p>This <code>Split</code> layer is available from the <a href="https://github.com/FluxML/Fluxperimental.jl">Fluxperimental.jl</a> package.</p></div></div></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../basics/">« Gradients and Layers</a><a class="docs-footer-nextpage" href="../../training/training/">Training »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.0 on <span class="colophon-date" title="Tuesday 3 December 2024 07:39">Tuesday 3 December 2024</span>. Using Julia version 1.11.2.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
end</code></pre><div class="admonition is-info"><header class="admonition-header">Note</header><div class="admonition-body"><p>This <code>Split</code> layer is available from the <a href="https://github.com/FluxML/Fluxperimental.jl">Fluxperimental.jl</a> package.</p></div></div></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../basics/">« Gradients and Layers</a><a class="docs-footer-nextpage" href="../../training/training/">Training »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.8.0 on <span class="colophon-date" title="Wednesday 4 December 2024 04:25">Wednesday 4 December 2024</span>. Using Julia version 1.11.2.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
Loading

0 comments on commit 8575a8e

Please sign in to comment.