Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-diff shape handling in norm layers #2363

Merged
merged 1 commit into from
Dec 20, 2023
Merged

Conversation

ToucheSir
Copy link
Member

@ToucheSir ToucheSir commented Dec 19, 2023

This reduces some latency when using Zygote.

using SnoopCompileCore
invalidations = @snoopr(using Zygote, Flux)
using SnoopCompile, AbstractTrees

loss(model, x) = sum(model(x))
tinf = let model = BatchNorm(3), x = rand(Float32, 16, 16, 3, 8)
    loss(model, x)
    @snoopi_deep gradient(loss, model, x)
end

julia> tinf
InferenceTimingNode: 2.846507/13.942681 on Core.Compiler.Timings.ROOT() with 139 direct children # master
InferenceTimingNode: 2.354729/9.970884 on Core.Compiler.Timings.ROOT() with 126 direct children # PR

PR Checklist

  • N/A Tests are added
  • Entry in NEWS.md
  • N/A Documentation, if applicable

This reduces some latency when using Zygote.
@ToucheSir ToucheSir merged commit 8718491 into master Dec 20, 2023
6 of 9 checks passed
@ToucheSir ToucheSir deleted the bc/norm-ad-ignores branch December 20, 2023 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants