Consider making the :expand option the default in @layer

[CarloLucibello]

It doesn't seem to be really needed, we could automatically detect if the layer contains only leaf types or is a nested container

[mcabbott]

No, we had this automatic detection, and it didn't work.

[CarloLucibello]

what is exactly that is not working? I'm testing now with making everything :expand and I don't see any problem

[mcabbott]

Every layer whose implementation contains other layers, or things which could plausibly be layers, but whose interface does not.

[CarloLucibello]

Ok, I see that we want:

julia> LayerNorm(10)
LayerNorm(10)       # 20 parameters

julia> MultiHeadAttention(64 => 1024 => 1024, nheads = 8)
MultiHeadAttention(64 => 1024 => 1024; nheads=8)  # 1_245_184 parameters

Maybe we should change the default then? I feel that 90% cases one would do with expand, since it cover both the layer with plain arrays and the container type.
I don't really like that Flux.@layer :expand Model would come out so often in code and since the meaning is not obvious one has to look in the documentation.

[mcabbott]

IDK, maybe the other default would have been better, but I'm against unnecessary churn.

My guess was that most people making layers are wrapping some parameter arrays, not building container layers. But I have no data.

The present default means that if you don't ever think about :expand, you get too-compact printing? Maybe I thought that was better than accidentally printing 100 pages of parameters. Although now I fixed that problem & it should print summary(x).

[CarloLucibello]

I think most people don't make layers, they make blocks and models.
In pytorch the default is something like :expand.
Since it only affects the printing, moving to expand as a default, and having another keyword (:noexpand or :compact?) for the layer print shouldn't be considered breaking I guess. If so, we can do it also after v0.15 although I would be in favor of doing it now.

[mcabbott]

Yea like I said, no data on which is more common. Certainly Flux's source has mostly compact ones.

Having another option (but and not breaking existing @layer :expand Layer code) and then changing the default seems OK. But seems polite to do on a breaking change, not just at some random time.