AttentionPoolLatent reference?

[billpsomas]

Hello all,

Is there any reference for AttentionPoolLatent (https://github.com/huggingface/pytorch-image-models/blob/main/timm/layers/attention_pool.py#L12). Which paper is this attention pooling from?

Thanks a lot!

[rwightman]

@billpsomas That impl is compatible with the attention pooling in SigLIP, but I had the layer coded up before that paper, based on some previous attention pooling ideas that were out there. I think maybe some of it was related to Perceiver but I forget exactly, the concept has been 'out there' for a while. When SigLIP was released, I tweaked it to be compatible, it supports some other options.

The '2d' variants in attention_pool_2d were based on the OpenAI CLIP ResNet attention pooling, but again had some additional options that I added.

[billpsomas]

Hello,

Thanks for answering that fast. Yep, my problem is that, if I am not wrong, it's not exactly the same as the SigLIP att pool (https://github.com/huggingface/transformers/blob/main/src/transformers/models/siglip/modeling_siglip.py#L363). It indeed has some ideas from Perceiver. So, there are a lot of attention poolings out there and I am making a study about them, so I will probably reference you haha for this SigLIP variant. :D

[rwightman]

@billpsomas I changed the defaults of that layer so that it does line up with SigLIP, it's used for the timm siglip encoders which match the original.

timm siglip use:

pytorch-image-models/timm/models/vision_transformer.py

Lines 580 to 589 in a49b020

	# Classifier Head
	if global_pool == 'map':
	self.attn_pool = AttentionPoolLatent(
	self.embed_dim,
	num_heads=num_heads,
	mlp_ratio=mlp_ratio,
	norm_layer=norm_layer,
	)
	else:
	self.attn_pool = None

original
https://github.com/google-research/big_vision/blob/ebb95a67192cd7cafd0c81b5a955810f100e5aa5/big_vision/models/vit.py#L163-L183

The transformers impl I recall differed in their usual ways but should be equivalent .