clip adaptor

[haoxiangzhao12138]

In the Clip Adapter, which specific model of OpenCLIP is used? When using torch.hub.load with adaptor_names='clip', it requires access to Hugging Face. Is there a way to load the Clip Adapter locally instead?

[mranzinger]

Hmm, are you sure it requires huggingface? It should be using open_clip:

RADIO/radio/open_clip_adaptor.py

Lines 23 to 29 in 354971d

	import open_clip
	self.oc_model = open_clip.create_model_from_pretrained(
	model_name=adaptor_config['model'],
	pretrained=adaptor_config['pretrained'],
	return_transform=False,
	)

And this is the specific model we use: https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv#L2

name and pretrained map to model_name and pretrained here:

RADIO/radio/open_clip_adaptor.py

Lines 26 to 27 in 354971d

	model_name=adaptor_config['model'],
	pretrained=adaptor_config['pretrained'],

[haoxiangzhao12138]

Thank you very much for your response. When I use torch.hub.load(... , ... , source='local') to load a local model and add the adaptor_name parameter, I encounter the following error:

requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /apple/DFN5B-CLIP-ViT-H-14-378/resolve/main/open_clip_pytorch_model.bin (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f422eedcb50>, 'Connection to huggingface.co timed out. (connect timeout=10)'))"), '(Request ID: cf1f5d30-1fe2-4417-abde-ed366c875d0a)')

From what I understand, it seems like there’s an issue accessing Hugging Face. Is there any additional parameter I might need to add to torch.hub.load to address this?

[mranzinger]

Ah, I see. So open_clip is pulling the model weights from huggingface. Do you have access to any environment that is able to access them? Do you need to text encoder for the clip model?