[CI][sharktank] Move Sharktank Data-Dependent Tests to OSSCI Cluster

[renxida]

We've been getting memory-allocation related errors possibly due to oversaturation on shark-mi300x-3.

[renxida]

Need #926; failing because no dir to cache hf downloads

[sogartar]

These tests require some data to be available on the runner.

The premise that we don't have enough resources on the machine seems wrong.
What we need is a proper allocation of GPUs to jobs/runners. Also users to adhere to a mandatory reservation system. Meaning that if you don't request a GPU you don't see any. For example when running iree-run-module --list_devices.
I sometimes catch myself not specifying the correct GPU.
Maybe we could make a service that utilizes the ROCR_VISIBLE_DEVICES env var. Or to utilize Linux groups to control access.

[sogartar]

I would also want to move these tests soon to nightly as we should strive to have only fast tests on every PR.