[BFCL] Error when consecutively generate from multiple oss_models with vllm

[YifanHao]

I have plenty of models and I want to test them all. So I run a bash script like this:

bfcl generate --model "model_1" --test-category simple
bfcl generate --model "model_2" --test-category simple
...

The first generation was good. But the second run and thereafter encountered the following error:

I suspect the reason is that VLLM_PORT still occupied by the previous run, so I got the output of ps -aux when the second run just began. However it seems the previous vllm server has already ended.

Now I have no idea why this isn't working, could you help me with this?
Or what's the recommended way if I want to run many models automatically?

Update:
Just after I open this issue, I find the error disappears if I wait long enough before the next run. (sleep 60 after each line).
But still, I wonder if there's a better way to do this?

[HuanzhiMao]

Hi @YifanHao,
I think the issue you are encountering is because vllm takes some time to shutdown the server and clean up after all the inference are done, and so the best approach would be probably like what you are doing (eg, add some time interval between the runs).