Reproducing repllama on passage retrieval.

[hengran]

Hello, thanks for sharing the great work!
I download the checkpoint of repllama from huggiface. I encode the query and passages with this checkpoint and find the result I got is MRR@10: 39.69, while in the paper is 41.2.

p_max_len=512
q_max_len=512
CUDA_VISIBLE_DEVICES=1 python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path $model_path \
  --lora_name_or_path $lora_model_save_path \
  --normalize \
  --encode_is_query \
  --fp16 \
  --per_device_eval_batch_size 64 \
  --passage_max_len $p_max_len \
  --pooling eos \
  --append_eos_token \
  --query_max_len $q_max_len \
  --dataset_path $dev_query_path \
  --encode_output_path $encode_path/dev_query_emb.pkl

sleep 10s
echo "====== encode corpus"
for s in 0 1 2 3 4 5 6 7;
do
gpuid=$s
CUDA_VISIBLE_DEVICES=$gpuid python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path $model_path \
  --lora_name_or_path $lora_model_save_path \
  --normalize \
  --fp16 \
  --per_device_eval_batch_size 64 \
  --pooling eos \
  --append_eos_token \
  --passage_max_len $p_max_len \
  --dataset_path $corpus_path \
  --query_max_len $q_max_len \
  --dataset_number_of_shards 8 \
  --dataset_shard_index ${s} \
  --encode_output_path $encode_path/corpus_emb.${s}.pkl   &
  # 等待最后一个循环完成
  if [ "$s" == "7" ]; then
      wait
  fi
done

[hengran]

Hello, thanks for sharing the great work! I download the checkpoint of repllama from huggiface. I encode the query and passages with this checkpoint and find the result I got is MRR@10: 39.69, while in the paper is 41.2.

p_max_len=512
q_max_len=512
CUDA_VISIBLE_DEVICES=1 python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path $model_path \
  --lora_name_or_path $lora_model_save_path \
  --normalize \
  --encode_is_query \
  --fp16 \
  --per_device_eval_batch_size 64 \
  --passage_max_len $p_max_len \
  --pooling eos \
  --append_eos_token \
  --query_max_len $q_max_len \
  --dataset_path $dev_query_path \
  --encode_output_path $encode_path/dev_query_emb.pkl

sleep 10s
echo "====== encode corpus"
for s in 0 1 2 3 4 5 6 7;
do
gpuid=$s
CUDA_VISIBLE_DEVICES=$gpuid python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path $model_path \
  --lora_name_or_path $lora_model_save_path \
  --normalize \
  --fp16 \
  --per_device_eval_batch_size 64 \
  --pooling eos \
  --append_eos_token \
  --passage_max_len $p_max_len \
  --dataset_path $corpus_path \
  --query_max_len $q_max_len \
  --dataset_number_of_shards 8 \
  --dataset_shard_index ${s} \
  --encode_output_path $encode_path/corpus_emb.${s}.pkl   &
  # 等待最后一个循环完成
  if [ "$s" == "7" ]; then
      wait
  fi
done

Since the paper only includes Recal@1000, I want to reproduce the model to obtain the values for Recall@100 and Recall@50, but I've found that the current reproduction results are different from the results of paper. It's possible that the parameters I've set for reproduction are not correct.

[MXueguang]

Hi @hengran, are you using the corpus downloaded from tevatron?

[MXueguang]

I used qlen=32 plen=156 during training and encoding repllama. Not sure if this can made that difference.

[hengran]

I used qlen=32 plen=156 during training and encoding repllama. Not sure if this can made that difference.

thanks for you reply, I try it.