Regarding the Fine Tunning The DPR on Our Custom DATA SET in the pdfs

[imtej]

My Question is that, How can We fine tune the existing DPR models on our custom Data Set in the pdf Format.

As the main problem arises in doing this is to create the dataset in the format of the of the DPR, I mean manually creating the Query- passage pair is a lot of labour works, and using the synthetic dataset wouldn't capture the nuances in our dataset and it would lead to overfitting.

[davidsbatista]

Hi @imtej,

I'm assuming that by DPR, you mean Dense Passage Retrieval. If that's so,Haystack 2.x doesn't support fine tuning of DPR, Haystack 1.x does, you can see a tutorial on how to do it here: https://haystack.deepset.ai/tutorials/09_dpr_training

[imtej]

Thanks @davidsbatista, But that tutorial is for the structured QA pairs not for the PDFs

[davidsbatista]

I would than suggest to extract segments of texts with possible answers from the PDFs, and keep track with a mapping to which PDFs they belong to. In that way instead of showing a text segment as an answer you show the user the PDF where that segment belongs.