Add paper analysis workflow & pdf loader #19
Conversation
| f"Paper analysis initiated", "yellow", attrs=["bold", "blink"] | ||
| ) | ||
| ) | ||
| pages = load_pdf(pdf_path) |
There was a problem hiding this comment.
The quality of retrieval might be slightly improved by splitting the documents into smaller sections and adding some chunk overlap.
There was a problem hiding this comment.
thanks for the feedback!
|
Could you please add the common sections of a paper? (Abstract, Introduction, Materials, Methods (sometimes they're in a single section called Materials & Methods), Results, Discussion, Supplemental Materials, References) I think a method to just extract the text from those sections would be SO useful on its own. |
Great idea! |
This PR adds a workflow that analyzes a given paper. The workflow extracts the main findings, methodology and limitations of the paper. The paper can be loaded as a PDF. For this, I added a new module "file_loaders" and a PDF loader that loads and splits PDFs by page. The PDF can load local and also remote PDFs if passed a URL (e.g. "https://arxiv.org/pdf/2302.03803.pdf").
Looking forward to your feedback!