How to ingest data from SharePoint Site?

[nhtkid]

Dear Community,

I have used Langchain to build QA bot with OpenAI, which works great.

I am iterating the app to change the data source to SharePoint Online, seeking some help here.

I am looking for a way to ingest data from SharePoint Pages content (.aspx) as well as SharePoint Document Libraries.

What would be the best way?

There is a workflow that I could create SharePoint Index with Azure Cognitive Services, which will be scheduled to run daily to bring in the delta. If Langchain can work with Cognitive Search Index, that would be fantastic.

The only problem with the SharePoint Index in Azure is that it can only index Document Libraries, not the page content because .aspx is not supported, which accounts for half of the intranet content.

Any ideas or suggestions are welcome.

Thank you all very much!