Can a ElasticsearchDocumentStore listen to a database for automatic indexing of new data?

[timpal0l]

What is the best practice to connect an ElasticsearchDocumentStore to an existing database? I have a document database on google cloud that retrieves new data on a daily basis.

Can you make the ElasticSearchDocumentStore listen to the database and once it notices new data, it updates the index with new embeddings automatically?

[tstadel]

Hi Tim,
thanks for your question :-)
As you can imagine, there would be a lot of document databases to support for listening to, which additionally might work in completely different ways.
That's why haystack supports a push-approach instead of a listening-approach, which I would also recommend here for decoupling the systems. You can use haystack's /file-upload API to implement that. Once you get a document you can send it via POST /file-upload to the API and haystack's indexing pipeline will take care of that (e.g. index it and update embeddings if you're using a dense retriver).