docs/introduction/saving-data

[giscus[bot]]

docs/introduction/saving-data

Your first steps into the world of scraping with Crawlee

https://crawlee.dev/docs/introduction/saving-data

[tteguayco]

In the case we want to save the scraped data into a database (.e.g Postgres, Mongo), does Crawlee incorporate anything for it or should we add our own custom code around the framework to handle such scenario?

[B4nan]

The storage layer is abstracted and you can implement your own, but the abstraction is a bit complex and there is no documentation around this currently. You could maybe try to extend the memory storage implementation, and override only the dataset methods. But it will be much simpler to do it outside of crawlee and just use your own helpers to store the data instead of the crawlee's dataset abstraction.

Going forward, we would like to simplify the storage interface some time in future (maybe v4), so the extension is simpler and maybe even provide more native storage implementations.

[tteguayco]

That sounds really nice, thanks!

[dbjpanda]

Hi, I am just wondering if it is now possible to implement a custom storage interface? Could not find any docs.

[maximilliangeorge]

The documentation says that Dataset is an append only store. What is the idiomatic approach to reconciling changed data? Is the idea to drop old records and append the updated version?

The documentation also mentions KeyValueStore. Is this a better data store to use if I want to update data over time?

[janbuchar]

You'd typically use the Dataset to gather homogeneous records in a single run of the crawler. If you're monitoring a website for changes, you could do that separately from the crawling when you have access to both the original dataset and the updated one. It is not possible to remove or update previously appended dataset items on the fly.

Regarding KeyValueStore, yes, that's a better fit if you need to update something as you go. You can either use the default KeyValueStore for each run of your crawler, or you can use a named one which will persist between runs - you need to chose whatever fits your need better.