How to use MemoryStorageClient with redis for Request Queue?

[abhichek]

Currently, as per source code, RequestQueue uses MemoryStorageClient by default. However, MemoryStorageClient uses file-based storage rather than in-memory storage. I am planning to use Redis as a storage backend for higher throughput.

Can anyone help me with an example of how to configure RequestQueue or MemoryStorageClient to use Redis as storage backend?

[janbuchar]

Hello! Your best course of action here would be to implement a custom RequestManager that would use Redis as the storage backend. Then, you can pass an instance of the new class to your crawler constructor, e.g., ParselCrawler(request_manager=RedisRequestManager()) and everything should work as expected. Feel free to reach out for more help!

[abhichek]

Thank you @janbuchar . That sounds like a good solution. Since I'm using PlaywrightCrawler(), I will try implementing your suggestion and reach out if I need further assistance. In the meantime, could you please let me know if it's possible to either connect to a remote browser or keep the browser open?

[janbuchar]

Glad to help. PlaywrightCrawler doesn't allow using playwright.connect as of now, so connecting to a remote browser will be difficult. Regarding keeping the browser open, what exactly do you have in mind? Would you like the crawler to keep waiting for new requests even if the queue is "empty"?

[janbuchar]

Connecting to remote browsers is tracked in this feature request for JS crawlee. Once we manage to implement that, a Python version is sure to follow. It may come sooner, too - we'll see 🙂

[vdusek]

Hi, I would like to add that in the future, implementing a custom storage client (e.g., for Redis) should become much easier, as we have a storage client redesign on our roadmap (#783, #92, #307).