You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
strongpauly opened this issue
Nov 8, 2024
· 0 comments
· Fixed by #2743
Assignees
Labels
featureIssues that represent new features or improvements to existing features.t-toolingIssues with this label are in the ownership of the tooling team.
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/core
Feature
When using the tieredProxyUrls option in ProxyConfiguration I would like to be able to start without a proxy, then escalate to using one if and when a request fails.
Motivation
I'm uncertain as to whether proxies will be required at all and I would like to minimise the cost of unnecessarily using them.
Ideal solution or implementation, and any additional constraints
Passing an empty array as the first tier seemed most obvious, e.g. [[], ["http://proxy-url"]] would start with no proxy, then escalate to "http://proxy-url" if and when that failed.
Alternative solutions or implementations
I could configure separate crawlers with and without the tiered proxy configuration, but seems like it would be a simple feature to add/document.
Other context
No response
The text was updated successfully, but these errors were encountered:
featureIssues that represent new features or improvements to existing features.t-toolingIssues with this label are in the ownership of the tooling team.
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/core
Feature
When using the
tieredProxyUrls
option inProxyConfiguration
I would like to be able to start without a proxy, then escalate to using one if and when a request fails.Motivation
I'm uncertain as to whether proxies will be required at all and I would like to minimise the cost of unnecessarily using them.
Ideal solution or implementation, and any additional constraints
Passing an empty array as the first tier seemed most obvious, e.g.
[[], ["http://proxy-url"]]
would start with no proxy, then escalate to "http://proxy-url" if and when that failed.Alternative solutions or implementations
I could configure separate crawlers with and without the tiered proxy configuration, but seems like it would be a simple feature to add/document.
Other context
No response
The text was updated successfully, but these errors were encountered: