Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to start with "no proxy" #2740

Closed
strongpauly opened this issue Nov 8, 2024 · 0 comments · Fixed by #2743
Closed

Add ability to start with "no proxy" #2740

strongpauly opened this issue Nov 8, 2024 · 0 comments · Fixed by #2743
Assignees
Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@strongpauly
Copy link

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/core

Feature

When using the tieredProxyUrls option in ProxyConfiguration I would like to be able to start without a proxy, then escalate to using one if and when a request fails.

Motivation

I'm uncertain as to whether proxies will be required at all and I would like to minimise the cost of unnecessarily using them.

Ideal solution or implementation, and any additional constraints

Passing an empty array as the first tier seemed most obvious, e.g. [[], ["http://proxy-url"]] would start with no proxy, then escalate to "http://proxy-url" if and when that failed.

Alternative solutions or implementations

I could configure separate crawlers with and without the tiered proxy configuration, but seems like it would be a simple feature to add/document.

Other context

No response

@strongpauly strongpauly added the feature Issues that represent new features or improvements to existing features. label Nov 8, 2024
@github-actions github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 8, 2024
@barjin barjin closed this as completed in 82f4ea9 Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants