Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dynamic signaling #38

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add dynamic signaling #38

wants to merge 1 commit into from

Conversation

n1kn4x
Copy link

@n1kn4x n1kn4x commented Mar 19, 2020

Adding and removing proxies is now possible through signals "ADD_PROXY" and "REMOVE_PROXY".

Additionally when a proxy is marked as dead/good a signal "DEAD_PROXY"/"GOOD_PROXY" is sent.

This enables the middleware to adapt proxies and dynamically spawn/kill proxies if the user wants.

@codecov-io
Copy link

Codecov Report

Merging #38 into master will increase coverage by 1.34%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master      #38      +/-   ##
==========================================
+ Coverage   51.68%   53.03%   +1.34%     
==========================================
  Files           5        5              
  Lines         238      264      +26     
  Branches       39       43       +4     
==========================================
+ Hits          123      140      +17     
- Misses        109      115       +6     
- Partials        6        9       +3     
Impacted Files Coverage Δ
rotating_proxies/middlewares.py 24.61% <0.00%> (-0.39%) ⬇️
rotating_proxies/expire.py 77.19% <72.00%> (-1.70%) ⬇️

@StasDeep
Copy link
Contributor

StasDeep commented Jun 7, 2020

Hi! Nice PR, thanks a lot! What do you think about using constants as signals instead of strings? The way it's done in Scrapy:

# scrapy/signals.py

spider_opened = object()
spider_idle = object()
spider_closed = object()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants