feat!: Enable additional status codes arguments to PlaywrightCrawler #959
Conversation
Since they exist now on all crawlers, move them to basic crawler level.
github-actions
bot
added
t-tooling
|
|
||
| if self._http_client.additional_blocked_status_codes != self._additional_http_error_status_codes: | ||
| raise ValueError( | ||
| 'Used `additional_blocked_status_codes` argument does not match with with ' |
There was a problem hiding this comment.
double with
Sorry, can't commit with the quick fix due to limited permissions.
| self._additional_http_error_status_codes = ( | ||
| set(additional_http_error_status_codes) if additional_http_error_status_codes else set() | ||
| ) | ||
| self._ignore_http_error_status_codes = ( | ||
| set(ignore_http_error_status_codes) if ignore_http_error_status_codes else set() | ||
| ) | ||
|
|
||
| self._http_client = http_client or HttpxHttpClient( | ||
| additional_http_error_status_codes=self._additional_http_error_status_codes, | ||
| ignore_http_error_status_codes=self._ignore_http_error_status_codes, | ||
| ) | ||
|
|
||
| if self._http_client.additional_blocked_status_codes != self._additional_http_error_status_codes: | ||
| raise ValueError( | ||
| 'Used `additional_blocked_status_codes` argument does not match with ' | ||
| f'{self._http_client.additional_blocked_status_codes=}. They have to be the same.' | ||
| ) | ||
| if self._http_client.ignore_http_error_status_codes != self._ignore_http_error_status_codes: | ||
| raise ValueError( | ||
| 'Used `ignore_http_error_status_codes` argument does not match with ' | ||
| f'{self._http_client.ignore_http_error_status_codes=}. They have to be the same.' | ||
| ) |
There was a problem hiding this comment.
Couldn't we just keep them only in the http_client instance? (PW Crawler has HTTP client as well)
There was a problem hiding this comment.
I was considering that option, but it felt like misuse to me, especially when it comes to PlaywrightCrawler. PlaywrightCrawler is not using HTTP client for page.navigate so it would be really strange if it would use some attribute of this unrelated component to decide whether response status code of page.navigate is ok or not.
(Mentioned : #953 (comment))
But I see it looks like unnecessary code duplication, so I am not 100% happy with this either.
There was a problem hiding this comment.
Yeah, I got it... However, having it duplicated seems like a worse option to me.
@janbuchar Your opinion please?
There was a problem hiding this comment.
I think we can think about taking this logic out of the http client. And put it in the BasicCrawler. Then it will work uniformly for any crawler and we will avoid code duplication
There was a problem hiding this comment.
I agree - in the long run, we want to have this logic factored out of the http client. I believe there was an issue to track that, but I only found #830.
It's probably fine to duplicate now and make an issue for refactoring this later.
There was a problem hiding this comment.
In that case, I don't see a problem if we keep the duplication of code at this point. It will be solved during refactoring.
vdusek
modified the milestones:
107th sprint - Tooling team,
108th sprint - Tooling team
…pify#959) Add `additional_http_error_status_codes` and `ignore_http_error_status_codes` to PlaywrightCrawler. Since they exist now on all crawlers, move them to `BasicCrawler` level. Do not use `_http_client` attributes for getting additional status codes related variables. **Breaking:** Remove `HttpCrawlerOptions` -> No unique options compared to `BasicCrawlerOptions` anymore. - Closes: apify#953
Description
Add
additional_http_error_status_codesandignore_http_error_status_codesto PlaywrightCrawler.Since they exist now on all crawlers, move them to
BasicCrawlerlevel.Do not use
_http_clientattributes for getting additional status codes related variables.Breaking: Remove
HttpCrawlerOptions-> No unique options compared toBasicCrawlerOptionsanymore.Issues