update queryRemoval rules for August 2024 #7
Conversation
| "pid", | ||
| "productName" |
There was a problem hiding this comment.
First off, this is 👍.
But also, looking at the website to verify that indeed the pid query parameter has no effect, I noticed that proper URL normalization (to detect duplicates before redirects) would need a new rule type to support treating https://www.bunnings.com.au/ozito-1800w-2030psi-high-pressure-washer_p0254158 as https://www.bunnings.com.au/_p0254158 (i.e. remove everything from the last URL path part before the last _, at least when the last path ends in _p<num>). It’s not the first time I see something like that in an e-commerce website.
There was a problem hiding this comment.
Thanks for raising @Gallaecio.
This normalization task seems to suit the SubpathRemovalProcessor but currently doesn't support substring removal in a given subpath. Created zytedata/duplicate-url-discarder#24 for tracking.
Note: Some rules have updated alphabetical ordering. It only happens when a given domain was seen lately. Otherwise, domains from old rules that weren't seen lately won't be touched at all. As time progresses, there'd be less diffs due to sorting.