feat: Integrate browserforge fingerprints #829

Pijukatel · 2024-12-18T14:26:24Z

Description

Integrate browserforge package and use it's fingerprint and header generation capabilities to enable fingerprint generation in PlaywrightCrawler

How can users use it?
Closely following JS implementation would lead to:

crawler = PlaywrightCrawler(
    headless=True,
    browser_pool_options={
        'use_fingerprints': True,
        'fingerprint_generator_options': {
            'browsers': 'firefox',
            'os': 'android',
            'screen': Screen(
                min_width=min_width, max_width=max_width, min_height=min_height, max_height=max_height
            ),
        },
    },
)

I suggest this instead:

from crawlee.fingerprint_suite import DefaultFingerprintGenerator, HeaderGeneratorOptions, ScreenOptions

fingerprint_generator = DefaultFingerprintGenerator(
    header_options=HeaderGeneratorOptions(browsers=['firefox'], operating_systems=['android']),
    screen=ScreenOptions(min_width=min_width, max_width=max_width, min_height=min_height, max_height=max_height),
)

crawler = PlaywrightCrawler(
    headless=True,
    fingerprint_generator=fingerprint_generator
)

Reasoning:

FingerprintGeneratorOptions are too complex to be represented by nested dicts - not very readable.
avoid having arguments use_fingerprints and fingerprint_generator_options. Instead have argument fingerprint_generator that is implicit for use_fingerprints, it contains all the fingerprint_generator_options and it also allows custom implementations of generator to be passed.
DefaultFingerprintGenerator is very convenient for us, as we can set it to browserforge adapter for now and change it later.

Currently we do not have our own injector, so we have to use browser forge injector. That means that temporarily we have to accept Fingerprint definition from browserforge as output that is expected from AbstractFingerprintGenerator.generate().
It is explicitly mentioned there as something that is subject to change.

Why doing it like this?
This change is already usable as it is and brings most value now. Common user can use some sort of fingerprint generation offered by browserforge.
Adding our own type Fingerprint and our own injector is some additional effort and has almost no value for external users and can be part of future refactoring. This refactoring would affect only very advanced users that created their own custom fingerprint generator (quite advanced for experimental feature that is not yet even available. So purely theoretical.)

Issues

Will be documented later by Create a new guide about how to not get blocked #481 , which already has decent PR docs: Add a new guide on how to avoid getting blocked #576
Closes: Add init script for Playwright browser context #549

TODO: Tests, JS page function for injecting fingerprint json.

Todo: Make it work with page.add_init_script

Added dev test TODO: Add proper tests Sync header generation with fingerprint generation (use gen from browserforge)

Added more tests.

Use browser_pool_options from crawler to pass fingerprint related stuff to be similar to JS

barjin · 2024-12-18T15:18:48Z

Good job! Why don't we use browserforge's injector module for this? Do we need more control over the fingerprint than browserforge can offer us?

src/crawlee/playwright_crawler/_playwright_crawler.py

Pijukatel · 2024-12-18T15:46:58Z

Good job! Why don't we use browserforge's injector module for this? Do we need more control over the fingerprint than browserforge can offer us?

I was not aware of that. Wow, that guy did everything. I will try to use that injector,

Pijukatel · 2024-12-19T09:42:02Z

src/crawlee/browsers/_playwright_browser_controller.py

@@ -152,8 +143,45 @@ async def _create_browser_context(self, proxy_info: ProxyInfo | None = None) ->
            else None
        )

-        return await self._browser.new_context(
+        if self._use_fingerprints:


(This can be simpler if we decide to rely purely on browserforge and remove our own header generation. )

src/crawlee/browsers/_playwright_browser_controller.py

…rge-fingerprints

Draft adapter and basic test.

TODO: Solve circular imports

Remove types unused in this change. Format, lint, type check.

(Hint about browserforge being just implementation detail and not core functionality.)

vdusek · 2025-01-17T09:30:59Z

How can users use it?

Why not expose the entire class directly, instead of creating an additional one just for the options and exposing that?

fingerprint_generator = FingerprintGenerator(
    header_generator=HeaderGenerator(...),
    screen=Screen(...),
)

crawler = PlaywrightCrawler(
    headless=True,
    fingerprint_generator=fingerprint_generator,
)

Keep our own header generation or rely on browserforge?

I would completely remove our implementation of HeaderGenerator and rely on the one from browserforge to avoid mixing the two.

Pijukatel · 2025-01-20T07:27:01Z

How can users use it?

Why not expose the entire class directly, instead of creating an additional one just for the options and exposing that?

At least for the HeaderGenaratorOptions that I think is necessary to avoid users mixing various header generators with fingerprint generators. HeaderGenerator is internal to FingerprintGenerator to ensure that generated fingerprint and headers are consistent.

FingerprintGeneratorOptions can be removed and the inputs defined in abstract class instead.

Pijukatel · 2025-01-20T12:49:52Z

Keep our own header generation or rely on browserforge?

I would completely remove our implementation of HeaderGenerator and rely on the one from browserforge to avoid mixing the two.

I agree, but that step I would rather take in separate follow up PR to not make this PR too large.

…rge-fingerprints

janbuchar

I don't know this part of the codebase that well, so I found just some minor design nitpicks

src/crawlee/fingerprint_suite/_fingerprint_generator.py

janbuchar

Looks good to me, but I'd prefer to wait for @vdusek's approval as well

vdusek · 2025-01-24T11:13:04Z

I agree, but that step I would rather take in separate follow up PR to not make this PR too large.

Do I understand it correctly that it will not require any changes on the interface?

Also, could you open an issue for it?

vdusek

Will be documented later by #481 , which already has decent PR #576

I would prefer to include the documentation along with this PR. It could provide a clearer understanding and demonstrate how it is being used. This is one of the reasons we keep the documentation in the same repository as the source code.

Otherwise it's great.

src/crawlee/fingerprint_suite/_fingerprint_generator.py

src/crawlee/browsers/_browser_pool.py

src/crawlee/browsers/_playwright_browser_controller.py

src/crawlee/browsers/_playwright_browser_plugin.py

vdusek · 2025-01-24T11:19:04Z

src/crawlee/fingerprint_suite/_fingerprint_generator.py

+    from browserforge.fingerprints import Fingerprint
+
+
+class FingerprintGenerator(ABC):


Missing class docstring? It should explain what it is and what it is for.

vdusek · 2025-01-24T11:19:08Z

src/crawlee/fingerprint_suite/_browserforge_adapter.py

+    from ._types import HeaderGeneratorOptions, ScreenOptions
+
+
+class BrowserforgeFingerprintGenerator(FingerprintGenerator):


Missing class docstring?

Co-authored-by: Vlada Dusek <[email protected]>

Pijukatel · 2025-01-27T07:43:33Z

I agree, but that step I would rather take in separate follow up PR to not make this PR too large.

Do I understand it correctly that it will not require any changes on the interface?

Also, could you open an issue for it?

Yes, I think it should be without any interface changes.
#937

Update docstrings. Add example code + doc page.

(To avoid some import race conditions downloads when running pytest with multiple processes)

Pijukatel · 2025-01-27T09:36:43Z

Will be documented later by #481 , which already has decent PR #576

I would prefer to include the documentation along with this PR. It could provide a clearer understanding and demonstrate how it is being used. This is one of the reasons we keep the documentation in the same repository as the source code.

Otherwise it's great.

I added example code + page.

I guess I should also make Python variant of this doc in other repo: https://docs.apify.com/academy/anti-scraping/mitigation/generating-fingerprints

Pijukatel added 6 commits December 16, 2024 14:50

Draft of integration of browserforge fingerprint generation.

b0d52f2

TODO: Tests, JS page function for injecting fingerprint json.

Works with page.evaluate.

be15847

Todo: Make it work with page.add_init_script

Use add_init_script

a9415ec

Added dev test TODO: Add proper tests Sync header generation with fingerprint generation (use gen from browserforge)

WIP

36727a1

Added more tests.

Fix format, type check and tests.

42eff80

Fix rootcause for flakiness in fingerprint generation

998cbb6

Use browser_pool_options from crawler to pass fingerprint related stuff to be similar to JS

Pijukatel added enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team. labels Dec 18, 2024

Pijukatel requested review from barjin, vdusek and janbuchar December 18, 2024 14:34

barjin reviewed Dec 18, 2024

View reviewed changes

src/crawlee/playwright_crawler/_playwright_crawler.py Outdated Show resolved Hide resolved

Use browserforge.injector code for fingerprints

e1025c8

Pijukatel changed the title ~~Integrate browserforge fingerprints~~ feat: Integrate browserforge fingerprints Dec 19, 2024

Pijukatel commented Dec 19, 2024

View reviewed changes

src/crawlee/browsers/_playwright_browser_controller.py Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/master' into integrate-browserfo…

33fdd6e

…rge-fingerprints

github-actions bot assigned Pijukatel Dec 19, 2024

github-actions bot added this to the 105th sprint - Tooling team milestone Dec 19, 2024

github-actions bot added the tested Temporary label used only programatically for some analytics. label Dec 19, 2024

Pijukatel and others added 8 commits December 19, 2024 11:02

Regenerate poetry lock after merge

85ba877

Remove unintentional change to headless test

6e35c1d

Merge branch 'master' into integrate-browserforge-fingerprints

3f96456

chore: revert React version bump

ddfabea

Merge remote-tracking branch 'origin/master' into integrate-browserfo…

3d37bca

…rge-fingerprints

Add ScreenFingerprint and NavigatorFingerprint

1b8e6a3

Add Fingerprint and their options types

9828a36

Draft adapter and basic test.

Add adapter tests

f733c07

Pijukatel added 5 commits January 13, 2025 17:33

Integrate into pw_crawler

97011d9

TODO: Solve circular imports

Further integration into our code.

debe900

Finalize draft.

3d8340c

Remove types unused in this change. Format, lint, type check.

Set fiongerprint generator as top level argument to pw crawler

3d9b170

Revert unnecessary change to function doc string.

25aa4e2

Pijukatel force-pushed the integrate-browserforge-fingerprints branch from 7bd985c to 25aa4e2 Compare January 14, 2025 10:13

Pijukatel added 2 commits January 14, 2025 11:26

Make test adapter-generic.

5e46b78

(Hint about browserforge being just implementation detail and not core functionality.)

Add types to __init__ if fingerprint_suite

69b6974

Remove FingerprintGeneratorOptions

27479be

Merge remote-tracking branch 'origin/master' into integrate-browserfo…

751f67c

…rge-fingerprints

Pijukatel marked this pull request as ready for review January 23, 2025 08:20

janbuchar reviewed Jan 23, 2025

View reviewed changes

src/crawlee/fingerprint_suite/_fingerprint_generator.py Outdated Show resolved Hide resolved

src/crawlee/fingerprint_suite/_fingerprint_generator.py Outdated Show resolved Hide resolved

Review commnets

1cbadb0

Pijukatel requested a review from janbuchar January 23, 2025 13:20

janbuchar approved these changes Jan 23, 2025

View reviewed changes

Handle inconsistent result from browserforge fingerprint generator

8e44acd

vdusek reviewed Jan 24, 2025

View reviewed changes

Apply suggestions from code review

d8001e7

Co-authored-by: Vlada Dusek <[email protected]>

Pijukatel added 2 commits January 27, 2025 10:04

Docs

07acbfa

Update docstrings. Add example code + doc page.

Make sure browserforge files are downloaded before tests.

866fe98

(To avoid some import race conditions downloads when running pytest with multiple processes)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Integrate browserforge fingerprints #829

feat: Integrate browserforge fingerprints #829

Pijukatel commented Dec 18, 2024 •

edited

Loading

barjin commented Dec 18, 2024

Pijukatel commented Dec 18, 2024

Pijukatel Dec 19, 2024

vdusek commented Jan 17, 2025 •

edited

Loading

Pijukatel commented Jan 20, 2025

Pijukatel commented Jan 20, 2025

janbuchar left a comment

janbuchar left a comment

vdusek commented Jan 24, 2025

vdusek left a comment •

edited

Loading

vdusek Jan 24, 2025

Pijukatel Jan 27, 2025

vdusek Jan 24, 2025

Pijukatel Jan 27, 2025

Pijukatel commented Jan 27, 2025

Pijukatel commented Jan 27, 2025

		from browserforge.fingerprints import Fingerprint


		class FingerprintGenerator(ABC):

		from ._types import HeaderGeneratorOptions, ScreenOptions


		class BrowserforgeFingerprintGenerator(FingerprintGenerator):

feat: Integrate browserforge fingerprints #829

Are you sure you want to change the base?

feat: Integrate browserforge fingerprints #829

Conversation

Pijukatel commented Dec 18, 2024 • edited Loading

Description

Issues

barjin commented Dec 18, 2024

Pijukatel commented Dec 18, 2024

Pijukatel Dec 19, 2024

Choose a reason for hiding this comment

vdusek commented Jan 17, 2025 • edited Loading

Pijukatel commented Jan 20, 2025

Pijukatel commented Jan 20, 2025

janbuchar left a comment

Choose a reason for hiding this comment

janbuchar left a comment

Choose a reason for hiding this comment

vdusek commented Jan 24, 2025

vdusek left a comment • edited Loading

Choose a reason for hiding this comment

vdusek Jan 24, 2025

Choose a reason for hiding this comment

Pijukatel Jan 27, 2025

Choose a reason for hiding this comment

vdusek Jan 24, 2025

Choose a reason for hiding this comment

Pijukatel Jan 27, 2025

Choose a reason for hiding this comment

Pijukatel commented Jan 27, 2025

Pijukatel commented Jan 27, 2025

Pijukatel commented Dec 18, 2024 •

edited

Loading

vdusek commented Jan 17, 2025 •

edited

Loading

vdusek left a comment •

edited

Loading