More questions about user_data_dir

[remcovos]

To preface - I'm entirely new to webscraping. I've built a basic scraper in a little over a week, capable of scraping dynamically generated (and updated) data using selenium. I've been steadily adding more and more datasources, until I ran into cloudflare's bot-protection. I've started migrating over to seleniumBase, so I can use Driver(uc=True).

The issue I run into is the cookiejar. The page loads succesfully, but it loads a cookie-message that needs to be addressed before page-data is loaded. Once addressed (manually or automatically) it takes roughly five seconds to load the content, whereas opening the next page of the same website takes less than a second. Because of this reason, I want my cookies to be stored in a user-profile so that I can start my session with the cookies accepted, lowering the time it takes to scrape all data significantly.

This earlier question put me on the right track. In Selenium I just used my own chrome-profile because I care less about being undetected, and more about getting the data fast. I now know I need a UC-generated profile. I tried defining a non-existant dir in my project as the u_d_d, but this would open a browser session and just sit there - not continueing to open the provided url(s).
This question is essentially the same question I have (I think?), but didn't explain what I need to do in order to get what I need.

To summarize - if I have read up correctly I need a UC-compatible user_data_dir to reference a UC-compatible chromeprofile if I want to be able to save the cookies between scraping-cycles. Because I can't find any (step-by-step) instructions on how to create either the u_d_d or the chromeprofile, I've been stuck running in circles between the same posts here and over at SO. Any help would be greatly appreciated.

Kind Regards

[mdmintz]

The user_data_dir will be automatically created if you set one and it doesn't already exist.
For UC Mode, make sure that UC Mode creates that directory (because one that wasn't created with UC Mode will be problematic).

Don't try to set the Profile Dir, which is different from the User Data Dir. The "Default" profile will be used automatically (it's a folder inside the User Data Dir).

[mdmintz]

What happens when you run the following script (exactly as it is) a few times?

from seleniumbase import Driver

driver = Driver(uc=True, user_data_dir="my_data_dir")
try:
    driver.open("seleniumbase.io/demo_page")
    driver.highlight("h2")
    driver.type("#myTextInput", "Automation")
    driver.click("#checkBox1")
    driver.highlight("img", loops=6)
finally:
    driver.quit()

[remcovos]

I figured it out. It wasn't my script or the u_d_d-declaration - I use chrome as my main browser. So I've had it open for the entirety of the time I have trying to figure this out - googling what the issue could be. I just accidentally closed my browser instead of minimizing it, and it ran as expected. I just got an error chrome can't write to it's directory './profiles', but I'll figure that one out.

Is this standard behaviour or something I can work around? I'd rather not have to swap browsers because I want to run my scraper every 15-30 minutes ..

[mdmintz]

The UC Mode browser isn't going to be the same as your regular one.
SeleniumBase opens a new browser each time you initialize Driver()

[remcovos]

Yes it opens a new browser, but it doens't actually do anything with that browser if I had an instance of Chrome already running. Is there a way it can function without me having to close all my open instances of Chrome?

[mdmintz]

I'm not seeing your issue. (I run SeleniumBase browsers all the time while I have a regular Chrome open.)

This is something you'll need to debug on your own. If that means closing your other Chrome instances first, then that's what it means for your situation unless you can figure out what's happening.