Skip to content

Commit

Permalink
fix: deploy on Railway
Browse files Browse the repository at this point in the history
  • Loading branch information
brunosduarte committed May 2, 2024
1 parent 510ecf8 commit 4e46885
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 1 deletion.
2 changes: 1 addition & 1 deletion docs/decisions/0000-crawling.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ Crawl the entire domain and extract all the links and assets of pages

## Decision Outcome

Chosen option: "Puppeteer", because The library Puppeteer employs the concept of a headless browser to fully access and interact with both static and dynamic web pages. It automates the opening of each webpage, efficiently extracting all internal links and associated assets. To preventing automated scraping, websites may employ captchas. And one important feature of this library is that Puppeteer can solve simple captchas or utilize external captcha-solving services, making it the preferable choice over Cheerio for tasks that require more than just simple HTML parsing.
Chosen option: "Puppeteer", because The library Puppeteer employs the concept of a headless browser to fully access and interact with both static and dynamic web pages. It automates the opening of each webpage, efficiently extracting all internal links and associated assets. To preventing automated scraping, websites may employ captchas, and one important feature of this library is that Puppeteer can solve simple captchas or utilize external captcha-solving services, making it the preferable choice over Cheerio for tasks that require more than just simple HTML parsing.
7 changes: 7 additions & 0 deletions server/src/infrastructure/services/ScrapperService.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,22 @@ export class ScrapperService implements IScrapperService {
'--no-first-run',
'--no-sandbox',
'--no-zygote',
'--incognito',
'--disable-gpu',
'--disable-webgl',
'--disable-dev-shm-usage',
'--disable-setuid-sandbox',
'--disable-features=IsolateOrigins',
'--disable-site-isolation-trials',
'--disable-accelerated-2d-canvas',
'--disable-plugins',
'--disable-sync',
'--disable-extensions',
'--disable-popup-blocking',
'--disable-infobars',
'--disable-client-side-phishing-detection',
'--enable-automation',
'--ignore-certificate-errors',
'--deterministic-fetch',
'--window-size=1600,900',
],
Expand Down

1 comment on commit 4e46885

@vercel
Copy link

@vercel vercel bot commented on 4e46885 May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.