Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Update package.json and add format script #1

Merged
merged 1 commit into from
Jul 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions .github/workflows/node.js.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: WebFreeze CI

on:
push:
branches: [ "main" ]
branches: ["main"]
pull_request:
branches: [ "main" ]
branches: ["main"]

jobs:
build:
Expand All @@ -15,17 +15,17 @@ jobs:
node-version: [14.x, 16.x, 18.x]

steps:
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Puppeteer dependencies
run: sudo apt-get install -y libgbm-dev
- name: Run linter
run: npm run lint || echo "Linting failed but continuing..."
- name: Run tests
run: npm test || echo "No tests specified"
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Install Puppeteer dependencies
run: sudo apt-get install -y libgbm-dev
- name: Run linter
run: npm run lint || echo "Linting failed but continuing..."
- name: Run tests
run: npm test || echo "No tests specified"
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
node_modules
32 changes: 16 additions & 16 deletions main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: WebFreeze CI

on:
push:
branches: [ "main" ]
branches: ["main"]
pull_request:
branches: [ "main" ]
branches: ["main"]

jobs:
build:
Expand All @@ -15,17 +15,17 @@ jobs:
node-version: [18.x]

steps:
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Puppeteer dependencies
run: sudo apt-get install -y libgbm-dev
- name: Run linter
run: npm run lint || echo "Linting failed but continuing..."
- name: Run tests
run: npm test || echo "No tests specified"
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: "npm"
- name: Install dependencies
run: npm ci
- name: Install Puppeteer dependencies
run: sudo apt-get install -y libgbm-dev
- name: Run linter
run: npm run lint || echo "Linting failed but continuing..."
- name: Run tests
run: npm test || echo "No tests specified"
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
"scripts": {
"start": "node webfreeze.js",
"lint": "eslint .",
"test": "echo \"No tests specified\" && exit 0"
"test": "echo \"No tests specified\" && exit 0",
"format": "prettier --write ."
},
"keywords": [
"web",
Expand Down Expand Up @@ -35,4 +36,4 @@
"url": "https://github.com/pudochu/webfreeze/issues"
},
"homepage": "https://github.com/pudochu/webfreeze#readme"
}
}
11 changes: 7 additions & 4 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,20 @@ WebFreeze is a powerful tool that allows you to capture and serve static version
cd webfreeze
```
3. Install dependencies:
```
npm install
```
```
npm install
```

## 📘 Usage

1. Open `webfreeze.js` and set your configuration:

- `SITE_URL`: The URL of the website you want to capture
- `SAVE_DIRECTORY`: The directory where files will be saved
- `isDownloadMode`: Set to `true` for initial capture, `false` for serving local files

2. Run the script:

```
npm start
```
Expand All @@ -43,7 +45,7 @@ WebFreeze is a powerful tool that allows you to capture and serve static version

## 🔧 Configuration

- `isDownloadMode`:
- `isDownloadMode`:
- `true`: Downloads and saves web resources
- `false`: Serves local files and blocks new requests

Expand All @@ -56,4 +58,5 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
Contributions, issues, and feature requests are welcome! Feel free to check [issues page](https://github.com/pudochu/webfreeze/issues).

---

Project Link: [https://github.com/pudochu/webfreeze](https://github.com/pudochu/webfreeze)
178 changes: 90 additions & 88 deletions webfreeze.js
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
/**
* WebFreeze: A tool to capture and serve static versions of dynamic web pages
*
*
* This script uses Puppeteer to capture a web page and its resources, saving them locally.
* It can then serve these local files, effectively "freezing" the web page at a specific point in time.
*/

const puppeteer = require('puppeteer');
const fs = require('fs');
const fsPromises = require('fs').promises;
const path = require('path');
const https = require('https');
const URL = require('url');
const puppeteer = require("puppeteer");
const fs = require("fs");
const fsPromises = require("fs").promises;
const path = require("path");
const https = require("https");
const URL = require("url");

// Configuration
const SITE_URL = 'https://example.com';
const SAVE_DIRECTORY = './files';
const SITE_URL = "https://example.com";
const SAVE_DIRECTORY = "./files";
const isDownloadMode = false; // true: download and save, false: use local files or block

/**
Expand All @@ -24,21 +24,23 @@ const isDownloadMode = false; // true: download and save, false: use local files
* @returns {Promise<void>}
*/
async function downloadFile(url, filePath) {
return new Promise((resolve, reject) => {
https.get(url, (response) => {
if (response.statusCode === 200) {
const fileStream = fs.createWriteStream(filePath);
response.pipe(fileStream);
fileStream.on('finish', () => {
fileStream.close();
console.log(`Downloaded: ${url}`);
resolve();
});
} else {
reject(`Failed to download ${url}: ${response.statusCode}`);
}
}).on('error', reject);
});
return new Promise((resolve, reject) => {
https
.get(url, (response) => {
if (response.statusCode === 200) {
const fileStream = fs.createWriteStream(filePath);
response.pipe(fileStream);
fileStream.on("finish", () => {
fileStream.close();
console.log(`Downloaded: ${url}`);
resolve();
});
} else {
reject(`Failed to download ${url}: ${response.statusCode}`);
}
})
.on("error", reject);
});
}

/**
Expand All @@ -47,90 +49,90 @@ async function downloadFile(url, filePath) {
* @returns {string} The local file path
*/
function getFilePath(url) {
const parsedUrl = new URL.URL(url);
let filePath = parsedUrl.pathname;
// Handle paths starting with _next
if (filePath.startsWith('/_next')) {
filePath = filePath.replace(/^\//, '');
}
return path.join(SAVE_DIRECTORY, filePath);
const parsedUrl = new URL.URL(url);
let filePath = parsedUrl.pathname;

// Handle paths starting with _next
if (filePath.startsWith("/_next")) {
filePath = filePath.replace(/^\//, "");
}

return path.join(SAVE_DIRECTORY, filePath);
}

/**
* Intercepts requests and handles them based on the current mode
* @param {puppeteer.Page} page - The Puppeteer page object
*/
async function interceptRequests(page) {
await page.setRequestInterception(true);
page.on('request', async (request) => {
const url = request.url();
const resourceType = request.resourceType();

if (['stylesheet', 'script', 'image'].includes(resourceType)) {
const filePath = getFilePath(url);
await page.setRequestInterception(true);
page.on("request", async (request) => {
const url = request.url();
const resourceType = request.resourceType();

if (isDownloadMode) {
// Download mode: Download and save files
try {
await fsPromises.access(filePath);
request.continue();
} catch (error) {
try {
const directoryPath = path.dirname(filePath);
await fsPromises.mkdir(directoryPath, { recursive: true });
await downloadFile(url, filePath);
request.continue();
} catch (downloadError) {
console.error(`Failed to download ${url}: ${downloadError}`);
request.continue();
}
}
} else {
// Block mode: Use local files or block requests
try {
await fsPromises.access(filePath);
const content = await fsPromises.readFile(filePath);
console.log(`Serving: ${filePath}`);
request.respond({
status: 200,
contentType: request.headers()['content-type'],
body: content
});
} catch (error) {
console.log(`Blocked: ${filePath}`);
request.abort();
}
}
} else {
if (["stylesheet", "script", "image"].includes(resourceType)) {
const filePath = getFilePath(url);

if (isDownloadMode) {
// Download mode: Download and save files
try {
await fsPromises.access(filePath);
request.continue();
} catch (error) {
try {
const directoryPath = path.dirname(filePath);
await fsPromises.mkdir(directoryPath, { recursive: true });
await downloadFile(url, filePath);
request.continue();
} catch (downloadError) {
console.error(`Failed to download ${url}: ${downloadError}`);
request.continue();
}
}
});
} else {
// Block mode: Use local files or block requests
try {
await fsPromises.access(filePath);
const content = await fsPromises.readFile(filePath);
console.log(`Serving: ${filePath}`);
request.respond({
status: 200,
contentType: request.headers()["content-type"],
body: content,
});
} catch (error) {
console.log(`Blocked: ${filePath}`);
request.abort();
}
}
} else {
request.continue();
}
});
}

/**
* Main function to run the script
*/
async function main() {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

await interceptRequests(page);
await interceptRequests(page);

console.log(`Navigating to ${SITE_URL}`);
await page.goto(SITE_URL, { waitUntil: 'networkidle0' });
console.log(`Navigating to ${SITE_URL}`);
await page.goto(SITE_URL, { waitUntil: "networkidle0" });

// Save the main HTML
const content = await page.content();
const htmlPath = path.join(SAVE_DIRECTORY, 'index.html');
await fsPromises.writeFile(htmlPath, content);
console.log(`Saved main HTML to ${htmlPath}`);
// Save the main HTML
const content = await page.content();
const htmlPath = path.join(SAVE_DIRECTORY, "index.html");
await fsPromises.writeFile(htmlPath, content);
console.log(`Saved main HTML to ${htmlPath}`);

// Keep the browser open for inspection
console.log('Browser will remain open. Close it manually when done.');
// Uncomment the next line to close the browser automatically
// await browser.close();
// Keep the browser open for inspection
console.log("Browser will remain open. Close it manually when done.");
// Uncomment the next line to close the browser automatically
// await browser.close();
}

main().catch(console.error);
Loading