Refactor Github Scraper Python to TypeScript (GSoC 2024 Mid-Term Evaluation)

[dgparmar14]

Description

GSoC Mid-Term Evaluation: Refactoring Scraper from Python to TypeScript

This task involves refactoring the existing Python-based scraper into TypeScript. The transition aims to enhance the codebase by introducing type safety and leveraging the capabilities of Octokit for more efficient GitHub interactions.

Fixes: #212

Week 1:

1. Refactor Github.py to Github.ts: Convert all scraper functionalities from the Python file to TypeScript.
2. Define Types: Define all necessary types for the TypeScript conversion.

Week 2:

1. Modularize Scraper: Break down the github.ts file into different modules for improved readability.

2. File Structure and Features:

   • index.ts: Entry point of the scraper containing main() and scrapGithub() functions.
   • fetchEvents.ts: Fetches all GitHub events and filters out blacklisted users (configurable via the .env file).
   • parseEvents.ts: Parses the events fetched by fetchEvents.ts based on required GitHub event types.
   • fetchUserData.ts: Fetches user-related data using fetchOpenPulls() and fetchMergeEvents().
   • config.ts: Handles Octokit authentication using GITHUB_TOKEN.
   • saveData.ts: Contains the mergedData() function to merge scraped data with previous contributor data.
   • types.ts: Contains all required types.
   • utils.ts: Contains common functions like calculateTurnaroundTime(), resolveAutonomyResponsibility(), loadUserData(), and saveUserData().

Week 3:

1. GitHub Discussions: Fetch all GitHub discussions of the organization.
2. Define Types: Define types for discussions.
3. Create discussion.ts: Fetches discussions and stores them in the data/github/discussion directory as discussion.json.
4. Update scraper-dry-run.yaml: Modify the dry-run file to work with a Node.js and npm environment.
5. Testing: Create the test file github-discussion-schema.test() for discussions.

How Has This Been Tested?

The refactored scraper can be tested using the following commands:

1. Build the project: pnpm build
2. Start the project: pnpm start org_name data_dir date(format:YYYY-MM-DD) num_days
3. Run all commands at once: pnpm dev org_name data_dir date(format:YYYY-MM-DD) num_days

(Default values will be used if date (current date) and num_days (1) are not provided).

[vercel]

@dgparmar14 is attempting to deploy a commit to the Open Healthcare Network Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
leaderboard	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 8, 2024 8:01am

[rithviknishad]

LGTM

[dgparmar14]

I guess there will be no type errors in the scraper. I checked multiple time if there still let me know.