Skip to content
This repository has been archived by the owner on May 27, 2022. It is now read-only.

feat: prepare data script + pull data into google spreadsheet #7

Open
wants to merge 76 commits into
base: master
Choose a base branch
from

Conversation

lionel-nj
Copy link
Contributor

@lionel-nj lionel-nj commented Mar 11, 2021

This PR provides changes to fetch and prepare data for visualization.

Data

The rest api provided by Github is heavily leveraged to get data from Github repositories

What do we get from there?

  1. comment counts / quarter-year
  2. number of opened issues / quarter-year
  3. number of merged PRs / quarter-year
  4. open issues count
  5. open PR count

This data is gathered for each of the following repositories:

  • MobilityData/gtfs-validator
  • MobilityData/transit
  • MobilityData/gbfs
  • NABSA/gbfs
  • google/transit

How is the data transformed into information?

Item from here are counted then classified by quarter of the year.

Technical

  • Calls from Github API are asynchronous
    • Octokit is the library used to proceed: it has been developed by Github, has great documentation and mechanisms to automatically paginate over HTTP responses.
    • It is necessary to authenticate when executing the implemented scripts in order to avoid HTTP error 403
  • aggregated data is exported as metrics.json files

GCP

This PR comes along with the implementation of a pipeline using Google Cloud Platform that enables automation of data crunching.

View data

@lionel-nj lionel-nj self-assigned this Mar 11, 2021
@lionel-nj lionel-nj linked an issue Mar 11, 2021 that may be closed by this pull request
@lionel-nj lionel-nj linked an issue Mar 11, 2021 that may be closed by this pull request
3 tasks
@lionel-nj
Copy link
Contributor Author

Per discussion with @carlfredl: it would be nice to have google/transit + MobilityData/transit data aggregated all together.

@lionel-nj
Copy link
Contributor Author

@carlfredl updated this PR following your suggestions.

@maximearmstrong: the PR in question

@heidiguenin according to this PR description, what other indicators could be relevant?

@lionel-nj
Copy link
Contributor Author

Please note that nice to have items from https://github.com/MobilityData/Tasks/issues/1170 are not implemented yet.

Copy link

@maximearmstrong maximearmstrong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall great job! Just be careful with the style consistency and when possible create constants for the keys/strings that often appear in the code.

src/scripts/AggregateData.js Outdated Show resolved Hide resolved
src/scripts/AggregateData.js Outdated Show resolved Hide resolved
src/scripts/AggregateData.js Outdated Show resolved Hide resolved
src/scripts/AggregateData.js Outdated Show resolved Hide resolved
src/scripts/AggregateData.js Outdated Show resolved Hide resolved
src/scripts/FetchRawData.js Outdated Show resolved Hide resolved
- declare symbolic constants
- leverage Standard JS
@lionel-nj
Copy link
Contributor Author

Thanks @maximearmstrong for the review! In addition to local installation of Standard JS, what do you think about adding a CI step to check the code syntax? Found this tool that could help us with that.

@lionel-nj lionel-nj marked this pull request as draft April 7, 2021 19:21
@lionel-nj lionel-nj changed the title feat: prepare data script feat: prepare data script + pull data in google spreadsheet May 26, 2021
@lionel-nj lionel-nj changed the title feat: prepare data script + pull data in google spreadsheet feat: prepare data script + pull data into google spreadsheet May 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pull metrics script into spreadsheet Create node script to prepare data from GitHub:
3 participants