mag_sample

The present repository is an extract from a project that uses data from Microsoft Academic Graph (MAG) and ProQuest. The programs work on a remote computer that has the data stored outside of the repository. The path to the data is defined in src/analysis/setup.R and src/dataprep/helpers/variables.py.

Directory structure

src/: data preparation and linking; analyze the publication careers of scientists.
output/: destination for tables and figures generated in src/.
snapshots/: contains files to reproduce the environments (for now, yml files for conda).

Contents

directory `src/dataprep`

The pipeline.sh script calls consecutively the scripts for

setting up the sqlite database with MAG data
preparing additional tables for analysis
loading the ProQuest data into the database
linking records between MAG and ProQuest
making some reports about the data and linking quality

directory `src/analysis`

The pipeline.sh script calls consecutively the scripts for

comparing graduation trends in ProQuest data and official statistics
assess the quality of current links
figures of publication dynamics over the career
analysis of career duration and becoming an advisor later in the career
supporting scripts are in setup.R and in helpers/

Notes

See open issues for some features that are currently lacking.

The python helpers are stored in a different location and we need to come up with a clean and simple way to make them accessible across the repository.
There are some redudancies between the Rscripts in analysis and dataprep because the scripts come originally from two different repositories.