stopwords

Code for the paper:

M. Gerlach, H. Shi, L.A.N. Amaral: "A universal information theoretic approach to the identification of stopwords" (2019)

Structure

• data.
  Contains the different language and gene datasets
• code
  • cluster_output
    Output files from running bulk analysis of topic models and document classification in cluster_scripts/
  • cluster_scripts
    Scripts run on Quest to do bulk analysis for topic models and document classification. submitted *_jobs.py. requires running the stopword-statistics beforehand (see scripts/). also requires installation of the different topic-models (see below).
  • figures
    figures from final analysis. run notebooks in figures_notebooks/
  • figures_notebooks create the final figures contained in the manuscript.
  • output
    Output files from running the stopword statistics in scripts/
  • src
    internal source-files
  • src_tm
    external source-files from topic models used in the analysis
  • tmp
    temporary folder needed to store intermediate results when running topic models.

installation steps

• install required packages
  • python (3.6.3)
  • gensim (0.13.1) for ldavb topic model
  • pandas (0.24.2)
  • jupyter (1.0.0)
  • scikit-learn (0.21.2)
• configure the topic models (code/src_tmp)
  • mallet
    $ cd code/src_tm/external/mallet-2.0.8RC3
$ ant
  • hdp
    cd code/src_tm/external/hdp-bleilab/hdp-faster
make

running

• Calculating stopword statistics of each corpus
  run the files in code/scripts/script_run-stopword-statistics_*.py. this will save statistics on stopwords (I, tfidf, ...) for each corpus in code/output
• Run topic models and document classification
  I submitted the jobs running the bulk analysis using the scripts in code/cluster_scripts/*_jobs.py to the cluster; this likely needs to be adapted if you want to run it somewhere else. The corresponding code is in code/cluster_scripts/*_code.py. The results are saved in code/cluster_output.
• make the figures
  The notebooks in code/figures_notebooks/*.ipynb use the results from the previous steps to generate the figures. they will be saved in code/figures/