index.Rmd

---
title: "Text mining in R for the social sciences and digital humanities"
author: "Andreas Niekler, Gregor Wiedemann"
date: "`r format(Sys.time(), '%Y-%m-%d')`"
output: 
  html_document:
    toc: true
    theme: united
    toc_float: true
    number_sections: yes
highlight: tango
bibliography: references.bib
csl: springer.csl
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
options(width = 98)
```

# Contents

In the digital age, large collections of text are an increasingly attractive data source for analysis in the social sciences. Corpora from thousands up to several millions of retro-digitized or natively digital documents cannot be investigated with conventional, manual methods alone. (Semi-)automatic computational analysis algorithms, also known as text mining, provide interesting opportunities for social scientists to extend their toolbox.

To realize complex designs in empirical social research, scientists need basic knowledge of computational algorithms to be able to select those appropriate for their needs. Specific projects may further require certain adaptations to standard procedures, language resources or analysis workflows. Instead of relying on off-the-shelf analysis software, using script programming languages is a very powerful way to fulfill such requirements. The course teaches an overview of text mining in connection with data acquisition, preprocessing and methodological integration using the statistical programming language R (www.r-project.org).

In sessions alternating between lectures and tutorials, we teach theoretical and methodological foundations, introduce exemplary studies and get hands on programming to realize different analyses. 

# Goals

Participants will learn about opportunities and limits of text mining methods to analyze qualitative and quantitative aspects of large text collections. With example scripts provided in the programming language R, participants will learn how to realize single steps of such an analysis on a specific corpus. We cover a range of text mining methods from simple lexicometric measures such as word frequencies, key term extraction and co-occurrence analysis, to more complex machine learning approaches such as topic models and supervised text classification. The goal is to provide a broad overview of several technologies already established in social sciences. Participants will be enabled to identify their own priorities and to lay foundations for further independent studying tailored to their individual needs.

# Requirements

We expect willingness to learn about algorithmic foundations of computational and statistical text analysis technologies. For the hands-on part, we rely on scripts in the programming language R. Thus, we strongly recommend some basic knowledge of R, to successfully take part in the tutorial sessions of the course. 

If you already have a certain amount of knowledge in another programming language, learning R will be easy for you. However, since R is a statistical programming language, some of its concepts largely differ from other languages. 

For participants without basic knowledge of R, we strongly recommend to learn at least a little in preparation of the course.

For a very brief overview of common R commands see: [Tutorial_0_R-Intro.html](Tutorial_0_R-Intro.html)

# Tutorials

The course consists of 8 tutorials written in R-markdown and further described in [this paper](http://gscl2017.dfki.de/proceedings.php). 

You can use *knitr* to create the tutorial sheets as HTML notebooks from the [R-markdown source code](https://github.com/tm4ss/tm4ss.github.io).

1. Web crawling and scraping
2. Text data import in R
3. Frequency analysis
4. Key term extraction
5. Co-occurrence analysis
6. Topic models (LDA)
7. Text classification
8. Part-of-Speech tagging / Named Entity Recognition

# License

This course was created by Gregor Wiedemann and Andreas Niekler. It was freely released under GPLv3 in September 2017. If you use (parts of) it for your own teaching or analysis, please cite

```
Wiedemann, Gregor; Niekler, Andreas (2017): Hands-on: A five day text mining course for humanists and social scientists in R. Proceedings of the 1st Workshop on Teaching NLP for Digital Humanities (Teach4DH@GSCL 2017), Berlin.
```

# Session info

The tutorials load some of the following packages.

```{r packagelist, echo=T, message=F, warning=F, error=F, results='hide'}
package_list <- c(
  "dplyr",
  "ggplot2",
  "igraph",
  "irr",
  "LDAvis",
  "LiblineaR",
  "Matrix",
  "NLP",
  "openNLP",
  "openNLPdata",
  "pals",
  "png",
  "quanteda",
  "readtext",
  "reshape2",
  "rvest",
  "topicmodels",
  "tsne",
  "webdriver",
  "wordcloud",
  "wordcloud2"
)
lapply(package_list, require, character.only = TRUE)
```

These are the package versions, we used for this tutorial. 

```{r sesinfo, echo=T}
sessionInfo()
```