README

================================================================================
TurboParser -- Dependency Parser with Linear Programming Relaxations.
Version 2.2.x

Written and maintained by André Martins (afm [at] cs.cmu.edu).
================================================================================

This file is part of TurboParser, a project started at the computational
linguistics research group, ARK (http://www.ark.cs.cmu.edu/), in Carnegie Mellon 
University.

This package contains a C++ implementation of the dependency parsers described 
in:

[1] André F. T. Martins, Noah A. Smith, and Eric P. Xing. 2009.
Concise Integer Linear Programming Formulations for Dependency Parsing.
In Annual Meeting of the Association for Computational Linguistics (ACL).

[2] André F. T. Martins, Noah A. Smith, and Eric P. Xing. 2009.
Polyhedral Outer Approximations with Application to Natural Language Parsing.
In International Conference on Machine Learning (ICML).

[3] André F. T. Martins, Noah A. Smith, Eric P. Xing, Pedro M. Q. Aguiar, and 
Mário A. T. Figueiredo. 2010.
TurboParsers: Dependency Parsing by Approximate Variational Inference.
In Empirical Methods in Natural Language Processing (EMNLP).

[4] André F. T. Martins, Noah A. Smith, Mário A. T. Figueiredo, 
Pedro M. Q. Aguiar. 2011.
Dual Decomposition With Many Overlapping Components.
In Empirical Methods in Natural Language Processing (EMNLP).

[5] André F. T. Martins, Miguel B. Almeida, Noah A. Smith. 2013.
Turning on the Turbo: Fast Third-Order Non-Projective Turbo Parsers.
In Annual Meeting of the Association for Computational Linguistics (ACL).

[6] André F. T. Martins and Mariana S. C. Almeida. 2014.
Priberam: A Turbo Semantic Parser with Second Order Features.
In International Workshop on Semantic Evaluation (SemEval), task 8:
Broad-Coverage Semantic Dependency Parsing. 

This package allows:
- learning the parser from a treebank,
- run the parser on new data,
- evaluate the results against a gold-standard.

This software has the following external dependencies: AD3, a library for
approximate MAP inference (http://www.ark.cs.cmu.edu/AD3/); Eigen, a template
library for linear algebra; glog, a library for logging; gflags, a library
for commandline flag processing. All these libraries are free software and are 
provided as tarballs in this package.

This package has been tested in several Linux platforms. It has also
successfully compiled in Mac OS X and MS Windows (using MSVC). 

Since version 2.2.x, the following is also provided:
- a Python wrapper for the tagger and parser (requires Cython 0.19);
- a semantic role labeler (TurboSemanticParser) implementing ref. [6] above.

If there are any problems running the parser please email: afm [at] cs.cmu.edu
I will only respond to questions not answered in this README.

We would like to thank Ryan McDonald and Jason Baldridge by MSTParser 
(available at http://sourceforge.net/projects/mstparser), in which the code in 
this package was partly based. 

================================================================================

TurboParser is free software; you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your
option) any later version.

TurboParser is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License
for more details.

You should have received a copy of the GNU Lesser General Public License
along with this program; if not, write to the Free Software Foundation, Inc., 
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.

================================================================================
Contents
================================================================================

1. Compiling

2. Example of usage
   a. TurboParser
   b. TurboTagger
   c. Scripts

3. Running the parser
   a. Input data format
   b. Training the parser
   c. Training the tagger
   d. Running the trained tagger/parser on new data
   e. Additional options

4. Installing the Python wrapper

5. Memory/Disk space and performance issues

6. Reproducing results in the ICML, ACL, and EMNLP papers

7. Reproducing results in the SemEval 2014 paper (TurboSemanticParser)


================================================================================
1. Compiling
================================================================================

To compile the code, first unzip/tar the downloaded file:

> tar -zxvf TurboParser-2.2.0.tar.gz
> cd TurboParser-2.2.0

Next, run the following command

> ./install_deps.sh

This will install all the dependencies (libraries gflags, glog, Eigen, and AD3).
Finally, type

> ./configure && make && make install

After these steps, a file named "TurboParser" and another named "TurboTagger" 
should have been created under the working folder.

Before starting to use TurboParser and TurboTagger, we need to add our local 
dependencies to the library path. This can be done via:

> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:`pwd;`/deps/local/lib:"


================================================================================
2. Example Usage
================================================================================

The directory data/sample contains small samples of training and testing  data. 
The data format is the one used in the CoNLL-X shared task, which we describe 
in the next section. The following sample files are provided:

sample_train.conll
sample_test.conll


================================================================================
2a. TurboParser
================================================================================

Before starting, we need to add our local dependencies to the library path:

> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:`pwd;`/deps/local/lib:"

These steps will train a parser on the training data, run it on the testing 
data, and evaluate the output against the gold standard:

> mkdir models

> ./TurboParser --train \
--file_train=data/sample/sample_train.conll \
--file_model=models/sample_parser.model \
--logtostderr

> ./TurboParser --test \
--evaluate \
--file_model=models/sample_parser.model \
--file_test=data/sample/sample_test.conll \
--file_prediction=data/sample/sample_test.conll.predicted \
--logtostderr

The results from running the parser are in the file 
data/samples/sample_test.conll.predicted and the trained model in 
models/sample_parser.model.


================================================================================
2b. TurboTagger
================================================================================

If you have not done this yet, add your local dependencies to the library path:

> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:`pwd;`/deps/local/lib:"

The input files in TurboTagger are not CoNLL files; they have the same tabular 
form, but should only have two columns, the first one for the words and the 
second one for the parts-of-speech. 
 
To test TurboTagger, run first the following script to convert the sample files
to this format:

> ./scripts/create_tagging_corpus.sh data/sample/sample_train.conll
> ./scripts/create_tagging_corpus.sh data/sample/sample_test.conll

This will create files sample_train.conll.tagging and sample_test.conll.tagging.
Then, run:

> mkdir models

> ./TurboTagger --train \
--file_train=data/sample/sample_train.conll.tagging \
--file_model=models/sample_tagger.model \
--form_cutoff=1 \
--logtostderr

> ./TurboTagger --test \
--evaluate \
--file_model=models/sample_tagger.model \
--file_test=data/sample/sample_test.conll.tagging \
--file_prediction=data/sample/sample_test.conll.tagging.predicted \
--logtostderr

The results from running the tagger are in the file 
data/sample/sample_test.conll.tagging.predicted and the trained model in 
models/sample_tagger.model.


================================================================================
2c. Scripts
================================================================================

Shell scripts are provided in the folder ./scripts that allow you to train, 
test, and evaluate the parser and the tagger with several options. 

If you type:

> cd scripts
> ./train_test_parser.sh sample
> ./train_test_tagger.sh sample

You will perform all the operations described above (the results are not
necessarily the same, since some parameter settings in the scripts may be
different).
 
We suggest you to look at these scripts and to edit them at your own needs.


================================================================================
3. Running the Parser
================================================================================

================================================================================
3a. Input data format
================================================================================

The data format is the same as the CONLL-X shared task. 
Here is a sample of two sentences from the Dutch dataset:

1   Cathy             Cathy             N     N     eigen|ev|neut                    2   su      _  _
2   zag               zie               V     V     trans|ovt|1of2of3|ev             0   ROOT    _  _
3   hen               hen               Pron  Pron  per|3|mv|datofacc                2   obj1    _  _
4   wild              wild              Adj   Adj   attr|stell|onverv                5   mod     _  _
5   zwaaien           zwaai             N     N     soort|mv|neut                    2   vc      _  _
6   .                 .                 Punc  Punc  punt                             5   punct   _  _

1   Ze                ze                Pron  Pron  per|3|evofmv|nom                 2   su      _  _
2   had               heb               V     V     trans|ovt|1of2of3|ev             0   ROOT    _  _
3   met               met               Prep  Prep  voor                             8   mod     _  _
4   haar              haar              Pron  Pron  bez|3|ev|neut|attr               5   det     _  _
5   moeder            moeder            N     N     soort|ev|neut                    3   obj1    _  _
6   kunnen            kan               V     V     hulp|ott|1of2of3|mv              2   vc      _  _
7   gaan              ga                V     V     hulp|inf                         6   vc      _  _
8   winkelen          winkel            V     V     intrans|inf                      11  cnj     _  _
9   ,                 ,                 Punc  Punc  komma                            8   punct   _  _
10  zwemmen           zwem              V     V     intrans|inf                      11  cnj     _  _
11  of                of                Conj  Conj  neven                            7   vc      _  _
12  terrassen         terras            N     N     soort|mv|neut                    11  cnj     _  _
13  .                 .                 Punc  Punc  punt                             12  punct   _  _

Please go to http://nextens.uvt.nl/~conll/#dataformat for more information 
about the meaning of each column field. 
The parser should still learn/run if some of the fields (like the FEATS field) 
are filled with "_".

Note: Some datasets are available at http://ilk.uvt.nl/conll/free_data.html.


================================================================================
3b. Training the parser
================================================================================

If you have a set of labeled data, first place it in the format described
above. For example, if your training data is in a file 
data/danish/danish.conll, and you want the model to be saved in a file called 
models/danish/danish_parser.model, then the parser can be trained by running 
the command:

> ./TurboParser --train \
--file_train=data/danish/danish_train.conll \
--file_model=models/danish_parser.model \
--logtostderr

This will train a second-order non-projective parser with features for arcs,
consecutive siblings and grandparents, using the AD3 algorithm as a decoder. 
The default training algorithm is (cost-augmented) MIRA, but there are other 
options (see section 3e below). It will also train a probabilistic model for 
unlabeled arc-factored pruning, which is employed to reduce the number of 
candidate arcs and speed up parsing.

If speed is more important than accuracy, you might want to use a simple 
arc-factored model, in which case a first order pruner is not necessary. 
You can set the flag --prune_basic=false to disable the first order pruner, and
--model_type=basic to use a simple arc-factored model. The command line is:

> ./TurboParser --train \
--file_train=data/danish/danish_train.conll \
--file_model=models/danish_parser.model \
--prune_basic=false \
--model_type=basic \
--logtostderr

In general, the --model_type flag is a string formed by the one or several of 
the following pieces: af enables arc-factored parts (required), +cs enables
consecutive sibling parts, +gp enables grandparent parts, +as enables
arbitrary sibling parts, +np enables non-projectivity parts, +dp enables
directed path parts, +hb enables head bigram parts, +gs enables grand-sibling
(third-order) parts, +ts enables tri-sibling (third-order) parts. The following
alias are predefined: "basic" is af, "standard" is af+cs+gp, "full" is
af+cs+gp+as+hb+gs+ts. Default is "standard".

In any case, after training ends, a model file "models/danish_parser.model" will
be created. To run the parser on new data (e.g. a file 
"data/danish/danish_test.conll"), issue the following command:

> ./TurboParser --test \
--evaluate \
--file_model=models/danish_parser.model \
--file_test=data/danish/danish_test.conll \
--file_prediction=data/danish/danish_test.conll.predicted \
--logtostderr

This will create a file "data/danish/danish_test.conll.predicted" with the 
predictions made by the parser, and will output some accuracy and speed metrics.

NOTE: the output is unlabeled accuracy (including all punctuation).

To ignore punctuation, which is standard for English (Yamada and Matsumoto 03), 
as well as the metric used in references [1] and [2] above, we provide 
in this package the PERL script eval.pl, used for evaluation at CONLL-X. 
Just run:

> scripts/eval.pl -b -q -g data/danish/danish_test.conll \
  -s data/danish/danish_test.conll.predicted | tail -5

This script also allows performing significance tests (see 
http://nextens.uvt.nl/~conll/software.html for details).

================================================================================
3c. Training the tagger
================================================================================

To train the tagger, one first needs to have input files with two columns, one
with words and the other with POS tags. Here is an example for the English
Penn Treebank corpus:

Ms.     NNP
Haag    NNP
plays   VBZ
Elianti NNP
.       .

Rolls-Royce     NNP
Motor   NNP
Cars    NNPS
Inc.    NNP
said    VBD
it      PRP
expects VBZ
its     PRP$
U.S.    NNP
sales   NNS
to      TO
remain  VB
steady  JJ
at      IN
about   IN
1,200   CD
cars    NNS
in      IN
1990    CD
.       .

Note: we include a script in this package (create_tagging_corpus.sh) to extract 
this information from CoNLL files. See section 2b above. 

Let english_ptb_train.tagging and english_ptb_test.tagging be the training and 
test files in the format above. Run: 

> ./TurboTagger --train \
--file_train=data/english_ptb/english_ptb_train.tagging \
--file_model=models/english_ptb_tagger.model \
--form_cutoff=1 \
--logtostderr

This will train a sequential trigram model for POS tagging. The default training 
algorithm is (cost-augmented) MIRA, but there are other options (see section 
3e below). For the sections 02-21 of the Penn Treebank, training will take only
about 5 minutes.

Note the flag --form_cutoff=1, which defines a cut-off on word frequency. 
Without setting this flag, performance will be substantially lower.

After training ends, a model file "models/english_ptb_tagger.model" will
be created. To run the tagger on new data (e.g. a file 
"data/english_ptb/english_ptb_test.tagging"), issue the following command:

> ./TurboTagger --test \
--evaluate \
--file_model=models/english_ptb_tagger.model \
--file_test=data/english_ptb/english_ptb_test.tagging \
--file_prediction=data/english_ptb/english_ptb_test.tagging.predicted \
--logtostderr

This will create a file "data/english_ptb/english_ptb_test.conll.predicted" with 
the predictions made by the tagger, and will output some accuracy and speed 
metrics. 

When training on 02-21 of the Penn Treebank, TurboTagger achieves an accuracy
of 96.9% in section 22 and 97.2% in section 23. Speed is about 40,000 tokens
per second, on a desktop PC with Intel Core i7 CPU @3.4 GHz and 8GB RAM.

NOTE: A script is included in this package for creating CoNLL files with the 
tags predicted by TurboTagger (rather than gold tags). This requires the 
original CoNLL file (assumed here to be called 
data/english_ptb/english_ptb_test.conll). To run this script, do
the following:

> cd scripts
> ./create_conll_predicted_tags_corpus.sh \
  ../data/english_ptb/english_ptb_test.conll \
  ../data/english_ptb/english_ptb_test.tagging.predicted

This will create the desired CoNLL file as 
"../data/english_ptb/english_ptb_test.conll.predpos".


================================================================================
3d. Running the trained tagger/parser on new data
================================================================================

This section assumes you have trained a tagger/parser model and it is stored in 
a file named models/my_parser.model (or models/my_tagger.model). 

First, format your data properly (section 3a).

It should be noted that the parser assumes words, lemmas, POS tags, etc. 
To generate POS tags for your data you can use TurboTagger, as described above,
or another tagger of your choice. If no lemmatizer is available, just let 
the LEMMA columns filled with "_", and the same for the morpho-syntactic 
features. However, remember that the training and test data should be generated 
the same way.

The parser also assumes that the arc label and parent index lines are
in the input. However, these can just be artificially inserted (e.g. with lines
of "LAB ... LAB" and "0 ... 0") since the parser will produce these lines
as output. The same strategy can be used for the tagger.

Hereafter, assume that the data is in a file called 
"data/my_language/my_corpus.tagging" (for the tagger) or
"data/my_language/my_corpus.conll" (for the parser). Assume also that you want 
to store the tagged sentences in a file named 
"data/my_language/my_corpus.tagging.predicted" and the parsed sentences in a 
file named "data/my_language/my_corpus.conll.predicted".

To tag, you need to run:

./TurboTagger --test \
--file_model=models/my_tagger.model \
--file_test=data/my_language/my_corpus.tagging \
--file_prediction=data/my_language/my_corpus.tagging.predicted \
--logtostderr

To parse, you need to run:

./TurboParser --test \
--file_model=models/my_parser.model \
--file_test=data/my_language/my_corpus.conll \
--file_prediction=data/my_language/my_corpus.conll.predicted \
--logtostderr

Pre-trained  models for English (both taggers and parsers) can be downloaded at 
http://www.ark.cs.cmu.edu/TurboParser/. 

A script "parse.sh" is provided in this package that allows you to tag and parse 
free text (in English, one sentence per line) with these models:

> ./scripts/parse.sh <filename>

where filename is a text file with one sentence per line. If no filename is 
specified, it parses stdin, so e.g.

> echo "I solved the problem with statistics." | ./scripts/parse.sh

yields

1    I    _    PRP    PRP    _    2    SUB
2    solved    _    VB    VBD    _    0    ROOT
3    the    _    DT    DT    _    4    NMOD
4    problem    _    NN    NN    _    2    OBJ
5    with    _    IN    IN    _    2    VMOD
6    statistics    _    NN    NNS    _    5    PMOD
7    .    _    .    .    _    2    P


================================================================================
3e. Additional Options
================================================================================

For additional options, type:

> TurboParser --help

and

> TurboTagger --help


================================================================================
4. Installing the Python wrapper
================================================================================

A Python wrapper for the tagger and parser is provided in the folder "python".
For installation instructions, see the README file in that folder.


================================================================================
5. Memory/Disk space and performance issues
================================================================================

This parser is memory and disk space intensive.

MEMORY ISSUES

To train a large dataset (like the Penn Treebank for English), a machine with 
at least 4Gb RAM is recommended.

DISK ISSUES

For English, using the Penn Treebank, model files are typically ~1GB in size.
Some cutoff options are available that can reduce the vocabulary size (type
"TurboParser --help").

PERFORMANCE ISSUES

Once a model has been trained, running the model on new data is pretty quick.
However, as with all discriminative trained parsers, it does take some time 
to train a parser (e.g. training on the Penn Treebank 02-21 with the default 
settings takes about 7-8 hours). This time is shortened to 1 hour using an 
arc-factored model.

Here is a quick comparison of parser models and rough speeds. This is when 
trained on 02-21 of the Penn Treebank; accuracies are unlabeled attachment 
scores (without punctuation) in section 23. Speeds were measured in a desktop PC 
with Intel Core i7 CPU 3.4 GHz and 8GB RAM.

- basic (arc-factored): 4,058 tokens per second, accuracy 91.05%.
- standard (af+cs+gp): 1,414 tokens per second, accuracy 92.81%.
- full (af+cs+gp+hb+as+gs+ts): 735 tokens per second, accuracy 93.07%.

================================================================================
6. Reproducing results in the ICML, ACL, and EMNLP papers
================================================================================

TurboParser 2.2.0 should reproduce the results in reference [5] above.
This software won't allow to reproduce exactly the results in the references 
[1,2,3,4] above, because a complete refactoring was made, and a different set 
of features is used.

================================================================================
7. Reproducing results in the SemEval 2014 paper (TurboSemanticParser)
================================================================================

Instructions for reproducing the results in [6] are in the README file in 
folder semeval2014_data.