paper.tex

% File clic2015.tex
% June 2015

%% Based on the style files for CLiC-IT-2014, which were, in turn,
%% Based on the style files for ACL-2014, which were, in turn,
%% Based on the style files for ACL-2013, which were, in turn,
%% Based on the style files for ACL-2012, which were, in turn,
%% based on the style files for ACL-2011, which were, in turn, 
%% based on the style files for ACL-2010, which were, in turn, 
%% based on the style files for ACL-IJCNLP-2009, which were, in turn,
%% based on the style files for EACL-2009 and IJCNLP-2008...

%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
% File clic2015.tex
% June 2015

%% Based on the style files for CLiC-IT-2014, which were, in turn,
%% Based on the style files for ACL-2014, which were, in turn,
%% Based on the style files for ACL-2013, which were, in turn,
%% Based on the style files for ACL-2012, which were, in turn,
%% based on the style files for ACL-2011, which were, in turn, 
%% based on the style files for ACL-2010, which were, in turn, 
%% based on the style files for ACL-IJCNLP-2009, which were, in turn,
%% based on the style files for EACL-2009 and IJCNLP-2008...

%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt]{article}
\usepackage[a4paper]{geometry}
\usepackage{clic2016}
\usepackage{times}
\usepackage{url}
\usepackage{latexsym}
\usepackage{textcomp}

%\setlength\titlebox{5cm}
\setlength\titlebox{6.5cm}

% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\usepackage[noinline,nomargin,noindex,draft]{fixme}
\fxsetup{theme=color,mode=multiuser}
\FXRegisterAuthor{n}{note}{NOTE} 
%\nnote[inline]{this is how to make regular notes...}
%\nerror[margin]{} or \fxerror[inline]{reserved for final paper}

\newcommand\BibTeX{B{\sc ib}\TeX}
\usepackage{xspace}
\newcommand\wtv{\texttt{w2v}\xspace}

\newcommand{\mytitle}{bot.zen @ EVALITA 2016 - A minimally-deep learning PoS-tagger \\
    (trained for Italian Tweets)}
\newcommand{\mypdftitle}{\mytitle}
\newcommand{\mypdfsubject}{}
\newcommand{\mystitle}{bot.zen - A minimally-deep learning PoS-tagger}
\newcommand{\mykeywords}{PoS tagger, word empbeddings, word2vec, neural
    network, RNN, LSTM, EVALITA 2016, PoSTWITA}
\newcommand{\mypdfkeywords}{\mykeywords}

%%%%% hyper & options
%\definecolor{darkred}{rgb}{0.5,0,0}
%\definecolor{darkgreen}{rgb}{0,0.5,0}
%\definecolor{darkblue}{rgb}{0,0,0.5}
%
%\usepackage[final=true, pdfstartview=FitH]{hyperref}
%\hypersetup{
%    pdftitle={\mypdftitle},
%    pdfauthor={egon w. stemle <egon.stemle@eurac.edu>},
%    pdfsubject={\mypdfsubject},
%    pdfkeywords={\mypdfkeywords},
%    setpagesize=true,
%    colorlinks=true,
%    linkcolor=darkred,
%    urlcolor=darkblue,
%    citecolor=darkgreen
%}

\title{\mytitle}

\author{Egon W.~Stemle \\
    Institute for Specialised Communication and Multilingualism \\
    EURAC Research \\
    Bolzano/Bozen, Italy \\
    {\tt egon.stemle@eurac.edu}}

\date{}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\maketitle

\begin{abstract} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\textbf{English.}  
This article describes the system that participated in the \emph{POS tagging
for Italian Social Media Texts} (PoSTWITA) task of the 5\textsuperscript{th}
periodic evaluation campaign of Natural Language Processing (NLP) and speech
tools for the Italian language EVALITA 2016.

The work is a continuation of \newcite{stemle:2016:WAC-X} with minor
modifications to the system and different data sets.  
It combines a small assertion of trending techniques, which implement matured
methods, from NLP and ML to achieve competitive results on PoS tagging of
Italian Twitter texts; in particular, the system uses word embeddings and
character-level representations of word beginnings and endings in a LSTM RNN
architecture.  
Labelled data (Italian UD corpus, DiDi and PoSTWITA) and unlabbelled data
(Italian C4Corpus and PAIS{\`A}) were used for training.

The system is available under the APLv2 open-source license.
\end{abstract}

\begin{abstract-alt}
\textrm{\bf{Italiano.}}
Questo articolo descrive il sistema che ha partecipato al task \emph{POS
tagging for Italian Social Media Texts (PoST-Wita)} nell'ambito di EVALITA
2016, la 5\textdegree~campagna di valutazione periodica del Natural Language
Processing (NLP) e delle tecnologie del linguaggio.

Il lavoro {\`e} un proseguimento di quanto descritto
in~\newcite{stemle:2016:WAC-X}, con modifiche minime al sistema e insiemi di
dati differenti.
Il lavoro combina alcune tecniche correnti che implementano metodi comprovati
dell'NLP e del Machine Learning, per raggiungere risultati competitivi nel PoS
tagging dei testi italiani di Twitter.
In particolare il sistema utilizza strategie di word embedding e di
rappresentazione character-level di inizio e fine parola, in un'architettura
LSTM RNN.
Dati etichettati (Italian UD corpus, DiDi e PoSTWITA) e dati non etichettati
(Italian C4Corpus e PAIS{\`A}) sono stati utilizzati in fase di training.

Il sistema {\`e} disponibile sotto licenza open source APLv2.
\end{abstract-alt}


\section{Introduction} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:intro}

Part-of-speech (PoS) tagging is an essential processing stage for virtually all
NLP applications.
Subsequent tasks, like parsing, named-entity recognition, event
detection, and machine translation, often utilise PoS tags, and benefit
(directly or indirectly) from accurate tag sequences.

Actual work on PoS tagging, meanwhile, mainly concentrated on standardized
texts for many years, and frequent phenomena in computer-mediated communication
(CMC) and Web corpora such as emoticons, acronyms, interaction words, iteration
of letters, graphostylistics, shortenings, addressing terms, spelling
variations, and boilerplate%
~\cite{androutsopoulos2007,BernardiniBaroniEvert2008,beisswenger2013} still
deteriorate the performance of PoS-taggers%
~\cite{giesbrecht2009,baldwin-EtAl:2013:IJCNLP}.

On the other hand, the interest in automatic evaluation of social media texts,
in particular for microblogging texts such as tweets, has been growing
considerably, and specialised tools for Twitter data have become available for
different languages. 
But Italian completely lacks such resources, both regarding annotated corpora
and specific PoS-tagging tools.\footnote{http://www.evalita.it/2016/tasks/postwita}
To this end, the \emph{POS tagging for Italian Social Media Texts} (PoSTWITA)
task was proposed for EVALITA 2016 concerning the domain adaptation of
PoS-taggers to Twitter texts.

Our system combined \texttt{word2vec} (\wtv) word embeddings (WEs) with a
single-layer Long Short Term Memory (LSTM) recurrent neural network (RNN)
architecture.
The sequence of unlabelled \wtv representations of words is accompanied by
the sequence of n-grams of the word beginnings and endings, and is fed into the
RNN which in turn predicts PoS labels.

The paper is organised as follows: We present our system design in
Section~\ref{sec:design}, the implementation in
Section~\ref{sec:implementation}, and its evaluation in
Section~\ref{sec:results}. 
Section~\ref{sec:conclusion} concludes with an outlook on possible
implementation improvements.

\section{Design} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:design}

Overall, our design takes inspiration from as far back as~\newcite{Benello1989}
who used four preceding words and one following word in a feed-forward neural
network with backpropagation for PoS tagging, builds upon the strong foundation
laid down by~\newcite{collobert:2011b} for a neural network (NN) architecture
and learning algorithm that can be applied to various natural language
processing tasks, and
ultimately is a variation of
% achieved an accuracy of $0.95$ on the Brown Corpus.
~\newcite{santos2014learning} who trained a NN for PoS tagging, with 
character-level and WE representations of words. 
%On the Wall Street Journal (WSJ) portion of the Penn Treebank they achieve
%accuracy scores of $0.9732$. 

Also note that an earlier version of the system was used in
\newcite{stemle:2016:WAC-X} to participate in the \emph{EmpiriST 2015 shared
task on automatic linguistic annotation of computer-mediated communication /
social media}~\cite{empirist:2016:WAC-X}.


\subsection{Word Embeddings} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Recently, state-of-the-art results on various linguistic tasks were
accomplished by architectures using neural-network based WEs.
\newcite{baroni-dinu-kruszewski:2014:P14-1} conducted a set of
experiments comparing the popular
\wtv~\cite{DBLP:journals/corr/abs-1301-3781,arXiv:1310.4546}
implementation for creating WEs to other distributional methods with
state-of-the-art results across various (semantic) tasks. 
These results suggest that the word embeddings substantially
outperform the other architectures on semantic similarity and analogy
detection tasks.
Subsequently,~\newcite{TACL570} conducted a comprehensive set of
experiments and comparisons that suggest that much of the improved results are
due to the system design and parameter optimizations, rather than the selected
method.  
They conclude that "there does not seem to be a consistent significant
advantage to one approach over the other".

Word embeddings provide high-quality low dimensional vector representations of
words from large corpora of unlabelled data, and the representations, typically
computed using NNs, encode many linguistic regularities and
patterns~\cite{arXiv:1310.4546}.


\subsection{Character-Level Sub-Word Information} %%%%%%%%%%%%%%%%%%%%%%%%%%%%

The morphology of a word is opaque to WEs, and the relatedness of
the meaning of a lemma's different word forms, i.e.~its different string
representations, is \emph{not} systematically encoded. 
This means that in morphologically rich languages with long-tailed frequency
distributions, even some WE representations for word forms of common
lemmata may become very poor~\cite{DBLP:journals/corr/KimJSR15}.

We agree with~\newcite{santos2014learning}
and~\newcite{DBLP:journals/corr/KimJSR15} that sub-word information is very
important for PoS tagging, and therefore we augment the WE representations with
character-level representations of the word beginnings and endings; thereby, we
also stay language agnostic---at least, as much as possible---by avoiding the
need for, often language specific, morphological pre-processing.


\subsection{Recurrent Neural Network Layer} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Language Models are a central part of NLP.
They are used to place distributions over word sequences that encode systematic
structural properties of the sample of linguistic content they are built from,
and can then be used on novel content, e.g.~to rank it or predict some feature
on it. 
For a detailed overview on language modelling research see~\newcite{mikolov2012}.

A straight-forward approach to incorporate WEs into feature-based
language models is to use the embeddings' vector representations as
features.\footnote{For an overview see,
e.g.~\newcite{Turian:2010:WRS:1858681.1858721}.}
Having said that, WEs are also used in NN architectures, where they constitute
(part of) the input to the network. 

Neural networks consist of a large number of simple, highly interconnected
processing nodes in an architecture loosely inspired by the structure of the
cerebral cortex of the brain~\cite{oreilly2000}.
The nodes receive weighted inputs through these connections and \emph{fire}
according to their individual thresholds of their shared activation function.
A firing node passes on an activation to all successive connected nodes.
During learning the input is propagated through the network and the output is
compared to the desired output. 
Then, the weights of the connections (and the thresholds) are adjusted
step-wise so as to more closely resemble a configuration that would produce the
desired output.
After all input cases have been presented, the process typically starts over
again, and the output values will usually be closer to the correct values.

RNNs are NNs where the connections between the elements are directed cycles,
i.e.~the networks have loops, and this enables them to model sequential
dependencies of the input.
However, regular RNNs have fundamental difficulties learning long-term
dependencies, and special kinds of RNNs need to be used~\cite{Hochreiter1991}; 
a very popular kind is the so called long short-term memory (LSTM)
network proposed by~\newcite{Hochreiter:1997:LSM:1246443.1246450}.

Overall, with this design we not only benefit from available labelled data but
also from available general or domain-specific unlabelled data.

\section{Implementation} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:implementation}

We maintain the implementation in a source code repository at
\url{https://github.com/bot-zen/}.  
The version tagged as {\tt 1.1} comprises the version that was used to generate
the results submitted to the shared task (ST).

Our system feeds WEs and character-level sub-word information into a
single-layer RNN with a LSTM architecture.


\subsection{Word Embeddings} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

When computing WEs we take into consideration \newcite{TACL570}: they observed
that one specific configuration of~\wtv, namely the skip-gram model with
negative sampling (SGNS) 
"is a robust baseline.  While it might not be the best method for every task,
it does not significantly underperform in any scenario. Moreover, SGNS is the
fastest method to train, and cheapest (by far) in terms of disk space and
memory consumption".
Coincidentally,~\newcite{arXiv:1310.4546} also suggest to use SGNS.
We incorporate \wtv's original C implementation for learning WEs%
\footnote{\url{https://code.google.com/archive/p/word2vec/}}
in an independent pre-processing step, i.e.~we pre-compute the WEs.
Then, we use gensim%
\footnote{\url{https://radimrehurek.com/gensim/}}%
, a Python tool for unsupervised semantic modelling from plain text, to load
the pre-computed data, and to compute the vector representations of
input words for our NN.


\subsection{Character-Level Sub-Word Information} %%%%%%%%%%%%%%%%%%%%%%%%%%%%

Our implementation uses a \emph{one-hot encoding} with a few additional
features for representing sub-word information.
The one-hot encoding transforms a categorical feature into a vector where
the categories are represented by equally many dimensions with binary values.
We convert a letter to lower-case and use the sets of ASCII characters, digits,
and punctuation marks as categories for the encoding.
Then, we add dimensions to represent more binary features like
\emph{'uppercase'} (was upper-case prior to conversion), \emph{'digit'} (is
digit), \emph{'punctuation'} (is punctuation mark), \emph{whitespace} (is white
space, except the new line character; note that this category is usually empty,
because we expect our tokens to \emph{not} include white space characters), and
\emph{unknown} (other characters, e.g.~diacritics).
This results in vectors with more than a single \emph{one-hot} dimension. 
%
% - alpha-characters are mapped to lower-case one-hot + 'is-uppercase'
% - digits are mapped to one-hot + 'is-digit'
% - punctuation marks are mapped to one-hot + 'is-punctuation'
% - whitespace (ecxept '\n') characters are mapped to one-hot +
% 'is-whitespace'
% - unknowns have their own one-hot
% * '\n' is treated as new-token-character

\subsection{Recurrent Neural Network Layer} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Our implementation uses Keras, a high-level NNs library, written in Python and
capable of running on top of either TensorFlow or Theano~\cite{chollet2015}. 
In our case it runs on top of Theano, a Python library that allows to define,
optimize, and evaluate mathematical expressions involving multi-dimensional
arrays efficiently~\cite{Theano2016}.

The input to our network are sequences of the same length as the sentences we
process.
During training, we group sentences of the same length into batches and process
the batches according to sentence length in increasing order.
Each single word in the sequence is represented by its sub-word information and
two WEs that come from two sources (see Section~\ref{sec:results}).
%Note that the WEs of the domain-specific corpus may have only been derived
%from a small corpus.
For unknown words, i.e.~words without a pre-computed WE, we first try to
find the most similar WE considering 10 surrounding words.  
If this fails, the unknown word is mapped to a randomly generated vector
representation.
In Total, each word is represented by $2,280$ features: two times $500$ (WEs),
and sixteen times $80$ for two 8-grams (word beginning and ending).
If words are shorter than 8 characters their 8-grams are zero-padded.

This sequential input is fed into a LSTM layer that, in turn, projects to a
fully connected output layer with softmax activation function.
During training we use dropout for the projection into the output layer,
i.e.~we set a fraction ($0.5$) of the input units to 0 at each update, which
helps prevent overfitting~\cite{Srivastava2014}.
We use categorical cross-entropy as loss function and backpropagation in
conjunction with the RMSprop optimization for learning.  
At the time of writing, this was the Keras default---or the explicitly
documented option to be used---for our type of architecture. 


\section{Results} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:results}

We used our slightly modified implementation to participate in the \emph{POS
tagging for Italian Social Media Texts} (PoSTWITA) shared task (ST) of the
5\textsuperscript{th} periodic evaluation campaign of Natural Language
Processing (NLP) and speech tools for the Italian language EVALITA 2016.
First, we describe the corpora used for training, and then the specific system
configuration(s) for the ST.


\subsection{Training Data for \wtv and PoS Tagging} %%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{DiDi-IT (PoS, \wtv)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\emph{didi-it}~\cite{frey-glaznieks-stemle:2016:didi}
(version September 2016) is the Italian sub-part of the DiDi corpus, a corpus
of South Tyrolean German and Italian from Facebook (FB) users' wall posts,
comments on wall posts and private messages.

The Italian part consists of around 100,000 tokens collected from 20 profiles
of Facebook users residing in South Tyrol.
This version has about 20,000 PoS tags semi-automatically corrected by a single
annotator.

The anonymised corpus is freely available for research purposes.


\subsubsection{Italian UD (PoS, \wtv)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Universal Dependencies (UD) is a project that is developing
cross-linguistically consistent treebank annotation for many
languages.\footnote{\url{http://universaldependencies.org/}}

\emph{italian-UD}%
\footnote{\url{http://universaldependencies.org/it/overview/introduction.html}}
(version from January 2015) corpus was originally obtained by conversion from
ISDT (Italian Stanford Dependency Treebank) and released for the dependency
parsing ST of EVALITA 2014~\cite{Bosco2014}.
The corpus has semi-automatically converted PoS tags from the original two
Italian treebanks, differing both in corpus composition and adopted annotation
schemes.

The corpus contains around 317,000 tokens in around 13,000 sentences from
different sources and genres.  
It is available under the CC BY-NC-SA 3.0\footnote{Creative Commons
Attribution-NonCommercial-ShareAlike 3.0 Unported, i.e.~the data can be copied
and redistributed, and adapted for purposes other than commercial ones. See
\url{https://creativecommons.org/licenses/by-nc-sa/3.0/} for more details.}
license.


\subsubsection{PoSTWITA (PoS and \wtv)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sssec:postwita}
\emph{postwita} is the Twitter data made available by the organizers of the ST.
It contains Twitter tweets from the EVALITA2014 SENTIPLOC corpus: the
development and test set and additional tweets from the same period of time
were manually annotated for a global amount of 6438 tweets (114,967 tokens) and
were distributed as the development set. 
The data is PoS tagged according to UD but with the additional insertion of
seven Twitter-specific tags.
All the annotations were carried out by three different annotators.
The data was only distributed to the task participants.


\subsubsection{C4Corpus (\wtv)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\emph{c4corpus}%
\footnote{\url{https://github.com/dkpro/dkpro-c4corpus}} is a full documents
Italian Web corpus that has been extracted from CommonCrawl, the largest
publicly available general Web crawl to date. 
See Habernal~\shortcite{Habernal.et.al.2016.LREC} for details about the corpus
construction pipeline, and other information about the corpus.

The corpus contains about $670m$ tokens in $22m$ sentences.
The data is available under the CreativeCommons license
family. 


\subsubsection{PAIS{\`A} (\wtv)} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\emph{paisa}~\cite{paisa2014} is a corpus of authentic contemporary Italian
texts from the web (harvested in September/October 2010).
It was created in the context of the project PAIS{\`A} (P{\'i}attaforma per
l'Apprendimento dell'Italiano Su corpora Annotati) with the aim to provide a
large resource of freely available Italian texts for language learning by
studying authentic text materials.

The corpus contains about $270m$ tokens in about $8m$ sentences. 
The data is available under the CC BY-NC-SA
3.0\footnote{\url{https://creativecommons.org/licenses/by-nc-sa/3.0/}} license. 


%An overview of the used training data can be found in Table~\ref{tab:corpora}.

%\begin{table}[h]
%\begin{center}
%\begin{tabular}{l|r|r}
%\hline
%\multicolumn{1}{c}{} & \multicolumn{1}{c}{Tokens} & \multicolumn{1}{c}{Sentences} \\ \hline
%corpus 1       & $10*10^3$     & $0.6*10^3$    \\ \hline
%corpus 2       & $0.9*10^9$    & $0.05*10^6$   \\ \hline\hline
%corpus 3     	& $2.0*10^9$    & $79*10^6$     \\ \hline
%%C4Corpus      & $$            & $$            \\ \hline
%\end{tabular}
%\end{center}
%\caption{\label{tab:corpora}Training data used for the shared
%    task. The PoS tags of the upper two corpora were used for training.
%} \end{table}


\subsection{PoSTWITA shared task} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{ssec:postwita}

For the ST we used one overall configuration for the system but three different
corpus configurations for training.
However, only one corpus configuration was entered into the ST: we used PoS
tags from \emph{didi-it + postwita} (run 1), from \emph{italian-UD} (run 2),
and from both (run 3).
For \wtv we trained a 500-dimensional skip-gram model on \emph{didi-it +
italian-UD + postwita} that ignored all words with less than 2 occurrences
within a window size of 10; it was trained with negative sampling (value 15).
We also trained a 500-dimensional skip-gram model on \emph{c4corpus + paisa}
that ignored all words with less than 33 occurrences within a window size of
10; it was trained with negative sampling (value 15).

The other \wtv parameters were left at their default settings%
\footnote{\texttt{-sample 1e-3 -iter 5 -alpha 0.025}}.

The evaluation of the systems was done by the organisers on unlabelled but
pre-tokenised data (4759 tokens in 301 tweets), and was based on a
token-by-token comparison.
The considered metric was accuracy, i.e.~the number of correctly assigned PoS
tags divided by the total number of tokens.

\begin{table}[h]
\begin{center}
\begin{tabular}{l|r|r}
\hline
\textbf{(1)} \textbf{\emph{didi-it + postwita}}   & $\textbf{76.00}$        \\ \hline
(2) \emph{italian-UD}           & $80.54$        \\ \hline
(3) \emph{didi-it + postwita + italian-UD} & $81.61$  \\ \hline\hline
Winning Team                    & $93.19$        \\ \hline
\end{tabular}
\end{center}
\caption{\label{tab:results}Official result(s) of our PoS tagger for the
three runs on the PoSTWITA ST data.
} 
\end{table}

We believe, the unexpectedly little performance gain from utilizing the much
larger \emph{italian-UD} data over the rather small \emph{didi-it + postwita}
data may be rooted in the insertion of Twitter-specific tags into the data
(see~\ref{sssec:postwita}), something we did not account for, i.e.~$18,213$ of
$289,416$ and more importantly $7,778$ of $12,677$ sentences had imperfect
information during training. 

%\fixme[inline]{The system is said to be trained on various widely available
%    corpora of German such as Tiger 2.2, DECOW148 etc. It is not discussed in
%    any detail and thus does not become clear why these corpora were selected
%    and especially what the systematic benefit of using these different data
%    sets was; more specifically, the paper does not systematically explore the
%    diversity of the data and their contribution to a learning improvement on
%    the system. A section with a systematic evaluation is missing and so is a
%    section with a more thorough discussion as to why this particular approach
%    was selected. It is suggested to the authors to provide more background on
%    these questions.}


\section{Conclusion \& Outlook} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\label{sec:conclusion}

We presented our submission to the PoSTWITA task of EVALITA 2016, where we
participated with moderate results. 
In the future, we will try to rerun the experiment with training data that
takes into consideration the Twitter-specific tags of the task.


\section*{Acknowledgments} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The computational results presented have been achieved in part using the Vienna
Scientific Cluster (VSC).


% References %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\clearpage
%\nocite{*}
%\small
\bibliographystyle{acl}
\bibliography{paper}
%\normalsize

%\appendix %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Supplemental Material}
%\label{sec:supplemental}
%Supplement

\end{document}