This is the accompanying repository to emoji2vec: Learning Emoji Representations from their Description the paper recently released by Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel.
In this repository, we present the code that we used to train our representations of emoji, the training data we used to do so, and several tools for analyzing the performance of the vectors trained.
NOTE: The
emoji_joined.txt
dataset was generated by combiningemoji.txt
(a dataset scraped from unicode.org) with a similar dataset scraped from iemoji.com. The data from iemoji.com contains short keyword descriptions for some of the lesser-used emoji, which improved coverage over the full emoji set. This source was mistakenly omitted from the original manuscript: all 6088 training examples were mistakenly attributed to unicode.org. In actuality, 2684 were scraped from unicode.org and 3404 were scraped from iemoji.com.
If you are interested in using the emoji vectors we used in our paper,
they can be found in Gensim text/binary format in ./pre-trained/
. The
pre-trained vectors are meant to be used in conjunction with word2vec
,
and are therefore 300-dimensional. Other dimensions can be trained
manually, as explained below. These vectors correspond with the following
hyperparameters:
params = {
"out_dim": 300,
"pos_ex": 4,
"max_epochs": 40,
"ratio": 1,
"dropout": 0.0,
"learning": 0.001
}
Once you've downloaded the pre-trained model, you can easily integrate emoji embeddings into your projects like so:
import gensim.models as gsm
e2v = gsm.Word2Vec.load_word2vec_format('emoji2vec.bin', binary=True)
happy_vector = e2v['😂'] # Produces an embedding vector of length 300
There are several prerequisites to using the code:
- You must supply your own pretrained word vectors that are compatible with the Gensim tool. For instance, you can download the Google News word2vec dataset here. This must be in the binary format, rather than the .txt format.
- To download tweets using Tweepy, you must create a Twitter application
at https://apps.twitter.com/, and place
the four generated keys in
secret.txt
in the directory where you run the Python script. However, you may not have to download the tweets, since they are stored raw in apickle
file in the repository.
Much of this code shares a common command line interface, which allows you to supply hyperparameters for training and model generation/retrieval as well as file locations. The following can be supplied:
-d
: directory for training data (default is./data/training
)-w
: path to the word embeddings (i.e. Google News word2vec)-m
: file where we store mapping between index and emoji, for convenient caching between runs-em
: file where we cache the vectorized phrases so we don't have to recompute each time, only change when you change the train, test, and dev files-k
: output dimension of the emoji vectors we are training-b
: number of positive examples in a training batch-e
: number of training epochs-r
: ratio between positive and negative training examples in a batch-l
: learning rate-dr
: dropout rate-t
: threshold for classification, used in accuracy calculations-ds
: name of the dataset we are training on, mainly for output folder
These are defined in parameter_parser.py
.
The Emoji2Vec model, as well as a class for passing in hyperparameters,
can be found in model.py
. The Emoji2Vec class is a TensorFlow
implementation of our model.
Important to note is that one can evaluate
the correlation between a phrase and an emoji in two ways: one can
either input a raw vector and an emoji index (for general queries),
or the index of a training phrase and the index of an emoji (indices
being the indices in the Knowledge Base). Typically, unless you are
training the model on a totally different set of training examples,
you'll want to use set use_embeddings
to False
in the constructor
of the model. Otherwise, you'll have to pass in embeddings generated
by the generate_embeddings
function in utils.py
.
In this initial release, the internals are a bit convoluted, so it would
probably behoove anyone using the codebase to use train.py
instead of
using the Emoji2Vec class directly.
The Phrase2Vec
class is a convenience wrapper to compute vector sums
for phrases. The class can be constructed with two different vector
sets simultaneously: a word2vec Gensim object and an emoji vector Gensim
object. Alternatively, you can provide two filenames to do so. Query
like so:
vec = phrase2Vec['I am really happy right now! 😄]
To train a single model, run train.py
with any combination of the
hyperparameters above. For instance,
python3 train.py -k=300 -b=4 -r=1 -l=0.001 -ds=unicode -d=./data/training -t=0.5
will generate emoji vectors with dimension 300, and will train in
batches of 8 (4 positive, 4 negative examples) at a learning rate of
0.001. ./data/training/
must contain train.txt
, dev.txt
, and
test.txt
, the format of each being a tab-delimited, newline-delimited:
beating heart 🍮 False
The program will output various metrics, including accuracy (at the
threshold provided), f1 score, and auc for a ROC curve. Additionally,
the program will generate a Gensim representation of the model, a
TensorFlow representation of the model, a TensorFlow tensorboard
folder, and a cache of the results of the model's predictions on the
train and dev datasets.
These results can be found in the following folder:
./results/unicode/k-300_pos-4_rat-1_ep-40_dr=0/
You can perform a grid search on a hyperparameter space one of two ways:
either directly modify the search_params
variable in grid_search.py
and running grid_search.py
, or from a separate file call grid_search
with supplied parameter set. In essence, this grid search will generate
results and embeddings in the same way as train.py
for each parameter
combination. The searchable parameters are represented as follows:
search_params = {
"out_dim": [300],
"pos_ex": [4, 16, 64],
"max_epochs": [10, 20],
"ratio": [0, 1, 2],
"dropout": [0.0, 0.1]
}
NOTE: The epochs parameter will not be explored exactly as input. Since larger batches take more epochs to converge, we scale the number of epochs by the batch size.
twitter_sentiment_dataset.py
contains a collection of helper functions
for downloading, processing, and reasoning about tweets. In general,
since tweets have already been downloaded and parsed and cached in
./data/tweets/examples.p
, a client shouldn't need to access these
functions unless they are running them on a new set of Tweets
TODO(beneisner): Clean up this library so that it's easier to run with new Tweets.
To generate a 2D visualization of the emoji embeddings, run:
python3 visualize.py {arguments}
This technique uses t-SNE to project from N-dimensions into 2 dimensions.
utils.py
contains several utility functions used in various files,
and generally need not be used externally.
We characterize the generated emoji embeddings in two files:
Results.ipynb
displays quantitative and qualitative metrics for a
given model. Change the hyperparameters near the top of the file to
evaluate a different model.
TwitterClassification.ipynb
contains an evaluation scheme for the
Twitter sentiment classification task outlined in the paper. It
implements two rudimentary classifiers.
Contact me at ben [dot] a [dot] eisner [at] gmail [dot] com
with questions about
implementation or requests.