Word2Vec

[YiQ-Zhao]

Hi,

It seems that the example didn't use Word2Vec. I am trying to load pre-trained vectors to represent these words. Do you think it will be helpful to to improve the bot performance? I can't figure out how to do that. Do you have any examples showing the similar procedures?

Thank you in advance!

[zsdonghao]

if you don't have the word2vec parameters and table, you can train one here https://github.com/tensorlayer/tensorlayer/blob/master/example/tutorial_word2vec_basic.py

then you can use the pre-trained parameters like that https://github.com/tensorlayer/tensorlayer/blob/master/example/tutorial_generate_text.py#L371

hope it helps

[YiQ-Zhao]

Thank you for your reply. I'm a newbie to deep learning. I didn't quite understand the code in these two examples. What I wanted is to use GloVe to represent these words and train the seq2seq model.

glove_weights_initializer = tf.constant_initializer(weights)
embedding_weights = tf.get_variable(
    name='embedding_weights', 
    shape=(VOCAB_LENGTH, EMBEDDING_DIMENSION), 
    initializer=glove_weights_initializer,
    trainable=False)

 net_encode = tl.layers.EmbeddingInputlayer(inputs=encode_seqs, vocabulary_size=xvocab_size, embedding_size=emb_dim, E_init=embedding_weights)

I'm quite sure this would not work. I saw in the tutorial_generate_text you used
tl.layers.assign_params(sess, [load_params[0]], emb_net) to load the existing embedding matrix. However, the net_encode is a part of model() function, how do I load the pre-trained parameters to it? Also, I don't know where the model_word2vec_50k_128.npy comes from. Is it possible to generate such a .npy file using glove.42B.300d.txt file?

Any help will be greatly appreciated!

[zsdonghao]

Hi, you need to make sure the lookup table you are using have the same dimension with your layer, otherwise, that parameters can't be assigned into the layer.

the model_word2vec_50k_128.npy is from the word2vec tutorial the link I send you, you can train one by simply run the code.

[YiQ-Zhao]

Thanks. Your suggestions were all very helpful to me.
谢谢！

[zsdonghao]

不客气