TA-x model reproduction.

[mer1988]

Hi Alberto, I have been trying for a while to replicate the results from the time aware link prediction paper using the published datasets using the re-implementation of your models from this repo, https://github.com/INK-USC/RE-Net I have carefully looked into the code it looks reasonable to your implementation. The only change I have made from this code and my version is that I change the way they calculate filter hits@10 -> they suppress every object from the ranked list that is observed in training/validation regardless of the time when it occurred while I only suppress those observed in the same timestamp, otherwise I keep it.

The best results I have gotten so far are

  | MRR . | H1 | H10
TA-DistMult | 0.351600377 | 0.2187883521 | 0.628807319
TA-TransE | 0.2173162281 | 0 | 0.6098404552

Do you have any thoughts?? or maybe an accessible implementation of your model?

Thanks!

[mniepert]

Dear Miguel,

Could you provide more context? What dataset is this? One difference in the code that I noticed is that they only use one negative sample (and binary crossentropy) and we use 500 negative samples. This has been shown to make a huge difference for scoring functions such as DistMult.

[mer1988]

Thanks Mathias,
I will take a look a close look at neg sampling. The results I show are for the icews14 dataset published in this repo.

Miguel R.

[AGDuran]

Hi miguel, did you apply that filter setting only to compute hits@10 or also for all other metrics? And yes, the loss is super important to get good performance. Alberto El mié., 11 sept. 2019 16:52, Miguel Rodríguez <[email protected]> escribió:

…

Thanks Mathias, I will take a look a close look at neg sampling. The results I show are for the icews14 dataset published in this repo. Miguel R. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACX4SD44SWL7QZV4MTBAWK3QJEA3XANCNFSM4IVN5M7A> .

[mer1988]

Hi Alberto, I apply the filter for all metrics. Thanks for the neg sampling suggestion, that I had totally missed.

[mer1988]

Hi Alberto, is there any possibility to use your implementation of the TA-xx models for research purposes?

[AGDuran]

Hello Miguel, I have a Keras implementation that is super dirty. I dont want to release that implementation without cleaning it up before, but I am busy at the moment with other projects. I can copy and paste the lines of the model though. It is a very simple code, so I hope you can understand it. Alberto ## TA-transE model ## act_fn = None hidden_state = True drop = 0.4 embedding_dim = 100 n_epochs = 500 num_negative = 250 # Inputs e1 = Input(shape=(num_negative + 1,), name="e1") e2 = Input(shape=(num_negative + 1,), name="e2") relation = Input(shape=(9,), name="rel") # Number of tokens in the relation sequence (depends on the dataset) # Entity embeddings ent_embedding = Embedding(numEnt, embedding_dim, name='ent_embeddings') e1_emb = Dropout(drop)(ent_embedding(e1)) e2_emb = Dropout(drop)(ent_embedding(e2)) # Time-enriched relation embedding rel_embedding = Embedding(numRel + numYear + numMonth + numDay, embedding_dim) rel_emb = Dropout(drop)(rel_embedding(relation)) date_embedding = LSTM(embedding_dim, activation = act_fn, return_state = True) output_state, state_h, state_c = date_embedding(rel_emb) i f cell_state: state = state_c else: state = state_h # Scoring function e2_emb_minus = Lambda(lambda x: -1 * x)(e2_emb) joint_emb = Add(name='joint_emb')([e1_emb, state, e2_emb_minus]) score1 = Multiply()([joint_emb, joint_emb]) score = Lambda(lambda x: -K.sum(x, axis=2), output_shape=(num_negative + 1,))(score1) # Loss score = Activation('softmax')(Reshape((num_negative + 1,), name='soft_out')(score)) adam = Adam() # Training model = Model(input=[e1, e2, relation], output=[score]) model.compile(loss='categorical_crossentropy', optimizer=adam) El mié., 25 sept. 2019 a las 20:08, Miguel Rodríguez (< [email protected]>) escribió:

…

Hi Alberto, is there any possibility to use your implementation of the TA-xx models for research purposes? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACX4SDYMRE7FPEPPIHV3HE3QLOSK3ANCNFSM4IVN5M7A> .