Why my results are so poor?

[sehbe]

I downloaded the code without changing anything, just to make sure it ran, I loaded the local bert_model

2023-10-26 12:59:12 INFO: - Step: 0/5001, span loss = 6.118810, type loss = 0.000000, time = 2.27s.
2023-10-26 12:59:53 INFO: - Step: 20/5001, span loss = 5.980084, type loss = 0.000000, time = 43.38s.
2023-10-26 13:00:36 INFO: - Step: 40/5001, span loss = 5.879471, type loss = 0.000000, time = 86.74s.
2023-10-26 13:01:17 INFO: - Step: 60/5001, span loss = 5.772162, type loss = 0.000000, time = 128.07s.
2023-10-26 13:01:57 INFO: - Step: 80/5001, span loss = 5.848635, type loss = 0.000000, time = 167.65s.
2023-10-26 13:02:36 INFO: - Step: 100/5001, span loss = 5.735874, type loss = 0.000000, time = 206.93s.
2023-10-26 13:03:15 INFO: - Step: 120/5001, span loss = 5.667262, type loss = 0.000000, time = 246.25s.
2023-10-26 13:03:55 INFO: - Step: 140/5001, span loss = 5.633423, type loss = 0.000000, time = 285.52s.
2023-10-26 13:04:34 INFO: - Step: 160/5001, span loss = 5.501272, type loss = 0.000000, time = 325.18s.
2023-10-26 13:05:16 INFO: - Step: 180/5001, span loss = 5.545497, type loss = 0.000000, time = 366.45s.
2023-10-26 13:05:55 INFO: - Step: 200/5001, span loss = 5.426091, type loss = 0.000000, time = 406.06s.

2023-10-26 13:08:26 INFO: - ***** Eval results inter-valid *****
2023-10-26 13:08:26 INFO: - f1 = 0.0
2023-10-26 13:08:26 INFO: - f1_threshold = 0.0
2023-10-26 13:08:26 INFO: - loss = tensor(1.1085, device='cuda:0')
2023-10-26 13:08:26 INFO: - precision = 0.0
2023-10-26 13:08:26 INFO: - precision_threshold = 0.0
2023-10-26 13:08:26 INFO: - recall = 0.0
2023-10-26 13:08:26 INFO: - recall_threshold = 0.0
2023-10-26 13:08:26 INFO: - span_f1 = 0.05210224202151717
2023-10-26 13:08:26 INFO: - span_p = 0.03787716670233228
2023-10-26 13:08:26 INFO: - span_r = 0.08343808925204142
2023-10-26 13:08:26 INFO: - type_f1 = 0.0
2023-10-26 13:08:26 INFO: - type_p = 0.0
2023-10-26 13:08:26 INFO: - type_r = 0.0
2023-10-26 13:08:26 INFO: - 0.000,0.000,0.000,3.788,8.344,5.210,0.000,0.000,0.000,0.000,0.000,0.000
2023-10-26 13:08:26 INFO: - ===> Best Valid F1: 0.0

[iofu728]

Hi @sehbe, according to the logs, it seems that you haven't trained for a sufficient amount of time. DecomposedMetaNER is a two-stage process that requires separate training for the Span Detector and Entity Typing. Our method is not zero-shot and needs to be trained on a few-shot dataset.

Additionally, it's normal for the type and final F1 to be 0 during the Span Detector stage. You'll need to wait until both classifiers are fully trained to see the final results. As also mentioned in #66.