Two classifier- based evalutation metrics

[yunzhel2]

Dear Jian:

Thanks for releasing your source code. It is really an interesting work and I want to follow and make further research. I have partly reproduced your experiments. The results are completely fitting your report.

While, there may be a bit ambiguous in coherence metrics: relatedness and sentence ordering. In this two metrics, you used two trained classifier to evaluate them. Could you explain more about them and(or) provide the post-trained classifier (just like the way you shared HINT.)

Look forward to your reply!

Thanks,
Yunzhe

[ghost]

Dear Jian:

Thanks for releasing your source code. It is really an interesting work and I want to follow and make further research. I have partly reproduced your experiments. The results are completely fitting your report.

While, there may be a bit ambiguous in coherence metrics: relatedness and sentence ordering. In this two metrics, you used two trained classifier to evaluate them. Could you explain more about them and(or) provide the post-trained classifier (just like the way you shared HINT.)

Look forward to your reply!

Thanks, Yunzhe

I meet the same question, have you get the ckpt you mentioned about ? BTW, the results of auto-metrics seem strange, e.g., PPL for WP datasets is much higher and LR-2 score for ROC datasets is much lower than the results reported in the paper. I wonder have you ever meet the same questions ?