I am implementing a Seq2Seq model. Each step of the decoder has |N| outputs (the number of unique words). Since |N| is huge, I am trying to speed up the training by negative sampling (n_sample=100). I can see the training time for one epoch is reduced by 25%. However, the performance (crossentropy loss) is not as good as full prediction. Is it normal?

1 Answer 1

$\begingroup$

$\endgroup$

Depending on your model and dataset, I think this type of result can be normal. If your vocabulary size is very large using negative sampling can offer much more than a 25% speedup. You would be able to compensate for the decrease in performance by training for more epochs and still come out ahead in terms of performance per computation time.