AI Stack Exchange
2023-11-12 12:36 UTC
By SAGALPREET SINGH
AI-110-20231112-social-media-49108cc8
Masking in Decoder of Transformer
I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input to the output of masked multi-head attention block adds some of the information from future time steps which is then used to construct the query matrix for multi-head attention block. Shouldn't some kind of mask be applied before adding the input values to the output of masked multi-head attention block as well? Asking this question in the context of training process.
I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input to the output of masked multi-head attention block adds some of the information from future time steps which is then used to construct the query matrix for multi-head attention block. Shouldn't some kind of mask be applied before adding the input values to the output of masked multi-head attention block as well? Asking this question in the context of training process.
Full article content could not be extracted automatically. Read the original below.
Source:
AI Stack Exchange
· ai.stackexchange.com