Attention and Transformers

One answer per question, one attempt per quiz — you cannot retake it.

Question 1 of 5

Why does the LSTM's cell state update help prevent vanishing gradients for long-term dependencies?

The cell state update can have a derivative of 1 when the forget gate is fully open.Sigmoid gates inherently prevent gradients from shrinking exponentially.Peephole connections allow gradients to bypass the recurrent weight matrices.The hidden state is recomputed from the cell state at every step.

Question 2 of 5

Why is it essential to scale the dot products by 1/√d in the Transformer's self-attention?

To prevent the softmax from saturating and causing vanishing gradients.To ensure that the attention weights sum to the sequence length.To normalize the value vectors before computing the weighted sum.To reduce the number of parameters in the attention computation.

Question 3 of 5

What is the main role of the softmax operation in the Bahdanau attention mechanism?

It normalizes scores so the context is a convex combination of states.It ensures the decoder selects a single most relevant encoder state each step.It makes the attention mechanism differentiable for backpropagation.It prevents the decoder from attending to the same position repeatedly.

Question 4 of 5

Why does the Transformer decoder use a causal mask during training?

To prevent the decoder from using future tokens during parallel training.To allow the decoder to generate all tokens simultaneously during inference.To reduce the computational cost of the attention mechanism.To force the decoder to learn a monotonic alignment with the encoder.

Question 5 of 5

What is the main advantage of using Byte-Pair Encoding (BPE) for tokenization in Transformers?

It balances vocabulary size and sequence length using subword units.It allows the model to represent every word as a single token, simplifying generation.It reduces the number of attention heads needed for long sequences.It avoids out-of-vocabulary issues by splitting all words into characters.

0 / 5 answered

Generated from lecture recordings by the autosummary pipeline.