Gradient Descent and General DL Remarks

One answer per question, one attempt per quiz — you cannot retake it.

Question 1 of 5

Which of the following is true about the ReLU activation function?

ReLU saturates for both negative and positive inputs, stopping learning.ReLU saturates only for negative inputs, where its gradient is zero.ReLU saturates only for positive inputs, due to large activations.ReLU never saturates, because its derivative is non-zero almost everywhere.

Question 2 of 5

Why is forward‑mode automatic differentiation impractical for training large neural networks?

It stores all intermediate values, making memory usage prohibitive.It requires a separate forward pass for each network parameter.It cannot handle non‑linear activation functions like ReLU or sigmoid.Its gradient estimates are biased and do not guarantee convergence.

Question 3 of 5

What does SGD with a constant learning rate achieve on a convex loss function?

It converges exactly to the global minimum after many iterations.It oscillates around the minimum but eventually settles at the exact point.It converges to a ball around the optimum, with radius proportional to the step size.It diverges because the stochastic noise accumulates and dominates.

Question 4 of 5

Why does standard momentum help in optimizing ill‑conditioned quadratics like x² + ρ y² (ρ ≪ 1)?

It adaptively scales the learning rate, increasing it along the shallow dimension.It dampens oscillations along the steep axis and accelerates along the shallow one.It normalizes gradient updates using the exponential moving average of squares.It computes the gradient at a lookahead point to anticipate the curvature.

Question 5 of 5

What is the primary advantage of Adam over standard SGD with momentum?

It incorporates Nesterov momentum for accelerated convergence.It maintains per‑parameter learning rates based on gradient history.It guarantees convergence even on non‑convex loss functions.It automatically finds an optimal learning rate schedule without tuning.

0 / 5 answered

Generated from lecture recordings by the autosummary pipeline.