The Mathematics of a Transformer Explained With Numbers You Can Check by Hand With Geometric Intutions Using Pytorch.Part-2 Attention Mechanism.
In the previous section, we watched Positional Encoding nudge our raw tokens in 3D space. Now, our position-encoded input tensor X enters the heart of the Transformer: Self-Attention.
