Member-only story
Positional Embeddings
Why so complex?
We already spoke about the nuances of the word token (sometimes referred as to a word, a number or a vector) in the previous article. And we shed light into that. But we intentionally left a glaring problem out of the equation: the order of the tokens in a sentence.
We, humans, read and write directionally, from left to right in English, but LLMs read all at once. Yes the ALL words at once. In fact, that’s what makes them so fast.
Therefore when you read “My name is Javier” there is an implicit order in your brain: [My, name, is, … ]. But the AI read all the sentence at once. This means for the machine the sentences “My name is Javier” and “Javier My name is” are exactly the same. Even when the 2nd sentence would have only be produced by a very old-and-wise Jedi master.
For this reason this article will focus on how to teach the AI not to speak like Yoda 🤣. Better said: this article will focus on how to inject sequence order into Transformers.
In the original paper of Transformers, Attention is All you Need, they injected the word ordering information using Positional Embeddings. That means that the same way we have the…
![Evolution of Attention [Part 2. Improved Alignment / Attention]](https://webproxy.poorya-velaei-d67.workers.dev/https://miro.medium.com/v2/resize:fit:1358/format:webp/0*Br6f2tUodE9R_u3w.jpeg)
