Chasing “Attention” (Part II) — Meddling with Sutskever’s paper
Twenty LSTMs, ten paired seeds, and a foolish attempt to find patterns in source reversal’s translation gains!
The Reversal Conundrum
If you followed Part I, the natural next step in our journey would be Bahdanau, Cho and Bengio’s attention mechanism. After all, we ended with a rather uncomfortable limitation of the original sequence-to-sequence architecture - an entire source sentence had to be represented through a fixed-dimensional state, from which the decoder would generate the translation without looking back at the original words.
Attention offered a remarkably different approach to that problem.
But before moving forward in this pipeline of chasing the concept of attention, there was something in Sutskever’s 2014 experiment that I couldn’t quite leave behind -
The magic of source reversal.
Authors said -
“While the LSTM is capable of solving problems with long term dependencies, we discovered that the LSTM learns much better when the source sentences are reversed (the target sentences are not reversed). By doing so, the LSTM’s test perplexity dropped from 5.8 to 4.7, and the test BLEU scores of its decoded translations increased from 25.9 to 30.6. While we do not have a complete explanation to this phenomenon, we believe that it is caused by the introduction of many short term dependencies to the dataset.”
With source reversal, some English words naturally move closer to their corresponding French words, while others move farther away. Yet the average distance remains unchanged. As the authors themselves noted -
“Normally, when we concatenate a source sentence with a target sentence, each word in the source sentence is far from its corresponding word in the target sentence. As a result, the problem has a large “minimal time lag” [17]. By reversing the words in the source sentence, the average distance between corresponding words in the source and target language is unchanged.”
At first glance, this seems like a zero-sum trade-off. If some dependencies become shorter while others become longer, and the average distance remains unchanged, why should reversing the sentence improve anything at all?
The authors offered the following explanation -
“However, the first few words in the source language are now very close to the first few words in the target language, so the problem’s minimal time lag is greatly reduced. Thus, backpropagation has an easier time “establishing communication” between the source sentence and the target sentence, which in turn results in substantially improved overall performance.”
Put simply, reversal doesn’t reduce the average distance; it changes which dependencies are short. Words appearing early in the original English sentence now lie much closer, computationally, to the early French predictions. These shorter paths could make it easier for learning signals to propagate during backpropagation, potentially improving optimization and contributing to the overall improvement in translation performance.
But beyond the question of overall performance, this rearrangement of computational paths raises another natural question. Shouldn’t the benefits of shorter computational paths be more pronounced in the earlier parts of the target sentence? After all, while reversal shortens some source–target dependencies, it lengthens others. One might therefore expect the strength in early predictions to gradually diminish, perhaps even becoming a disadvantage for later predictions.
Interestingly, the authors themselves had initially anticipated precisely this positional trade-off, as they acknowledged in the paper —
“Initially, we believed that reversing the input sentences would only lead to more confident predictions in the early parts of the target sentence and to less confident predictions in the later parts.”
But their experiments also revealed something unexpected along another dimension — sentence length.
“However, LSTMs trained on reversed source sentences did much better on long sentences than LSTMs trained on the raw source sentences (see sec. 3.7), which suggests that reversing the input sentences results in LSTMs with better memory utilization.”
That last observation caught my attention.
Performance on long sentences and prediction quality at later French positions concern different aspects of the model’s behaviour. Better translation of long sentences does not establish how the advantage is distributed within each translation. It nevertheless raises a broader question about how reversal changes the model’s ability to learn from and use the source sentence.
The positional trade-off we had just reasoned through seemed intuitive, and the authors themselves had initially expected it. Yet they said they saw better translation performance even on long sentences. What could explain this broader improvement?
One possible explanation lies in the decoder itself. After all, it is autoregressive. Each new word is predicted using the words generated before it. If reversal helps the network make more accurate predictions early in the translation, those early successes could provide a more reliable context for subsequent predictions, reducing the risk of errors accumulating as generation proceeds. If that were the whole explanation, perhaps the puzzle would end there.
But what if the advantage went beyond that? What if reversing the source also changed how the network learned to represent and use information from the English sentence, improving its predictions independently of any advantage carried forward from earlier generated words?
Reproducing the Experiment
To investigate this, I set up a controlled comparison involving twenty LSTM-based sequence-to-sequence models across ten random seeds. For each seed, one model was trained on source sentences in their original order, while its counterpart was trained on the same sentences with their source tokens reversed. Everything else was kept identical - the architecture, dataset, training procedure, and evaluation protocol. This gave us ten paired comparisons rather than a single potentially misleading result. The objective was not simply to establish whether reversal improved translation performance, but to understand whether any advantage was consistent across runs and, more importantly, what might be responsible for it.
Recreating the Original Architecture
The first decision was how closely to follow Sutskever, Vinyals, and Le’s original architecture. Their 2014 system used a four-layer LSTM encoder and a four-layer LSTM decoder, with 1,000-dimensional hidden states and word embeddings. Most importantly, there was no attention mechanism. The decoder was initialized using the encoder’s final hidden and cell states, meaning that the entire English sentence had to be represented through these states before French generation could begin. I deliberately preserved this architectural constraint. At the end of the day, if the purpose was to understand why reversing the source sentence helped information travel through the recurrent network, introducing attention would fundamentally change the problem we were trying to investigate.
Data
The next challenge was the dataset. Authors of the original paper had trained their models on approximately 12 million English–French sentence pairs from WMT14, a scale that would be difficult to reproduce on my single GPU. To strike a balance between having sufficient training data and working within my computational limitations, I turned to the IWSLT17 English–French TED-talk corpus. After all, the objective was not to match their absolute translation performance, but to investigate the effect of source reversal under controlled conditions. After removing duplicate training pairs and excluding training sentences whose English sources appeared in the evaluation sets, I retained -
230,595 sentence pairs for training,
890 for validation, and
8,597 for testing,
drawn from six evaluation sets spanning 2010 to 2015. Importantly, I preserved the original dataset partitions rather than creating a new random split. The corpus was considerably smaller than the original, but both source-order conditions would encounter exactly the same training examples and be evaluated against the same French references.
Next came tokenization. I used Moses tokenization to split the English and French sentences into words while preserving case and punctuation, rather than breaking them into smaller subword units using BPE. This was intentional, since the original experiment involved reversing complete words, and reversing subword fragments would introduce an additional complication. After examining the vocabulary-frequency distributions, I retained the 32,000 most frequent English words and 48,000 French words, covering 98.97% and 99.18% of word occurrences in the training corpus, respectively. On the validation set, these vocabularies still covered 98.17% of English and 98.03% of French word occurrences.
Words outside the vocabulary were mapped to <unk>, while three additional special tokens—<pad> for padding, <bos> for the beginning of a sentence, and <eos> for its end—were reserved. Each English sentence was then prepared in two forms - its original word order and its exact reverse, while the corresponding French sentence remained unchanged. Importantly, <eos> was appended only after reversal, ensuring that it remained at the final encoder position in both configurations.
Training the Twenty LSTMs
With the dataset prepared, the next step was training. I followed the optimization procedure described in Section 3.4 of the original paper as closely as my hardware would allow. Following the authors, I initialized the model parameters using a uniform distribution between -0.08 and +0.08, except for the padding embeddings, which were set to zero. Each of the ten random seeds produced a different initialization, but within every seed pair, the normal and reversed models began with exactly the same parameter values - verified using SHA-256 fingerprints. I also retained the original optimization settings - stochastic gradient descent (SGD) without momentum or weight decay, an initial learning rate of 0.7, gradient clipping at a global norm of 5, and a fixed training duration of 7.5 epochs.
The learning rate remained constant for the first five epochs and was halved every half-epoch thereafter. The original experiment used batches of 128 sequences across multiple GPUs. To accommodate my 16 GB GPU, I used smaller batches of 8 sentences and accumulated their gradients to maintain an effective batch size of 128. All computations were performed in 32-bit floating-point precision (FP32). Within each pair, the models also encountered the training examples in exactly the same order, ensuring that source-word ordering was the only intended experimental difference.
All twenty models were trained to the same predetermined endpoint of 7.5 epochs. I also evaluated their behaviour on the same 890 validation sentences at epochs 1, 2, 3, 4, 5, 5.5, 6, 6.5, 7 and 7.5. At each stage, I recorded teacher-forced negative log-likelihood, probabilities assigned to the correct French words, and translation quality under greedy decoding. These measurements allowed me to follow the development of the reversal advantage without using validation performance for early stopping or checkpoint selection. The test evaluation used the models obtained at the fixed 7.5-epoch endpoint.
With all twenty models trained, it was time to see whether reversing the English sentences had actually made a difference. I evaluated the models obtained at the fixed 7.5-epoch endpoint on all 8,597 test sentences, using greedy decoding to generate the French translations and SacreBLEU to measure their quality. Averaged across the ten random seeds, the models trained on English sentences in their original order achieved a BLEU score of 8.17, while those trained on reversed sentences reached 9.91 - an improvement of 1.74 BLEU points. More interestingly, the reversed models outperformed their normal-order counterparts in every one of the ten seed pairs. Sutskever and colleagues had reported an increase from 25.9 to 30.6 BLEU in their original experiment. Although our absolute scores are not directly comparable because of differences in the dataset and evaluation procedure, the direction of the improvement was the same. Reversing the source words had once again produced better translations.
The Meddling
So, ten random seeds and twenty LSTMs later, the magic of source reversal repeated. But reproducing the improvement was never quite what I had in mind. I kept coming back to the explanation Sutskever and his colleagues had offered - that reversing the English sentence introduced shorter dependencies, making it easier for the network to establish communication between corresponding English and French words. There was something about this explanation that I wanted to investigate more closely. After all, as seen, reversing a sentence doesn’t bring every English word closer to the decoder. It brings some words closer while pushing others farther away. Yet the improvement we had just measured was expressed as a single BLEU score, telling us little about how those opposing changes played out within the translation. I wondered whether we could look beneath that aggregate improvement and follow what happened to individual words when their positions in the encoder were reversed.
I started thinking about the problem at the level of individual words rather than entire sentences. Suppose we take an English word and the French word corresponding to it. In the original source order, information from that English word must travel through a certain number of recurrent steps before the decoder produces the French word. Reverse the English sentence, and that number changes - sometimes becoming smaller, sometimes larger. But what happens to the model’s prediction of that particular French word? Does it assign a higher probability when the recurrent path becomes shorter, or does the relationship turn out to be more complicated?
This gave me two quantities worth examining together - how much the recurrent distance changed and how much the probability assigned to the correct French word changed. Rather than relying solely on the overall BLEU improvement, I wanted to see whether these changes were related across individual word pairs.
Quantity-I
To put this idea into mathematical terms, consider an English sentence containing n words. Let i represent the original position of an English word and j the position of its corresponding French word. In the normal source order, information from that English word must pass through the remaining English words, the encoder's <eos> step, and the decoder steps leading to the French word. We can represent this recurrent distance as
When the English sentence is reversed, the same word moves to position n + 1 - i, giving us -
Subtracting the two, we obtain -
ΔD represents the change in recurrent distance for a word when the English source sentence is reversed.
The French position j cancels when the two distances are subtracted. For a fixed English sentence length, the change is therefore determined by the English word’s original position. In a five-word sentence, reversal shortens the first word’s path by four recurrent steps, leaves the middle word’s path unchanged, and lengthens the last word’s path by four steps. This measures the change in temporal separation under our counting convention. It does not directly measure how much source information survives those steps or how strongly the decoder uses it.
But before I could put this idea into practice, there was something fundamental I needed to establish -
Which French word corresponded to which English word?
As generally known, English and French do not necessarily follow the same word order, and a single word in one language may correspond to multiple words in the other. Simply knowing their positions in their respective sentences would not be enough. What I needed was a word-level alignment algorithm that could identify these correspondences and allow me to construct a lookup table for each sentence - recording an English word and its original position i, along with the corresponding French word or words and their positions j.
For this, I turned to OmniAlign, a pretrained multilingual word-alignment model. Its role was to provide plausible English-French word correspondences independently of our twenty LSTMs. With these positional mappings available, I could connect each aligned word pair to the recurrent-distance change ΔD we had just defined. Importantly, these alignments were used only for analysis and had no influence on how either LSTM was trained.
OmniAlign, introduced by Yang et al. (2026), is a multilingual alignment model built on an encoder-only Transformer architecture, with approximately 305 million parameters and a backbone derived from gte-multilingual-mlm-base. Its word-alignment approach relies on contextual representations - rather than comparing isolated words, it represents them in the context of their respective sentences and uses cross-lingual similarity to identify possible correspondences. For my experiment, I used the pretrained OmniAlign checkpoint without modifying its weights. Each English-French sentence pair was processed using Moses word tokenization, followed by OmniAlign's subword tokenizer. I extracted contextual embeddings from the model's seventh hidden layer, normalized them, and calculated cosine similarities between English and French.
These cosine similarities, however, were calculated between subtokens, whereas my investigation required correspondences between complete English and French words. A single word could be divided into several subtokens, so I needed a way to combine their similarities without losing the original word boundaries. I constructed a word-to-word similarity matrix by considering every possible subtoken pairing between an English word and a French word, retaining the highest cosine similarity as the score for that word pair.
To understand the approach I took, consider -
English sentence - “The black cat sleeps”
and its
French translation “Le chat noir dort”.
Moses tokenization first separated the sentences into four words each, preserving their original positions. Both sentences were then processed independently through OmniAlign, producing contextual representations for their respective subtokens. Now consider the English word black. Rather than assuming that its French equivalent was noir, I compared its subtoken representations against those belonging to every French word — Le, chat, noir, and dort. For each French word, I retained the highest cosine similarity across all possible subtoken combinations. Repeating this procedure for The, cat, and sleeps produced a 4 x 4 similarity matrix, containing sixteen scores — one for every possible English-French word pair.
To make this clearer, consider the following illustrative subtoken-level similarity matrix for our two sentences. Suppose black is split into two subtokens and noir is also split into two. The other words remain as single tokens. Each entry represents the cosine similarity between an English subtoken and a French subtoken.
Table 1 - Subtoken-level cosine similarities
Notice what happens with black and noir. Because both words have two subtokens, there are four possible comparisons, producing similarity scores of 0.91, 0.72, 0.66, and 0.81. We retain the maximum, 0.91, as the similarity between the complete words black and noir. Repeating this aggregation for every English-French word pair converts the subtoken-level matrix into the following word-level matrix.
Table 2 - Word-level similarity after maximum aggregation
Thus, we compared subtokens across the two entire sentences, but ultimately assigned one similarity score to each pair of complete words. The bold entries suggest the expected English-French correspondences, although they are not automatically accepted as alignments. That required the additional filtering procedure, which we can explain next.
Now that I had a word-to-word similarity matrix, the next question was how to decide which scores represented plausible correspondences. Simply selecting the highest score in each row would force every English word to have a French counterpart, even when none of the candidates was particularly convincing. Instead, I applied two conditions. First, an English-French word pair had to achieve a cosine similarity of at least 0.80. Second, looking at the matrix from the opposite direction, the English word had to appear among that French word's three highest-scoring English candidates.
Crucially, I did not restrict each English word to its single highest-scoring French candidate. Instead, I examined every English-French word pair in the similarity matrix and retained every pair that satisfied both conditions. This meant that if an English word passed the checks with two or more French words, all those correspondences were preserved. Conversely, if none of its French candidates satisfied both conditions, the English word remained unaligned.
Mathematically, an English-French word pair (eᵢ, fⱼ) was retained only when both conditions were satisfied -
Here, S(eᵢ, fⱼ) represents the cosine similarity between the English word eᵢ and the French word fⱼ, while rank(eᵢ | fⱼ) represents the English word’s position when all English candidates for fⱼ are ranked by decreasing similarity.
Diagramatically,
Consider black and noir in our example. Their similarity of 0.91 satisfies the first condition, and black also ranks among the top three English candidates for noir, satisfying the second. The pair is therefore retained. Applying these conditions across the matrix gives us four accepted correspondences - The–Le, black–noir, cat–chat, and sleeps–dort. Although the four accepted correspondences in this example happened to be one-to-one, that was not a restriction imposed by the filtering procedure.
Once the alignments had been identified, I organized them into a sentence-level lookup structure. For every English-French sentence pair, I recorded a unique sentence identifier, the lengths of both sentences, and the original positions of the aligned words. Each English word was stored alongside its position i, together with the French words and their corresponding positions j that survived our filtering procedure.
Importantly, these positions referred to the original English sentence, before any reversal, and the original French reference. The structure also preserved multiple French correspondences for an English word, while words without an accepted alignment were recorded with an empty list rather than being assigned an arbitrary match. Returning to our example, assuming the four expected correspondences survived filtering, the resulting positional lookup can be illustrated as follows.
Table - Illustrative English-French positional lookup
English - “The black cat sleeps” (n=4)
French - “Le chat noir dort” (m=4)
Notice that black occurs at position 2 in English, but its corresponding French word noir occurs at position 3. The lookup therefore preserves the word correspondences independently of word order.
The example above illustrated the structure for a single sentence pair, but the actual lookup was considerably larger. Applying the same alignment procedure across the complete corpus produced positional records for 240,140 English-French sentence pairs, covering more than 4.28 million English-word occurrences. The resulting alignment coverage is summarized below.
Table — English-French Word Alignment Coverage
The number of accepted word-pair links exceeded the number of matched English words because a single English word could correspond to multiple French words. With these correspondences available, I could now return to the question that had motivated this entire exercise. For each aligned English-French word pair, the original English position i and sentence length n allowed me to calculate the change in recurrent distance ΔD, while the French position j identified the corresponding word whose prediction I wanted to examine. But the lookup itself told me nothing about how well either LSTM predicted that French word. To investigate whether changes in recurrent distance were associated with better or worse predictions, I needed a second measurable quantity -
The difference in probability assigned to the same correct French word by the normal and reversed models.
Quantity-II
To measure this, I returned to our twenty trained LSTMs, but instead of asking them to generate complete French translations, I examined the probability each model assigned to the correct French words in the reference sentences. There was an important complication, however. If the normal and reversed models generated translations independently, they could produce different preceding words, making it difficult to determine whether a later prediction differed because of source reversal or because the models had already followed different translation paths.
Get Pisapati Sri Venkata Sumanth’s stories in your inbox
Join Medium for free to get updates from this writer.
To avoid this, I used teacher forcing during evaluation. At each French position j, both models were given the same correct French words preceding that position, and I recorded the probability each assigned to the actual French word at j. The normal model received the English sentence in its original order, while its paired counterpart received the reversed English sentence. I then compared their log-probabilities, subtracting the normal model’s value from the reversed model’s. A positive difference meant that the reversed model assigned a higher probability to the correct French word, while a negative difference favored the normal model.
To express this mathematically, let the original English sentence be x = (x₁, x₂, …, xₙ), and its reversed version be xᴿ = (xₙ, xₙ₋₁, …, x₁). Similarly, let the correct French reference be y = (y₁, y₂, …, yₘ), where yⱼ represents the French word at position j, and y_{<j} represents all the correct French words preceding it. For each paired random seed s, the normal and reversed LSTMs assign the following probabilities to the same correct French word -
The French word being evaluated and its preceding French context remain identical in both expressions. Only the English source order and the corresponding trained model differ. I therefore defined Quantity II as the difference between the log-probabilities assigned by the reversed and normal models -
Using the logarithmic identity that the difference of two logarithms equals the logarithm of their ratio, this can also be expressed as -
This formulation is useful because it tells us not merely which model assigned the higher probability, but also how their probabilities compare proportionally. A positive value indicates that the reversed model assigned a higher probability to the correct French word, a negative value indicates that the normal model assigned a higher probability, and a value of zero indicates that both assigned the same probability.
There is also a direct connection to the negative log-likelihood used during training and evaluation. Since the negative log-likelihood of the correct word is simply the negative of its log-probability, Quantity II can equivalently be written as -
Thus, a positive Quantity II also represents a reduction in the correct French word’s negative log-likelihood under the reversed model.
Bringing the Two Quantities Together
With both quantities now defined, each accepted English-French correspondence could be associated with a pair of measurements -
Here,
Quantity I measures how reversing the English sentence changes the recurrent distance from the aligned English position i to French position j.
Quantity II measures how the probability assigned to that correct French word changes between the paired models.
The first quantity depends on the word’s position in the source sentence, whereas the second is obtained directly from the trained LSTMs and calculated separately for each random seed.
Before examining the relationship between these two quantities, however, I had to address a complication arising from our word-alignment procedure. Recall that our earlier filtering conditions did not enforce one-to-one correspondences. This meant that a single French word could be aligned with multiple English words. Why was this a problem? Quantity II measured the change in probability assigned to a particular French word at position j, whereas Quantity I depended on the original position i of its corresponding English word. If two English words at different positions were aligned with the same French word, we would have one French-word probability difference associated with two different recurrent-distance changes.
To avoid this ambiguity, I introduced an additional restriction specifically for the statistical analysis - each French word had to have exactly one accepted English correspondence. Importantly, this restriction did not apply in the opposite direction. A single English word could still correspond to multiple French words, provided each of those French words had only one accepted English match. The original OmniAlign lookup remained unchanged; I was simply selecting the correspondences suitable for comparing our two quantities. I also excluded word pairs where either word had been mapped to the unknown vocabulary token <unk> and retained only sentences containing at least two eligible aligned word pairs, since the subsequent statistical analysis would examine variation within individual sentences.
Testing the Relationship - A Within-Sentence Regression
Having obtained both quantities for our eligible English-French word correspondences, I could now formulate the hypothesis that would guide the statistical investigation. The intuition behind source reversal was that shortening the recurrent path between an English word and its French correspondence might make that French word easier for the decoder to predict. However, as we had already established through Quantity I, reversal did not shorten every path. Some became shorter, while others became longer. If recurrent distance was indeed related to prediction quality, these differences should be reflected in Quantity II.
My working hypothesis was therefore that -
ΔD and Δlog P would exhibit a negative relationship.
In other words, English words experiencing greater reductions in recurrent distance - represented by more negative values of ΔD - should, on average, be associated with larger improvements in the probability assigned to their corresponding French words - represented by more positive values of Δlog P.
Importantly, this was not a claim that every shortened path would produce a better prediction, nor that path shortening necessarily caused the improvement. It was a prediction about the overall pattern across aligned word pairs. To investigate whether that pattern existed, I needed to move beyond individual examples and examine the relationship statistically.
To test this hypothesis, I used our regular go-to tool ordinary least-squares (OLS) regression, with Quantity I (ΔD) as the explanatory variable and Quantity II (Δlog P) as the outcome. However, there was a complication. Reversal might benefit some sentences more than others for reasons unrelated to the recurrent distance of any particular word. If I simply combined all word-level observations, such differences between sentences could influence the estimated relationship. To address this, I used sentence fixed effects, allowing each sentence to have its own baseline probability difference. The regression therefore examined whether words experiencing greater path shortening also tended to show larger probability improvements relative to other eligible words in the same sentence, rather than relying on comparisons between different sentences.
However, introducing sentence fixed effects did not mean that every word within a sentence was now directly comparable. While this allowed me to account for differences shared by words belonging to the same sentence, individual words could still differ in ways that might influence how much they benefited from source reversal. I therefore considered three additional characteristics that could be relevant to our investigation.
First
The first control concerned the position of the French word within the target sentence. Teacher forcing ensured that the normal and reversed LSTMs received exactly the same correct French prefix when predicting a particular word. In our familiar translation, “Le chat noir dort”, both models received Le when predicting chat, and Le chat when predicting noir. This made the comparison between the models fair for each prediction. However, the regression compared many different predictions, each made at a particular stage of decoding.
Why could this matter for the reversal advantage? Near the beginning of a French sentence, the decoder had relatively little French context available. Later, the supplied prefix could itself provide substantial clues about the next word. Meanwhile, the decoder state carrying information from the English source had undergone further recurrent updates. These changing conditions need not affect the two trained models equally. Reversal could therefore provide different advantages at different French positions, even when both models received the correct history. Teacher forcing removed differences arising from previously generated mistakes, but it did not make the resulting log-probability difference, Y, independent of decoding position.
This possibility became relevant to the distance analysis because French position could also be associated with ΔD. Within a source sentence, ΔD was determined by the original English position. Wherever earlier English words tended to align with earlier French words, shortened paths would also tend to accompany earlier French predictions. If reversal provided a larger advantage at those predictions, a relationship between shorter paths and larger Y could partly reflect the French positions at which those paths occurred. Such a pattern could recur across many sentences and remain present when their aligned words were pooled together.
I therefore included the relative French position, j/m, where j denoted the French word’s position and m the total French sentence length. The fraction expressed decoding progress on a comparable scale across sentences. Sentence fixed effects accounted for sentence-wide differences, while this control accounted for an average linear positional trend in Y within sentences. The resulting distance coefficient measured the association between ΔD and the reversal advantage after accounting for that trend.
Second
The second control concerned the training frequency of the aligned English word. Some English words occurred repeatedly throughout the training corpus, while others appeared relatively infrequently. Their frequency therefore described the amount of exposure the models had received to each source word before the French predictions were evaluated. I represented this characteristic using the English vocabulary’s frequency ordering, in which lower ranks indicated more common words and higher ranks indicated less common words.
Within each model pair, every English word had been encountered equally often under normal and reversed source order. However, reversal changed the positions and preceding English contexts in which those encounters occurred. Equal exposure between the two models therefore did not guarantee that reversal would produce the same advantage across words with different frequencies. Frequently encountered words had more opportunities to be learned under both processing orders, while less frequent words had fewer. The change in source order could consequently matter differently across these levels of exposure, without implying in advance which words would benefit more.
The relevant concern was whether frequency was associated with the reversal advantage itself. Quantity II compared the models’ predictions of the same French word under the same correct French prefix. Nevertheless, the resulting difference, Y, could systematically differ between predictions aligned with common English words and those aligned with uncommon ones. If English frequency also varied with source position, it could be associated with ΔD, since source position determined the distance change within a sentence. The regression could then combine a distance association with differences in the frequency composition of the aligned words being compared.
I therefore included the logarithm of the aligned English word’s vocabulary rank as an additional control. This allowed the regression to account for an average relationship between log frequency rank and Y, alongside the other included variables. The distance coefficient consequently addressed whether ΔD remained associated with the reversal advantage after accounting for this potential variation with English training frequency.
Third
The third control concerned the training frequency of the French word being predicted. Each occurrence of a French word provided the decoder with another example of the circumstances in which that word should be generated. Common words supplied many such learning opportunities, while uncommon words supplied fewer. Both models received the same target sentences, but the source representations accompanying those predictions were learned under different English processing orders. The resulting reversal advantage could therefore differ across French words with different levels of training exposure.
Teacher forcing ensured that both models predicted the same French word after receiving the same correct prefix. However, this comparison was repeated across many different words. If reversal provided a small advantage for frequently predicted words and a larger advantage for less frequently predicted words, that pattern would remain present in Y. Comparing the models fairly for each word did not make their difference independent of French frequency.
This became relevant to the distance analysis if French frequency was also associated with ΔD. Through the English-French alignments, different English positions could correspond to different mixtures of common and uncommon French words. Predictions associated with shortened paths could consequently differ in frequency composition from those associated with lengthened paths. If reversal’s advantage also varied with French frequency, the distance slope could partly capture that overlapping pattern. I therefore included the predicted French word’s log vocabulary rank to account for an average French-frequency trend in Y.
For both languages, vocabulary ranks represented frequency orderings rather than exact occurrence counts. I used their logarithms to compress the broad numerical range and model an average relationship between Y and log rank. Together, the three controls addressed plausible variation associated with French decoding position, English training frequency, and French training frequency. Their purpose was to examine the association between ΔD and Y after accounting for these specified trends. Their inclusion did not establish that these characteristics explained the observed relationship.
Conceptually, the regression could therefore be written as -
Here, X remained Quantity I (ΔD), Y remained Quantity II (Δlog P), and b remained the coefficient of primary interest. The three additional control variables were -
C₁ = j/m - Relative French word position, accounting for differences associated with the stage of decoding.
C₂ - Logarithm of the aligned English word’s vocabulary rank, accounting for differences associated with English word frequency.
C₃ - Logarithm of the French word’s vocabulary rank, accounting for differences associated with French word frequency.
Their corresponding coefficients, γ₁, γ₂, and γ₃, captured the associations between these variables and the outcome. The intercept a represented the baseline, which I allowed to differ between sentences through sentence fixed effects, while ε represented the remaining unexplained variation.
In other words, I was still investigating the relationship between ΔD and Δlog P, but now accounting for sentence-specific differences, French decoding position, and vocabulary characteristics. In the actual implementation, I also divided ΔD by 10, so that the main coefficient would represent the associated change in log-probability difference for a ten-step increase in recurrent-distance change.
For each of the ten paired random seeds, I estimated the following regression separately at every recorded validation stage. The equation describes one seed pair at one stage, with the training-stage index omitted for readability. The aligned words, distance values and control variables remained fixed across these evaluations, while the model-derived probability differences changed. I then applied the same specification to the final test predictions.
where,
k — English-French sentence pair.
j — Position of the French word being predicted.
s — Identifies one of the ten paired random seeds.
i(j) — Original English position uniquely aligned with French word j.
mₖ — Total number of French words in sentence k.
αₖ⁽ˢ⁾ — Sentence fixed effect, accounting for sentence-specific baseline differences.
β⁽ˢ⁾ — Main regression coefficient measuring the relationship between Quantity I and Quantity II.
γ₁⁽ˢ⁾, γ₂⁽ˢ⁾, γ₃⁽ˢ⁾ — Coefficients of the three additional control variables.
rₖ,ᵢ₍ⱼ₎ᴱᴺ — Vocabulary rank of the aligned English word.
rₖ,ⱼᶠᴿ — Vocabulary rank of the French word being predicted.
εₖ,ⱼ⁽ˢ⁾ — Residual variation not explained by the regression model.
I calculated standard errors clustered by TED document, allowing observations within the same talk to be statistically dependent. The validation regressions used 10,052 eligible links from 831 sentence occurrences across eight documents. The final test regressions used 98,810 links from 8,177 sentence occurrences across 89 documents. Each seed pair was analysed separately.
I also examined whether the pooled relationship concealed differences associated with sentence length. For this secondary analysis, I fitted the same regression separately to short English sentences containing 1–19 words, medium sentences containing 20–39 words, and long sentences containing 40 or more words. Length was measured before adding special tokens. I additionally examined a normalized distance predictor, ΔD/(n−1), and a specification containing a quadratic distance term.
Normalization expressed the distance change relative to the source sentence’s positional range. Because sentences were divided by different lengths, this changed their relative influence on the pooled fit. Agreement between the absolute and normalized specifications could therefore be examined rather than assumed.
Observations
I first examined how the relationship between recurrent-distance change and reversal’s predictive advantage developed throughout training. These measurements were made on the same validation set at every recorded checkpoint. While the models continued learning from the training data, the validation examples provided a fixed population on which to observe how their predictions evolved. I then examined the final models, obtained after 7.5 epochs, on the test set.
Under the primary regression, which used the absolute distance change divided by ten alongside sentence fixed effects and the three specified controls, the validation coefficients followed the pattern below.
Diagrammatically,
At epochs 1, 2 and 3, all ten seed pairs produced negative validation coefficients. After accounting for sentence fixed effects, French relative position and both vocabulary-rank controls, greater path shortening was associated with a larger reversal advantage. This direction was consistent with my working hypothesis.
However, the relationship did not remain in that direction.
As training progressed, the coefficients gradually changed sign. By epoch 5, eight of the ten seed pairs had positive coefficients. At epoch 6, all ten were positive, and at the final checkpoint, nine remained positive. The transition did not occur simultaneously across seeds, and individual trajectories were not perfectly monotonic. Nevertheless, the overall pattern revealed a broadly shared change from negative to predominantly positive associations.
What made this particularly interesting was that the geometry of the validation examples had not changed. The original English positions, accepted word alignments, sentence lengths and resulting values of ΔD remained identical across checkpoints. The regression also retained the same eligible validation population and controls. What changed were the models themselves.
As their parameters developed through training, the relationship between the fixed positional rearrangement and the models’ relative predictions changed. The coefficient trajectory therefore described an evolving association between source position and reversal’s learned predictive advantage, rather than a change in the underlying recurrent-distance calculation. There was another important distinction. A negative regression coefficient did not imply that reversal was harmful overall, just as a positive coefficient did not imply that reversal benefited only words whose paths became longer.
Across every recorded validation checkpoint and seed pair, the descriptive mean of Y was positive for both words whose paths became shorter and words whose paths became longer. At epoch 1, the mean reversal advantages, averaged across seeds, were approximately +0.20335 and +0.08029 for the shortened and lengthened groups, respectively. At epoch 7.5, these means were approximately +0.12866 and +0.23275.
These descriptive summaries used the 10,082 eligible validation links before the removal of sentences containing fewer than two eligible links for regression. Thus, reversal’s average predictive advantage was present in both groups. What changed throughout training was how that advantage was distributed across different distance changes. The regression coefficients described this adjusted relationship, rather than simply indicating whether reversal improved predictions on average.
I next examined whether the predominant positive association observed at the final validation checkpoint was also present in the test data. At epoch 7.5, all ten seed pairs produced positive coefficients under the primary absolute-distance specification. Their mean coefficient was approximately +0.01497 nats per ten-step increase in ΔD. Nine of the ten corresponding 95 percent confidence intervals, calculated using standard errors clustered by TED document, lay entirely above zero.
This was the opposite direction to my original hypothesis. Under the fitted specification, larger increases in recurrent distance were associated with larger improvements in the probability assigned to the correct French word, after accounting for the included controls. The final test association therefore agreed in direction with the predominant late-stage validation pattern. However, the test regressions were performed only at the predetermined 7.5-epoch endpoint. They did not establish whether the same transition had occurred across earlier checkpoints on the test set.
The final models also retained an overall advantage under teacher forcing. Averaged across the ten seed pairs, the test negative log-likelihood decreased from approximately 2.8964 for the normal models to 2.7657 for the reversed models.
This mattered for the original conundrum. Both models received the same correct French prefix when predicting each target word. Consequently, the observed difference in conditional prediction quality could not be attributed solely to one model having generated a better sequence of preceding French words during evaluation.
Reversal had produced an advantage even when the models were compared under a shared correct target history. However, this result did not independently identify whether the improvement arose from the encoder’s learned representation, the decoder’s use of that representation, or their interaction.
The positive pooled test coefficient raised another question. Was this relationship broadly shared across sentences, or was it concentrated within particular sentence lengths? To examine this, I fitted the primary absolute-distance regression separately within the three predefined sentence-length groups.
The results revealed that the positive pooled association was not uniform across sentence lengths. Medium-length sentences produced the clearest positive relationship, with all ten coefficients positive and nine document-clustered confidence intervals lying entirely above zero. Short and long sentences, however, predominantly produced negative coefficients, although most of their individual confidence intervals included zero. Interestingly, reversal still improved greedy BLEU in every seed pair across each of the five evaluated sentence-length bands - 1–9, 10–19, 20–29, 30–39 and 40 or more English words. The corresponding average improvements were approximately +1.74, +2.23, +2.14, +1.39 and +0.77 BLEU points, respectively.
Moreover, the normalized-distance specification produced positive coefficients in nine seeds for short sentences, all ten for medium sentences and nine for long sentences. The relationship between distance change and predictive advantage therefore depended on both sentence length and the distance specification, while improvements in overall translation quality remained observable across the evaluated length bands.
Limitations
The findings should be interpreted within the scope of the experimental setup, the evaluation populations and the statistical measurements used.
Future Scope
The next investigation would examine how source reversal changes the way a trained encoder-decoder preserves and uses source information. For each aligned word pair, I would measure the gradient of the correct French word’s loss with respect to the embedding of its English word occurrence, tracing sensitivity through the complete encoder-decoder computation. Alongside this, gradients with respect to the encoder’s final hidden and cell states would reveal how the decoder’s local sensitivity is distributed across layers, state types and French positions. Relating these measurements to distance change, prediction advantage and sentence length could reveal whether the positional patterns observed in the regressions have corresponding patterns of sensitivity within the network. The saved 7.5-epoch models would provide a starting point, while the limited earlier gradient probes could offer historical context without supporting an equivalent analysis across training. Carrying out the full gradient and intervention investigation across all twenty models and a large population of aligned words would, however, exceed the computational budget I can currently allocate to this project. I therefore leave this extension as future work.
These measurements would describe local sensitivity, rather than directly quantify retained information or establish its causal use. Assessing retention would require separate tests of what source information could be recovered from the final states, while carefully controlled perturbations within each model could examine how predictions depend on particular source occurrences or state components. Such comparisons would need to respect the possibility that separately trained networks use different internal coordinates. Together, these analyses could help distinguish changes in source representation from changes in the decoder’s dependence on that representation, without assuming that larger gradients imply better memory. This would extend the investigation from counting recurrent steps to examining the learned computations associated with reversal’s advantage.
Conclusion
In Part I of Chasing Attention, I ended by pointing towards Bahdanau, Cho and Bengio’s work on attention. Before moving there, however, I wanted to spend a little more time with Sutskever, Vinyals and Le’s paper. Source reversal was a surprisingly simple idea, yet its explanation raised an interesting question — could the improvement actually be understood through the changing recurrent distances between corresponding source and target words?
This experiment was more of a fun, curiosity-driven attempt to understand that question, rather than an attempt to challenge the original findings. Source reversal improved translation quality in my experiments, but the relationship between recurrent distance and word-level prediction advantage was not as straightforward as I initially expected. The observed association changed during training, differed across sentence lengths and depended on how distance was measured. These findings raised more questions than they answered, particularly about how the encoder preserves source information and how the decoder learns to use it.
I do not consider this investigation a complete explanation of why source reversal works. It was an opportunity to move beyond reading a paper, formulate a question, test an intuition and confront results that did not entirely support my expectations. Understanding the underlying mechanisms would require looking more closely at gradient sensitivity, information retention and the decoder’s dependence on the encoder’s representations. For now, I will leave those questions open for future investigation.
There is still a long way to go in this series, and many fascinating ideas ahead of us. Perhaps another question will prove too tempting to leave unexplored, leading to an occasional experimental detour like this one. But the larger journey remains the same — to follow the ideas, understand the problems that motivated them and discover how each development brought us closer to the attention mechanisms we know today.
That brings me back to where Part I ended. The fixed-state encoder-decoder architecture had demonstrated that complete sentences could be translated without explicitly aligning source and target words during generation. But it also left an important architectural question unanswered — what if the decoder did not have to rely entirely on the encoder’s final states, and could instead access different parts of the source sentence whenever it needed them?
That is where Bahdanau, Cho and Bengio’s Neural Machine Translation by Jointly Learning to Align and Translate enters the story, and where the next part of Chasing Attention will begin.
