Ground-Up VLA: Self-Attention

From GPU basics to robot foundation models

Previously on Ground-Up VLA, we talked about neural networks, which are the type of machine learning technique that enables deep learning. Large language models are a family of deep learning models used for language tasks. Today, we’ll approach a core concept that was key to unlocking practical large language models: self-attention (which leads to transformers, but we’re not there yet).

Thinking about thinking

Traditionally, thinking about thinking would put us in the realm of philosophers. However, the type of thinking you need for neural networks is more physical than metaphysical.

Consider: how does the brain think about interpreting a movie scene? It takes cues from a few different things:

1. What is being said?

2. What is the expression of the person who is speaking?

3. What is the tone of voice, inflection, and speed of the person speaking?

Using all this information, the brain comes up with an interpretation of how the listener should feel about what was said. A movie scene is incredibly complex, however, since you have to understand visual sights, sounds, and the actual words of sentences.

This is supposed to be an introductory post, so let’s zoom in on just the first problem. If we assume that the speaker literally means what they are saying, then the problem narrows down to wondering how to understand the words that are being said — or, to make it simpler from a machine perspective, what words are written.

So, in the interest of taking a written example, below is an excerpt from a clay tablet sent to Ea-nāṣir (𒂍𒀀𒈾𒍢𒅕) by Nanni (𒈾𒉣𒉌) in 1750 BCE.¹

You have withheld my money bag from me in enemy territory²

How would we go about understanding this sentence? We pay attention to key words in the phrase and what they indicate about one another. A word, like “bag”, on its own, is able to convey that it is some kind of deformable entity capable of holding things. When we prepend it with “money”, we probably think the size of the bag is on the smaller side, and it should be sturdy to hold coins.³ The word “withheld” and “you” are both critical to understanding the sentence, because they tell us what action was taken and by whom. In contrast, “territory” probably needs to pay a lot of attention to “enemy”, but perhaps not that much attention to “have”.

For each word, we can define a set of importance scores describing how much it should care about every other word. A neural network layer that learns these scores over a sequence is called self‑attention.⁴ We’ll get to the math of it later, but hopefully this provides a high-level picture of what both we and language models do to understand sentences.

High-level math

Attention is based on three types of matrices: the query matrix Q, the key matrix K, and the value matrix V. All three matrices are computed by multiplying an input matrix X with learned sets of weights W_Q, W_K, and W_V, that let each word of a text learn more about what it means in the context of the surrounding text.

• Q = XW_Q: what kind of information would help understand each word in context?

• K = XW_K: what kind of information does each word offer to others?

• V = XW_V: what information does a word provide if another word decides to pay particular attention to it?

Let’s consider this with an example from our Mesopotamian clay tablet. Ignore most of the sentence for the moment, and focus in on “money bag”. The matrix Q, when applied to “bag”, tells us what kinds of information is useful to know, like the bag size, bag contents, actions taken on the bag, etc. The matrix K tells us that for the word “bag”, the word “money” is rather useful. The matrix V tells us that if the word “money” is something that “bag” pays attention to, then the sentence as a whole can be construed to have some kind of financial context.

We next compute compatibility scores. Recall that a row i of Q tells us “what does the i^th word want to know?” and that a row j of K tells us “what kind of information can the j^th word provide?” Suppose we take the dot product of the row vectors Q_i and K_j. This, intuitively, should tell us if word j can provide information that is useful for the types of queries that word i wants to ask. The larger the dot product, the more likely it is that we get useful information for word i from word j. Let’s consider our example from earlier. When we multiply the row Q_bag by the column (K_money^T), we get a large number because “money” provides important context for the contents of the bag. Expanding this concept to all the words in the sentence, we can compute S = QK^T.

Usually, we perform what’s called a softmax on S to make the outputs fall into the range [0, 1]. That’s not super important for the intuition, however, so we’ll come back to that, and instead focus on what we do now that we have S.

The final step of the math is to compute Y = SV. In the case of “bag” paying attention to “money”, this will cause the output row Y_bag to acquire a monetary context.

Y is the final output of a self-attention block in a neural network, so this would be the input to the next layer.

Math details

We first considered an intuitive idea of why self-attention makes sense, conceptually. Then we looked at a high level idea of what the different mathematical parts of attention provide. Now, let’s try to follow a small example with more concrete mathematical details. This is going to use a fair amount of linear algebra, so if you’re looking for just the high-level concepts, feel free to skim this part.

To start, let’s say the following:

• We have the input sequence “You have withheld my money bag from me in enemy territory”, which is n=11 words long.

• Let’s assume that the model we are making has d_model = 10 dimensions.

Embeddings

In large language models, we have a predefined dictionary that uses a vector of size d_model to represent words. In other words, “bag” and “money” have vector representations, or embeddings, that we can look up in some dictionary.

With this lookup done, we can produce the X matrix from earlier examples. Here, X is going to have dimensions n x d_model = 11 x 10.

Query, key, and value matrices

Recall that we compute Q, K, and V with the learned weight matrices W_Q, W_K, and W_V. In the simplest case, each of these weight matrices have size d_model x d_model. Since X has dimensions n x d_model, and because Q=XW_Q, K=XW_K, and V=XW_V, the matrices Q, K, and V will all have dimensions n x d_model. More concretely, for our example sentence: Q, K, and V will all have dimensions 11 x 10.

Note: There are more complex concepts, like multi-head attention, which change these dimensions. However, this post is aimed at keeping things simple, so the pre-trained weight matrices are assumed to have dimensions d_model x d_model^.

Compatibility scores

Now that we have the query, key, and value matrices, we can compute the compatibility scores, which is what we defined when we said S = QK^T. Since Q has dimensions n x d_model and K^T has dimensions d_model x n, the dimensions of S will be n x n. With our running example, this makes the dimensions of S 11 x 11.

One part we did not discuss earlier is how these compatibility scores get scaled. When we compute S, we typically scale each element by dividing it by the square root of d_model. We do this to try to preserve nuance.

\(S = \frac{QK^\top}{\sqrt{d_{model}}}\)

When we compute QK^T, the value of any individual element is the sum of d_model multiplications (i.e., the dot product of two vectors), and this value can quickly grow large (especially if we stack many self-attention layers), and dividing by the square root of the model size keeps the compatibility score from growing too quickly.

For a more detailed look at this scaling operation and other ways of computing compatibility scores, take a look at Chapter 11.3.1 and its neighbors in Dive into Deep Learning.

Attention matrix

Remember how we said we’d revisit something called softmax? That falls into place right here. Given the compatibility scores in S, we want to scale the results so that each row represents a probability distribution. This normalization allows for attention to generalize across many different types of contexts. In other words, this is part of what allows a neural network to correctly process the disparate sentences “You have withheld my money bag from me in enemy territory” and “My brain has too many tabs open”.

Consider the S matrix that we’ve already computed. Take a particular row of this matrix, which represents the compatibility scores of all words in a sentence with respect to one particular word — e.g., how important each word of “You have withheld my money bag from me in enemy territory” is to the word “bag”. Softmax is computed as:

\(A_{ij} = \frac{e^{S_{ij}}}{\sum_{k=1}^ne^{S_{ik}}}.\)

Here, i could be the row representing the word “bag”, and j ranges over all the columns — i.e., we are computing what the row A_i should be. The numerator represents the compatibility score that the combination of word j and word i produced. The denominator is the sum of the exponentiation of each compatibility score that word i had with all other words in the sentence. The outcome is that A_i is a row that sums to value one, and has values that fall in the range [0,1].

Since A just scales S, it has the same dimensions: n x n—in our running example, this is an 11 x 11 matrix.

We won’t get into why exactly it’s this particular softmax function, that is a bit beyond the scope of this post. There’s also non-softmax functions we could use, but softmax is among the more common normalization methods as of early 2026.

Attention outputs

Now we’re on the last step. We have our normalized attention matrix A, and we have our values matrix V. Earlier, when we talked in a hand-wavy manner about computing output matrix Y, we said we would skip normalization and compute Y = SV. Now we know better, because SV could result in scaling problems as the compatibility scores grow. Instead, we will compute Y = AV.

Since A is an n x n matrix, and V is a n x d_model matrix, the output Y = AV will have dimensions n x d_model, which is the same as the original input X. In our running example, this means that the dimensions of both X and Y are 11 x 10 matrices.

And that’s the full self-attention block!

Recap

This was originally going to be a post about transformers, but the self-attention part itself got pretty long, so I made that the core focus. We’ll try to piece this into the larger transformer architecture next time.

In this post, we primarily covered self-attention, which is a technique by which, e.g., the words of a sentence can understand how important each word is with respect to each other word in a sentence. It also applies to other contexts, like understanding images, but we’ll get to that later.

First, we tried to develop a high-level intuitive understanding of why we’d want to perform self-attention. Second, we looked at the high-level math to see why the matrix operations we perform lead to some kind of meaningful result. And finally, we walked through all the parts of self-attention to see how an input matrix X can be used by a self-attention mechanism to produce a context-enriched output matrix Y.

The flow looks like the following.

1. Get a sentence, and convert the words of the sentence into embeddings. The embeddings get packed together into an input matrix X.

2. Pre-trained matrices W_Q, W_K, and W_V, are used to compute query, key, and value matrices Q, K, and V, respectively, when multiplied by X.

3. The compatibility scores are computed as S = QK^T. These are unnormalized attention scores.

4. The compatibility scores are normalized by the softmax function: A = Softmax(S). A is the attention matrix.

5. The output is produced by multiplying the attention matrix with the value matrix: Y = AV.

1. I asked Gemini, Perplexity, and ChatGPT about how to spell Nanni in Old Babylonian Akkadian. Gemini and ChatGPT said that it was 𒈾𒉌 based on how short names were typically spelled using pure phonetics. Perplexity said that the two-character variant would really spell “Na-ni” and skip over the extra “n”. Perplexity suggested 𒈾𒉣𒉌 or 𒈾𒀭𒉌, but also said the former seems more plausible.

2. Pearce, Victor and A. Leo Oppenheim. “Letters from Mesopotamia : official business, and private letters on clay tablets from two millennia.” (1968).

3. It was unlikely to be paper money — the first known use of paper money is the Song dynasty in the 11th century AD, with the concept originating further back in the Tang dynasty in the form of merchant receipts. See: Daniel R. Headrick (2009). Technology: A World History. Oxford University Press. Or Wikipedia.

4. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in Proc. 3rd Int. Conf. Learn. Representations (ICLR), 2015.