Attention and the Transformer make it possible to recognize that the probability of “black” applying to the cat is much much higher than to the mat due to the phrasing “which is” in between the cat and black.

Thank you. So this is based on the training data, I assume.

It is a lot harder to take the black out of the cat than it is to take the mat out from under it.

Humans know that, how does transform know that? Based on training data?

Sort of. Part of the training for a model includes telling it which parts of a sentence are important... a human points and clicks.

This is extremely important to know. That the relationships between words in the sentence are actually trained by human evaluation.

They are not.

No, thats incorrect. The connections are automatically deduced from the training data (which is just vast amounts of raw text).