Why do Transformers handle long-range dependencies better than LSTMs despite lacking explicit recurrence?

Recurrent architectures such as LSTMs and GRUs were originally designed to address the vanishing gradient problem and capture long-range dependencies in sequential data. However, in recent years Transformer-based architectures have largely replaced RNN-based models in many domains such as natural language processing, time-series modeling, and even reinforcement learning. One commonly cited explanation is that Transformers allow parallel computation and avoid sequential processing, which improves training efficiency. However, this does not fully explain why they often outperform LSTMs in modeling long-range relationships. From a modeling perspective, I am trying to understand the following points: In LSTMs, the cell state is explicitly designed to propagate information across time steps. In Transformers, there is no recurrence or persistent state between tokens. Why does the self-attention mechanism still capture long-range dependencies more effectively? Is the improvement mainly due to the attention mechanism allowing direct connections between distant tokens, or are there additional factors such as representation capacity and optimization dynamics? Are there known theoretical explanations or empirical studies comparing the ability of Transformers and LSTMs to capture long-range dependencies? Are there scenarios (for example streaming data or low-resource environments) where recurrent architectures still outperform Transformers? I would appreciate references to research pape…