I'm having a little trouble justifying something in my head and was hoping someone could provide some intuition? I understand for LSTM models or models that maintain some state about a sequence that training auto regressively makes sense to help a model learn to manage its context and states. What I am struggling with is if we used a model that did not maintain some hidden state.

For this example lets say that we have some time series x and we apply an AR model to the series. Now say we replace the parameters of the AR model with multiple fully-connected layers.

I believe training the model on one step ahead predictions and applying it to multi step would introduce compounding errors, but I cannot justify this intuitively to myself in my head. Could anyone shed light on this?

I keep coming back to the idea that basically one step ahead training here is just like teacher forcing techniques for LSTMS.

Any insight is much appreciated!